Published
November 22, 2017
Author
Rob Meyer (@robfromca)
Category
Comments
Time To Read
Estimated reading time: 1 minute

re:Invent 2017 – Building a Mission Critical, Serverless Photo Pipeline for 100 Million Photos

By Rob Meyer (@robfromca) in General on November 22, 2017 | Responses (1)

SRV315: How We Built a Mission-Critical, Serverless File Processing Pipeline for over 100 Million Photos

Monday, November 27th 2:30 PM, Aria, Level 3, Juniper 4

Mike worked on HomeAway‘s image processing pipeline using Lambda functions, SNS, and DynamoDB to handle an ever growing collection of over 100 million property photos uploaded by the property owners. He’ll share the from scratch evolution of the service, walking the audience piece by piece through the overall architecture and sharing some of the key insights. You’ll learn:

  • Where Lambdas work and where they don’t
  • Best practices for limiting Lambda processing time,
  • Some surprising results about Lambda memory optimization,
  • How important using the dead letter queue is to reliable and scalable pipelines.

A diagram of the overall image pipeline architecture.

 

Abstract: In this session, principal architect Mike Broadway describes how HomeAway built a high-throughput, scalable pipeline for manipulating, storing, and serving hundreds of image files every second with Lambda, Amazon S3, DynamoDB, and Amazon SNS. He also shares best practices and lessons learned as they scaled their mission-critical On Demand Image Service (ODIS) system into production. Lambda functions form the backbone of ODIS, which handles over 100 million photographs that are uploaded to HomeAway’s vacation rental platform. HomeAway is a vacation rental marketplace with more than 2 million rentals in 190 countries and is part of Expedia.

See the complete list of Expedia team members speaking at re:Invent 2017.

1 Comment

  • Srinivasan - November 27, 2017

    @Mike – Loved your session at the AWS re:Invent 2017. We built a similar application to backup photos & videos from mobile devices and process these images in a system similar to IRIS on AWS. It was vindicating to see lots of similarities in the architecture and appreciate the differences.

    Loved your analogy of Jeff Goldblum from Jurassic Park and the note that this architecture might be a thing of the past in a year from now given the rate at which AWS is innovating.

    Reply

Leave A Comment