The Inside Scoop on Primer, Expedia’s Internal Cloud Deployment Tool
In early 2013, Expedia decided it was time to more heartily explore cloud solutions. At the time, the company’s web services structure was a monolith. The team was nearing the end of a major platform overhaul, changing the way the entire core of Expedia operated. The cloud was becoming too interesting a tool to ignore. But how does a company move a proverbial rock uphill?
A small group of us came together to look at moving a microservice to the cloud. We focused on an application within the shopping path — starting on a segment of the business that wouldn’t immediately impact the booking path. We knew we wanted to run everything through an automated path. For us, if developers had to SSH to the boxes, then automation had failed.
To kick things off, we used Chef as a provisional cloud server deployment tool. It served as a model for moving over to Amazon Web Services (AWS) as a longer-term solution. As we started building microservices we also built modules to help teams with things like monitoring, bot-detection, site resolution, and site experimentation. By observing these modules and making small tweaks along the way, we were able to effectively deploy our first set of microservices to AWS. This enabled us to provision an Elastic Cloud Compute (EC2) instance and deploy Chef’s Cookbook on it with monitoring and logging infrastructure. This was our first iteration of deployment automation to the cloud.
After few months we added capability to our automation to do blue green deployments. Using these two techniques, teams were able to deploy to AWS successfully — we rolled out about 12 services that first year. However, it was still a time consuming process: in order to take advantage of AWS and its automation, teams had to wire in platform modules, write a Chef Cookbook, and create infrastructure files and deployment configuration files. Since we recognized much of this work could be automated itself, in May 2014 we decided to build Primer. Primer is a tool that automates creation of a HelloWorld service. In just five minutes, it wires in a platform module, creates a Chef Cookbook, builds infrastructure configuration files and deployment configuration files, and creates monitoring dashboards.
We also created Jenkins Jobs for continuous integration and continuous delivery (CI/CD) for the teams. Every time a check-in happens to a GitHub repository, the service is deployed automatically to AWS.
Here’s a glimpse at our CI/CD tooling pipeline for cloud:
- Ruby scripts using ruby which interact with AWS SDK
- GitHub repository which has Infrastructure and deployment configuration files
The workflow involves:
- A commit pushed to the GitHub repository’s master branch triggers Jenkins Job Builder
- Jenkins build runs compile, test and triggers deployment
- The deployment is parameterized so any application can deploy to multiple environments in AWS in multiple AWS regions by press of a button
- The deployment process involves: using custom Ruby scripts we spun up a EC2 instance, deploy the artifacts onto it using chef, once deployment verification tests succeed we create an image of the instance and give it a AWS CloudFormation to spin up a full blown stack, the stack involves creation of multiple instances with ability to scale up and scale down and it also has a ELB in front of it. We use Route53 DNS to create CNAME which point to the ELB
- The workflow also allows teams to deploy at anytime and choose to release (take traffic) at a later time. We use Route53 weighted routing policy to achieve this.
The workflow for CloudFormation/Chef based Primer template is illustrated below
Primer supports the following app types:
- Spring MVC
- Spring boot
- Drop wizard
- Nodejs + Express
- Ruby Sinatra
- Spark cluster creation
- Storm Topologies
- AWS Lambda Java and Node
- And many more, as members of the dev community can submit new templates
With Primer in place, cloud adoption within Expedia skyrocketed. We currently have more than 1000 services we’ve tested and created. We continue to learn from our efforts in the cloud space and are currently working on a next generation version of Primer, revamping how the deployments work so we can make every part of the process be a REST API call. I will share the detailed architecture of Primer in a later post. We’ve expanded our footprint of cloud developers within Expedia, so there is a large community who understands how the underpinnings work.
We’re pleased by the early success from moving some of the smaller parts of our business to the cloud. Step by step, we’re exploring the right ways to continue this journey.
In a later post, I will share how we deploy docker containers on a cluster of machines using Primer.