When we deploy our application into the world, we generally want to avoid downtime. This can be avoided by deploying the new version in parallel and then convincing your router or reverse proxy to change where it’s pointing to. This is a deployment automation process described by Martin Fowler as the blue-green deployment. So the theory being sound, how would you actually achieve this in practice?
Getting the application ready
First step is to make sure your application infrastructure allows you to have two instances running in parallel. This step might be trivial to some projects, however this might uncover some tangled up parts in your application. Containerization or at least deployment in an isolated environment might help you our here. At Findmypast we have adopted docker containerization for our deployment, meaning each application deployment results in an isolated docker container, to which our reverse proxy can routed to.
When you decide to go down the docker path you quickly realize that docker alone will leave your infrastructure in a bit of a mess. So to help with configuration management and orchestrating (tying together) the docker containers we use docker-compose. This, in turn, allows us to define a name for your project, project here being a reference to a collection of services working together. Using a unique project name allows to create multiple containers running the same version of the service, and therefore bring up blue and green containers at the same time.
Service discovery
To allow your service proxy (or router) to know where to point, a form of service discovery is required. A popular set of tools for this is HashiCorp’s Consul and the registrator docker container. Together they automatically provide a dynamic index of your running services. Along with a name, services can have “tags” metadata values. This is very useful if you need to keep track of containers running different testing environments, and on top that it can help us to separate blue and green instances. The control of the tags as well as project name is done via the container’s environment variables such as SERVICE_TAGS and SERVICE_NODE. These variables are then read by registrator.
When the values in consul change, a handy tool called consul-template can monitor and update template files based on this data. The template file could be part of a reverse proxy (in our case nginx) configuration. So once a container is deployed and registrator container registers it with consul, the reverse proxy can automatically start routing traffic to it.
Consul has a convenient feature which allows storage of arbitrary key-value pairs for user-defined data. Using this store we keep track of the current live colour per service.
Deployment sequence
Using our task runner tool usher, it’s possible to tie multiple command sequences together. This is where we can control our deployment commands which include the blue-green switch. Of course the deployment steps do not have to be implemented in a tool like usher or be defined as a command sequence; however, a degree of automation is highly recommended! Alternative tools/frameworks like capistrano, rake, grunt, gulp, or just plain bash scripts can help out.
Regardless of implementation, the sequence of actions related to deploying with a blue-green switch can be described as:
- Determine the new colour to be deployed, default to something, if nothing is deployed already.
- Deploy a new application instance, while marking (e.g. tagging) it with this new colour.
- Run tests against the new instance.
- If the tests pass, change the active colour to be the new one.
Tooling
At Findmypast, we make use of tools as soon as we come across a step which could be automated. This approach certainly encourages us to not only develop a task runner like usher, but to go even further and abstract communication with various services. Since our state of the service colours is in a store in Consul, we rely on communicating with its API. This is exactly the case where the specific API calls (to read and manipulate key-value pairs) can be abstracted in a command line tool. Enter twoface.
This command line tool has a very simple job of looking up the opposite value colour to the one in Consul as well as changing it. This functionality is exposed by two commands: twoface peek and twoface flip. Given the two possible values, peek prints the opposite value to standard out and flip writes the opposite value to Consul’s key-value store. So replacing the steps in the sequence mentioned above with twoface usage:
NEW_COLOUR=$(twoface peek -H my.consul.com -k myservice/colour -b blue -g green)- Deploy a new application instance, while marking (e.g. tagging) it with this new colour.
- Run tests against the new instance.
twoface flip -H my.consul.com -k myservice/colour -b blue -g green
Finding the approach right for you
The aforementioned technologies are our personal choice which you do not have to agree with. You are free to implement blue-green development using your favorite approach, however one piece of advice on your journey would be to make this (and every) part of your deployment as transparent and as automated as you can. The reason for the twoface tool was to encapsulate communication with consul and make it reusable. But regardless of where or how you choose to keep track of your state, make sure it keeps your deployment streamline! The easier it is to add and maintain blue-green deployment to your service or application, the less overhead it will add while you get to reap the benefits of rapid and more reliable delivery.