Usher CLI: A Node.js CLI for stitching together command line interfaces

At Findmypast we needed a way to reduce the complexity of our deployment scripts. We built usher-cli to simplify and standardise our application deployment automation scripts.

Firstly let me take you back three years.

At Findmypast we stored all our release build configuration and deployment scripts in Teamcity. This was breaking one of the big rules of continuous delivery “Application code and app & system configuration all in source control”. Breaking that rule manifested itself negatively for us in two ways. Firstly our teamcity occasionally suffered outages which meant we couldn’t release. The second and worse problem was that anyone could change the configuration and there was no change management so configurations were regularly broken. This resulted in a lot of developer time being spent from within the Teamcity interface tweaking our configuration, trying to get our builds working again and unfortunately delivering no value to our customers.

From here we started looking more at scripting the various steps and maintaining those scripts within our project source repositories. We’ve scripted with powershell (too windows specific), ruby-rake (concurrency in ruby melted my brain) and laterly gulp (Node.js). Gulp was a great step forward because it took all the automation logic out of Teamcity and moved it to within the application’s codebases. Our engineers were now more easily able to test their changes to the automation and also have a full history of changes through version control.

There were some downsides to Gulp though. It was hard to test our scripts because we never took the time to fully abstract them away from Gulp itself. Between our various projects there was a lot of repetition of similar Gulp tasks and a few times we ended up in the situation that we had multiple different places to apply a fix when we found a bug. We broke out a few libraries to individual NPM packages but we never got the seams quite right so that they weren’t that reusable and transparent.

Recently we started a large re-architecture of our Findmypast product and moved over to Docker containers for hosting our applications and Consul for service discovery. At this point we thought about everything that had gone before and the tooling available to us with the Docker ecosystem. We decided that the best point of abstraction was to build individual command line applications in order to encapsulate various tasks that we needed to perform as part of our deployment automation.

We tried out a few tools. First we tried Ansible which we thought was great but a lot of our devs are running Windows as their OS meaning they couldn’t run Ansible without some elaborate VM setup. We thought about straight bash scripts but we liked the model of CI tools like CircleCi and Travis where there’s a file at the root of your repository that describes how to put your application into production. This also gives a nice standardisation across all our projects; if you want to know where the deployment commands are, a developer simply goes and looks in usher.yml. At this point we decided to build Usher.

Usher is built in Node and allows you to write your deployment scripts in yaml in a very similar way to CircleCi etc. We chose Node for a few reasons: it’s really easy to build small command line applications; we can make use of lots of other Node command line applications that are available in the ecosystem; finally a Node cli usually works quite well cross platform as it handles spawning child processes for the environment that you’re running under be that Windows, Unix or Mac.

Have a look at our code in github or the npm package.

Here’s an example of a small Usher file with a task for Elixir and Node unit tests and also an aggregate task to run all your unit tests. Usher supports full templating erb syntax, examples in our repository documentation.

elixir_unit_tests:
  - description: Run elixir unit tests
    cmd: docker-compose -f docker-compose/dev.yml run --rm --no-deps api mix test
    environment:
      - MIX_ENV=test

node_unit_tests:
- description: Run node unit tests
  cmd: docker-compose -f docker-compose/dev.yml run --rm --no-deps web npm test

unit_tests:
  - description: Run unit tests inside a container (with automatic build)
  - task: node_unit_tests
  - task: elixir_unit_tests

Then you can run commands like so

usher run node_unit_tests

For future features we’re looking at being able to do some form of file inheritance or inclusion so that you can import various standardised tasks and allow some form of centralisation of deployment configuration.

Sound off in the comments!