DevOps at Findmypast: How Spanners implement DevOps Practices

Introduction

There are many ways to define DevOps and even more ways to implement its core principles within an organisation. To the Spanners team, DevOps is an iterative and collaborative approach to the development, testing, deployment and monitoring of software services. Overall, there is a great deal of alignment with the thoughts outlined by Mueller and Willis regarding what DevOps is, and how it is best implemented.

Spanners see our role within the wider engineering cohort at Findmypast as advocates for DevOps principles and practices, and the vehicle through which those practices can be implemented day-to-day. Spanners operate as an autonomous team, but our efforts centre around providing feature teams with the processes and tools they need to reap the rewards of following a DevOps approach when developing and maintaining services.

Source: Atlassian

Team Goals

Our goals always trace back to wider organisation goals that in turn align with the expected company and department behaviours. For Spanners our quarterly team goals generally focus on how we improve five key areas in terms of software delivery across the engineering teams:

Speed of Deployment – The rate of code deployments and the speed with which code can be built, tested and deployed to our environments.
Collaboration – Enable teams to investigate issues and monitor services and systems using a well-defined and trusted series of tools. Providing a framework for teams to take Ownership of the service pipelines (sometimes described as “You build it, you run it”).
Quality – Enable teams to continually improve upon their service/s through automation in the form of CI/CD pipelines, and quickly identify issues and shortcomings in those implementations.
Reliability – Reduce variables when it comes to the building, testing and deployment of code changes, and provide near real-time feedback (measurement) of the effectiveness of those code changes.
Security – We strive to manage everything as code and provide the tools and means for all engineers to do their jobs with minimal friction, building security and accountability into everything we deliver.

The Team

The Spanners team comprises eight engineers, supporting a wider engineering cohort of roughly 120 engineers and management. Backgrounds within the team vary, some members having more of an operations background, whilst others have more of a development background. The Spanners team is truly multi-disciplinary, with a variety of skill levels ranging from senior to junior, and a variety of specialisms within the team. This breadth of knowledge enables Spanners to provide a one-stop-shop for support of the product build pipelines and underpinning technologies.

Spanners operate like any other feature team within the wider engineering group. We follow an agile methodology built around a fortnightly sprint cycle, and follow a regular cadence of planning, refining and retrospective meetings.

To ensure queries from the wider engineering group are answered in good time, we operate a support rota, whereby a team member is assigned as an initial point-of-contact for queries. This ensures that the team is always on hand to collaborate with a feature team on a request or an incident.

The Challenge

One underappreciated aspect of software development is the need to integrate new team members into the engineering cohort. Without a smooth on-boarding process both the new and existing team members can become bogged down in the configuration of systems, tools and certificates. Ultimately, this results in a negative impact to the speed of deployment, reliability and security key areas that we as a team are constantly striving to improve.

The Solution

From our initial investigations we identified that all systems that teams were interfacing with during the on/off-boarding process could be interacted with using RESTful APIs, so the obvious solution was to wrap these interactions in a higher-level API provided by a single service.

A sample of the systems and tools we configure for new starters

The catalyst for prioritising the development of the service was a rapid growth in the engineering team cohort, which included new engineers joining the Spanners team. This meant that we began to see first-hand the shortcomings in the existing processes, the pressure this placed upon feature teams, and the detrimental effects for new starters.

Working in sprint cycles meant we had allocated time to define, refine and plan for tickets that would deliver upon the prescribed solution to reduce friction for the on-boarding and off-boarding processes. Quickly T-shirt sizing the initial tickets we could see this would be more than just a trivial task, and as such we defined an Epic, and made it our quarterly goal.

We started by engaging stakeholders responsible for the on/off-boarding processes, and development got underway at the beginning of the next quarter. Within four weeks we had our minimal viable product (MVP) ready for demonstration to stakeholders.

Service Swagger Interface

Scaffolding up the service, repository and build pipeline was very straightforward thanks to a pre-existing service developed by Spanners. This service bootstraps a microservice from nothing but a name to a fully deployable project, interfacing with Github and Teamcity (our CI/CD solution) to define a repository, populate it with templates and define a build pipeline. This scaffolding service is another example of how Spanners have applied DevOps principles and practices to find a solution to a business problem.

The on/off-boarding MVP configured access to several of our systems (Jira, VictorOps, Launch Darkly, Github and Teamcity), as well as configured the user’s device, installing applications and CLI tools (VS Code, NPM, docker etc.) required for them to do their job on either Linux, Mac or Windows OS.

installHelm() { 
cd /tmp 
# Check for installed version of Helm 
CUR_HELM_VER=`/usr/local/bin/helm version 2> /dev/null | grep "Version:" | cut -d "," -f 1 | cut -d ":" -f 2 | sed 's/"//g'` 
if [[ $CUR_HELM_VER == $HELM_VER ]] 
then 
log "Helm already installed, and at the correct version." 
else 
log "Installing helm: ${HELM_VER}" 
curl -LO https://get.helm.sh/helm-${HELM_VER}-linux-amd64.tar.gz 
tar -zxf helm-${HELM_VER}-linux-amd64.tar.gz 

# Install Helm executable  
sudo mv ./linux-amd64/helm /usr/local/bin 
log "Helm successfully installed to version ${HELM_VER}." 
fi 
} 

Extract of our Linux (Ubuntu) setup script

It also fulfilled the off-boarding requirement, enabling administrators to clean-up user accounts when an engineer left the company.

In a commitment to “shift left”, most of the service codebase was developed with unit and integration tests from the very start. Unit tests are run using the jest framework, and triggered locally on a pre-push hook, whilst integration tests are run on each and every build. All build pipeline configuration to run tests is pre-configured in our build pipelines, and all of it bootstrapped by our scaffolding service.

As the services need to interface with systems (both internal and external) at an administrative level securing the credentials to interface with these systems is of course required. This was not much of an issue to implement though, as we already operate an on-premise Vault cluster and our scaffolded build pipelines have pre-defined secure methods to interact with this Vault instance.

Spanners regularly deliver workshop sessions to the Engineering cohort in which we take a deep dive in to recent or upcoming changes to processes and technology. Following completion of the MVP we delivered one such workshop session regarding the new service. The feedback gathered resulted in several tickets being defined, and as they were executed incremental improvements were made to the scripts and endpoints of the service. Due to changes being applied directly to the trunk branch, and test automation in the build pipeline, each change could be verified by stakeholders, and benefits realised immediately.

module.exports.createAccount = async ({ username }) => {
  const response = await callApi(
    `members`,
    'POST',
    JSON.stringify([
      {
        email: `${username}${fmpEmail}`,
        role: 'writer',
      },
    ])
  );

  if (![200, 201].includes(response.status)) {
    const error = await response.text();
    throw new StaffAPIError({
      message: `Failed to setup new joiner ${username} with access to Launch Darkly`,
      error,
      statusCode: response.status,
    });
  } else {
    logger.info(`A Launch Darkly account for new joiner ${username} has been made successfully!`);
  }
};

Example on-boarding API call

The service was shortly thereafter demonstrated to the wider engineering team, and flagged as the replacement to previous on/off-boarding procedures. Documentation throughout our codebase was updated to signpost the new service as the one-stop-shop for on/off-boarding staff.

Migration to use of the script was very smooth, due to the engineering cohort now being able to self-serve on/off-boarding requests.

What are Spanners working on now?

Working on the service opened the teams eyes to security loopholes and blind spots within our environment, which put us on the path to improving the role based access controls (RBAC) provided to the engineering cohort. We are in the process of improving security within our network by implementing RBAC for engineer access to servers (via SSH & RDP) and Kubernetes resources.

Conclusion

I hope the above has given you an idea of how Spanners implement DevOps principles within the wider Engineering cohort at Findmypast. The values held by the company and the behaviours expected from staff, lay a great foundation for the adoption of DevOps practices by the entire Engineering cohort. Behavioural expectations at the organisational level such as “We use data to fail fast and learn how to do it differently next time”, and “We share our knowledge to empower the people around us”, really resonate with the DevOps principles of continuous improvement and collaboration respectively. In practice these principles mean that new engineers very easily adopt DevOps practices. Yielding improved outcomes for both the individual and the business, with teams acting autonomously to experiment and deliver product enhancements at a rapid pace.

We may be looking for engineers to join our team. If you’re interested contact us, or check out our current vacancies.