Technology

Deploying software regularly

Give feedback about this page

From:
Technology community (web operations)

You must release updates to your software regularly so you:

  • avoid making changes to one part of your software system that cause other parts to fail (these are known as ‘breaking changes’)
  • have constant feedback from users: this will help you design and build your system in an iterative manner
  • create a resilient and easy to change system
  • reduce the risk of creating vulnerabilities in your system

Your software release cycle should be based on how your team can support updates.

Some teams release weekly, some release daily and some release constantly throughout the day.

You’ll have to explain how you’re doing this at your service assessments.

Principles for deploying regularly

When deploying software you need to:

  • deploy little and often
  • deploy quality software
  • use auditable deployments
  • use zero downtime deployments where possible

Deploy little and often

You should deploy software often and with small changes, so that:

  • users benefit from new features or improvements quickly, instead of waiting months for an update
  • users can give you feedback quicker, allowing you to iterate your software to meet their needs
  • your team gets better at releasing because they do it often
  • you identify bugs and cyber security issues more easily because fewer things are changing at the same time: this means they’re cheaper to fix
  • you can react quickly to urgent fixes, such as security updates and last minute changes
  • your team can move faster through the Build-Measure-Learn loop needed for iterative development

You should measure how long it takes for code changes to hit production so you are aware of how regular your deployment cycle is.

Deploy quality software

To deploy quality software, make sure:

  • deployments are consistent across environments: this gives you confidence that your testing is always relevant
  • you automate the testing process of the software to routinely check for issues

Deployment should be a low-risk process. By the time you’re deploying a new version of software into production, you should be sure it’ll work.

To check the new version will work, you can:

Regular deployments which use the same tools and technologies repeatedly and reliably give you confidence that your deployment mechanism works. This is because you’re constantly testing deployment.

Use auditable deployments

To keep track of deployment history and get immediate feedback on your deployment, you should:

  • know which version of your service is running in each environment at all times
  • be able to trace changes back to the initial code commits in the source code repositories after a deployment hits production

If you combine this approach with small and frequent releases, you can:

  • narrow the source of any problems down to a small number of commits
  • roll back to a previous version more easily
  • confidently ‘roll forward’ with a code change to fix a production issue: you can be a lot more confident your fix will work if you’re making frequent releases

Aim for zero downtime deployment

Deployments that minimise or limit service disruption are known as zero downtime deployments.

These can take place during the normal business day, which means staff do not have to work unsociable hours.

Zero downtime deployments can:

  • increase user satisfaction
  • lower service operation costs: release windows are often in unsociable hours, and require hands-on skills from operations staff
  • prevent difficulties when negotiating emergency releases - for example, for security or critical issues

You may think that setting up your application to do zero downtime deployments involves lots of work, but downtime can cost you money. You should find out the cost of downtime for your service and decide if you can accept that cost.

Find more information and resources about deployment.

How to deploy regularly

It’s easier to deploy software regularly if you:

  • build a single artifact rather than variations for different environments
  • have multiple deployment environments
  • manage variable configuration
  • secure passwords and keys
  • use smoke tests: software tests that check if the most important functions are working

Building a single artifact

An artifact can be:

  • the compiled binary and its dependencies
  • a .jar file for JVM languages
  • a compressed archive of the source: not including compilation artifacts
  • an entire virtual machine image with the application pre-deployed

You should try to build a single application artifact that’s deployed into each environment, from development machines through test into production.

Having the same application artifact deployed everywhere means that you:

  • can be confident the artifact is tested in the same way as in production
  • have a single artifact to represent a deployment, meaning it’s auditable and easy to roll back or upgrade

Teams often add debugging symbols, optimisations or test code solely to their pre-production environments. This makes it hard to have confidence in the artifact that will go to production.

Managing artifacts

You should keep all your artifacts in a central repository. Having one place for your artifacts allows you to:

  • give your team access to a shared set of artifacts
  • promote your artifacts across environments
  • rollback or upgrade more easily

You’ll need to host your repository yourself or choose a service to host it for you. For example, some teams at GDS use Aptly or Nexus Repository OSS.

Multiple deployment environments

You should have multiple deployment environments so you can phase the rollout of your software and ensure it’s adequately tested.

At the very least you need a:

  • development environment running the latest version of the software
  • production environment hosting live users

You may also have a staging environment prior to production, or environments dedicated to:

  • exploratory testing
  • user testing
  • performance testing
  • security testing

Using a pipeline

You may choose to manage the deployment of your software to your environments using a pipeline.

This will make sure a version of software is not deployed to a later environment before it has been deployed and tested at an earlier stage.

For example, GOV.UK deploys to its staging environment first so the development team can be confident the software does what it should before it’s deployed to production.

Deciding which environments to deploy

You also need to decide which of your environments you must deploy to in order to have confidence in your product. Emergency fixes can then skip unnecessary stages that are not part of the normal deployment pipeline so that development is not held up.

For example, a product owner should not have to sign off a security fix in a user acceptance environment.

The order of your environments does not have to be strictly linear - you can run some sets of software tests in parallel, for example user acceptance tests and performance tests.

However, you may find there’s a single production environment that’s later than all others and a single entry point that precedes all others.

Managing variable configuration

To build a single artifact, your deployment mechanism should provide a way to inject configuration that varies between environments, like URLs of dependent services. For example, passwords and locations for external services like databases.

Your application should follow the 12 factor principles so that you can inject specific configuration for your environments.

Securing passwords and keys

To secure your software, you must take extra care when managing database passwords or TLS (Transport Layer Security) private keys. Make sure:

  • confidential information cannot be accessed outside of the environment it’s intended for
  • confidential information is known only by the machines that need to know it

For example, in a 3-tier app with database, application and web servers:

  • the database server does not need to know the TLS private keys for the site
  • the web server does not need to know the database credentials

Learn more about Secure by Design principles for government services.

Using smoke tests after you deploy

Once you’ve deployed your software, you should check whether it’s working as expected using a ‘smoke test’.

If it’s not working, you can cancel or roll back the deployment.

A good smoke test should:

If your software needs a database to be present to operate effectively, your smoke test should exercise an application code path that will fail if the database is not present or returns an error.

If a smoke test fails

You should plan what to do if a smoke test fails - the simplest option is to manually roll back to a previous version of the application.

You could also have a system that automatically detects smoke test failures and cancels the deployment, or rolls back to the previous version.

Ideally the system you have will not add the updated application to the production load balancer until it has been smoke tested and verified (if the application fails the smoke test, it’s discarded).

This ensures no rollback is necessary, and no interruption in service happens. This works particularly well with the immutable server pattern.

Get help

Get in touch with the web operations community directly to:

  • discuss this page
  • share ideas across government
  • find support from teams that have worked on similar services

You may also find the Uptime and availability guide useful.

From:
Technology community (web operations)
Last updated
29 days ago
Last update:

Integrated elements on Discovering vulnerabilities and Implementing a vulnerability management process.

  1. Changed reference from SSL to TLS as this was out of date.

  2. Restructured bullet points and added guidance on managing artifacts in response to recent feedback in user testing.

  3. Guidance first published