Releasing changes
Releasing changes means getting new features and fixes onto the live service, safely and often. The surprising part, for anyone who finds releases nerve-wracking, is that releasing small and often is safer than saving everything up for a big launch. Releasing changes pulls together four habits: release in small, frequent batches; run every change through an automated pipeline that tests it before it goes out; roll each change out gradually and watch how it behaves; and keep the ability to roll back quickly if something goes wrong. The people side of the same change is change management.
THE CORE OF RELEASING CHANGES
What good looks like
Changes go out in small, frequent batches, rather than being saved up for big, risky releases.
Every change runs through an automated pipeline that builds and tests it before it can be released.
Each release is tested before it reaches the public, and a person gives the final go-ahead on the releases that need one.
New changes are rolled out gradually, to a small share of users first, watched, and continued only if they behave.
A release can be rolled back quickly when something goes wrong, so a bad change is a short problem rather than a lasting one.
Before deploying to the cloud, the Government of Canada's security guardrails are in place.
The team tracks whether releasing is healthy: how often it releases, how long a change takes to reach users, how often a release breaks something, and how fast it recovers.
Why it matters
Big, infrequent releases are where risk collects. When months of changes go out at once, there is more that can break, it is harder to tell which change caused a problem, and users wait a long time for fixes and improvements. Releasing in small batches turns that around: each change is small enough to test, to understand, and to undo, so a problem is caught early and contained. This is the Government of Canada's expectation, its standard is to iterate and improve frequently, releasing small batches and building in automated testing so new changes do not introduce new problems. The evidence backs it up: teams that release often and recover fast are also the ones that break things least, so speed and stability come together rather than trading off. For a service in the cloud, releasing also rides on a baseline of security: the GC Cloud Guardrails must be implemented, validated, and reported within the first 30 business days of getting a cloud account.
Whose job it is
Releasing changes is shared across the team, with each role holding a different part:
- Developers keep each change small, write the automated tests, and build and maintain the pipeline that delivers them.
- Operations and release engineers run the deployments, watch each rollout, and roll back when something misbehaves.
- Security specialists make sure the security checks and guardrails ride the pipeline and that patches go out promptly.
- The business owner of the application makes sure releasing is frequent and funded, accepts that small frequent releases are the safer path, and gives the go-ahead on releases that need a sign-off.
A closer look
Comparison
Two ways to release changes
Pax
Meet Pax, a service manager. They ran the tax-filing service on small, frequent releases:
- released small changes every week through an automated pipeline that tested each one
- rolled each change out to a small share of filers first, watched it, then expanded
- could roll back to the last good version in minutes
The result: the few problems that slipped through hit a handful of users, were spotted fast, and were undone before most people noticed, and filing stayed up through the busy season.
What Releasing changes looks like in each phase
How a service releases changes shifts across its life.
The release habits are set before launch. The team builds the pipeline and the automated tests, decides how it will deploy and roll back, and, for a cloud service, puts the GC security guardrails in place and validates them within the first 30 business days. Choosing to release small and often from the start is far easier than retrofitting it onto a service built for big launches.
The official instruments behind releasing changes
Everything official this subject brings with it, and where in a service's life each one comes up. The full detail, including who does the work and what the business owner personally does, is in the table on the home page.
The service team's own recovery arrangements: how this system gets back up, in what order its parts are restored, and proof from testing that the restore actually works. This is the part the team owns, as distinct from the departmental business continuity plan, which is the department's.
- BetaFill
- StabilizationKeep current
- MaturityKeep current
Further reading
See also
Assumptions this page makes
You are already working to the Government of Canada Digital Standards, design with users, iterate and improve frequently, work in the open, use open standards, address security and privacy, build in accessibility, empower staff, be good data stewards, design ethical services, and collaborate widely, and to the law on privacy, security, official languages, and accessibility. The standards say how the government works in the digital world. The six Government of Canada digital competencies say what every public servant has to be able to do to work that way, and the team page covers them. This guide builds on those.