Enable javascript in your browser for better experience. Need to know to enable it? Go here.
/content/dam/thoughtworks/images/photography/banner-image/insights/in_banner_blogs.jpg

Avoid the release rush: a playbook for smooth production releases

Releasing a product or service is hardly straightforward. Even a brilliant team, good test coverage, solid continuous integration and a robust release pipeline cannot save you from the ‘release rush’ ( rush-like feelings and actions when closing in on the finish line for a product release). This happened to me quite often. Somehow releases always turned out to be more stressful than they needed to be. For instance, teams would start early and often work without break until midnight. 

 

Over time, to prevent this, we studied the system and devised a plan to improve our release process. This blog documents my learnings and is meant to be a practitioner’s guide that can help avoid the release rush.

 

 

Pre-release checks

 

Before a release, we’d recommend listing all the knowns, the unknowns and performing the following checks: 

 

  • Completion of the accepted features

  • Acceptance testing

  • Sign-off from the product owner and agreement on the release date

  • Approval from other stakeholders such as security, licensing, IT, etc.

  • Feature completion of upstream or downstream systems

  • Feature toggles to be enabled/disabled

  • Bug bash

  • Regression testing — manual or automated

  • Hotfix plan, if something goes wrong

  • NFR/CFR checks like performance, browser and platform compatibility, etc.

  • Appstore/Play Store publishing

  • Creation of a new environment

  • Backend and frontend release

  • Data upload

  • Third party licenses, renewals, contracts

  • Documentation — release notes, manual, known issues, etc

 

This list is not exhaustive but is a good starting point. For your project, you might need a different list or you might want to  evolve this to make it foolproof over time.

 

 

Post-production checks

 

In one of my projects, we thought the release was successful. Only to later realize that we had accidentally uploaded a previous version of the data. Such mistakes can also be avoided by following a thorough checklist. Some essential items to check are:

 

  • Core functionality, especially if the features are entirely new

  • Performance across different supported devices

  • Data integrity, if it involves SaaS or a major migration

  • Logs, in case of integrations with new upstream or downstream systems

  • Performance dashboards, if you are expecting heavy loads (like a booking system)

  • Other monitoring dashboards or Sonarqube

Automation

Automation is good, so long as you know exactly where to put the machine.
Eliyahu Goldratt
Renowned author and management consultant

To ensure consistency and reduce dependencies, automate anything that can be automated — whether it’s a complex process or a mundane task. Here are a few things you can automate: 

 

  • Build and pipeline

  • Deployment process

  • Testing — regression, sanity, performance, etc

  • Calendar reminders for renewals

  • Data upload

  • App publishing

  • Reminders for document publishing

  • Post-production sanity

  • Sonarqube

 

Release template

 

Document the checklists in a place that is accessible to everyone involved. Ideally, create a template and use it each time a release is scheduled. Documenting is a breeze in confluence; you can use wiki too. You can also group the list based on teams, tools, or processes for ease of use.

 

 

Backtracking dates

 

These days it is common for teams to schedule release dates even before deciding what is being released. Use the release template to create a timeline by which each task needs to be completed.

Project accountability

 

Set up a meeting with all the stakeholders involved with the release of the product and ensure:

 

  • Clear list of teams/people involved and corresponding dependencies

  • Understanding of timelines and events

  • Sharing of suggestions/objections/changes etc.

  • Discussion around possible RAIDs

 

Then, invite people to sign up for tasks on a copy of the original document/template. For instance, a product owner needs to know when to give the go-ahead. The IT team needs to know when to renew a license, create new servers/environments, etc. The security team needs to plan its final audit. And so on.

 

By the end of the meeting, you need two things: A date (of completion) and a person accountable. The person accountable can, of course, seek help from others. But it is essential that an individual is accountable to ensure the task gets done by the said date.

 

 

Following the plan

 

Set up the cadence for a periodic check-in on the plan. A meeting for every iteration is a good start, but keep a daily tab when you are in the last iteration of the release. The initial few meetings might be longer, but after a few iterations, it becomes quick. Even async communication modes like chat/emails could work. 

 

Use these periodic tabs to solidify the plan. Recalibrate dates/tasks as you need. Make this process as organic as possible.

Release 15, September 2021

 

Sl.no Task name Assigned to Date Completed

Code releases

 

1 Front end (web) and back end release Jaya (QA, Dev team) 14 September  
2 Andriod app release SPD (QA, Dev team) 10 September Done

Testing

 

3 Android app regression SPD (QA, Dev team) 9 September Done
4 Front end and back end regression Jaya (QA, Dev team) 12 September Delayed. ETA 13th.

Other dependent teams

 

5 Final security audit Simon (Security team lead) 8 September Done
6 Upstream service release Rajesh (QA, Upstream team) 5 September Done

Post release checks

 

7 Post release sanity Sneha (BA, Dev team) 14 September  
8 Monitor Dashboard Shakier (Dev, Dev team) 15 September  

A sample of a working plan. Please assume today’s date is 12, September 2021.

 

While doing this, keep the following in mind:

 

  • Whenever possible, add a buffer to your deadlines 

  • If you foresee a task going beyond the buffer date, let all your stakeholders know.

  • Setup a meeting to see what can be leveraged; for instance, you could cut back on a user story, if ‘feature completion’ is causing the delay 

  • Consider if the release can be postponed by a day or two — it is better to deliver a quality product late than a buggy one on time

 

Following the plan is fundamental to avoiding the release rush. It will stop you from ending up in a release rush or worse, uncover a high priority production issue after release. All cadence and plan follow-ups are to make sure you’re in a position to anticipate problems. 

 

From experience, I’ve learned that the best way to avoid last-minute anxiousness — or the release rush — is to have complete visibility and ongoing conversations. This practitioner’s guide is designed to help you track the release and encourage open discussions. Feel free to customize it to the needs of your projects and use it in ways that’s most helpful to your work.

Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.

Keep up to date with our latest insights