Focus on mean time to recovery

Traditionally operations groups look to improve the mean time between failures. While avoiding failures is obviously still important, lessons from cloud computing have taught us to expect failure and instead to focus on mean time to recovery. Continuous Delivery automation makes rolling out rapid fixes easier and we are also seeing a growth in monitoring techniques to spot failures quickly through a ‘production immune system’. Teams are also successfully using semantic monitoring and synthetic transactions to exercise production systems in non-destructive ways. This combined focus allows teams to move rapidly with higher confidence, it can also reduce the emphasis on expensive test-execution in pre-production environments and is particularly important in responding to the ever-growing list of security vulnerabilities that are being discovered.

In DevOps-savvy organizations delivery teams often configure production monitoring and respond to incidents themselves. This visibility and access into production environments allows those teams to make changes to their systems to improve their ability to recover quickly when something goes wrong. This focus on mean time to recovery improves quality of service overall, and allows teams to safely deploy more frequently. This can also reduce the emphasis on expensive test execution in non-production environments. Techniques we've used include end-to-end 'semantic monitoring' or reconciliation of real business transactions, and the injection of 'synthetic transactions' which exercise systems in non-destructive ways in production.

In previous radars we recommended arranging automated acceptance tests into longer journeys and, in what we call semantic monitoring, running these tests continuously against a production environment. We still believe that this is an important technique for scenarios the team can anticipate in advance. A variation of this approach, seen especially with startups, is to reduce the number of tests while increasing monitoring and automatic alarms. This shifts the focus from avoiding problems that can be anticipated to reducing mean time to recovery for all problems.