Cloud
How We Moved to the Cloud
Cloud and Platform-as-a-Service (PaaS) offerings can provide significant business advantages when deployed in a way that accelerates application delivery. However, placing the decision-making authority of cloud purchases solely in the infrastructure department often achieves incremental efficiencies at the cost of long-term business results. The greatest benefit of these technologies comes from the organizational changes they allow.
Many of these organizations—including Microsoft—have woken up to the dangers and are now using their financial clout to embrace the next disruption. But this doesn’t guarantee success. The sheer size and complexity of large, successful organizations make this kind of change hard.
Those who have done it most successfully have done so by focusing on the technological underpinnings of a delivery platform independently of innovation itself. Amazon was one of the most successful eCommerce sites in the world when it introduced Amazon Web Services (AWS) in 2006 as a completely unrelated infrastructure offering. AWS emerged from the infrastructure department’s desire for decentralization to better support internal development. It’s now Amazon’s most profitable division.
Netflix disrupted Blockbuster using a fairly low-tech approach: mailing movies to customers instead of trying to attract customers to stores. Netflix built a delivery platform that has spurred repeated innovations through a laser focus on technical agility. This included introducing streaming movies, despite the disruption to its own business model. Netflix architects credit much of their technical dexterity to the migration to the AWS cloud ecosystem in 2010.
The researchers confirmed the standard expectation that companies with high alignment between business and IT do quite well in the presence of an effective IT organization, with an average of growth rate of 35% over three years and reduced overall IT spending. However, high alignment proved disastrous in the absence of a reliable engine of execution in IT, with a negative top-line growth and significantly more IT spending. The data strongly suggest that focusing on IT effectiveness should take priority over focusing on business alignment.
Forrester’s analysis is backed up with top-line business results. For the past few years, Puppet Labs has used surveys backed with statistical analysis to look for correlations between application delivery approaches and various technology and business results. Their 2015 State of Devops Report showed that “high-performing IT organizations deploy 30x more frequently with 200x shorter lead times; they have 60x fewer failures and recover 168x faster”.
That’s a stark contrast to McKinsey’s popular two-speed IT advice that you have to choose between speed and stability. The Puppet Labs report found no statistical significance to differences in results between greenfield systems of engagement and back-end systems of record (including COTS packages), further challenging McKinsey’s assumptions. The previous year’s report showed even more eye-opening top-line business correlations: “firms with high-performing IT organizations were twice as likely to exceed their profitability, market share, and productivity goals”. This reinforces the results seen in the effective-IT quadrants of the Alignment Trap, but provides additional insights into the tradeoffs we make to get there.
Based on such data and its own analysis, Forrester recommends abandoning the idea of two-speed IT. It argues that DevOps, combined with loosely coupled architectures and cross-functional organizational structures, are the keys for improving both the speed and quality of delivery. These are the building blocks of a solid execution engine for innovation. These are the principles of your delivery platform.
Far too many organizations focus on cost optimizations, which ensures low IT effectiveness despite their best efforts. Business and technology leaders need to start by optimizing for speed and quality first, and cost second, if they want to improve the performance of technology. Lost opportunity costs you more than increased unit costs of delivery activities. This simple tradeoff has important implications on how we define our cloud strategy in our delivery platform.
Remember the key principles of your delivery platform: DevOps, loosely coupled architectures, and cross-functional teams. The primary reason so many organizations find it challenging to embrace these principles is because, as Conway’s Law predicts, their communication structures—such as the ticketing system used for infrastructure provisioning—constrain them to solving yesterday’s problems when IT was seen as simply a cost center.
It’s not pay-for-use that makes AWS revolutionary: it’s transforming ticket-based infrastructure support to development self-service. By exposing APIs for developers to provision their own virtual infrastructure, the cloud supports a fundamental re-routing of the communication structures of an organization.
By redefining the boundary between infrastructure and development teams, infrastructure-as-a-service (IaaS) and PaaS support cross-functional teams at scale. The traditional approach of application teams creating tickets to shared services for infrastructure created artificial silos between infrastructure and application.
Monitoring, for example, was often considered an infrastructural concern, and database schema design was often a battleground between the DBAs who owned the operations of the databases and the application teams that used them. These silos significantly slowed down application delivery, as now, per Conway’s Law, the application itself has had to adjust to the increased complexity of the communication structures of the organization.
Unfortunately, as long as we look at software delivery through traditional organizational structures, such dysfunctions remain the inevitable outcomes, and our application will spend more time waiting to be released than in development. Layering a PaaS into your IT department while maintaining the same organizational silos is completely missing the point.
At the most fundamental level, it is not cloud or a PaaS that unleashes your delivery platform potential; it is the organizational restructuring those technologies unlock. Reading ACM’s interview with Amazon’s CTO Werner Vogels from 2006 (the same year AWS was launched) reinforces this point. Many low-performing IT organizations would relate to the problems Vogels describes in the early days of Amazon’s explosive growth.
It’s clear that many of the transformative changes Amazon went through had one key goal: fundamentally restructuring the organization to allow more ownership and isolation to unlock delivery potential. The famous Amazon push for service orientation defined a cross-functional ownership model that is still controversial ten years later.
"The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service" Werner Vogels, 2006.
That ownership model—reinforced by cloud-based self-provisioning—provided development teams tremendous flexibility. As Vogels said: “Developers of our services can use any tools they see fit to build their services. Developers themselves know best which tools make them most productive and which tools are right for the job.”
It is this mindset that birthed the cloud.
To truly embrace cloud technologies is to also embrace radical shifts in divisions of labor within your IT department. In large organizations, this will never happen by default. It requires executive intervention and the willingness to fight turf wars and break up managerial fiefdoms. This requires both courage and vision, as well as the persistence to continue after early missteps.
By fundamentally changing the role of infrastructure in application delivery, the cloud offers us a case study of disintermediation through technology, but in this case it is disintermediation of departments within your organization. Picking a cloud offering, and a platform-as-a-service on top, can provide just the technical levers needed to build the foundational elements of a delivery platform, but only if you’re willing to change your organization in the process.
To learn more, visit our Digital Platform Strategy page.
Disintermediation through technology is one of the great themes of our times. Even historically "safe" markets are being disrupted, from taxis (Uber) to banks (Bitcoin) to cable companies (Netflix) to auto dealerships and gas stations (Tesla). Only 61 companies on the Fortune 500 from 60 years ago remain on it today. Software is eating the world, and changing consumer expectations. No company is immune from these changes.
Many of these organizations—including Microsoft—have woken up to the dangers and are now using their financial clout to embrace the next disruption. But this doesn’t guarantee success. The sheer size and complexity of large, successful organizations make this kind of change hard.
Those who have done it most successfully have done so by focusing on the technological underpinnings of a delivery platform independently of innovation itself. Amazon was one of the most successful eCommerce sites in the world when it introduced Amazon Web Services (AWS) in 2006 as a completely unrelated infrastructure offering. AWS emerged from the infrastructure department’s desire for decentralization to better support internal development. It’s now Amazon’s most profitable division.
Netflix disrupted Blockbuster using a fairly low-tech approach: mailing movies to customers instead of trying to attract customers to stores. Netflix built a delivery platform that has spurred repeated innovations through a laser focus on technical agility. This included introducing streaming movies, despite the disruption to its own business model. Netflix architects credit much of their technical dexterity to the migration to the AWS cloud ecosystem in 2010.
Determining the technology approach to deliver innovation is a central concern for modern business executives.
One of the most common approaches is to align IT and business initiatives. While this is indeed a worthwhile goal, an MIT Sloan Management Review article called Avoiding the Alignment Trap in IT shows the unexpected consequences of such an approach taken naively.The researchers confirmed the standard expectation that companies with high alignment between business and IT do quite well in the presence of an effective IT organization, with an average of growth rate of 35% over three years and reduced overall IT spending. However, high alignment proved disastrous in the absence of a reliable engine of execution in IT, with a negative top-line growth and significantly more IT spending. The data strongly suggest that focusing on IT effectiveness should take priority over focusing on business alignment.
Figure 1: Typical impacts of the alignment trap
Building a strong delivery capability is therefore as much a business concern as it is an IT concern. As Amazon and Netflix have demonstrated, the cloud plays an important role, but before defining how the cloud fits into your technology strategy, it’s important first to define the tradeoffs successful platforms make.Optimize for Speed Over Cost
Forrester wrote about these tradeoffs in a report called Faster Software Delivery Will Accelerate Digital Transformation, starting with the advice to “go fast or go home.” The authors point out that “application delivery capability has now become the essential enabler of an organization’s digital business strategy".Forrester’s analysis is backed up with top-line business results. For the past few years, Puppet Labs has used surveys backed with statistical analysis to look for correlations between application delivery approaches and various technology and business results. Their 2015 State of Devops Report showed that “high-performing IT organizations deploy 30x more frequently with 200x shorter lead times; they have 60x fewer failures and recover 168x faster”.
That’s a stark contrast to McKinsey’s popular two-speed IT advice that you have to choose between speed and stability. The Puppet Labs report found no statistical significance to differences in results between greenfield systems of engagement and back-end systems of record (including COTS packages), further challenging McKinsey’s assumptions. The previous year’s report showed even more eye-opening top-line business correlations: “firms with high-performing IT organizations were twice as likely to exceed their profitability, market share, and productivity goals”. This reinforces the results seen in the effective-IT quadrants of the Alignment Trap, but provides additional insights into the tradeoffs we make to get there.
Based on such data and its own analysis, Forrester recommends abandoning the idea of two-speed IT. It argues that DevOps, combined with loosely coupled architectures and cross-functional organizational structures, are the keys for improving both the speed and quality of delivery. These are the building blocks of a solid execution engine for innovation. These are the principles of your delivery platform.
Far too many organizations focus on cost optimizations, which ensures low IT effectiveness despite their best efforts. Business and technology leaders need to start by optimizing for speed and quality first, and cost second, if they want to improve the performance of technology. Lost opportunity costs you more than increased unit costs of delivery activities. This simple tradeoff has important implications on how we define our cloud strategy in our delivery platform.
Your Delivery Platform Requires A Different Organization
Nearly 50 years ago, Melvin Conway wrote that “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations". Now known as Conway’s Law, this insight dictates how we should allocate responsibility to increase our application delivery capability, and the role that cloud and PaaS offerings play.Remember the key principles of your delivery platform: DevOps, loosely coupled architectures, and cross-functional teams. The primary reason so many organizations find it challenging to embrace these principles is because, as Conway’s Law predicts, their communication structures—such as the ticketing system used for infrastructure provisioning—constrain them to solving yesterday’s problems when IT was seen as simply a cost center.
It’s not pay-for-use that makes AWS revolutionary: it’s transforming ticket-based infrastructure support to development self-service. By exposing APIs for developers to provision their own virtual infrastructure, the cloud supports a fundamental re-routing of the communication structures of an organization.
By redefining the boundary between infrastructure and development teams, infrastructure-as-a-service (IaaS) and PaaS support cross-functional teams at scale. The traditional approach of application teams creating tickets to shared services for infrastructure created artificial silos between infrastructure and application.
Monitoring, for example, was often considered an infrastructural concern, and database schema design was often a battleground between the DBAs who owned the operations of the databases and the application teams that used them. These silos significantly slowed down application delivery, as now, per Conway’s Law, the application itself has had to adjust to the increased complexity of the communication structures of the organization.
Unfortunately, as long as we look at software delivery through traditional organizational structures, such dysfunctions remain the inevitable outcomes, and our application will spend more time waiting to be released than in development. Layering a PaaS into your IT department while maintaining the same organizational silos is completely missing the point.
At the most fundamental level, it is not cloud or a PaaS that unleashes your delivery platform potential; it is the organizational restructuring those technologies unlock. Reading ACM’s interview with Amazon’s CTO Werner Vogels from 2006 (the same year AWS was launched) reinforces this point. Many low-performing IT organizations would relate to the problems Vogels describes in the early days of Amazon’s explosive growth.
It’s clear that many of the transformative changes Amazon went through had one key goal: fundamentally restructuring the organization to allow more ownership and isolation to unlock delivery potential. The famous Amazon push for service orientation defined a cross-functional ownership model that is still controversial ten years later.
"The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service" Werner Vogels, 2006.
That ownership model—reinforced by cloud-based self-provisioning—provided development teams tremendous flexibility. As Vogels said: “Developers of our services can use any tools they see fit to build their services. Developers themselves know best which tools make them most productive and which tools are right for the job.”
It is this mindset that birthed the cloud.
Use The Cloud To Change Your Organization
It’s vital to keep this point in mind during the selection process of a private cloud or a PaaS. Often, this decision is handed to one department (usually infrastructure) to evaluate the offerings on the market. This treats the cloud as just another product, no different than the latest hypervisor or orchestration tool, and all but ensures minimal changes to existing ownership structures. Placing the decision in the hands of your development organization has the advantage of coming at the decision from a customer-centric viewpoint. Even so, this will still result in a suboptimal choice if viewed through existing organizational boundaries.To truly embrace cloud technologies is to also embrace radical shifts in divisions of labor within your IT department. In large organizations, this will never happen by default. It requires executive intervention and the willingness to fight turf wars and break up managerial fiefdoms. This requires both courage and vision, as well as the persistence to continue after early missteps.
By fundamentally changing the role of infrastructure in application delivery, the cloud offers us a case study of disintermediation through technology, but in this case it is disintermediation of departments within your organization. Picking a cloud offering, and a platform-as-a-service on top, can provide just the technical levers needed to build the foundational elements of a delivery platform, but only if you’re willing to change your organization in the process.
To learn more, visit our Digital Platform Strategy page.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
