The role of IT in the business world has changed dramatically over the past decades. New technologies and techniques allow enterprises to get much more out of IT, while at the same time increasingly sophisticated business models have pushed IT to investigate and deliver novel solutions. These may be based on new technology; however, larger breakthroughs require both new technologies, as well as a rethinking of how people are organized to make use of the technology.
Agile software development led the way, and now the DevOps and DesignOps movements are hitting the mainstream. In my opinion, IT in businesses is now entirely a team activity, and while there is, as ever, a need for experts with deep technical knowledge, we must focus our attention on how we get people from all disciplines working together effectively.
Technology in a supporting role
In the early days of IT in business, the 1960s, IT mostly played a supporting role. IT systems were usually introduced to make existing processes faster or to reduce the cost of executing a given process. The focus was on the bottom line, and the goal was to improve operational efficiency.
A major shift came with the PC. It was more powerful than green-screen terminals, and it found its way onto more desks. As a result, new areas of the business were supported by IT, including areas where no off-the-shelf software existed. So, in response, enterprises started asking their internal IT departments to build custom applications.
There was obviously some collaboration between the people commissioning and using the IT systems — often simply referred to as the business — and the people building the systems, the IT department. However, IT was still considered a tool, and IT departments were treated as a cost center. This led organizations to a project-centric view of IT and to introducing a clear separation between development and operations. The project-centric approach created a culture in which the ultimate goal became delivery of a fixed set of requirements, on-time, and on-budget. IT departments created more formal and elaborate processes to prove that they held up their end of the bargain, and I’ve seen more than a few cases where that was deemed more important than creating actual value for the organization as a whole.
The PC brought another innovation: office packages, including word processors and spreadsheets. Through macros, embedded programming capabilities and integration mechanisms, these packages became a powerful platform that the business side used to implement complex solutions on their own. One of the key drivers for this was that the IT departments had come up with the rigid processes mentioned above, effectively discouraging the business side to interact with them, when actually, the business needed closer collaboration.
Building systems outside the IT department are so common that a term was coined for it: shadow IT. Unfortunately, while improving collaboration, shadow IT is also expensive; studies have found that 30–50% of the IT spend can be on shadow IT, and it introduces significant business risk.
Technology becomes the business
Closer collaboration gave some business people an important insight: while they could treat IT as a cost center, they could, instead, also view IT as an enabler. When used well, the technology could give their organization a competitive advantage in the market, not because they could deliver existing products more efficiently, but because they could offer something new. They had hit on the concept of technology-driven differentiation. They were less interested in figuring out how to reduce the cost of delivering an IT project and became more interested in finding out how much money they could make with a new technology-led product.
With the rise of the dotcom economy, this trend continued, and companies sprung up for whom technology was the business. They didn’t differentiate their products through the use of technology; they only existed because of technology.
Figure 1. The changing role of technology
For companies that adopted a view of IT as an enabler, the then-dominant project-focused view of building software did not work well. Some organizations experimented with different approaches, culminating in the Manifesto for Agile Software Development in 2001.
From personal experience, I can say that one industry that saw the new potential of IT early was investment banking. In trading especially, technology became a key differentiator. As a response, many front-office groups adopted agile software development in the early 2000s.
Another industry that visibly used a collaborative and product-focused approach were the start-ups of the dotcom boom. Certainly, the dotcom crash, as well as stories about haphazard engineering and operations practices, did much to discredit the working mode of these start-ups, but in hindsight, it seems obvious that they’d been onto something.
Cycle time is key
When an organization relies on technology to differentiate itself or when technology is at the core of its business, cycle time becomes key. The quicker an organization can take a new business idea and turn it into software running in production, the better.
Previously, organizations had focused mostly on two factors: on reliability, in the sense of a reliable process to turn business requirements into working software; and on throughput, that is delivering the maximum number of features with a given development capacity. Analogous to the saying "never underestimate the bandwidth of a station wagon loaded with tape", high throughput isn’t linked to short cycle time and a lot of organizations had improved, or attempted to improve, reliability and throughput at the expense of cycle time.
Conversely, organizations who focus on cycle time cannot simply ignore reliability and throughput, which means that they have to come up with an approach that delivers all three. Close collaboration lies at the heart of the solution.
Agile software development promotes collaboration and introduces short feedback cycles on multiple levels. By releasing a version of the software that’s still incomplete but already usable, real user/customer feedback becomes available. This feedback is then used to direct the development effort, which increases the reliability of the process to deliver what is needed. Throughput is increased, too, because little to no effort is put into unneeded features. This concept is often described as “waste minimization”, a term borrowed from lean manufacturing.
Businesses that have technology at their core often don’t know with certainty what they need to deliver next in order to stay ahead of their competition. Product managers merely have hypotheses that need validation. In this situation, the effect of fast feedback is even more pronounced. Release cycles common in project-centric organizations, measured in quarters or even years, would be prohibitively long.
More than agile software development
Following many implementations of agile development approaches, around 2008, it became apparent that there were shortcomings on both ends of the software development process that still had to be addressed in order to reduce cycle time. On one end, having well-tested, continuously-integrated software in version control doesn’t help the business. The software has to be in production, and the path to production must be short and free of obstacles, which it usually is not. On the other end, business people often cannot describe the requirements for the systems they need to the development teams, whether they are agile or not.
From the internet start-ups, some of whom have evolved into giants, we've learned to shift from project-focused to product-focused thinking. Rather than separating the lifecycle of software into distinct phases, with separate teams for each, the team boundaries are redrawn, bundling responsibility for the continued evolution of software and its operation in one team; an approach that Werner Vogels, CTO of Amazon, famously summarized as "you build it, you run it". These cross-functional teams are normally kept small, around eight to 12 people, so that they can maintain a focus on a specific aspect of the overall product. Spotify was one of the early companies to implement these concepts, also exploring strategies to scale them [pdf].
While this approach was relatively easy to follow in start-ups, larger enterprises struggled to transition to this new world. It had been difficult to learn agile development. Replacing teams grouped by functional specializations such as databases, frontends or backends with small, cross-functional teams caused further friction and resistance. And bringing together development and operations was exceptionally hard because in many cases they were explicitly separated, to the point where different vendors, sometimes located on different continents, could be responsible for either.
DevOps and DesignOps
At the same time, practitioners from development and operations teams felt increasing levels of pain and frustration as their priorities collided more and more often. Development teams had been taught to focus on on-time and on-budget delivery, which they often did at the expense of stability and maintainability, because they knew they wouldn’t be measured on the latter, never mind held accountable. Operations teams, on the other hand, were measured on stability and cost of operations, which made them uncomfortable with any new release and led them to introduce complicated formal processes, resulting in less collaboration and long cycle times.
Similar to the early stages of agile software development, it was practitioners who joined in a grassroots movement, called DevOps, to resolve the conflict. The devopsdays conference series, which began in 2009, played a central role, and reports show that while the problem was well understood, practical solutions were not commonplace. Nonetheless, practitioners began to explore how they could, in their organizations, transition to a model that not only allowed but promoted close collaboration between development and operations teams.
DevOps and, more specifically, the collection of techniques and tools known as Continuous Delivery (CD) cleared the path to production. In the best case, individual chunks of functionality, captured by user stories, can be developed and deployed into production in a day or two, not in weeks or months. Today, sizeable IT departments, comprising many product teams, can manage hundreds of releases of related pieces of software into production every week.
This possibility of dramatically reduced cycle time motivated many organizations to trial DevOps. For decision makers who still needed convincing, the 2014 State of DevOpsreport provided evidence that “high IT performance correlates with strong business performance”. The authors later provided more insights into why they were confident to make this assertion.
With cycle time, we should measure the time between an idea and its realization as software in production. User stories in agile development are already much more than an idea, and many organizations that introduced agile development struggled with writing appropriate user stories. Oftentimes, large backlogs of detailed stories were created, bearing an eerie resemblance to detailed specification documents of days past. In response, the focus turned to improve where the user stories came from [pdf].
In a movement almost symmetrical to DevOps, organizations started to include user experience into their IT departments, to ensure that the actual needs of the business were represented in the user stories. The user experience experts brought new techniques such as design thinking and user research that added further feedback cycles. They also pushed for explicitly capturing hypotheses about features and validating them with data gathered from actual use of the system, a technique known as hypothesis-driven development. And, quickly picking up insights from the DevOps movement, designers and developers are taking steps to collaborate closely. They use new tools that enable them to work jointly on the same technical artifacts. This is sometimes known as DesignOps and is practiced at Airbnb for example.
Enabling techniques and technologies
While this article focuses on the organizational changes that allowed businesses to get more out of IT, it’s important to note that a number of breakthroughs in technology and software engineering techniques went hand-in-hand with the organizational changes.
The practices collected in eXtreme Programming, most notably test-driven development (TDD) and continuous integration (CI), gave development teams the necessary boost in confidence to release software more frequently and, because of the absence of unpredictable merges, made the process more reliable.
Service-Oriented Architectures (SOA), later refined by the concept of microservices, provide independent evolvability. With such an architecture the small, cross-functional teams described above can each work independently on their part of the overall IT landscape. Techniques such as consumer-driven-contract testing and micro-frontends often allow teams to release services into production without testing them in an integration environment first. In fact, for organizations that release software many times a day, it’s simply not possible to run extensive integration tests in a pre-production environment.
Another important innovation was the virtualized infrastructure. Rather than manually deploying an application on an operating system manually installed on a physical server, modern services are running in virtual machines or containers, and each step of their deployment is fully automated. In the past, the lifecycle of an operating system installed on a server was longer than that of the applications running on it. New versions of an application would be installed onto existing servers, often sharing the same operating system with other applications. Today, the lifecycle of the entire (virtual) machine is linked to the lifecycle of a single version of a service, and a machine usually exists with only one service deployed on it. This greatly reduces dependencies and improves the reliability of deployments.
When developers and operations people started working closely together, the developers brought more automation with them. Rather than configuring servers manually, they saw infrastructure as code. Scripts that can set up the entire deployment infrastructure, including software-defined networking, are managed just like the source code of the services running on them. Rebuilding the infrastructure is treated in the same way as releasing a new version of a service, and it can be done with the same speed and reliability.
Conclusion and outlook
When technology becomes the business, or when a business relies on technology to offer differentiated products, collaboration is key. We have seen a shift to small, cross-functional teams where engineers with different specializations collaborate. The DevOps and DesignOps movements bring closer collaboration between all roles involved in the life of an IT solution. In combination with agile software development, this allowed the business and IT to collaborate efficiently.
Today, organizations that have mastered the concepts described in this article consider their IT landscape a digital platform. Business-centric services that can evolve quickly and independently, combined with frequent and reliable releases, finally put the old dream of reusable and re-combinable components in reach for them. The organizations can innovate and move fast because their whole approach to IT allows them to experiment at scale.