In order for you to thrive in the digital environment, you need to understand the implications of the changing technology landscape on your organization. This is the first article in Technology Radar Echoes, a new series where authors share their insights and experience on the technology problems and solutions driving business differentiation for enterprise leaders. Now in its sixth year, the Thoughtworks Technology Radar is an assessment of trends significantly impacting software development and business strategy created by an international advisory board of technologists.
Not too many years ago, before social media, mobility, the Internet of Things (IoT), cloud, and big data muddied the field, technology stacks (the different layers of technology required to implement an application) were simpler and often vendor-specific. A good example is the Microsoft stack: .NET languages for programming, IIS as a web and application server, and the SQL Server database.
Today, technology stacks have exploded. We have platforms such as Microsoft Nano Server, Deis, Fastly, Apache Spark and Kubernetes. New tools pop up every week including Docker Toolbox, Gitrob, Polly, Prometheus and Sleepy Puppy. Programming languages and new frameworks such as Nancy, Axon, Frege, and Traveling Ruby are introduced. Advanced techniques such as the Data Lake, Gitflow, Flux and NoPSD mature. This list goes on and on.
On every Thoughtworks Technology Radar we include approximately 120 items and cull almost as many for lack of space. In the past five years we have seriously considered more than 2,000 tech “blips” for the radar. While a small subset of recommended technologies would apply to a given application, technology stacks varieties exhibit exponential growth in the last five or so years.
So why would you, an executive or manager, care? Why would you as a CEO care? Isn’t this the job of the technologists to manage? No! If software and software development are keys to your digital future, then you should care—not about the details, but what the overall trend means to your organization.
Technology, in many forms, drives differentiation and disruption—executives and managers must understand the implications of that technology landscape to make relevant decisions.
Implications:
The technology landscape exists in a state of dynamic equilibrium. As new technologies and approaches appear, the ecosystem adopts them to create a new equilibrium.
Here are some recent landscape shifts:
Single vendor solutions are a thing of the past
In the recent past, even some large organizations maintained a single-vendor solution to their tech stack. In the mid-2000s a friend of Jim’s worked on a data warehouse project for a company that was completely an IBM shop. For their data warehouse project the client insisted on an IBM tech stack (data warehouse) on aging AS/400 hardware. The team tried to convince the client that in this case a Microsoft stack would be much more effective. After missing deadline after deadline using the IBM hardware and tech stack, building a prototype using the Microsoft stack on Windows servers, and meeting after meeting, the development team finally convinced management to change the tech stack. Progress improved dramatically. The company’s single-vendor solution had failed as they moved into a new application area.
That’s a trend that’s growing in momentum. There’s so much happening, so quickly, that even a company the size of Microsoft or Oracle can’t keep up any more. Not just is the volume of change too fast, we also see that many of these changes are disruptive, thus causing existing products to be obsolete—and it’s hard for any company to walk away from what has been their jewels.
The rise of open-source tools further fuels this. Open-source products facilitate deeper collaborations across companies, allowing a vendor to build new tools on top of a dozen other technologies without having to agree to multiple licenses. This encourages smaller, more focused companies, who can specialize in a single technology area. There’s little incentive now for large technology corporations to consolidate except, and here’s a killer irony, when the technology is dying out.
Mobile development means dealing with multiple operating system vendors for different devices. Big data development means dealing with different types of database management systems, many that are either open-source or from newer vendors with short track records; the days of a single data solution have ended.
If you want to be a digital enterprise, increasing tech stack complexity means increasing vendor complexity—with all its associated headaches, and opportunities.
Team composition and organizational structures need to change
As tech stack complexity grows, organizing your teams based on technology layers becomes less and less effective. In the early Internet era of only a few tech layers—business logic, database, middleware, presentation layer—teams organized around technology layers weren’t pretty, but they worked to some degree. Today, those degrees are shrinking to zero. The wide diversity of skills required to build today’s applications means collaborative teamwork, not dysfunctional silos. There are fewer and fewer instances in which one or two people have all the requisite skills to design, build, and deploy an application. Furthermore, as team sizes get larger to acquire the skills required, diverse teams have to collaborate more closely to be successful.
In the past, operating cost and complexity encouraged companies to create technical silos such as developers, QA, and operations; but Cloud and a focus on DevOps culture give us the ability to provision machines automatically and deliver new product features daily. These capabilities give us a new architectural opportunity. Rather than partition your architecture in technical layers, teams increasingly organize around domain concepts like Customer or Order, encapsulating the technical layers within a service. This trend towards domain (or product) organizations rather than technical layer organizations is accelerating as companies go digital. If companies continue to organize along technical silo lines, the number of siloed functions with their necessary interactions will increase dramatically causing more severe integration problems.
Collaborative teams become an absolute necessity
As teams become more complex, collaboration becomes more critical. As noted above, today’s teams require a wider range of skills, from pure tech stack layer expertise through to operations, product design and product management. This diversity makes it both harder to collaborate effectively and more necessary. One of the benefits of the agile and lean movements has been to encourage the empowered, collaborative, cross-functional teams.
Building software delivery capabilities just got much more difficult
Engineering matters. More and more companies are realizing that their digital strategies remain fallow without world-class software delivery capabilities.
“As digital technologies relentlessly reshape competition, products and services increasingly depend on software for differentiation and performance. Despite the mission-critical nature of software, it gets surprisingly little attention in the C-suite. New research suggests, however, that companies pay a price when they undervalue the strategic importance of producing excellent software.” [“The perils of ignoring software development,” McKinsey Quarterly, 2015]
There are five key factors that impact an organization’s ability to build a world-class software delivery capability: breadth of skills, depth of skills, quick learning, innovation, and decision-making.
The breadth of skills required to build applications today has increased dramatically as tech stacks become more complex. To keep up, you may need to develop Internet of Things (IoT) skills as well as software delivery skills.
But not only has the breadth of skills increased, but the depth of skill required (advanced versus basic) has increased also. Cutting-edge applications require cutting-edge skills. Just as a 5.10 (difficulty level) rock climber doesn’t have a chance of ascending a 5.13 route, an average skilled software team has little chance of delivering a gnarly software application.
As we have seen with the tech stack’s rapid pace of change, the skills and knowledge required tomorrow will be different, meaning that not only do you need people who have today’s skills but people who can quickly acquire tomorrow’s.
In a long-quoted observation, a manager laments that he’s reluctant to send developers to training because he’s afraid they’ll leave for another company, and the reply is: “What if you don’t train them and they stay?”
On top of the technical skills themselves, individuals need to apply those skills in innovative ways. There needs to be a sense of adventure and exploration in your organizational culture.
Finally, but not least important, your organization needs better decision-making around capabilities and skills. You need to decide questions such as: What skills you need? Do you need to build each skill internally? What depth do you need in each skill set? How do you balance formal training versus experiential skill acquisition methods? How do you keep your skills updated? Do you build skills internally, or buy (or borrow) external resources?
Outsourcing versus in-sourcing versus co-sourcing options are more complicated and no longer cost-driven
Many organizations are finding that they just can’t build and maintain the technology and software delivery capabilities required today. Some can’t attract the required talent because today’s best developers are in high demand and want to live in global “tier one” cities such as New York or London, while many corporate IT departments are in lower cost, less glamorous locations. Some can’t afford the talent on an ongoing basis because they don’t have a continuous pipeline of digital business projects. During the cost-cutting, efficiency-focused strategies of the prior decade many companies outsourced so much of their capability that rebuilding it to fit today’s needs would take too long and leave them at a competitive disadvantage. Today’s digital business sourcing decisions need to be focused on innovation, delivery cycle time, and value generation—not on cost savings.
Today, outsourcing technology development is not driven by cost reduction but on capability acquisition. Furthermore, you should be looking for partners who not only have the current capability you need, but have demonstrated the ability to build new capabilities rapidly.
While bringing in capability from the outside solves some pressing opportunities, it also carries the same problems as cost-based outsourcing, only the problems are magnified. For example, outsourcing always carries potential knowledge transfer problems if you ever want to bring the outsourced applications back in-house. Tech stack complexity magnifies those knowledge transfer problems. Increasing the number of vendors and new types of contracts adds additional challenges. For example, contracting can be more difficult as developing cost-control oriented contracts are better understood than innovation-centric, value-oriented contracts.
Building a cutting-edge technology organization is costly
We hear from clients that they are afraid to adopt modern technology stacks (e.g. Hadoop) because they don't think their people can up-skill sufficiently. At the same time they want to attract new young talent and know they have to adopt modern technologies that excite them. This paradox cannot be resolved by the technology organization—it requires business executive leadership.
There is no way around it, if you want to build a digital savvy enterprise, you can’t do it with a cost-cutting mentality. With the complexity of the tech stack, the need to innovate at the technology, business model, and product levels, and the war for talent—undertaking a strategy to build a cutting-edge technology organization should not be made lightly. The decision to move forward on this front must be made from an overall business strategy (building an entrepreneurial company, creating a digital business) and a focus on value, not cost. Cost is important as a constraint, not as an objective.
Technology change can be overwhelming, but you have to keep up to keep ahead
Thriving in today’s competitive environment requires four key capabilities: creating an effective digital strategy; implementing a dynamic, value-driven portfolio management process; building a fast, iterative software delivery capability; and growing an adaptive, innovative culture throughout the organization.
The bottom line is that technology changes are prescribing other changes in your business, changes that will have a long-term impact on business viability. In recent years, the complexity of the technology stack required for digital business applications has soared leaving businesses on the horns of a dilemma. On one hand there is the necessity of meeting the digital business challenge and on the other is the unanticipated difficulty in building the technology skills required to meet that challenge. As the pace of change of technology accelerates, tech strategy must inform your business strategy. The only meaningful way to do that is to foster an organization that can assess, adopt and adapt to new technology.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.