Macro trends in the technology industry | October 2021

Twice a year we create the Thoughtworks Technology Radar, an opinionated look at what’s happening in the enterprise tech world. It’s a detailed collection of tools, techniques, languages and platforms and we generally call out over 100 individual ‘blips.’ Creating the Radar involves over 20 of our senior technologists from around the globe, and as we discuss individual Radar blips we also talk about bigger trends. This article is a consolidation of those “macro trends” that we see in the tech industry today.

Securing the software supply chain

Earlier this year, details emerged around a well-coordinated hack that made headlines for months. Hackers had broken into SolarWinds’ systems and added malicious code to their Orion software. The reason this was such a juicy target? Orion is a network management system that monitors servers and runs with a high level of privilege across all the sensitive parts of a company’s systems. Microsoft, Intel, Cisco as well as governments across the world were compromised in the attack.

What was interesting, though, is that this wasn’t a hack against running instances of Orion, like many remote exploits are. It was a hack against SolarWinds’ build environment and “path to production” and instead injected malicious code into the software directly, which was then digitally signed and distributed to customers. They downloaded and installed a “bug fix” release of Orion which then left them vulnerable to the hackers.

Since then, awareness has been increasing of the importance of securing not just your software but also the “software supply chain” — all of the software and processes that contribute to creating, building, testing, distributing and running that software. We spoke to Mike Ensor and Jim Gumbley on the Thoughtworks Technology podcast and they highlighted that the Biden administration had issued an executive order on cybersecurity, including directives focused on the software supply chain problem. In this edition of the Radar we blipped Software Bill of Materials (SBOM), a technique mentioned in the executive order, which says that all software should come with a machine readable list of component names, version numbers and source vendors. When a vulnerability is discovered, you can scan the SBOMs of all your software, know what’s affected, and take action to patch your systems quickly. We also blipped Cosign, a tool for signing container images to improve security across the software supply chain.

Punctuated equilibrium in the platform space

In this edition of the Radar, we noticed far fewer Platform blips than usual and so we discussed what this might mean. Certainly the platform concept is still critically important, with organizations that can effectively build and leverage platforms able to use them as a leg up over competition, accelerating their ability to get software into production and delivering value to customers. But why aren’t we excited by platform tech at the moment?

Our hypothesis is that for the most part, organizations have chosen a cloud vendor (or two) and have standardized on Kubernetes and Kafka. While the major cloud platforms continue to release new services, there’s nothing particularly earth shattering about those new features. And we haven’t seen a genuinely ‘new’ platform creating traction, mostly just new features and use cases for existing platforms.

We don’t think this is necessarily a bad thing. The pace of technological innovation is uneven; not everything moves fast all the time. We suspect that for platforms, at least, we are in a period of equilibrium where innovation has slowed a little and platform adoption is the main force driving us forward. Of course, this could all change tomorrow, which is one of the reasons it’s always interesting working as a techie.

Careful decisions required in the face of complexity

Shooting yourself in the foot is, it seems, more tempting than ever. Mounting system complexity means that making a convenient but shortsighted decision can cause “future you” significant pain down the road. It’s important to tackle this problem head on because, left to their own devices, software systems tend to complexity and entanglement and teams tend towards doing the easy thing over the right thing.

Examples we discussed while assembling this edition of the Radar included:

• BFFs (Backend-for-Frontend) growing from simple service adaptors and aggregators into complex services orchestrating many back-end services. This often happens because a customer experience focused team has more direct control over the BFF layer and can get something done quickly there, whereas doing a change in the ‘right’ place — the underlying business services — requires more coordination between teams

• GraphQL being used as though it were defining a RESTful interface, replete with poor query patterns such as n+1 selects or multiple repeated calls to retrieve details about objects in the graph (which is the whole point of GraphQL in the first place!)

• The kong-js-pdk project that adds a plugin development kit to Kong, enabling developers to bury complex logic in what should be a simple part of your infrastructure

The “right thing” in each of these cases is to carefully consider the architecture of the system you’re trying to build, and to put the functionality and logic into a sustainable location. That might mean paying closer attention to architectural guidance and maybe moving a little slower in the short term, but the quality of your software will be a benefit in the long term.

Clever developers are often solving the wrong problem

Neal Ford has a saying that “meta work is more interesting than work” and related to that we often observe whole departments of very clever people solving the wrong problem. One client I worked for over a decade ago had very strict controls around their test environments, which were so locked down developers couldn’t deploy code to them. But the client valued automated tests, so they had built a system where developers could submit a request to run their latest code changes in the special test environment. The system accepted those changes, built the software, deployed it to the test environment, and ran many tests in parallel across the environment. Very clever, but ultimately solving the wrong problem: why not make the tests run on developer’s workstations or on lightweight cloud environments that can be easily spun up and down?

There’s an important distinction here between the inherent complexity of the task — test my code so I can have confidence to deploy to production — and the accidental complexity of the problem developers ended up solving. Examples we discussed recently include Airflow and Prefect which are very clever tools that manage spaghetti-code data pipelines or the plethora of tools such as Nx that manage the problems caused by monorepos. 

Any time a tool seems to be solving something other than the inherent problem at hand, you should be at least a little skeptical and ask why we need it. Instead of jumping into a technological solution, consider whether you could change some other part of your approach such that a clever tool would not be needed. And when you do write something clever, remember to think about how you’ll retire it in future, too.

Conway’s is still the law

We’re still talking about Conway’s Law. Regular readers are probably aware of this concept by now, but most organizations still aren’t explicitly aware of it and are surprised and confused when their organizational design approach results in a substandard outcome. This could be avoided if a few more people understood the concept, and indeed, we’re seeing more and more situations where it applies.

Conway’s Law states that the structure of any system will mirror the communication structure of the organization that built it. Practically, this means that software architecture will tend to end up looking like the teams that built it — large, centralized teams tend to build monolithic code, and smaller distributed teams tend to build more modular, loosely coupled code. There’s also a strong implication for the ease or difficulty of making changes and getting things done. An organization containing a lot of committees and red tape will frequently have systems that are difficult to change, and competing departments within an organization will often have a lot of friction between their software components.

But Conway’s Law can be a helpful change agent, if you leverage it well. Organizations embarking on their digital transformation journeys should re-organize their engineering teams, including culture, reporting-structure and incentive programs, in alignment with how they want to evolve their architecture and technology strategy. As Rebecca Parsons said in our press release for the Radar, “Organizations are far better served leveraging Conway’s Law as a positive force by paying attention to the people who build the software and creating a product-centric operating model and engineering culture.”

Creativity in hybrid and flexible working models may be COVID’s silver lining

While the pandemic continues to be a global issue with a tragic human cost, it has probably accelerated our adoption of distributed and remote working practices by at least a decade. These changes were likely coming eventually but the pandemic has forced us to adopt new techniques much quicker. Work from home has been proven a viable arrangement for many, and a full return to the office isn’t necessary to keep people productive. Indeed, for many people a return to office working could actually prove counterproductive. This is reflected in the demands of the tech sector workforce – many skilled workers explicitly seek support from a prospective employer on hybrid working patterns with a mixture of office days and remote work.

In discussing this edition of the Radar, we noted many creative accommodations in the remote work space. Asynchronous video walkthroughs are a technique to record showcases, product demonstrations, or even architectural decisions so that others can access them on-demand later. Remote pairing tools continue to evolve at pace, with most major IDEs incorporating first-class collaboration features such as shared editor windows, environments, and even live debugging in a shared session. And some organizations have embraced an explicit timezone alignment in how they structure their teams and systems, allowing for flexibility in where people are located but reducing the stretch that cross-timezone collaboration can cause.

That’s about all we have time for in this Macro Trends round up. I’d like to thank Chris Willis, Lakshminarasimhan Sudarshan, Scott Davies and Srinivasan Raguraman for their input to this article.