Humans have always faced complex problems, from basic survival to the pursuit of technological progress. But even as we have pushed the boundaries of what is possible, we have always been constrained by the limits of our own cognitive abilities. Nowhere is this more evident than in the field of software engineering, where the rapid pace of technological change has created complexity that can be overwhelming for even the most brilliant minds. Make no mistake: the threat posed by these limitations is significant. One small error or oversight can lead to disastrous consequences, from unmaintainable software and system failures to even more serious issues such as lost revenue and damaged reputations.
Yet despite these challenges, we have managed to achieve remarkable success by leveraging our strengths and developing a wide array of tools and techniques to achieve engineering effectiveness. By creating and implementing effective software engineering practices that are tailored to the unique challenges we face, we can ensure that our businesses remain competitive and agile, even as the complexity of our operating environments continues to grow. It is only by doing so that we can hope to remain at the forefront of technological progress and continue to push the boundaries of what is possible.
How do we solve complex problems?
Humans have two distinct modes of thinking: concrete and abstract. Concrete thinking centers on tangible objects and sensory experience, while abstract thinking focuses on concepts and relationships. Balancing both is vital for effective problem-solving, as different problems require switching between them. Doing so will ultimately lead to a broader perspective which, in turn, will lead to more effective solutions. For instance, sorting elements in software engineering requires concrete thinking to move different elements appropriately, while abstract thinking makes it possible to build scalable and reusable concepts that create clear and maintainable code, adding business value.
When faced with complexity, we typically expect to be able to break it down into smaller parts. This means, the thinking goes, that we can solve small problems and then recombine them to build a complete solution. Indeed, even the etymological definition tells us the same thing: complexity is “composed of interconnected parts formed by a combination of simple things or elements.” Unfortunately, however, this isn’t always the case — sometimes things can’t be easily broken down into component parts. And often, when they are it changes the whole nature of the problem we thought we were dealing with.
Limitations of our own thinking
In software engineering, our cognitive limitations affect problem-solving and can pose various risks. Recognizing this is important: failing to attend to these limitations will lead to weaker outcomes, decreased efficiency and a decline in overall business value.
Here are some of the ways our cognitive abilities are limited:
We have a limited capacity to manage complexity: Our working memory, which temporarily stores and processes information, can only hold a certain amount of data. The fragility of our memory, susceptible to degradation and interference, further limits our ability to comprehend and solve problems with multiple interrelated variables and elements.
Oversimplification: Our tendency to oversimplify problems may give us a false sense of understanding. This can lead us to creating a suboptimal solution that overlooks important details.
Overgeneralization: This can lead to a one-size-fits-all approach, which neglects important nuances and details of a problem. This may result in a solution becoming — more complex, making it difficult and expensive to maintain in the long run.
Confirmation bias: Choosing to only consider information that aligns with our preconceptions, while disregarding information that challenges them, can lead us to overlook alternative solutions or critical issues that could compromise the success of a project.
Finite attention span: Our attention span will always pose a challenge for us when we are trying to focus on a complex problem. This can lead to mistakes, missed chances, and an insufficient understanding of the complexity at hand. Emotions, stress and fatigue can also impact this.
How can we mitigate the limitations of our own thinking?
Although our cognitive abilities may limit our effectiveness in solving complex problems, there are still various tools and methods available to alleviate these limitations and reduce related risks and maximize business value.
In our toolkit we have two powerful instruments at our disposal: abstraction and composition. Both are fundamental techniques for breaking down complex problems into smaller, manageable parts.
Abstraction involves filtering out unnecessary details and emphasizing what's critical. It simplifies the management of a problem’s complexity by reducing the amount of information that’s processed; this allows our minds to function more efficiently. Composition, on the other hand, involves combining simpler components to form more sophisticated and abstract ideas. Together, abstraction and composition aid in the description of complex problems ensuring that our brains are not overwhelmed.
Abstraction and composition require clear language. In software engineering a typical example is a programming language where we model abstract concepts using constructs such as classes, methods or interfaces. In software architecture, we might use a diagramming language that employs standard symbols for services, data stores and message brokers to represent abstract concepts. In both cases we are able to describe complex systems: an application and software system respectively.
Creating easily composable abstractions is critical, as they can be combined in various ways to form new concepts. It's also important to choose a language that allows us to represent abstract concepts effectively. This makes it easier to work with them.
Besides abstraction and composition — which are essential for managing complexity and overcoming cognitive limitations — software engineering offers numerous proven techniques and tools to achieve similar advantages. While some of these techniques are more context-specific, here is a list of some of them:
Design patterns: Design patterns represent a commonly-occurring solution to a problem in software design that can be reused in various scenarios. They aim to reduce complexity by naming and defining abstractions. As a matter of fact, the idea of being "reusable" suggests simplifying excessive details to create a general solution applicable in multiple scenarios.
Refactoring: Refactoring involves changing the code's internal structure without altering its functionality. One goal of refactoring is to enhance code readability by reducing code complexity. This results in a better understanding of the code and makes it easier to visualize in the mind.
Test-driven development: Test-driven development drives the evolution of code's business logic through continuous verification against pre-written tests. This approach often leads to better structured code and controlled complexity as developers stay focused on what's essential from a business viewpoint.
Collaboration: This practice ensures knowledge and expertise is shared, resulting in new perspectives and breakthroughs. By bringing together people with different backgrounds and specialisms — like developers, testers or architects — you can uncover new approaches to solving problems. We might even say that collaboration can improve our collective cognitive skills, expand expertise and increase attention span.
Agile practices: The purpose of agile practices is to simplify the software development process by encouraging rapid feedback and collaboration. Additionally, the iterative approach to delivery helps ensure that development teams are focused and closely aligned to the priorities of stakeholders.
Platform thinking: Platform thinking focuses on abstracting common infrastructure concerns and creating an environment for development teams to easily use that shields them from the growing complexity of the underlying infrastructure and at the same time enabling them to build quality software.
The possibility of a new cognitive revolution in the age of technology
Despite our limited cognitive abilities, we've made significant advancements in our ability to solve complex problems. Technological innovation proves this: it demonstrates we can constantly produce novel solutions to problems and challenges. To improve the efficiency of our problem solving, however, it's crucial to maximize our strengths and mitigate weaknesses. This is particularly true in software engineering, where failure to do so will impact product and service quality.
Human history spans tens or even hundreds of thousands of years. We evolved cognitive abilities for survival needs such as gathering food, hunting or building shelters. Over time, we have honed these skills. However applying them to new fields like software engineering poses a challenge as we haven't had enough time to adapt. The development of abstract thinking may have been a result of environmental changes that required us to adjust. Currently, we are facing problems that are both new and more complex than ever before. This could spark a new "cognitive revolution," accelerated by technology and artificial intelligence, but this time it may not require thousands of years of natural evolution.
We must continue to utilize our knowledge and developed tools to optimize our potential, while preparing for a future where new cognitive advancements may redefine the limits of our abilities.
Further readings
Milewski, Bartosz, Category Theory for Programmers, published 2019
Seemann, Mark, Code That Fits in Your Head: Heuristics for Software Engineering (Robert C. Martin Series), Addison-Wesley Professional, published 2021
Le Cunff, Anne-Laure, Cognitive bottlenecks: the inherent limits of the thinking mind, blog post at https://nesslabs.com/cognitive-bottlenecks
Le Cunff, Anne-Laure, The art and science of abstract thinking, blog post at https://nesslabs.com/abstract-thinking
Cherry, Kendra, What Is the Confirmation Bias?, blog post at https://www.verywellmind.com/what-is-a-confirmation-bias-2795024, published 2022
Wayman, Erin, When Did the Human Mind Evolve to What It is Today?, blog post at https://www.smithsonianmag.com/science-nature/when-did-the-human-mind-evolve-to-what-it-is-today-140507905/, published 2012
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
