Agile - Theory vs. Practice

When does Agile fail?

I’ve worked on a number of Agile projects throughout my 13 year career, and my experience with agile has been bittersweet. I was thus quite excited by the opportunity to work with the geeks that literally wrote the book on agile, and have since gained a more well rounded understanding of the concepts and practices along with their pros and cons.

I have come to believe that the main reason many efforts to go agile fail is because practitioners tend to forget that the manifesto is a statement of utopian ideals to aspire to. Any prescriptive practices that come out of it like Scrum and Kanban are simply practical attempts to stride towards this ideal however, due to the nature of the principle, none of these practices can ever actually get to the ideal. Some get nearer than others, but absolute agility can never be achieved in practice.

In practice being agile is simply an optimal compromise with ambient constraints. When many managers new to the principle hit pitfalls and frustrations in their application of Agile practices (e.g. pair programming, TDD, Iterations etc.) they unfortunately get discouraged. They then wrongly believe that these frustrations can be resolved by rigidly enforcing even more prescriptive activities to the team. Slowly the project finds its way down the vicious spiral towards abysmal failure. What do they blame? Agile.

An understanding of the contextual limitations of Agile practical methodologies in achieving the utopian principles allows for an understanding of how and why these projects fail. To understand the why, when and how of these Agile” failures, let’s examine some of the core principles of Agile, and see where the clash occurs. As a statement on the overarching principles that should govern the delivery of IT systems, the Agile Manifesto can be summarized under three broad concerns: Transparency, Inspection and Adaptation.

#1 Transparency

Through transparency we endeavor to ensure that all aspects affecting the outcome of the project are visible and known throughout the delivery process to the project managers. However, in many real-world scenarios, transparency into the intricate and complex details becomes, at best, more costly than beneficial, and, at worst, impossible to achieve across the myriad of teams and individuals involved.

Lets consider transparency while delivering a National Health System (NHS). By the project’s very nature, the entire population of the country is now a user. It would be cost-prohibitive to do befitting user-research that would be representative of the innumerable views and opinions of the country’s population. So we decide to bury our heads in the sand about the issue by interacting with members of government purporting to represent the citizens. From an Agile manifesto perspective we have failed to completely engage all stakeholders. Faced with negative public sentiment we are wholly dependent on the politicians to shield us from the consequences. However, they too have their own ulterior motives and transparency is not commonly known as a defining trait of politicians. Even after that there are the myriad private and public clinical facilities that need to be brought on board. Can we really in effect maintain transparency with a nation of tens of millions, their organizational entities, political affiliations and family concerns in order to deliver the required solution? The UK government decided that such a situation is best handled by solutions like Prince2 and only brings in agile for well-defined software deliveries. That said this position appears to be changing.

Consider the development of Linux which takes this issue to the next level of complexity - maintaining transparency across all software vendors, driver manufacturers, software users, operating system technicians (effectively the entire population of the world) - a task that will mean the project never gets off the ground. What Linus Torvalds did was place himself at the nucleus of the problem and gradually accrue converts until he had momentum to bring everyone else on board. He was initially the only stakeholder.

And this isn’t necessarily a problem with scale, but with culture as well. I’ve worked on smaller scale projects where the team was eager to adopt agile but the businesses perspective was less technology focused/dependent and more compliance/regulatory-centric. In this scenario the eager technology managers were introducing a new technology delivery process that the business was not convinced was required. Participation from business was essential to get the transparency required to develop this process. Unfortunately, the business regarded it as less important than committee/policy meetings and decisions. Going Agile in this atmosphere was largely painful and wholly unsuccessful.

#2 Inspection

Now lets consider inspection. We aspire that various aspects of the project must be inspected frequently enough to identify unacceptable variances from the process. We would like to 'fail fast'. Using the example of the NHS above, software is usually only a small insignificant part of the project, thus making it difficult to engage a critical mass of the stakeholders in its regular inspection, which in effect dilutes the exercise.  Even defining what an “acceptable” process/result of the inspection should be becomes difficult. In addition, Variances of the expected process are likely to be erratic and cannot be normalized, even if distilled from a successful inspection process, due to the magnitude and complexity of the project. Figuring out what is unacceptable then becomes next to impossible due to the compounded issues above, so how in the world can you manage to fail fast? Is failing fast even defined in these circumstances? Furthermore, the process of Inspection can also have unintended consequences on the inspected. For example, high level gaming, and inspection for the sake of inspection, an issue I’ve often seen even with small-scale projects.

#3 Adaptation

Now, suppose the problems of transparency and inspection are solved, we still have one more, adaptation.  If inspection determines that one or more aspects of the process are outside the acceptable limits for producing an acceptable product then the inspector must be able to adjust the process quickly and effectively. How do you set about successfully adjusting a process that involves millions of stakeholders along with their complex interactions at individual, political and organizational levels? In this case, any adaptive decision made is at best a dictatorship of the loudest voices over those that cannot be heard. At worst, it is an abrogation of knowledge that does not exist or cannot be attained, leading the entire process into anarchy.

There are results that could not be attained using Agile project management methodologies like the proving of Einstein's theory of Relativity but in essence the result matured in an agile and self-organized environment. Others, like Linux, are inherently agile without any prescriptive agile methodology that can be understood by project managers. Agile is a great utopian aspiration however it inherently cannot be realized completely and flawlessly through doctrinal practice. Higher-level project management systems exist within which agile is allowed to concentrate on well-defined areas/features e.g. Prince2/PMBOK. The reasons that many project managers fail to succeed in agile is because they are looking for something to DO to achieve it. Being agile is not about keeping control of a project through Scrum/Kanban, it’s about hiring the right people for the job and allowing them to naturally discover the optimal project configuration to deliver successfully.