The word that took the tech world by storm: Returning to the roots of agile

I. How agile was born

In 2001, a group of software developers gathered in the picturesque ski resort town of Snowbird, Utah, to discuss emerging approaches to their trade. They hit on some ideas that they felt were good enough to commit to paper. They certainly had ambitions. But they had no clue how far their words would go.

Their central belief was that software development needed to be more adaptive and responsive to change - or ‘agile’ as the group decided to call it. The Agile Manifesto transformed this core concept into a software industry rallying cry. The group continued to collaborate, producing a succinct statement of supporting principles that emphasizes spontaneous collaboration, simplicity, and the pursuit of technical excellence to meet changing demands and achieve goals.

Seventeen years later, these principles have penetrated virtually every industry and market. They are prescribed by consultancies and business schools as the antidote to a variety of enterprise ailments. And they are regularly dissected through books, management seminars, and online forums. Not bad for a project the founders intended only to galvanize the software sector - and that some were unsure would succeed even in that.

Time, it seems, continues to prove the naysayers wrong. While many management buzzwords die a quick death (remember holacracy? If not, you’re not alone), agile has proved remarkably persistent. Google Trends data shows interest in terms like “agile” and “agile certification” has remained on a steady upward track worldwide since 2004. Agile has also been the focus of recent articles in publications as diverse as Forbes, CIO, Global Healthcare and Harvard Business Review exploring its applicability across functions from marketing to human resources.

By most measures, then, the movement born in Utah all those years ago can be considered a success. But success can change ideas the same way it changes people, and in the eyes of some of its earliest proponents, agile has strayed too far from its original meaning and mission. The time has come to return to the roots of agile - and in doing so, ensure it reaches its potential in spurring the successful digital transformation of businesses for decades to come.

II. The rise of fake agile

The constant and exhaustive discussion around agile has resulted in what Fowler terms “semantic diffusion": the spreading around of a term to an extent that its original definition is weakened and eventually lost.

While this can be an unpleasant process to watch for champions of a concept, in Fowler's view, it's an inevitable by- product of an idea gaining popular currency. “It’s the price of success," he said. “You only get this problem when people have realized the idea is valuable and want to take it on."

At the same time, the agile trend means more people and organizations claim to aspire to it, but “don't actually want to learn the hard part of doing it. They just want someone to wave a wand and give them the thing." These forces have given rise to versions of agile that may look and sound like the real thing, but are fake agile in much the same way fashion labels contend with fake brands. And fake agile has become, to use Fowler's words, an “industrial enterprise."

"The whole world of [the Scaled Agile Framework] and certifications, the tool vendors that basically say buy this training course and you will become agile - that's been very problematic," Fowler says. This is less to do with agile frameworks or courses being inherently flawed than with organizations deciding the course or framework is all they need to do.

"This was always going to be a big struggle with agile because the agile mindset involves changing the way people work," Fowler said. "It's not a purely technical thing, like object orientation, therefore it's always going to be a much harder change to make.”

Fake agile is often characterized by “blithely implementing practices; just doing it," said Sunil Mundra, Principal Consultant at Thoughtworks and the author of Enterprise Agility - Being Agile in a Changing World. “There's nothing wrong with starting that way - there are processes, ceremonies that are there to implement - but if you just do that the inertia that is still there from the systemic issues in the organization will overpower any green shoots you create at the team level, and you’ll be back to your original state. If you don't understand the 'why' of what you’re doing, it's not going to give you any benefit," Mundra said.

It's this focus on superficial change that is behind the relatively high number of unsuccessful agility experiments. Executives at the Dutch financial giant ING have expressed dissatisfaction with the early outcomes of high-profile agile initiatives. Studies have shown a relatively high rate of agile information technology projects end in some degree of failure, and that CIOs are increasingly cynical about the concept.

This makes it critical that the focus returns to the original version of agile which is capable of driving genuine, and positive, outcomes. As Fowler notes, "The battle has shifted. Ten years ago it was trying to persuade people that agile was actually something worth doing. Now we need to define what agile is really about."

III. What real agile looks like

So how do enterprises know 'real agile' when they see it? As Fowler has pointed out, a defining characteristic is that it involves transformation - a process that is often painful for companies to undertake. Thus many enterprises settle for ‘doing’ agile: adopting some established agile practices, instead of ’being' agile, which means developing an entirely new mindset. Both can produce benefits, but it's in ‘being’ agile where many of the greatest innovation and customer satisfaction dividends lie.

Several pervasive stumbling blocks can prevent organizations from getting there. The first is the persistence of the command and control mindset, or the traditional management structure where plans are instituted in a top-down method and must be adhered to at any cost. "There are so many organizations that have been built on the idea that your competence is based on how much you control and predict things," said Fowler. “That takes a huge shift to get rid of."

"The fixation on predictability and sticking to old ways of planning is getting in the way of truly iterative development and resulting in wastage in terms of software getting built that is not hitting the mark” agrees Sriram Narayan, Vice President of Advisory for Thoughtworks and author of Agile IT Organization Design.

There is also a tendency for hierarchies to emerge even in nominally 'flat‘ organizations, with negative consequences for the free flow of information that real agile requires. “A lot of organizations are very much power struggles, which affects people's ability to deal with honest communication. That doesn‘t work at all with agile, which demands a much greater degree of transparency," Fowler says.

Finally, the evolution of agile into a general management discipline means organizations often underestimate the bedrock of technical skills needed to support processes like dynamically adjusting solutions in response to customer feedback.

"This is specific to the software world but there are parallels in the non-software world as well," Fowler explains. Organizations are not giving the necessary attention to things like testing and development, refactoring and discretion, or the specific technical skills in their area. These really enable the shift between the first and second levels of the Agile Fluency™ Model, where technical expertise is required to get you to the levels of productivity and effective execution you're trying to attain."

Smaller, newer organizations, starting with a relatively clean slate typically find it easier to navigate these roadblocks as they have less in the way of legacy processes and hierarchies to contend with. As Fowler puts it, “It’s much harder to ‘unlearn' something you know and are fluent in, than it is to learn something from scratch." Despite these difficulties, examples of agile success can come in all shapes and sizes.

Let's take a look at agile in practice.

Daimler: While the German auto giant has a long history, it has been actively exploring new, agile ways of working. They've notably implemented a ‘swarm organization' initiative that aims to bring 20% of employees globally into agile and flexible work structures by 2020. The ‘swarm' is based on teams that bring together game changers from around the world, blending functions and seniority levels to generate insights on company transformation.

Daimler's increasingly fluid and employee-led approach has informed its strategies in markets like China, where innovation and digitalization have proven a key source of competitive advantage. The company's Mercedes-Benz E-commerce (MBE) platform is one of the first of its kind in China's luxury car industry, allowing customers to browse models, configure and order their choice of Mercedes-Benz entirely online. Designed in collaboration with customers and dealers, the platform was launched in just seven months. Daimler has also developed a mobile solution for Mercedes-Benz dealers, 'One Touch Retail', that manages all aspects of the sales process from lead management to delivery.

REA Group: REA is the operator of the world’s largest property portal. In 2011 the Australian organization initiated a re-platforming project for its core systems to enhance its ability to deliver innovative new services at speed and scale. This was achieved with a new offshore centre of excellence that uses collaborative technologies to work with REA’s onshore teams, enhancing capacity as needed and injecting more diversity into the design process. The centre is also helping REA explore potential applications for emerging technologies like robotics, augmented reality and data science.

REA’s agile development approach has helped it notch up several impressive accomplishments, including becoming one of the few multinationals to thrive in China's notoriously opaque online market. REA Chief Inventor Nigel Dalton has been called the “Godfather of agile" - but has also noted agility needs to be supported by strong management and organizational resiliency.

Trainline: Aiming to be a one-stop shop for European rail and coach services, Trainline’s online platform sells tickets on behalf of over 140 rail and coach companies and offers real-time travel information across 36 countries. As with many other firms, Trainline was seeking ways to release and push new services to customers faster. Key to this was the streamlining of communication among Trainline's development teams in different locations, who were often pursuing projects independently, complicating management and support.

This issue was addressed by enhancing functions such as backend web services, mobile optimization and data processing. They also established mechanisms to clarify ownership over development projects and set clear procedures for knowledge transfer. The ‘structured decentralization’ approach, along with the introduction of automation in key parts of the development process, has accelerated the delivery of solutions from distributed teams to an average of 10 releases a day, supporting Trainline‘s plans for a more mobile-focused customer strategy and the launch of corporate ticketing solutions.

IV. How agile fuels digital transformation

Since the Agile Manifesto was conceived, the pace of technological change has accelerated massively, as has the rate at which market demands evolve. This has made the speed and responsiveness that characterize the agile enterprise even more of a competitive necessity.

Like agile, digital transformation is a mindset rather than an end-goal. It's defined by constantly striving to improve customer satisfaction and speed to market; the ability to quickly and smoothly launch new services and solutions; and robust lines of communication that allow organizations to rapidly assess, and even anticipate the impact of their own or broader changes.

Two key factors distinguish the agile, and therefore digital transformation-ready enterprise. One is the presence of cross-functional teams that are effectively self-organizing and operate with a high degree of autonomy - yet also work towards the same defined goal or purpose. This means leadership has a role “to not only ensure teams are moving in the right direction, but to remove impediments, and ensure that they have all they need in order to perform,” as Mundra puts it.  

As some degree of hierarchy is all but inevitable (and indeed necessary), the goal should not be to eliminate it altogether but to manage it in a way that harnesses its positive characteristics. As Narayan notes, most firms have distinct hierarchies around communication, reporting, and decision-making. To support digital transformation, the former two need to be minimized. “You can’t have 15 levels of reporting between the coal-face and the CEO,” Sriram said. And the latter should be acknowledged but sufficiently decentralized, to ensure teams are empowered to act independently, but also, in Sriram's words, to “maintain alignment with overall strategy and learn from previous decisions." 

Another important agile characteristic is the absence of organizational barriers. According to Fowler at the initial stages of agile fluency enterprises tend to focus on creating effective teams rather than tackling more comprehensive enterprise-wide change. This can result in the organization being structured around multiple high-performance teams that effectively operate in their own silos. 

This is especially problematic when there are no direct lines between development teams and the rest of the business. This chokes the flow of customer feedback that should fuel future stages of the development process. 

“Teams that can produce software very effectively but aren’t necessarily properly connected run into the danger of producing the wrong thing very rapidly,” Fowler said. “That happens a great deal when people put up barriers between business and software, and have the mindset that software is something you just kind of toss over a wall. Digital transformation is about breaking that down and realizing development has to be put at the centre of everything you do.”  

This approach should also extend to developing shared goals for development teams and the rest of the business, and eliminating the barriers to interaction between the enterprise and its customers. 

“’Good" agile is where delivery teams are completely transparent with their customers, respond to customer needs for change and make decisions jointly with the customer,” Mundra notes. “If they encounter a problem they work with the customer to find a solution, even if it means they go over time and over budget.”

Measuring results and moving to the next level

Defining success in the pursuit of agile principles and moving to the higher stages of agile fluency is not easy. But there are telltale signs that show when an organization is on the right track. Pioneering agile organizations stay on top of technical debt, elevate technical excellence and their teams continuously monitor progress.

As Fowler has noted, once companies develop the necessary base of technical skills, they can move on to the higher levels of agile fluency. This enables digital transformation, characterized by the “really cool stuff,” like teams capable of monitoring their own outcomes.

That said, any transformation will inevitably face roadblocks and resistance, and will need to be measured, justified and perhaps even restructured based on its results. This raises the question of how enterprises can gauge levels of agile achievement. According to the experts the following are signs of a high level of agility: 

The ability to cope with tightening feedback cycles 

“(Agile) is not an easy thing to measure, but one of the largest things we’ve looked at is cycle time: how quickly you can get from idea to production,” Fowler said. “The other aspect of this is how rapidly you can learn from things that are happening to generate ideas for the next thing to do.”

The capacity, and confidence, to try and fail  

At the later levels of the Agile Fluency™ Model, enterprises are so quick to recognize and act on changes that they essentially anticipate, rather than respond to market developments – giving them more time to test and develop solutions. “You get a great competitive advantage because you’re constantly one step ahead,” Fowler said. “It also means you have the capacity to try six things, accept that five will fail, find out which one of them succeeds and double down on it. That’s where the real promise of agile lies.” 

High levels of customer and team satisfaction 

The true measure of your success as an organization is that you’re making your customers more effective,” Fowler said. “But another sign is on the people side. Do people get to decide what process they follow? Do good people want to work in your organization because they see you as a leader in the field? Are you having problems with staff being poached to take leadership roles by other organizations, because you’re seen as a great place to have come from? That’s how you know you’re geared to succeed.”

VI. The evolution of the agile enterprise 

Metrics such as these need to be constantly examined because agility and transformation are ongoing processes rather than destinations companies can reach, then rest on their laurels. They also show that the parameters by which enterprises are assessed and success defined – many of which date back to the manufacturing era - are due for a broader overhaul. In a traditional enterprise, the focus is on efficiency and the maximum utilization of resources. However, in an agile enterprise, the ability to react to and anticipate change is prioritized.

“Organizing has to be done with a mindset of responsiveness, whether signals and feedback from customers or directives from regulators – and that’s very different than organizing based on cost efficiency,” Mr. Narayan explains. “Efficiency is a big reason companies take on projects beyond their capacity, which creates backlogs, so they start more projects and finish fewer.”

Ultimately these new values, and the enduring longevity of agile as a concept, are based on the recognition that “it’s a very uncertain world and you’ve got to live with that," Fowler said. "Which means you have to switch into this feedback-driven style of working instead of constantly laying out plans. That, together with the recognition of technological skill, are the two real underpinnings of what agile is about.”

By Daniel Pallozzi