The methods used to build solutions with software haven’t changed significantly for 30 years. And even though we’ve lived in the The Age of Agile for almost 20 of those years, the time it takes to deliver solutions, not to mention quality and efficacy, are still falling short of expectations.

Beyond Agile, examined why software development failed to meet the lofty aspirations of the Agile Manifesto founders. The book also explained how to avoid some of the classic mistakes of Agile software development.

A talk given by one of the creators of the Manifesto for Agile Software Development, Dave Thomas, underlines the dysfunction caused when the principles of the manifesto are misinterpreted or distorted to fit an enterprise IT agenda. He half-jokingly attributes the downfall as our tendency to turn adjectives into nouns; so instead of the title Manifesto for Agile Software Development, people turn it around a truncate it to — Agile Manifesto, thus losing its meaning and purpose.

Beyond Agile explores many of the noteworthy lightweight methodologies in use today, some of which pre-date Agile. This provides context and explains why software development often fails to meet expectations. Another founder of the Manifesto for Agile, Alistair Cockburn, puts it simply:

“The Agile Manifesto was the product of seventeen people from different schools and backgrounds. No one person is responsible for the words we came up with — it is clear that it was the product of all seventeen people. The addition or removal of any one person would have changed the outcome, something we recognised and discussed at the end of that meeting.

Whether you think ‘Agile’ saved the world or poisoned it, be sure always to recognise that it grew from a rich compost (joke intentional) of backgrounds. The next time you read a would-be history of the Agile movement, look for all those names. If you don’t see them, it is not a history; it is one person’s recounting of their journey, years after the event (as indeed, this one is).”

Cockburn’s statement gets to the heart of why Agile is such an enigma: few people bother to understand or respect it for what it is, choosing instead to follow second-hand interpretations of the manifesto or, worse still, adopt a consultant’s cheap imitation of it.

Agile often fails, not because it is inherently flawed or lacking in some respect, but how people interpret and apply the principles. The Agile Manifesto and its core principles are, in fact, quite logical and straightforward. The problem with Agile is it was released into the world and then distorted by people unwilling and unable to think about the true meaning and purpose of the method. Kent Beck, the creator of Extreme Programming and another of the creators of the Agile Manifesto, calls this the ‘staring dog problem.’

“If you try to point something out to a dog, it will look at your finger,” he explains. “If you explain an idea in terms of concrete practices — like test-driven development, pair programming, continuous integration — people will fixate on the practices and stop thinking.”

Kent is calling out that often we’re unable to make the cognitive leap between practice and application. Scrum and other so-called enablers of Agile fail precisely because they compensate for lack of intellectual integrity. They shroud important ideas like user-centricity and fast iteration in rituals and catchy names. The Agile manifesto left too much room for interpretation and not enough specific rules. It feels like we’re back where we started in the late twentieth century with powerful technology and tools but no idea how to make them relevant to the ever changing needs of our enterprise users and customers.

Beyond Agile, describes three outcomes Agile practised well delivers:

1. Eliminates (or at least reduce) waste in software development;
   
   
2. Creates software that people use; and
   
   
3. Delivers projects on time and budget.

An analysis of fifty thousand software projects conducted by The Standish Group in 2015 found 29% of projects was successful (meaning they were delivered on time, on budget, and to a satisfactory standard). The remainder, 71% were either failures or failed to meet expectations. Just think about that for a moment: starting a large-scale technology project, there’s only a 1 in 3 chance of success. Not very good odds. The same report in 2018 showed the situation got worse, with only 23% of projects successful and 77% unsuccessful.

Other sources corroborate these stats. In a paper presented to the 7th International Conference on Knowledge Management in Organisations, Stanley and Uden proved that software projects typically overrun their budgets by two hundred per cent, and exceed their scheduled completion dates by fifty per cent.

But it’s the cost of project failures that’s the real issue here. It’s nothing short of tragic:

– The cost of re-worked and abandoned systems costs the US economy an estimated $75 billion per year;

– In Australia, $5.4 billion is wasted each year on IT projects that don’t deliver value or are abandoned completely;

– One project alone — the infamously abandoned National Health Service patient record system in the United Kingdom — cost taxpayers £10 billion.

To put those numbers into perspective, the United Nations estimates that a yearly investment of $267 billion would end world hunger by 2030. A meagre $175 billion per year would eliminate extreme poverty globally.

It’s been almost two decades since the Agile Manifesto was written, and even longer since Winston Royce published Managing the Development of Large Software Systems. But little has changed and no single methodology has improved software development; it’s still a hugely expensive and wasteful endeavour that rarely meets expectations. And that’s hard to accept, in any era.

It shouldn’t be beyond humans to come up with a better, more reliable way of developing software solutions. Just as garment makers and car manufacturers in past centuries industrialised their domains, so too software coders need to do the same. Digitisation begets automation.

Methods like Beyond Agile address some of the shortfalls, but it’s by no means a silver bullet. The method requires skilled people, disciplined practices, full engagement with users and problem owners.

When I Googled ‘Agile’ I was hoping for the answer to some of the more taxing questions surrounding the method. What I came away with was this: to make Agile software development successful, there are four areas that align closely to the original manifesto principles:

1. Start by defining the challenge or problem to be fixed
2. Define the project outcome in terms of a quantifiable measure
3. Describe the first 6–12 features or functions the solution must have
4. Identify the first half a dozen users you’ll engage at the outset and throughout.

Rinse and repeat.

Digital Village is a community of IT professionals dedicated to providing flexible IT project team solutions in an authentic Agile way to enterprises in Australia and New Zealand.