Whoops! They did it again!

Making the rounds is an article in The Register with the delightful title, Study finds 268% higher failure rates for Agile software projects .

So of course I had to dig in and find out what’s going on because almost every time I read an article like this, it turns out to be by some company selling their alternative to agile. This one was no exception. Though I can’t find the actual study, I did find a write-up of the study’s findings .

It’s by a company called Engprax . I’ll skip most of what’s in there and instead focus on one key paragraph:

Today, new research conducted for a new book, Impact Engineering, has shown that 65% software projects adopting Agile requirements engineering practices fail to be delivered on time and within budget, to a high standard of quality. By contrast, projects adopting a new Impact Engineering approach detailed in a new book released today only failed 10% of the time.

Oh! That’s pretty horrifying! If true, we need to dump Agile and immediately switch to Impact Engineering!

Hmm, what’s Impact Engineering ? I’ve never heard of it. It turns out that it’s a new approach to project management detailed in a book published by “Engprax Ltd.”

OK, so the study’s findings are very self-serving, but to be fair, it doesn’t mean they’re wrong. So let’s look at a key line from that paragraph:

65% software projects adopting Agile requirements engineering practices fail to be delivered on time and within budget, to a high standard of quality.

First, what’s quality? I’ve been consulting for many years and “quality” is one of the worst catch-all terms I hear. It’s like “dysfunctional,” a word that many use but few define.

Is quality lower defect rates? Higher performance? Satisfied customers? Perfectly designed code? Who knows? Not one of those metrics requires or excludes the others.

Engprax did not define quality.

So what about that “on time and within budget”? That’s pretty damning, eh? But what does it mean? A day late and a dollar short? Who knows?

Because Engprax failed to show real numbers, for all we know, Impact Engineering might be more likely to be “on time and within budget,” but have everything cost twice as much and take twice as long. Because they didn’t link to the research, apparently you have to buy their book to find out. I’m not going to buy their book because I’ve seen so many of these fad methodologies come and go.

What is Agile, Lean, and Impact Engineering?

To understand what’s going on, we need to know what agile, lean, and waterfall are. If we don’t have working definitions, we probably aren’t talking about them same thing. Fortunately, here Engprax provided working definitions.

Agile: Development starts before clear requirements, no complete specification, significant changes late in development.
Lean: Only working on one project at once.
Impact Engineering: Use of all engineering practices studied which increase success rates.

This is so jaw-droppingly dishonest that I hope whoever wrote this has a sense of shame. It’s a straw man relying on the fact that agile snake-oil salesman have so sufficiently muddied the waters that we can’t see the bottom.

I have been working in software for decades, using Agile, Lean, and Waterfall. While they don’t cover waterfall, I can address the agile and lean definitions: they’re total BS.

First, read the Agile Manifesto . Read through their agile principles . In particular, read the part where they write:

We abhor clear requirements.

Have you wondered why they wrote that? What? You can’t find where they wrote that? Because they didn’t.

Out of the twelve principles they cite, this is, for our purposes, the most important:

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

“Change” is key word here. The clear requirement is software that satisfies the customer’s needs, but those needs are not always clear up front. But to understand what that means, we have to go back to some basics.

What is Project Management?

There are many people who think that project management is about delivering projects on time and on budget. This is wrong and a simple thought experiment should make this clear.

You’re an experienced project manager working for Acme Corp. After completing your fifth project, your pointy-haired-boss pulls you aside and informs you that you’re about to be fired. He’s run the numbers and all of your projects have run an average of 50% over budget. You’re costing the company money.

So you failed as a project manager, right?

Then the CEO steps in, livid, pointing out that while this is true, all of your projects have been wildly successful in terms of customer happiness, revenue, and employee retention. Your projects have driven the financial success of the company.

Of course you need to control costs, so project management is about delivering successful projects while controlling costs. And “successful” needs meaningful metrics to judge it. That metric is not “X amount over budget.”

You build your project once. You keep your customers for years. It’s the latter point which defines success, not the former.

So where’s the disconnect? And what costs are we controlling?

The following is, like many things, a gross oversimplification.

Agile

Speculative projects are going to risk having to pivot quickly when uncertainties arise. If they weren’t, they wouldn’t be speculative. This is where agile shines. I’ve been on plenty of these and yes, the best ones have clear requirements ... until they get into the areas where we don’t know how the customer is going to respond.

The biggest cost threat is change and uncertainty. It’s appropriate when you are trying to create a new market or a novel product in a market.

Lean

By contrast, lean projects are where the market and customer behavior are fairly well-known and you’re creating a proven product, but adding differentiators to distinguish your product from the competitors. So you build a well-defined MVP as your base and then iterate like mad, based on customer feedback, based on how they receive the differentiators.

The biggest cost threat is waste: building something you don’t need. Hence, the focus on a well-defined MVP as the base of your work.

Structured

Also known as “waterfall”, it’s appropriate where the market has solidified (ossified?) and you absolutely must not fail to meet requirements. Many utility services fall into this category. Telecoms, electricity, ISPs, and others which must have incredibly high reliability and predictability.

The biggest cost threat is deviation from requirements. At this stage, your requirements are highly detailed, up front, and there is very little change or uncertainty. Agile needs to “fail fast,” but that’s probably not a great mantra when building a nuclear reactor.

We can sum these up with the following image:

A graph showing agile with the most uncertainty, lean with middle, and structured with the least uncertainty. — The height of each bar conceptually illustrates the appropriate uncertainty level.

Thus, for a given cost threat, you choose an appropriate project management solution for that threat. If I’m an employee talking to HR, I don’t want a lot of change and uncertainty. I also don’t want that with the janitorial staff. Of course, there are janitors who have to deal with lots of change and uncertainty. They’re called Navy SEALs.

This also has another implication: you have many different projects and those projects have different cost threats, so don’t try to force everyone in the company to operate off of the same project management model.

Enter The Agile Industrial Complex

Jon Kern, one of the authors of the Agile Manifesto, argues that today we’re struggling with what he calls the Agile Industrial Complex . I’ve seen them in action and they have several key faults:

Overemphasis on certification over practical experience
Proliferation of “Agile Coaches” with limited real-world experience
Misinterpretation and rigid application of Agile principles

A key driver behind the organizations who are all promoting different versions of “agile” is obvious: money. One well-known Certified Scrum Trainer® (CST) wrote about how he lost his CST when he pointed out that Scrum is not always appropriate. This was apparently perceived as a threat to the Scrum Alliance and they have a clause in the license that you will not disparage them. However, as far as I can tell, you’re not allowed to read the license until you pay them a lot of money for one of their courses. (I can’t link to his writing because he quietly deleted that blog post)

Others are churning out highly-paid consultants specializing in SAFe , a framework I’ve never worked with, so I can’t directly comment on it, but it has many detractors, most focusing in the fact that SAFe isn’t agile. That being said, maybe it’s a solid approach. I’m not qualified to say if SAFe is good, but I’m qualified to say that it’s not agile.

Solution

Pick an appropriate project management solution.

Is your project highly speculative and you’re unsure how your customers will react? Go agile.

Is it a quick internal project that can be safely corrected if it gets a few details off? Go lean.

Is it a cloud service utility to take on Amazon and Google? Go structured.

None of these mean unclear requirements or specifications. However, the detail of your specification vis-a-vis areas of uncertainty will change depending on your approach.

It’s true that agile focuses on outcomes more than process. It’s true that agile focuses more on building value than controlling costs. But if you choose a project management approach that emphasizes controlling the cost of change and uncertainty when your project has little change or uncertainty, it’s not agile that failed. It’s you.

The Agile Industrial Complex