The Agile Revolution

Reduce project failure rates through adoption of Agile

The number of high-profile IT project failures is well-documented, with the oft-quoted figures for failure rates coming somewhere between 50 percent and 60 percent. Alarmingly, even when delivered, up to 65 percent of functionality is rarely or never used. Other analysis has suggested the situation is worse. The Royal Academy of Engineering and the British Computer Society found that only around 16 percent of IT projects could be considered truly successful in their research last April.

"At your company, when application software projects fall, what is most likely the root cause?"

Source: Adopting Agile Development Processes, Forrester Research, Inc., March 2004

Whatever the actual figure, the fact is that UK firms are throwing away billions on software projects that fail to meet requirements. But despite the frequent warnings about the high rates and costs of failure, firms are still willing to take the risk and commit the huge outlay required for new technology projects. Organisations seem to be tackling their software projects from the wrong angle, and measures for improving their chances of success are eluding them. A change in approach is clearly required.

Issues with traditional approaches

Creating a software system is a highly complex task. Very detailed and extensive methodologies have been established for firms to use as step-by-step guides to the development process. These methodologies usually prescribe a "waterfall" approach to developing software, wherein the requirements are fully defined and 'locked down', then a design is fully created, the code is completely developed, and all testing testing takes place at the end. Each phase is a complete activity that leads to the next. These traditional development methodologies take a granular approach, attempting to detail every aspect of the future development process right at the start. This type of heavyweight methodology has documentation and rigidity at its heart. The project team is expected to conform to a structured plan, to go away and work from this as a separate unit from the customer, and to return with a completed piece of software within the timeframe and budget conforming to initial expectations.

But therein lies the problem. Initial expectations are likely to have changed dramatically over the course of the development, which could typically be anything from one to three years under the waterfall method. Indeed, by the time a customer has got to the stage of fully defining the detailed requirements, business needs may have changed. Whether through a change in business requirements or a change in market conditions, any company is going to be moving in a different direction - however slight - a month or so down the line, let alone a year or more. And just as businesses are not static entities, nor are their software projects.

Traditional waterfall methodologies require that customers tell the developers everything that they might want in the system before the next waterfall steps can occur. This usually results in customers including everything that might have any possible value, however slight or unlikely. Along with business changes, this contributes to the 65 percent of delivered functionality rarely or never used. A waterfall approach does not allow either for the design process to overlap with the requirement process, or influence it in any way. This means issues arising out of the design process, which create the need for changes, can only be taken on board by the project team if a rigid and constricting change control process is followed. There is also no opportunity to measure progress along the way, as the customer is just handed the end product - by which stage any mistake will have evolved to a monumental problem driven deep down into the heart of the system.

In response to these rigid methodologies, iterative, lightweight development processes were born - now referred to as Agile.