Principles Over Process

Agile - a failed development methodology

Agile is a failed development methodology. Now, that’s a bold statement, so keep reading to get to the nuance of why I say that. In order to support this claim, let me start with an overview of several crises in the history of the software development industry. The origin of the term “software engineer” is not clear but many claim that the term was coined in the Fall of 1968 at the Garmisch-Partenkirchen conference as a challenge to the software community to get its act together and start rationalizing the software production process. From the 1970’s into the 1990’s software development was equated to physical engineering and practitioners attempted to borrow as much as possible from existing methodologies. This approach led to clearly defined phases from requirements to deployment and it was termed "waterfall" because teams complete one step, completely, before moving on to the next.

In the early 1990s, as PC computing began to proliferate, software development faced another crisis but this one was of productivity. Industry experts estimated that the time between a validated business requirement and an application in production was about three years. Of course, business requirements changed much faster than that and thus the applications were almost always out of date. Contrastingly, projects in civil or mechanical engineering rarely change over the course of a decade or more. The design of a bridge is very likely to not require modification in a year or two. Mary Poppendieck, in her talk Six decades of software development, shares that pre-1990s software development was actually more iterative and incremental than what it became. 

In February 2001, at a ski resort in Utah, seventeen people representing different development practices such as Extreme Programming and Feature-Driven Development, gathered to discuss an alternative to the document heavy processes that existed at the time. What they developed was a set of principles, such as:

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

• Business people and developers must work together daily throughout the project.

• Working software is the primary measure of progress.

Agile picked up where Waterfall left off. The promise of Agile (uppercase) was that the software development could change with changing product requirements allowing it to be more agile (lowercase). Agile as originally intended, a set of principles, was brilliant. However, putting a process-first approach doomed it to failure. What Agile turned into was a set of processes or ceremonies that supposedly if a team followed they would be agile. These ceremonies include sprint planning, stand-ups, retrospectives, etc. While nothing is inherently wrong with these ceremonies, they shouldn’t replace understanding and adhering to the principles. Applying Agile ceremonies to a Waterfall project, which is very common, just leads to frustration. This problem of ceremony over principles was noticed as early as 2008 when James Shore from the XP community wrote about the cargo culting of Agile. 

For these reasons and more, I’m a huge advocate for product development teams following a set of principles and not a methodology. Principles that are important include things like: 1) start with customer outcomes that are grounded in insights 2) state your hypotheses and assumptions about customer outcomes 3) leverage customer insights to explore multiple paths and guide product direction and 4) learn efficiently, adjusting course when necessary.