The Future of Software Development - Thoughts on Agile/Extreme Programming

So much has been written about Agile and Extreme Programming that it's probably impossible to say anything new or interesting about them. But I'll try anyway, because I think there are two things worth calling out.

The critical contribution that Agile/Extreme makes to the new software is that it creates a culture of testing. Whether you buy into the notion of test-first development or not as the right way to do test-driven development, it's impossible to argue with the value of codified, disciplined, and repeatable unit testing as an essential part of the coding process. In a very short time this meme has become so firmly entrenched that coding practice has been changed, probably forever. To be sure, more people are still talking about test-driven development (TDD) than doing it; but peer pressure and the desire to be on the cutting edge will fix that.

To put it another way, testing is cool. It's hard to be taken seriously by other hot-shot code jocks if you can't point to a battery of unit tests for your code. It's hard to be taken seriously as a development organization if your build system doesn't automate the running of those unit tests as part of the build. It's hard to be taken seriously by your peers if your automated unit tests fail and break the build on a regular basis. The meme of test-driven development - more than any specific practice of testing - is the big win from Agile/XP.

A side effect of TDD, and it's almost as important as the cultural shift, is that the act of writing effective unit tests requires developers to write code that can be easily tested. Making code testable frequently means breaking it down into small, discrete, coherent chunks instead of writing long, procedurally-oriented methods. This, of course, is refactoring - another cultural shift that is changing the practice of coding. (Side note: of course, the practices of unit testing and refactoring have been around since people started writing code. The real change is the culturally-driven codification of these practices - suddenly, they're "just the way you do things"). The practice of writing testable units of code, combined with refactoring and unit testing, will start producing higher-quality code (at least, in the hands of a reasonably skilled developer). It may even produce truly reusable code.

A have one quibble with TDD, and it's only a quibble. The emphasis on unit tests tends to obscure the equally important need for component-level, system-level, and stress/performance testing. The standard Agile/XP answer is to use mock objects, but I don't think this is sufficient. We need to codify the entire hierarchy of testing, not just the lowest level. If you do that, then our QA departments could actually start doing actual Quality Assurance instead of just being our bug catchers.

Unfortunately, the Agile/Extreme guideline of avoiding planned, up-front design plays to the worst instincts of the coding cowboys who just want to start writing code as soon as possible. I'm aware that Beck, Fowler, et. al., advocate a light-weight process of incremental and evolutionary design, not the abandonment of design. But because Agile/Extreme spends so much time talking about the development process and so little about design, that message gets lost. It doesn't help that the hypesters have seized on XP as the latest silver bullet that will magically solve every software development problem and allow you to deliver full-blown systems in some ridiculously small increment of time. The result is that as I write this, there are a lot of projects out there that are trying to do Agile/XP and failing.

That isn't an indictment of Agile/XP per se; but it is a critical flaw in the process. What's worse is that failing to address this flaw probably dooms a project right out of the gate. No set of programming practices, no matter how effective, can save a bad design, much less a non-existent one.

We need a set of guidelines for Agile Design that describe how to execute an lightweight, adaptive, evolutionary, and rapid design process that accomodates changing requirements and market conditions, gets the big decisions right, and seamlessly integrates with Agile/Extreme Programming methods.

Of course, almost every software design methodology developed over the last 40 years has attempted to meet pretty much the same goals. And most have failed, to a greater or lesser degree.