The Future of Software Development...Boehm flames Agile/XP 15 years before they exist

[As a supplement to my earlier rant about Agile/XP]

...evolutionary development also has its difficulties. It is generally difficult to distinguish it from the old code-and-fix model, whose spaghetti code and lack of planning were the initial motivation for the waterfall model. It is also based on the often-unrealistic assumption that the user's operational system will be flexible enough to accomodate unplanned evolution paths...evolutionary development projects have come to grief by pursuing stages in the wrong order: evolving a lot of hard-to-change code before addressing long-range architectural and usage considerations.

Barry Boehm, "A Spiral Model of Software Development and Enhancement", IEEE Computer, vol.21, #5, May 1988

New Blogs to Add
Joanna Rothman
Thomas Barnett

I want to find more weblogs like these two and replace some of the popular blogs that everyone reads. Too often, the so-called "top-tier" weblogs are second- or third-hand echoes of the ideas of the really innovative and interesting thinkers out there.

Listening to podcasts of the PopTech 2004 conference really opened my eyes to this; here were a bunch of people, most of whom I'd never heard of, talking about subjects they knew best and were the real thought leaders in.

So I want to read/hear more of these voices directly instead of through the filter of the most popular bloggers.

Free Trade vs. Mercantilism

Masahunyam Monani on the how protectionism is a form of tyranny, in Bubble Generation:

This is the real scandal of "free" trade today : the vast majority of Ricardian gains are pocketed by a small section of the Western population at the expense of some of the most vulnerable people around the world. Developed nation have consistently used the WTO to push the developing world into reducing trade barriers, but heavily protected their domestic markets against competition...As my b-school prof puts it, nobody has a right to deprive others of their comparative advantage. Stand up for Free Trade to End World Hunger : how's that for a modern version of Hasta La Victoria Siempre?

There is a place for protectionism and subsidies, and that's to protect nascent or developing industries from unfair or overwhelming competition. But most protectionism is devoted to mature industries with political clout.

The Future of Software Development - A Rant About Agile/XP

Here's what really bugs me about Agile/XP.

The introduction of every new software engineering fashion to us teeming masses is like Athena springing out, full-grown and dressed in armor, from Zeus's forehead; the new methodology bursts onto the scene full-grown and ready for action. There's no sense of a period of gestation or ancestry. It's just there one day, big as life and twice as natural, sweeping away all that came before it.

And XP is just like the RUP/UML which was just like OOD&A which was just like JAD which was just like RAD which was just like...well, you get the idea.

Except that Agile/XP seems to have a lot of similarity to Barry Boehm's Spiral Model, which Boehm wrote papers about back in the mid-80's. The Agile/XP mavens mostly deny this. Granted, there are some significant differences which make Agile/XP both a subset and superset of the Spiral Model. But there's also an unwillingness to acknowledge, much less build upon, the contributions of the past - particularly if they come from Computer Science.

Computer Science? We don't need no stinkin' computer science. This is engineering, boys! We've got to be practical and get things done!

I think this attitude is one of the reasons why a viable discipline of software engineering has been so slow to emerge. Let's face it: we have more in common with the fashion industry than we do with real sciences. If we tried to be an applied science instead of a fashion mill, we'd acknowledge and build upon the past. We'd make hypotheses about what works, test those hypotheses, evaluate the results, then rinse and repeat instead of throwing it all away. And we'd build up an effective body of practice whose roots are based in sound, tested theory.

Of course, this would depress the software engineering fad industry. And wouldn't that just be too bad.

The Future of Software Development - Thoughts on Agile/Extreme Programming

So much has been written about Agile and Extreme Programming that it's probably impossible to say anything new or interesting about them. But I'll try anyway, because I think there are two things worth calling out.

The critical contribution that Agile/Extreme makes to the new software is that it creates a culture of testing. Whether you buy into the notion of test-first development or not as the right way to do test-driven development, it's impossible to argue with the value of codified, disciplined, and repeatable unit testing as an essential part of the coding process. In a very short time this meme has become so firmly entrenched that coding practice has been changed, probably forever. To be sure, more people are still talking about test-driven development (TDD) than doing it; but peer pressure and the desire to be on the cutting edge will fix that.

To put it another way, testing is cool. It's hard to be taken seriously by other hot-shot code jocks if you can't point to a battery of unit tests for your code. It's hard to be taken seriously as a development organization if your build system doesn't automate the running of those unit tests as part of the build. It's hard to be taken seriously by your peers if your automated unit tests fail and break the build on a regular basis. The meme of test-driven development - more than any specific practice of testing - is the big win from Agile/XP.

A side effect of TDD, and it's almost as important as the cultural shift, is that the act of writing effective unit tests requires developers to write code that can be easily tested. Making code testable frequently means breaking it down into small, discrete, coherent chunks instead of writing long, procedurally-oriented methods. This, of course, is refactoring - another cultural shift that is changing the practice of coding. (Side note: of course, the practices of unit testing and refactoring have been around since people started writing code. The real change is the culturally-driven codification of these practices - suddenly, they're "just the way you do things"). The practice of writing testable units of code, combined with refactoring and unit testing, will start producing higher-quality code (at least, in the hands of a reasonably skilled developer). It may even produce truly reusable code.

A have one quibble with TDD, and it's only a quibble. The emphasis on unit tests tends to obscure the equally important need for component-level, system-level, and stress/performance testing. The standard Agile/XP answer is to use mock objects, but I don't think this is sufficient. We need to codify the entire hierarchy of testing, not just the lowest level. If you do that, then our QA departments could actually start doing actual Quality Assurance instead of just being our bug catchers.

Unfortunately, the Agile/Extreme guideline of avoiding planned, up-front design plays to the worst instincts of the coding cowboys who just want to start writing code as soon as possible. I'm aware that Beck, Fowler, et. al., advocate a light-weight process of incremental and evolutionary design, not the abandonment of design. But because Agile/Extreme spends so much time talking about the development process and so little about design, that message gets lost. It doesn't help that the hypesters have seized on XP as the latest silver bullet that will magically solve every software development problem and allow you to deliver full-blown systems in some ridiculously small increment of time. The result is that as I write this, there are a lot of projects out there that are trying to do Agile/XP and failing.

That isn't an indictment of Agile/XP per se; but it is a critical flaw in the process. What's worse is that failing to address this flaw probably dooms a project right out of the gate. No set of programming practices, no matter how effective, can save a bad design, much less a non-existent one.

We need a set of guidelines for Agile Design that describe how to execute an lightweight, adaptive, evolutionary, and rapid design process that accomodates changing requirements and market conditions, gets the big decisions right, and seamlessly integrates with Agile/Extreme Programming methods.

Of course, almost every software design methodology developed over the last 40 years has attempted to meet pretty much the same goals. And most have failed, to a greater or lesser degree.