The Future of Software Development (and my future)...

I've been thinking a lot about this topic lately, especially since I'm just finishing up a very intense 4-month development cycle. You can't call a 4-month project a death march, really - perhaps death sprint is a more appropriate term?

At any rate, I think that we're seeing the beginnings of a shift in software development. You have to navigate through the hype from the gurus and the quacks to see it, but if you cull out what's interesting and important from current fads and fashions, you start to see something emerging from the following trends:

• Extreme/Agile Programming


• Open Source


• Outsourcing


• Web Services


• Generative Programming


• Software "Factories"


• The Rise of Dynamic Languages


• New Internet-Based Innovations (e.g., RSS, Podcasting)


• Microsoft's Slide Into Irrelevance

On the face of it, a rather cliched list. But each of these trends will contribute something crucial to the art/craft/science/business of creating software - and those contributions will not be the aspects that are getting the most hype right now.

Each trend deserves serious consideration, and I'll do that over the next few months. But for right now, I'll give you both the punch line (since every attempt to predict the future is a joke) and the setup.

The punch line:

There will never, ever be a substitute for talented software developers. No technology, no process, no other innovation will ever change this.

The demand/need for software will continue to accelerate. Software will disappear into the fabric of our everyday lives as virtually every man-made device becomes a computer. This will fuel the demand for effective integration of every thing.

The level of abstraction will continue to rise and so will the capabilities of our applications - but this will not significantly (or even slightly) lower the complexity of creating usful software.

The new software will be designed and created at an extra level of indirection, and we will have to evolve from programming systems to meta-programming of meta-systems.

This is good news for talented people who keep their skills sharp and relevant. Only really smart people with really good skills can tame software complexity.

The Setup

Christopher Alexander's single overriding rule for A New Theory of Urban Design:

Every increment of construction must be done in such a way as to heal the city.

I've quoted this before, in a slightly different context. At the time I thought it only applied to software maintenance. But I've come to an important realization:

Dividing development from maintenance is artificial and arbitrary. All software creation is software maintenance, and vice versa.

Alexander again:

...the word "heal" must be understood in its old sense of "make whole". It includes not only the repair of existing wholes which are there already, but the creation of new wholes.

Thus, Alexander's overriding rule is directly applicable to software. All increments of software creation must be done in such a way as to heal the software - that is, to make it whole..

Refactoring is only the best and most obvious example of this. In fact, all aspects of software creation should be indirectly driven by this overriding rule. Test-driven development is a way to heal the software. Inspection is a way to heal the software. Patterns are a way to heal the software. Good software engineer processes are a way to heal the software. Aspect-Oriented Programming is a way to heal the software.

All of the worthwhile techniques of software creation are deeply interrelated parts of a process that produces an organic wholeness in our software. When I say "process" , I don't mean it in the way we usually use it in software engineering. I mean something more fundamental - not a series of procedures, like a manufacturing process, but a set of actions, functions, and changes that advance us towards a goal. Exactly what this process is remains unknown; we can see and understand fragments of it - the trends listed above represent those fragments - but we don't know how to describe it yet, much less practice it.

This process will be interpretative, not mechanical, and thus can only be discovered and executed by talented software developers. All of the mechanistic aspects of the process will be automated, leaving only those aspects that require taste, judgement, and talent.

Funny/Weird Lyric of the Day

I wanna be like Captain Kirk
Get up every day and love to go work
Don't wanna be like Mr. Spock
I want to kick out the jams and rock the block

Bob Schneider, "Captain Kirk", I'm Good Now

PopTech 2004
On a less incendiary note, if you really want to hear what smart people are thinking about the present and the future, check out these podcasts from the PopTech 2004 conference on IT Conversations. Strongly recommended:

• Adrian Woolridge on why connected politics is not necessarily left-wing politics


• Richard Florida on the rise of the creative class


• Thomas Barnett on the Pentagon's new map


• Joel Garreau on fundamental changes in human nature

Very cool stuff. They're all good, but Garreau's telling of how DARPA has created telekinetic monkeys will blow your mind.

Night of the (Shrill) Generals
I missed this last week on the Shrillblog. Read the whole thing; it's a shocking indictment of the Bush Administration's military strategy by 7 of our most eminent retired military leaders. Here's one quote, from Lt. Gen. Claudia Kennedy (Army deputy chief of staff for intelligence, 1997-2000), that really jumped out at me:

Rumsfeld proudly announced that he had told General Franks to fight this war with different tactics in which they would bypass enemy strongholds and enemy resistance and keep on moving. But it was shocking to me that the secretary of defense would tell the Army how to fight. He doesn't know how to fight; he has no business telling them...As he [Rumsfeld] was being briefed on the war plan, he was cherry-picking the units to go. In other words, he didn't just approve the deployment list, he went down the list and skipped certain units that were at a higher degree of readiness to go and picked units that were lower on the list -- for reasons we don't know.

Unbelievable. It's like LBJ picking bombing targets in Vietnam. It's like Churchill telling his commanders (pre-Montgomery) how to fight Rommel. It's like Hitler moving little flags around the map as the Red Army rolled from the Volga to the Vistula. In short, it's like every other damn-fool politician telling their generals how to fight throughout history; a recipe for failure, even disaster. Rumsfeld's incompetence, fortunately, had no effect on their ability to do the job they were trained and equipped for - defeating the Iraqi military. But it then doomed them to a job they're neither trained for nor equipped to do well.

Brad DeLong puts it best:

They are the finest and best-equipped battlefield soldiers in the history of the world. THEY ARE NOT ARABIC-SPEAKING MILITARY POLICE! THEY SHOULD NOT BE USED AS IF THEY WERE ARABIC-SPEAKING MILITARY POLICE!!

A Little Perspective
This thoughtful article in today's San Jose Mercury News caught my attention:

Let me say at the outset that if you're one of those people ready to scream because American sensibilities seem to be swinging hard to the religious right, your target is not evangelical Christianity, it's evangelical politics. The problem with missing the distinction: If the left keeps attacking and marginalizing the religion, it will only make things worse.

Many overlooked the political power of conservative evangelical churches largely because they could not imagine so many people could have such a different worldview. That kept them from understanding how the culture of evangelical churches -- set up, after all, to convert people -- can be an ideal political mobilizing machine. There's a regular time and place to meet, a sense of mission, small communities accustomed to teamwork, and leaders who are often strong communicators.

Karl Rove got it. He incorporated the agenda laid out by conservative evangelical leaders, and those leaders in turn encouraged their network to support Republican causes and candidates. Other political groups seem to have given up on evangelicals altogether, and some critics have resorted to ridicule, which has worsened feelings of alienation among evangelicals.

I wrote about moral fashion a few days ago, and I still think it's a relevant point. But that is clearly only one side of the story. The majority of evangelicals across this country aren't marching in lockstep behind Rove, Dobson, and Bob Jones. They're mostly ordinary, middle-class folks with the same problems and concerns as most other ordinary, middle-class folks. They also happen to believe that the tenets of their faith are necessary not only just to cope with those problems and concerns but are necessary to solve them as well. That doesn't necessarily mean that they all want to erase the separation between church and state, or overturn Roe v. Wade, or ban gay marriage. They're not stupid, they're not ignorant, they're not ill-informed, and they're not racist bigots. And they don't like being labeled as being any of those things any more than moderate democrats and progressives like being labeled as radical, unpatriotic, immoral, and elitist.

All that alienating them accomplishes is to drive them further into arms of the far right. But because they're not stupid, and they're not fanatics, and they're not ignorant or ill-informed, chances are good they'll realize that the economy is continuing to falter, that the war in Iraq is continuing to not go well, that the costs of medical care and education are continuing to rise, and that in general things aren't improving for them despite the pandering of the administration and the far right.

And when they do, will democrats and progressives treat both their values and concerns with respect, find common ground with them, and start working together on a shared agenda? Or will they continue to dismiss their values and ignore their concerns? Because, if it's the latter, the Democratic party will continue to lose elections just like they lost this last one.

Yet Another Map
This one really illustrates the urban-rural split. Of America's major cities, only Houston, Dallas, Phoenix, San Antonio, and Denver went red; 32 of the top 50 (by my unofficial count) were blue. (A Tip o'the Hat to Social Tonic).

The Lighter Side of the Insurgency
Juan Cole, after discussing incidents all over Iraq yesterday:

On a lighter note, it is hard to avoid observing that al-Baghdadi castigated Bush's administration as "fundamentalist" and "right-wing." When even the Sunni Salafis of Mosul consider you too fundamentalist and right-wing, you have probably gone too far.

Sore Winners: The Winner
This loathsome, bile-spewing rant takes first prize. Money quote, as the avatars of the hate brigade like to say:

If anyone needs to work to “bring the country together” it’s those on the left who have divided it so badly. Those who sought to destroy this great man should get down upon their knees and beg the victors for mercy. And maybe, just maybe, we’ll let a few of them linger on for the simple reason that they amuse us. My life’s goal is to see the Democratic Party virtually obliterated and left as a rump of people like Stephanie Herseth who both mostly agree with us anyways and are easy on the eyes.

That’s the future of the Democratic Party: providing Republicans with a number of cute (but not that bright) comfort women.

Memo to the rest of us: It's ok - in fact it's a sign of a thinking adult - to loathe both Ann Coulter and Barbara Streisand, to dismiss both the Swift Boat liars and Michael Moore, to reject both Fox News and IndyMedia, and, especially, to condemn bigoted cretins like this creep.

Sore Winners, A Continuing Series...
Here's a two-for-one special. Courtesy of Matt Welch, who's convinced me to wear my coastal elitistism as a badge of honor.

The State of the Onion
The Onion tells it like it is. Again.

A Contrarian Viewpoint
Paul Graham has a very different take on what decided this election.

Maps
Electoral maps are all the rage these days. But the very best one (aesthetically speaking) can be found at one of truth and beauty's favorite rest stops, the always compelling 101-365.

Moral Fashion
The right-wing meme o'the week last week, coming from everyone from Rush Limbaugh to Instapundit, was that this election was not about gay-bashing. It was about morals and values, and the American people picked the candidate who embodies those morals and values - one of which just happens to be that gay marriage is a threat to the sanctity of heterosexual marriage and the nuclear family.

And the right's spin does indeed contain a grain of truth. A key differentiator in this election was the acceptance or rejection of a certain set of values. The exemplars of those values are right-wing Christian extremists, who've won a propaganda war that asserts that their values are morally superior and everyone else's are morally inferior. And propaganda is exactly the right word; the assertion of moral superiority has been sold, using all the techniques of large-scale public relations and advertising. And it has found a very receptive audience in a group of people looking for certitude in an era they find increasingly incomprehensible.

Paul Graham, in an essay entitled What You Can't Say, writes:

What scares me is that there are moral fashions too. They're just as arbitrary [as consumer fashions] and just as invisible to most people. But they're much more dangerous. Fashion is mistaken for good design; moral fashion is mistaken for good. Dressing oddly gets you laughed at. Violating moral fashions gets you fired, ostracized, imprisoned, or even killed.

Neither Graham nor I am suggesting that imprisonment or death is the consequence of the latest moral fashions. What this is really about is a struggle for power, as was the English Reformation or the Thirty Years War or our new, modern-day Islamic Reformation. Dressing it up as a struggle between God-fearing Christians and atheistic, homosexual-loving, liberal moral relativists is a more than convenient rationalization tailor-made for a segment of the elite that seeks to gain unencumbered political control.

I'm not trying to trivialize anyone's faith. I've a deep respect for the truly devout, who actually try to live their lives according to Christ's/Yahweh's/Mohammed's/Buddha's teachings. But, like Ken Layne, I've seen and known too many people whose trumpeting of their faith was a cover for the hypocrisy of their daily lives.

The cure for fashions that lack real aesthetic or moral values is to repudiate them. Hold them up to the light of day, examine them, criticize them, and (especially) mock them. If there is anything good and true hidden inside the ephermal mores of the moment, it will emerge all the stronger for the being challenged. And what is dishonest and false will be consigned to history's dustbin.

Sore Winners, cont.
Matt Welch, on the healing voices of the right.

Electronic Voting Glitches
A few isolated incidents? Or the beginning of a steady stream of reports? Note that in the third link, it's a county that heavily favored John Kerry. Not counting votes accurately is not counting votes accurately, regardless of who it helps.

There's no evidence of any kind of widespread fraud in the election and therefore no reason to give any credence to loony conspiracy theories. But we still have electronic voting machines that are insecure, unreliable, and wide open to fraud. And we have a number of non-technical problems that we're still failing to address - registration, polling place lines, and ballot design.

This article says it best:

What worries voting reformers more is that Congress, the White House and the states will see the lack of a 2004 election meltdown as vindication of America's voting system and neglect the tools of democracy another four years.

"There's a huge danger," said Ted Selker, a Massachusetts Institute of Technology computer scientist who co-directs the Caltech/MIT Voting Technology Project.

"If you're an election official, it's going to be very hard to go to Congress and say we need more money," said Doug Chapin, executive director of Electionline.org, a nonpartisan clearing house for voting reform information. "What they could say is we need to finish the job we started."

Now that election-year pressure is off, will a sharply divided nation forget about shoring up voting systems poorly suited for sharp divisions?

Voting reformists are crossing their fingers and hoping not.

"Is the testing of voting machines satisfactory?" asked Selker. "No. Are the design standards appropriate? No. Are they all improving? Yes. But do we have good mechanisms for improving them? No."

Beyond those issues, the Elections Assistance Commission needs to find the best way to count provisional ballots, develop reliable registration databases and verify electronic votes.

Not Red, Not Blue. Purple

This is fantastic. Edward Tufte would be proud.

What To Expect

There will be a smokescreen of uniter-not-divider rhetoric for a short time, but I think it will quickly be replaced by a mean-spirited triumphalism. One of the first manifestations will be an all-out attack on the mainstream media. Here's a clue - keep in mind that these guys are sore winners, too.

Now, the mainstream media deserves plenty of criticism for its faux-objectivity, laziness, and status quo elitism. But that's not what will be under attack. Instead, the objective is to neutralize any journalist and any story that departs from the well-established story lines of the Republican mainstream. If they can't be coerced into being Fox-lite, then they'll be battered into submission.

Jay Rosen has a fascinating, and more optimistic, take on this. And I think he's right. There's still 49% of us who want to hear the other side of the story. That means there's a real opportunity for anyone who can speak truth to power and do it well.

This Election, a Coda

Damn. Damn. Damn.

A few observations:

• Gay-bashing and fear-mongering is a winning electoral strategy


• Placing your hopes in voters aged 18-29 is a losing electoral strategy


• This election was a referendum on the Bush administration, and they won.


• 51% is sufficient to declare a mandate, especially if you cement your control of Congress.


• Karl Rove, not GWB or Dick Cheney, is now the most powerful man in America. This is his victory.

The last point, I think, deserves a little more explanation. I think it's safe to say that a lot of Republicans were worried they would lose this election. Lots of folks on the other side, like me, believed that they would. What seems to have put the GOP over the top was the, ahem, values thing. And as far as I know, that was Rove's strategy. So I think everyone who got elected, from Bush down, is feeling both grateful and indebted to Rove.

Four more years, indeed. Welcome to the new intolerance.

The Worst Possible Outcome...

The idea that North American society can exacerbate its drive to economic bimodality without serious fissuring, up to and including a constitutional crisis in the U.S., is not likely...

W.R. Clements, Quantum Jump, 1998

The VoteMaster Revealed
It turns out the guy behind www.electoral-vote.com is Andrew Tanenbaum, who geeks like me know as the author of the Minix operating system (a precursor of Linux). He's currently a Professor of Computer Science at the Vrije Universiteit in Amsterdam, The Netherlands (he's American, his wife is Dutch).

There seems to be a lot of interesting stuff coming out of the Netherlands these days.