Learning

"The best thing for being sad," said Merlyn, beginning to puff and blow, "is to learn something. That is the only thing that never fails. You may grow old and trembling in your anatomies, you may lay awake at night listening to the disorder of your veins, you may miss your only love, you may see the world about you devastated by evil lunatics, or know your honor trampled in the sewers of baser minds. There is only one thing for it then - to learn. Learn why the world wags and what wags it. That is the only thing which the mind can never exhaust, never alienate, never be tortured by, never fear or distrust, and never dream of regretting."

T.H. White, The Once and Future King

In Memoriam

Mr. Cracker April 1993 - June 25, 2005

He was beautiful. He had a gentle grace and dignity that made everyone love and admire him. And everyone who ever petted him remarked on how soft his fur was and how calm and sweet he was. Sooner or later, the same phrase came off of everyone's lips: "Sweet boy".

I brought him home the same weekend Sara and I had our first date. He rapidly became the center of our lives. Every day started with him and ended with him, and he brought a kind of joy into our lives that no words can really describe. It's hard to believe that so much happiness could come from the simple acts of walking him, feeding him, and taking care of him.

He was the aristocrat of Town Lake, the bad boy of the dog park (where he discovered the sport of Weimaraner tipping), the unofficial mascot of the K-State girls rowing team when they came for their yearly regatta, and an ambassador of goodwill for Greyhounds. He hated the rain but loved the water, shied away from crowds but would walk right up to people if they smelled or felt right, was fiercely independent but came on command when he needed to, was reserved and regal but silly and uninhibited when he played.

We would go for walks past the Four Seasons on Town Lake along the hike and bike trail, and every day people would stop and stare and ask me about him. I soon developed my standard speech about him and about greyhounds, and maybe somewhere along the line we influenced someone to adopt a greyhound.

We moved to San Francisco, and he discovered the joys of the ocean and of the mountains. He loved to run along the shore and then sit in the water almost up to his neck to cool off. I use to worry about him getting a chill - greyhounds supposedly being susceptible to that kind of thing - but we decided he knew what he was doing. When we took him to Yosemite and Mount Shasta and introduced him to snow, he was enraptured. He ran in it, rolled in it, tasted it, danced in it, and generally had the time of his life. His usual reserve disappeared at the beach and in the snow, and he expressed his joy and love of life.

His life got even better when we moved to Santa Cruz, because now there was a yard for him to lay in and catch the smells on the breeze and see and hear life go by. When we bought a house, his life was complete because now there was a backyard full of beautiful cool grass to lay in. There's a spot in the center of the backyard where the no-longer-functional sprinkler system leaks and it's cool and damp and the grass grows greener and lusher than anywhere else in the yard, and that became his favorite spot of all. I like to think he also loved the smells of the wisteria, honeysuckle, trumpet flowers, roses, and jasmine in spring and summer. It was his oasis, and we were delighted to have pleased him so.

His generosity of spirit allowed us to bring in another greyhound whose personality was 180 degrees different from his and who brought a load of anxieties and problems from having been ill-treated as a racer. He accepted her, made room for her, and never acted jealous or hurt, even when she tried to bully him. Instead, his personality rubbed off on her slowly and surely, and she became calmer, a little less anxious, and more loving as time went on.

He made us better people, too. Sara treated him like a king, and he responded by being even more loving and lovable. He was a finicky eater, but he never acted spoiled or petulant. Both acceptance and decline had a graciousness that I, awkward and graceless, admired without envy. I'm not ashamed to say that, like the other role models in my life, I wish I was more like him.

His decline was slow and gradual and then came swiftly at the end. Two winters ago, he stopped running. We thought that the arthritis he'd developed was the cause. But he still loved his walks, especially after we discovered the trails at DeLaveaga park. We could see that he was slowing down and that his back legs were getting stiff and awkward, so it was no surprise to learn that he had degenerative myelopathy that was robbing him of the ability to control his back legs. But he developed a limp and when Sara took him to the vet 2 Fridays ago, we discovered that he had cancer in his leg and his lungs. Mercifully, his suffering did not last long. We almost lost him the next week, but shifting to a steroid-based medicine got him walking again and gave him 6 more good days with us. He let us know that he was ready to go, and we called the vet and put him at peace.

I wish he were still here so I could rub his chest, kiss his head, look in his eyes and have him reach out his paw to me to tell me to rub him some more. This house feels so empty without him that I can hardly stand it. Yesterday after the vet left I though my chest would explode. But my sadness is tempered by the knowledge that he had a long, happy, and beautiful life and that the love he gave us will eventually fill the holes in our hearts back up.

I do believe that all creatures have souls and I believe in an afterlife, although I have no idea what form it takes. But I like to imagine that he's running on a beach somewhere with all his greyhound buddies that went before him, and that nearby is a snowbank that he can just dive into whenever he feels like it.

Goodbye, Mr. Cracker. We loved you so much, and you gave us so much more.


...do you know what love is?

Sure I know.
A boy loves his dog.

Harlan Ellison, A Boy and His Dog

Only in Santa Cruz

News from the land that time and neo-conservatism forgot:

Headline in today's Santa Cruz Sentinel:

Drug Suspects Rolling in Dough, Deputies Allege
Marijuana, Mushrooms, Hash, Cheesecake, and Cookie Dough found

The subtitle is only in the print version, not on-line, and I may have not gotten it exactly right. But I'm not making up the cheesecake part. Here's what it says in the 2nd graf:

...deputies found butter, cheesecake, chocolate chip cookies, cookie dough and a nut ball all laced with THC...

Thank God we're safe from these Cookie- and Cheesecake-altering fiends. I know I'll sleep better tonight.

Then, from an article (registration required, grrrr) on yesterday's record heat in the San Jose Mercury News:

Beaches from Santa Cruz to San Francisco were full of revelers, some of them nude, until the fog rolled in.

So at least they had something to cover them.

More Old-School Thinking

The very first step towards the management of disease was replacement of demon theories and humours theories by germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical solutions. It told workers that progress would be made stepwise, at great effort, and that a persistent, unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering today.

Fred Brooks, No Silver Bullet - Essence and Accident in Software Engineering

The Future of Software Development - Open Source

I think I've flogged Agile/XP enough, so let's move on to Open Source Software (OSS) development.

Once again, this is a subject that's been done to death, so it's hard to say anything new. But I can think of at least two facets of OSS development nobody seems to talk much about even though they're pretty central:

1. The Absence of Senior (Non-Technical) Management, i.e., No Suits


2. The Absence of Originality

It's hard for me to see the first as anything but positive - perhaps I've spent too long in the trenches - but it is unfortunately a partial cause of the second. On the other hand, the lack of originality in most OSS products is not necessarily a bad thing. It's even something of a boon in the area of programmer tools, as I'll explain farther down.

But first I want to take about what No Suits means to OSS. I think I can sum it up in one word: freedom. The 80/20 rule, that fundamental law of the universe that masquerades as a useful heuristic, applies to the Suits as well as it does to anything else. About 20% are competent enough not to be actively harmful to the process of developing good software. (A very small percentage are competent enough to provide the real leadership, motivation, and skilled management that actively aids and improves the process, and I have been lucky enough to work for that small percentage a couple of times in my life). The other 80% are the incompetent, egomaniacal, soul-crushing bastards we've all the bad luck to work for too often in our careers. "Pointy-haired boss" is far too forgiving a term for these fools. And doing good work in the environments they create is a Sisyphean travail.

So getting rid of the suits is a big win at least 80% of the time. And most self-selecting teams can manage themselves as well or better than the merely competent portion of the remaining 20%. The freedom gained from removing management overhead that is at best neutral but more often malignant results in software that works considerably better than the software created in for-profit enterprises. This freedom results into the ability to release new versions of the software when its creators think it's ready. It results into the ability to recast or reinvent the project, the team, or the product at a moment's notice. It results into the ability to continually refine a software product until it meets a desired level of quality and customer satisfaction.

The usual counter-argument to this is that OSS development lacks the spur of the competition that is provided by the commercial market. This point of view asserts that higher-quality software is created as a result of companies competing for customers. The ultimate driver here is the customer, and only commercial competition provides the impetus to create what the customer wants.

This might be true if in fact your average software company was driven by what customers want. In fact, they're more usually driven by a fragmented view of what the company's leaders think the customers want, which is further distorted by what the suits think they need to do to a) retain control and b) make money (usually the best way to accomplish a) is b), but it isn't always necessary).

The absence of the suits in OSS means that the development team has to deal directly with their customers. All of the usual institutional filters get removed, and suddenly developers are getting direct, unmediated feedback from real users. This kind of disintermediation can have some really positive benefits - OSS projects don't succeed unless they have users, and they don't get them unless users get something they find truly useful. So they either listen to their customers and respond to them, or they get ignored.

So it shouldn't surprise us that better software results when the feedback loop is tighter, more direct, and less subject to distortion, when the process can be as flexible as it needs to be, and when the developers have greater creative freedom in a less restrictive and less hierarchical environment. And it shouldn't surprise us when users respond by replacing commercial software packages with OSS equivalents.

Now, I don't want to give a completely Polyanna-ish view of OSS. From the standpoint of an ordinary, non-technical user most OSS products are effectively inaccessible. Even products intended for regular people are often too difficult to install, too difficult to use, or too far out of the mainstream to be comprehensible. The very best OSS products are the ones produced for technical users, especially programmers, not for people to whom a computer is a tool that helps them do their real work.

Here's where the absence of that very small percentage of highly competent executive leaders hurts. These are the people who are capable of bringing together the innovaters, the creators, the customers, the marketers, and the money to create truly innovative products. I'm not saying that OSS can't produce brilliantly innovative products. But it's a lot easier to produce products when all of the requirements are known and when the market is well understood. So OSS produces a lot of "me-too" products that are effectively commoditizing what were once profitable innovations - think operating systems (Linux, BSD *nix), programming languages (gcc, perl, python), personal productivity software (Open Office, Star Office), browsers (Firefox, Mozilla, Opera), and databases (MySQL, Postgres). I'm hard-pressed to think of any truly innovative OSS products. Maybe Apache, although that's a little bit of a stretch.

I mentioned earlier that this lack of originality is mostly a positive when it comes to programmer tools. That's because developers work in teams and usually need to use all the same tools. A decade or more ago, this was a real problem. The first IDEs, like the C++ environment from CenterLine, cost big bucks. Compilers, modeling tools, source code control systems, and the like also came with big price tags. OSS changed that. Today, anyone with access to the Internet can quickly download every tool they'll ever need to develop software. It's still true that many of the commercial tools are better than OSS ones - e.g., Idea IntelliJ and Borland's JBuilder are better than, say, Eclipse or NetBeans, and Perforce is clearly superior to CVS - but this gap is narrowing all the time. Eventually the differences in features, quality and support will diminish to the point where it's unreasonable to pay for the commercial tool. In this situation, developers don't need or even want constant, radical innovation in their toolsets. They want continuous evolution among a set of alternatives - so that they can both grow with the tool without having to confront steep learning curves on a regular basis and so that they can easily switch when something more convenient comes along.

Convenience, of course, is one of the hallmarks of commoditized consumer products. And that's what we're talking about with OSS, and that's why OSS is no threat to innovative commercial software. It is a threat to the stagnant commodization of software, and that is an entirely good thing; the alternative is having MS selling Windows, Office, and IE for the next 20 years with no more than the occasional "update".

So what does all this have to do with the future of software development?

First, both the success of OSS without suits and its inability to innovate are clues. We don't want to eliminate the suits - just subtract the losers and keep the value and innovation added by the winners. In an ideal world, we'd eliminate that 80% of managerial incompetents and keep the competent 20%. Fortunately, there is already a widely accepted way to do this - by flattening organizations and their management structures. Flattening the organization eliminates a considerable number of the suits. And that might end at least some of the internal political jockeying and infighting that, among many other negative consequences, tends to choke and distort communication in the organization. If organizational flattening can be accompanied by tearing down the walls between the executive suit and the cube farm, we might find ourselves with companies that are more able to produce quality software that their customers want.

The second thing is that the standard programmer's toolset for mainstream development will be almost completely composed of OSS products. There will be the occasional vendor who makes a splash with a great new tool that redefines the market but this will become increasingly rare, and chances are good the vendor will come from somewhere other than the US - for example, Idea is a Czech company. The one big exception to this is .NET, but that will only be true if Microsoft manages to litigate or intimidate any attempts to clone .NET as OSS - and up to this point, they have not been able to kill Mono.

Third, the success of OSS will increasingly be held up as an example of how to produce higher-quality software. The obvious counter-example is Microsoft, again; the contrast between the security, reliability, and performance of Linux/Apache/FireFox versus Windows/IIS/IE does not favor MS. The comparison of turnaround times for patches and fixes does not favor commercial products. The walls thrown up by marketing, support and PR in most companies versus the transparency of OSS - where anybody can query the bug database, enter a bug, write an email to the project leader, or even submit a fix to a bug - does not favor commercial companies. The only factor commercial vendors have in their favor is the ability to leverage innovation into breakthrough products. Very few vendors will have the vision and skill to do this; but they will be very successful even as OSS commoditizes yesterday's innovations.

A few good software companies - especially those that have already embraced OSS - will follow at least one of two strategies. The first strategy will be to build product suites around OSS that their professional services organizations sell and support. The second will be to copy the environment and attitude of OSS projects in their own development organizations, giving development teams unprecedented freedom and control (note that this will include the freedom to fail spectacularly). Companies like IBM will be able to do both, and augment the suites sold by their services organization with products produced by their developers.

The Future of Software Development...Boehm flames Agile/XP 15 years before they exist

[As a supplement to my earlier rant about Agile/XP]

...evolutionary development also has its difficulties. It is generally difficult to distinguish it from the old code-and-fix model, whose spaghetti code and lack of planning were the initial motivation for the waterfall model. It is also based on the often-unrealistic assumption that the user's operational system will be flexible enough to accomodate unplanned evolution paths...evolutionary development projects have come to grief by pursuing stages in the wrong order: evolving a lot of hard-to-change code before addressing long-range architectural and usage considerations.

Barry Boehm, "A Spiral Model of Software Development and Enhancement", IEEE Computer, vol.21, #5, May 1988

New Blogs to Add
Joanna Rothman
Thomas Barnett

I want to find more weblogs like these two and replace some of the popular blogs that everyone reads. Too often, the so-called "top-tier" weblogs are second- or third-hand echoes of the ideas of the really innovative and interesting thinkers out there.

Listening to podcasts of the PopTech 2004 conference really opened my eyes to this; here were a bunch of people, most of whom I'd never heard of, talking about subjects they knew best and were the real thought leaders in.

So I want to read/hear more of these voices directly instead of through the filter of the most popular bloggers.

Free Trade vs. Mercantilism

Masahunyam Monani on the how protectionism is a form of tyranny, in Bubble Generation:

This is the real scandal of "free" trade today : the vast majority of Ricardian gains are pocketed by a small section of the Western population at the expense of some of the most vulnerable people around the world. Developed nation have consistently used the WTO to push the developing world into reducing trade barriers, but heavily protected their domestic markets against competition...As my b-school prof puts it, nobody has a right to deprive others of their comparative advantage. Stand up for Free Trade to End World Hunger : how's that for a modern version of Hasta La Victoria Siempre?

There is a place for protectionism and subsidies, and that's to protect nascent or developing industries from unfair or overwhelming competition. But most protectionism is devoted to mature industries with political clout.

The Future of Software Development - A Rant About Agile/XP

Here's what really bugs me about Agile/XP.

The introduction of every new software engineering fashion to us teeming masses is like Athena springing out, full-grown and dressed in armor, from Zeus's forehead; the new methodology bursts onto the scene full-grown and ready for action. There's no sense of a period of gestation or ancestry. It's just there one day, big as life and twice as natural, sweeping away all that came before it.

And XP is just like the RUP/UML which was just like OOD&A which was just like JAD which was just like RAD which was just like...well, you get the idea.

Except that Agile/XP seems to have a lot of similarity to Barry Boehm's Spiral Model, which Boehm wrote papers about back in the mid-80's. The Agile/XP mavens mostly deny this. Granted, there are some significant differences which make Agile/XP both a subset and superset of the Spiral Model. But there's also an unwillingness to acknowledge, much less build upon, the contributions of the past - particularly if they come from Computer Science.

Computer Science? We don't need no stinkin' computer science. This is engineering, boys! We've got to be practical and get things done!

I think this attitude is one of the reasons why a viable discipline of software engineering has been so slow to emerge. Let's face it: we have more in common with the fashion industry than we do with real sciences. If we tried to be an applied science instead of a fashion mill, we'd acknowledge and build upon the past. We'd make hypotheses about what works, test those hypotheses, evaluate the results, then rinse and repeat instead of throwing it all away. And we'd build up an effective body of practice whose roots are based in sound, tested theory.

Of course, this would depress the software engineering fad industry. And wouldn't that just be too bad.

The Future of Software Development - Thoughts on Agile/Extreme Programming

So much has been written about Agile and Extreme Programming that it's probably impossible to say anything new or interesting about them. But I'll try anyway, because I think there are two things worth calling out.

The critical contribution that Agile/Extreme makes to the new software is that it creates a culture of testing. Whether you buy into the notion of test-first development or not as the right way to do test-driven development, it's impossible to argue with the value of codified, disciplined, and repeatable unit testing as an essential part of the coding process. In a very short time this meme has become so firmly entrenched that coding practice has been changed, probably forever. To be sure, more people are still talking about test-driven development (TDD) than doing it; but peer pressure and the desire to be on the cutting edge will fix that.

To put it another way, testing is cool. It's hard to be taken seriously by other hot-shot code jocks if you can't point to a battery of unit tests for your code. It's hard to be taken seriously as a development organization if your build system doesn't automate the running of those unit tests as part of the build. It's hard to be taken seriously by your peers if your automated unit tests fail and break the build on a regular basis. The meme of test-driven development - more than any specific practice of testing - is the big win from Agile/XP.

A side effect of TDD, and it's almost as important as the cultural shift, is that the act of writing effective unit tests requires developers to write code that can be easily tested. Making code testable frequently means breaking it down into small, discrete, coherent chunks instead of writing long, procedurally-oriented methods. This, of course, is refactoring - another cultural shift that is changing the practice of coding. (Side note: of course, the practices of unit testing and refactoring have been around since people started writing code. The real change is the culturally-driven codification of these practices - suddenly, they're "just the way you do things"). The practice of writing testable units of code, combined with refactoring and unit testing, will start producing higher-quality code (at least, in the hands of a reasonably skilled developer). It may even produce truly reusable code.

A have one quibble with TDD, and it's only a quibble. The emphasis on unit tests tends to obscure the equally important need for component-level, system-level, and stress/performance testing. The standard Agile/XP answer is to use mock objects, but I don't think this is sufficient. We need to codify the entire hierarchy of testing, not just the lowest level. If you do that, then our QA departments could actually start doing actual Quality Assurance instead of just being our bug catchers.

Unfortunately, the Agile/Extreme guideline of avoiding planned, up-front design plays to the worst instincts of the coding cowboys who just want to start writing code as soon as possible. I'm aware that Beck, Fowler, et. al., advocate a light-weight process of incremental and evolutionary design, not the abandonment of design. But because Agile/Extreme spends so much time talking about the development process and so little about design, that message gets lost. It doesn't help that the hypesters have seized on XP as the latest silver bullet that will magically solve every software development problem and allow you to deliver full-blown systems in some ridiculously small increment of time. The result is that as I write this, there are a lot of projects out there that are trying to do Agile/XP and failing.

That isn't an indictment of Agile/XP per se; but it is a critical flaw in the process. What's worse is that failing to address this flaw probably dooms a project right out of the gate. No set of programming practices, no matter how effective, can save a bad design, much less a non-existent one.

We need a set of guidelines for Agile Design that describe how to execute an lightweight, adaptive, evolutionary, and rapid design process that accomodates changing requirements and market conditions, gets the big decisions right, and seamlessly integrates with Agile/Extreme Programming methods.

Of course, almost every software design methodology developed over the last 40 years has attempted to meet pretty much the same goals. And most have failed, to a greater or lesser degree.

The Future of Software Development (and my future)...

I've been thinking a lot about this topic lately, especially since I'm just finishing up a very intense 4-month development cycle. You can't call a 4-month project a death march, really - perhaps death sprint is a more appropriate term?

At any rate, I think that we're seeing the beginnings of a shift in software development. You have to navigate through the hype from the gurus and the quacks to see it, but if you cull out what's interesting and important from current fads and fashions, you start to see something emerging from the following trends:

• Extreme/Agile Programming


• Open Source


• Outsourcing


• Web Services


• Generative Programming


• Software "Factories"


• The Rise of Dynamic Languages


• New Internet-Based Innovations (e.g., RSS, Podcasting)


• Microsoft's Slide Into Irrelevance

On the face of it, a rather cliched list. But each of these trends will contribute something crucial to the art/craft/science/business of creating software - and those contributions will not be the aspects that are getting the most hype right now.

Each trend deserves serious consideration, and I'll do that over the next few months. But for right now, I'll give you both the punch line (since every attempt to predict the future is a joke) and the setup.

The punch line:

There will never, ever be a substitute for talented software developers. No technology, no process, no other innovation will ever change this.

The demand/need for software will continue to accelerate. Software will disappear into the fabric of our everyday lives as virtually every man-made device becomes a computer. This will fuel the demand for effective integration of every thing.

The level of abstraction will continue to rise and so will the capabilities of our applications - but this will not significantly (or even slightly) lower the complexity of creating usful software.

The new software will be designed and created at an extra level of indirection, and we will have to evolve from programming systems to meta-programming of meta-systems.

This is good news for talented people who keep their skills sharp and relevant. Only really smart people with really good skills can tame software complexity.

The Setup

Christopher Alexander's single overriding rule for A New Theory of Urban Design:

Every increment of construction must be done in such a way as to heal the city.

I've quoted this before, in a slightly different context. At the time I thought it only applied to software maintenance. But I've come to an important realization:

Dividing development from maintenance is artificial and arbitrary. All software creation is software maintenance, and vice versa.

Alexander again:

...the word "heal" must be understood in its old sense of "make whole". It includes not only the repair of existing wholes which are there already, but the creation of new wholes.

Thus, Alexander's overriding rule is directly applicable to software. All increments of software creation must be done in such a way as to heal the software - that is, to make it whole..

Refactoring is only the best and most obvious example of this. In fact, all aspects of software creation should be indirectly driven by this overriding rule. Test-driven development is a way to heal the software. Inspection is a way to heal the software. Patterns are a way to heal the software. Good software engineer processes are a way to heal the software. Aspect-Oriented Programming is a way to heal the software.

All of the worthwhile techniques of software creation are deeply interrelated parts of a process that produces an organic wholeness in our software. When I say "process" , I don't mean it in the way we usually use it in software engineering. I mean something more fundamental - not a series of procedures, like a manufacturing process, but a set of actions, functions, and changes that advance us towards a goal. Exactly what this process is remains unknown; we can see and understand fragments of it - the trends listed above represent those fragments - but we don't know how to describe it yet, much less practice it.

This process will be interpretative, not mechanical, and thus can only be discovered and executed by talented software developers. All of the mechanistic aspects of the process will be automated, leaving only those aspects that require taste, judgement, and talent.

Funny/Weird Lyric of the Day

I wanna be like Captain Kirk
Get up every day and love to go work
Don't wanna be like Mr. Spock
I want to kick out the jams and rock the block

Bob Schneider, "Captain Kirk", I'm Good Now