The Future of Software Development - Open Source

I think I've flogged Agile/XP enough, so let's move on to Open Source Software (OSS) development.

Once again, this is a subject that's been done to death, so it's hard to say anything new. But I can think of at least two facets of OSS development nobody seems to talk much about even though they're pretty central:

1. The Absence of Senior (Non-Technical) Management, i.e., No Suits


2. The Absence of Originality

It's hard for me to see the first as anything but positive - perhaps I've spent too long in the trenches - but it is unfortunately a partial cause of the second. On the other hand, the lack of originality in most OSS products is not necessarily a bad thing. It's even something of a boon in the area of programmer tools, as I'll explain farther down.

But first I want to take about what No Suits means to OSS. I think I can sum it up in one word: freedom. The 80/20 rule, that fundamental law of the universe that masquerades as a useful heuristic, applies to the Suits as well as it does to anything else. About 20% are competent enough not to be actively harmful to the process of developing good software. (A very small percentage are competent enough to provide the real leadership, motivation, and skilled management that actively aids and improves the process, and I have been lucky enough to work for that small percentage a couple of times in my life). The other 80% are the incompetent, egomaniacal, soul-crushing bastards we've all the bad luck to work for too often in our careers. "Pointy-haired boss" is far too forgiving a term for these fools. And doing good work in the environments they create is a Sisyphean travail.

So getting rid of the suits is a big win at least 80% of the time. And most self-selecting teams can manage themselves as well or better than the merely competent portion of the remaining 20%. The freedom gained from removing management overhead that is at best neutral but more often malignant results in software that works considerably better than the software created in for-profit enterprises. This freedom results into the ability to release new versions of the software when its creators think it's ready. It results into the ability to recast or reinvent the project, the team, or the product at a moment's notice. It results into the ability to continually refine a software product until it meets a desired level of quality and customer satisfaction.

The usual counter-argument to this is that OSS development lacks the spur of the competition that is provided by the commercial market. This point of view asserts that higher-quality software is created as a result of companies competing for customers. The ultimate driver here is the customer, and only commercial competition provides the impetus to create what the customer wants.

This might be true if in fact your average software company was driven by what customers want. In fact, they're more usually driven by a fragmented view of what the company's leaders think the customers want, which is further distorted by what the suits think they need to do to a) retain control and b) make money (usually the best way to accomplish a) is b), but it isn't always necessary).

The absence of the suits in OSS means that the development team has to deal directly with their customers. All of the usual institutional filters get removed, and suddenly developers are getting direct, unmediated feedback from real users. This kind of disintermediation can have some really positive benefits - OSS projects don't succeed unless they have users, and they don't get them unless users get something they find truly useful. So they either listen to their customers and respond to them, or they get ignored.

So it shouldn't surprise us that better software results when the feedback loop is tighter, more direct, and less subject to distortion, when the process can be as flexible as it needs to be, and when the developers have greater creative freedom in a less restrictive and less hierarchical environment. And it shouldn't surprise us when users respond by replacing commercial software packages with OSS equivalents.

Now, I don't want to give a completely Polyanna-ish view of OSS. From the standpoint of an ordinary, non-technical user most OSS products are effectively inaccessible. Even products intended for regular people are often too difficult to install, too difficult to use, or too far out of the mainstream to be comprehensible. The very best OSS products are the ones produced for technical users, especially programmers, not for people to whom a computer is a tool that helps them do their real work.

Here's where the absence of that very small percentage of highly competent executive leaders hurts. These are the people who are capable of bringing together the innovaters, the creators, the customers, the marketers, and the money to create truly innovative products. I'm not saying that OSS can't produce brilliantly innovative products. But it's a lot easier to produce products when all of the requirements are known and when the market is well understood. So OSS produces a lot of "me-too" products that are effectively commoditizing what were once profitable innovations - think operating systems (Linux, BSD *nix), programming languages (gcc, perl, python), personal productivity software (Open Office, Star Office), browsers (Firefox, Mozilla, Opera), and databases (MySQL, Postgres). I'm hard-pressed to think of any truly innovative OSS products. Maybe Apache, although that's a little bit of a stretch.

I mentioned earlier that this lack of originality is mostly a positive when it comes to programmer tools. That's because developers work in teams and usually need to use all the same tools. A decade or more ago, this was a real problem. The first IDEs, like the C++ environment from CenterLine, cost big bucks. Compilers, modeling tools, source code control systems, and the like also came with big price tags. OSS changed that. Today, anyone with access to the Internet can quickly download every tool they'll ever need to develop software. It's still true that many of the commercial tools are better than OSS ones - e.g., Idea IntelliJ and Borland's JBuilder are better than, say, Eclipse or NetBeans, and Perforce is clearly superior to CVS - but this gap is narrowing all the time. Eventually the differences in features, quality and support will diminish to the point where it's unreasonable to pay for the commercial tool. In this situation, developers don't need or even want constant, radical innovation in their toolsets. They want continuous evolution among a set of alternatives - so that they can both grow with the tool without having to confront steep learning curves on a regular basis and so that they can easily switch when something more convenient comes along.

Convenience, of course, is one of the hallmarks of commoditized consumer products. And that's what we're talking about with OSS, and that's why OSS is no threat to innovative commercial software. It is a threat to the stagnant commodization of software, and that is an entirely good thing; the alternative is having MS selling Windows, Office, and IE for the next 20 years with no more than the occasional "update".

So what does all this have to do with the future of software development?

First, both the success of OSS without suits and its inability to innovate are clues. We don't want to eliminate the suits - just subtract the losers and keep the value and innovation added by the winners. In an ideal world, we'd eliminate that 80% of managerial incompetents and keep the competent 20%. Fortunately, there is already a widely accepted way to do this - by flattening organizations and their management structures. Flattening the organization eliminates a considerable number of the suits. And that might end at least some of the internal political jockeying and infighting that, among many other negative consequences, tends to choke and distort communication in the organization. If organizational flattening can be accompanied by tearing down the walls between the executive suit and the cube farm, we might find ourselves with companies that are more able to produce quality software that their customers want.

The second thing is that the standard programmer's toolset for mainstream development will be almost completely composed of OSS products. There will be the occasional vendor who makes a splash with a great new tool that redefines the market but this will become increasingly rare, and chances are good the vendor will come from somewhere other than the US - for example, Idea is a Czech company. The one big exception to this is .NET, but that will only be true if Microsoft manages to litigate or intimidate any attempts to clone .NET as OSS - and up to this point, they have not been able to kill Mono.

Third, the success of OSS will increasingly be held up as an example of how to produce higher-quality software. The obvious counter-example is Microsoft, again; the contrast between the security, reliability, and performance of Linux/Apache/FireFox versus Windows/IIS/IE does not favor MS. The comparison of turnaround times for patches and fixes does not favor commercial products. The walls thrown up by marketing, support and PR in most companies versus the transparency of OSS - where anybody can query the bug database, enter a bug, write an email to the project leader, or even submit a fix to a bug - does not favor commercial companies. The only factor commercial vendors have in their favor is the ability to leverage innovation into breakthrough products. Very few vendors will have the vision and skill to do this; but they will be very successful even as OSS commoditizes yesterday's innovations.

A few good software companies - especially those that have already embraced OSS - will follow at least one of two strategies. The first strategy will be to build product suites around OSS that their professional services organizations sell and support. The second will be to copy the environment and attitude of OSS projects in their own development organizations, giving development teams unprecedented freedom and control (note that this will include the freedom to fail spectacularly). Companies like IBM will be able to do both, and augment the suites sold by their services organization with products produced by their developers.