Code Reads #8: The Cathedral and the Bazaar
This is the eighth edition of Code Reads, a series of discussions of some of the central essays, documents and texts in the history of software. You can go straight to the comments and post something if you like. Here’s the full Code Reads archive.
The Cathedral and the Bazaar, Eric Raymond’s classic essay on open-source software development, is now 10 years old. I first read it soon after its publication. At that time the term “open source” was newly minted, and the movement was in the news chiefly because (a) it seemed to offer a threat to Microsoft, and (b) Netscape, at a late stage of its war with Microsoft, had decided to release its browser code under open-source license. I was editing Salon’s technology section in those days, and one of our central projects was Andrew Leonard’s thorough coverage of the open-source phenomenon. (He interviewed Raymond in early 1998.)
I think it’s fair to say that today CatB (as Raymond and others now call the essay) has proved its importance independent of its place in the long-concluded browser-war saga. I’ve reread it several times over the years since, and each time find something new and valuable.
In CatB Raymond identifies two different styles of open-source development:
I believed that the most important software (operating systems and really large tools like the Emacs programming editor) needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time. Linus Torvalds’s style of development — release early and often, delegate everything you can, be open to the point of promiscuity — came as a surprise. No quiet, reverent cathedral-building here — rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches (aptly symbolized by the Linux archive sites, who’d take submissions from anyone) out of which a coherent and stable system could seemingly emerge only by a succession of miracles. The fact that this bazaar style seemed to work, and work well, came as a distinct shock.
One of the most common misconceptions about the essay is to equate the “cathedral” approach with closed source. The line from cathedral to bazaar is actually orthogonal to the line from open source to proprietary software. There are plenty of examples of open-source “cathedrals.” And though it’s hard to imagine instances of true closed-source “bazaars,” there are enough examples of extensible proprietary software products with thriving communities of users building new functionality on top of the proprietary code to see the “bazaar” principle at work in some limited form, even in the closed-source world.
Raymond argued that the bazaar style exemplified by the Linux project represented a new methodological advance for software development. With a horde of volunteer developers collaborating across the Net, Brooks’ Law could be transcended, and large quantities of programmer brainpower could be harnessed without getting stuck in the tar pit. “Given enough eyeballs, all bugs are shallow.”
It’s important to note that Raymond explicitly excluded the initial work of a project from this principle:
It’s fairly clear that one cannot code from the ground up in bazaar style [IN]. One can test, debug and improve in bazaar style, but it would be very hard to originate a project in bazaar mode. Linus didn’t try it. I didn’t either. Your nascent developer community needs to have something runnable and testable to play with.
When you start community-building, what you need to be able to present is a plausible promise. Your program doesn’t have to work particularly well. It can be crude, buggy, incomplete, and poorly documented. What it must not fail to do is (a) run, and (b) convince potential co-developers that it can be evolved into something really neat in the foreseeable future.
Today the concepts in CatB — like ” Every good work of software starts by scratching a developer’s personal itch,” “Good programmers know what to write. Great ones know what to rewrite (and reuse),” and “Release early, release often” — have become commonplaces in the field. And a significant number of large open source projects have built on them.
Today’s rapid evolution of Web-based applications seems often to be built on open-source infrastructure foundations (the “LAMP” stack, with variations like Ruby on Rails). But it’s not clear to me that “Web 2.0″ has produced many bazaars itself. Or maybe they just haven’t attained sufficient scale or prominence to have landed on my radar.
In a recent blog posting, Ted Leung (a developer and manager on the Chandler project that served as the central story of Dreaming in Code) discussed the many looming difficulties of developing what he calls “Rich Internet Applications” (RIAs). Leung, a longtime participant in the Apache project and a confirmed believer in open source methods, considers the available toolsets for RIA developers: AJAX is popular but, he says, a “nightmare” for cross-platform development. Adobe’s Flash/Flex is far less nightmarish. Trouble is, it’s proprietary.
To me, this suggests that, whatever other benefits the open source bazaar approach may have conferred on the world of Web development today, it has not provided solutions to the next wave of problems today’s developers confront. Not yet, anyway!
Wordyard RSS feed
March 3rd, 2007 at 4:22 pm
My response here:
http://my.opera.com/Vorlath/blog/2007/03/04/code-reads
March 5th, 2007 at 3:31 am
This is an important article indeed. Really, an open source phenomenon is a fascinating achievement of twentieth century. All we knew about “right” software development management has crashed above the rocks of open source community. Just think about it:-
2. Suddenly incompetent managers have become obsolete. True professional leadership which was backed down before has finally taken over their deserved place of glory.
3. ISO 9000, TQM, Sigma 7, Word Documents and all these bureaucracy Things has become instantly a joke. They are proved as inefficient, unnecessary, motivation killers.
4. Holy Grail of “pure motivation drive” has been achieved. I mean these people in open source community doing far better work that within companies that are paying a lot of money for it.
And list goes on…and on…
Shortly speaking, the lessons of open source community are far more reaching. It exposes the unbelievable inefficiency of “well managed” software development organizations. And I do believe that lessons of OS can be applied not only in software industries.
In addition I like to compare programming discipline with fractal drawings of infinite depth. Suppose you see fractal drawing from a high distance. You can recognize the overall patter (system architecture), this understanding gives you a false impression that you understand how it works. However when you try to zoom in into the picture in you suddenly understand that the pattern consisting of yet another drawings completely different from what you already know. And the process is going on and on and on. The funny thing is that knowledge of overall pattern gives you only a small advantage over knowing smaller parts of the system because of its infinite complexity.
That’s why cathedral model would never work in programming. One man cannot rule over “smaller” parts of the system because of human brain limitations in comprehension of such artifacts as modern enterprise systems.
However, I do believe, that foundations for great software must be laid by one or maximum two developers.
March 5th, 2007 at 2:30 pm
I also read this not long after it was first released. It’s interesting to compare my memory with the updated version.
Here’s an interesting comparison/companion to the article:
ESR gives up on Fedora.
I have some other points to make, but I’ll need to get my notes in order first.
March 6th, 2007 at 12:09 pm
Cleo, thanks for that great post in response. I want to quote one paragraph:
“Coming up with new and useful ideas is just one part of the battle in producing something successful (by whatever definition you use). Another part is giving programmers and the users a way to take part in something bigger than themselves, yet still feel like they have some control. Too often, I submit a contribution or try and discuss a useful feature and it gets torn down immediately. All these projects are now dead or inactive. It’s not because they didn’t accept my contributions. It’s because they hardly accepted any contributions. This is no longer a bazaar. It’s not even a cathedral. It’s more like a hostage situation.”
I think that would call for a retitling as “The Cathedral, the Bazaar and the Prison”!
You go on to talk about how, in order to build a bazaar, you’ve got to create the actual mechanism for people to interact. In the open source world this has usually been a matter of mailing lists, source-code repositories, bug trackers and so on. The mechanisms seem to exist; they also often determine the shape and nature of contributions — another example of Mitch Kapor’s aphorism “architecture is politics” at work, I thin.
Then, you write: “While there are great tools such as forums, mailing lists and software repositories, I find that there’s no equivalent for software interaction….I’m betting that any project that enabled software interoperability will have a huge advantage over all other projects.”
If I understand what you mean, don’t standards like XML (and RSS), and any sort of open APIs (like the kind Web services like Google and Amazon expose) all qualify as “the equivalent for software interaction”? Is that what you mean, or are you getting at something else?
March 6th, 2007 at 1:45 pm
Hi. I wrote an essay for IBM.com a few years ago that critiqued CatB. It is here…
http://www.chc-3.com/pub/manage_themselves.htm
and here is Eric Raymond’s response to it…
http://www.chc-3.com/pub/manage_themselves_r1.htm
and my response to Eric….
http://www.chc-3.com/pub/manage_themselves_r2.htm
Chuck Connell
March 11th, 2007 at 12:28 pm
[...] Original post by Scott Rosenberg [...]
March 13th, 2007 at 12:28 pm
Scott, sorry for the late reply. RSS doesn’t notify me of responses here.
About software interaction, as I said, I feel that there exists tools in the form of software for human interaction. But I don’t feel there is anything for software to software interaction.
You mention XML and RSS. RSS is fascinating because it exemplified this missing interaction. If XML and the web were sufficient, why did it take this long to take hold? RSS is a tool fo human interaction. It isn’t one built for software interaction.
People believe that web apps have an advantage, and always will, over desktop apps. This is not true. The web has a bazaar. There’s a way to communicate. Desktop apps don’t have that. If they did, desktop apps would be WAY superior to web apps. I believe this to be the big secret of the past two decades. But doing this decentralises thing, so that’s why I believe MS has a tough situation on their hands. The Amiga had it and it was incredible what you could do software-wise. It’s 20 years later and I still haven’t seen anything close to it, but it was underused back then. We took it for granted. The web and the Internet were nowhere what they are today. The web didn’t even exist at first.
Speaking of decentralisation, the Amiga often had several small companies create their own small apps and someone else would come around that would create another app that used all these other smaller apps together. (You can’t do that today with ALL apps on the market. You could with the Amiga via software. There was a standard published API plus a custom one for custom functionality.) You could buy only the apps you wanted for the functionality you wanted. It often displaced large companies because many small companies usually cared much more about their code and I remember it causing a lot of fuss. It was also much cheaper. Graphic and video editing apps were often done like this. The way Lightwave 3D works with plugins and tools reflects its roots on the Amiga with its piecemeal framework of having separate tools all work together. Software bazaar anyone?
A communication protocol isn’t enough. There has to be some form of understanding between all apps just as there is for web apps. We’ve had sockets for years, but it’s devoid of meaning. Even XML is devoid of meaning. But HTML isn’t. As bad as people think it is, HTML has a definition. Some forms have meaning too. Desktop apps have no way for another app to control it and thus it’s impossible to expand and build on top of it. But web apps can build off each other even if it’s rather limited right now. That’s what I want more of. That’s what I want to see be expanded because I’ve seen what’s possible. That’s why RSS had a problem. The current communication channel wasn’t adequate and RSS is a big waste of bandwidth most of the time.
The best ideas are almost never the ones that take hold. If you want good ideas, look in the past. It’s a gold mine.
March 13th, 2007 at 1:34 pm
I first read CatB about three or four years ago while working for a Fortune 100 company with several IT campuses in four states. Consequently, there was a lot of duplicated effort - for example, we had at least four different logging libraries used just on the web site.
I proposed developing an internal “open-source” repository where we could all share such utility code. This was to be a simple, low-maintenance, low-expectation kind of engineering effort - some common logging and analysis tools, some standard hacks around issues in our environment, nothing forced on anyone, freedom to link in whichever version of a library you needed, etc. Although every developer and architect I spoke to supported the idea, thought it would save money for the company, and make their lives easier, in the end there was too much attachment to “our code” to make the project work. Nobody was willing to either update their code for anyone else, or allow anyone else to add changes the code. Even when we explored some of the tools from the open-source world for managing change and keeping people informed, nobody was willing to take the leap. After a few months of trying to get it to work, I finally admitted defeat.
Has anyone else tried to use this kind of bazaar approach in corporate software development? Has it ever worked?