I only just caught up with Joel Spolsky’s amusing and insightful Yale talk posted last December — a return-of-the-prodigal-son sort of thing for this Yale graduate. (Here’s parts one, two and three.)

These quotes were worth highlighting:

Time and time again, you’ll see programmers redefining problems so that they can be solved algorithmically. By redefining the problem, it often happens that they’re left with something that can be solved, but which is actually a trivial problem. They don’t solve the real problem, because that’s intractable.

This is a failing, in one sense — it’s why certain Big Problems in the field never seem to get solved. On the other hand, in the face of deadlines or business pressures, we can surely see the value in a programmer’s ability to take some problem that’s impossible to solve (given available resources) and redefine it as a job that can actually be accomplished.

And:

Being able to write clearly on technical topics is the difference between being a grunt individual contributor programmer and being a leader.

The hardest problems facing most programmers don’t involve communicating with the computer; they arise in the course of communicating with people — colleagues, customers, users.

Earlier this week the Open Source Applications Foundation — the organization developing Chandler, whose work I followed for three years and whose story I tell in Dreaming in Code — announced what it called a “restructuring,” which meant laying off roughly 2/3 of its employees. (Infoworld’s story has the basics.)

So does this mean that Mitch Kapor, OSAF’s founder and chief funder, is “pulling the plug” on OSAF, as Techdirt has it, or “bailing on” Chandler, as a CNET blogger put it?

Kapor is indeed leaving the OSAF board and handing the rest of the reins to Katie Capps Parlante, who’s been running the project day-to-day for some time. That’s a big change. From what I can tell, though, this isn’t so much a “plug-pulling” as an amicable parting of the ways.

Kapor’s financial withdrawal is not a big surprise. He had told the Chandler developers several months ago that he intended to fund the project through the end of 2008 but no further; that decision had been discussed on the project’s public mailing list, so the looming changes couldn’t have come as a total shock to anyone who’d been following the story. (I write about these changes in a new epilogue to DREAMING IN CODE that will be included in the forthcoming paperback edition, due out in late February.)

What actually happened at the end of 2007, according to Parlante, who I interviewed today, is that OSAF and Kapor agreed that it would be healthy for the project to move out from under Kapor’s wing faster than they originally envisioned. Over the past year, Kapor has been less involved with Chandler and more focused on new projects (such as Foxmarks), and Parlante says that the project leaders felt that if they were headed towards independence anyway, it made sense to move faster.

“We’ve been joking that we laid off Mitch,” she says.

At that point, Kapor agreed to provide transitional funding for the group — less than he’d originally planned when envisioning paying the entire project’s bills for another year. That triggered the staff cuts.

How much exactly did Kapor commit? Referring to the deal that launched the Mozilla Foundation in 2003, which Kapor brokered when AOL was shutting Netscape down, Parlante says, “He’d told AOL to give Mozilla enough support for it to become viable on its own, and he decided he should follow his own advice with us.” (In 2003, AOL put up $2 million, and Kapor himself put up $300,000, to fund the Mozilla Foundation, which is now a hugely successful enterprise, based on Firefox’s revenue from Google.)

Chandler started in 2002 as a high-profile project that aimed to produce a novel personal-information manager and demonstrate that the open source development methodology could produce innovatively designed desktop software. But the work proceeded at an agonizingly slow pace, and it took about five years for OSAF to ship a usable “Preview” edition last fall.

So there’s no question that the cutbacks represent a come-down for a project that started out with such grand ambitions and golden prospects. But surely there are better metaphors for what’s happening than “pulling the plug”: “leaving the nest,” perhaps. In fact, as of the end of the month, OSAF will move out of its longtime home at Kapor’s offices on Howard Street in San Francisco’s SOMA district and become a virtual team.

From the start of OSAF, Kapor had made clear that he did not envision the project as an open-ended philanthropic obligation, but rather as a test-bed for new ideas in software design and project organization. During the time I spent at OSAF, he would regularly repeat his belief that OSAF ultimately needed to become financially self-sustaining.

Parlante says OSAF will use its transitional funding period to explore lots of different business models — everything from selling advertising or charging for service to community support (donations and fundraising) to business partnerships or deals to bundle the Chandler Hub server with other products. “Open source business models are always a little up in the air, they’re always changing — that’s true even of Mozilla,” Parlante says.

Can Chandler survive on its own? Right now I’d give it good odds for continuing in some form: as long as there are developers interested in continuing work on it, there’s no reason for it not to. It will be harder, but by no means impossible, for the organization to find enough money to support a small full-time staff beyond the transition.

The obituary writers are already chomping at the bit. And of course many of the criticisms of OSAF’s mistakes are accurate. Still, those mistakes are now history. Since shipping the Preview version of Chandler its team has sped up the flow of new releases, new features and bug fixes, suggesting that the pressure of looming independence has already made a difference. The question is, does Chandler today offer enough innovative value to build a thriving community and win support — both volunteer development effort and cash?

Given the history, it would be foolhardy to say “yes” for sure; but I think it’s also a mistake to say “no way.”

I’m a veteran of a company that spent roughly three years disproving premature reports of its demise. Salon is still around and doing good work. It’s certainly going to be challenging for OSAF to continue on its own, and it’s entirely possible that the organization will be gone within a couple of years. But writing it off today seems wrongheaded to me.

Of course, in spending so much time and thought on telling Chandler’s story, I invested something of myself in it, too, so maybe I’m just unwilling to let go. But there are other indicators that the project is unlikely to simply evaporate.

At the very moment that OSAF announced its cutbacks and the obituary writers jumped the gun, geek-hero blogger Cory Doctorow posted an enthusiastic endorsement of Chandler on the extremely popular BoingBoing. Doctorow’s post might be part of an effort to rally support for Chandler among its users now that its future isn’t guaranteed. That’s what happens when an enterprise that people care about is threatened. Or maybe it was a spontaneous coincidence — that happens too!

The big question is whether enough people care enough about Chandler to keep it afloat. In a sense, that is the final experiment in the OSAF lab, and as long as Kapor continued to fund the project out of his own deep pockets, it would never be able to get results.

* * *I’ve posted this before I’ve had a chance to talk with Kapor, because this story is already generating a lot of commentary. I’ll update as needed. This is what he told Infoworld:

“I would say I had a lot of ambitions that we wound up, for very good and practical reasons, scaling back on,” Kapor said in an interview Thursday. He described the outcome as “a working subset of a grand vision.”

Kapor said his interest in continuing waned. “We found ourselves in the situation that the team wanted to continue on very much,” he added. “I found myself in a different place. I did not have that same level of commitment and desire, because I had the original dream in mind.”

UPDATE: I did catch up with Mitch Kapor late today. He confirmed the accuracy of this account. “It’s time for me to move on,” he said. See also Kapor’s own blog post on this topic.

I’ve also received a couple of anonymous emails from people involved in the project, suggesting what I’m sure is just the tip of the iceberg of disagreements over Chandler’s direction, its leadership, and the personnel choices involved in the layoffs. Having lived through the difficulty of such events more than once myself, I know that they’re deeply emotional — and that there is often more than one “right” view of the situation. I haven’t been following Chandler closely enough for the last couple of years, since I wrapped up the book, to gauge the accuracy of these perspectives.

I also don’t yet know exactly who’s leaving. I’ll try to link to the participants’ own reports. Here’s Ted Leung’s announcement of his departure. Here’s Matt Eernisse’s. And Mike “Code Bear” Taylor’s. Taylor and Eernisse have both joined Seesmic, Loic LeMeur’s new startup. Brian Moseley, who for a long time was the central developer for the Cosmo Server (later Chandler Server), is also leaving.

Funny footnote: Moseley and I crossed paths a decade before we met at OSAF; he was part of a small development team of Cornell students who wrote the first, somewhat disastrous version of Salon’s Table Talk in 1995. He’s plainly become a very different developer in the interim. His colleagues went on to fame and fortune during the dotcom boom before going bust.

Bryan Cantrill has now fleshed out his critique of Dreaming in Code that I recently addressed. I want to thank him for returning to the topic and giving it serious consideration. I’m going to respond at length because I think, after clearing away a lot of smaller points, we’ve actually found an interesting point to dispute.

As far as I can tell, Cantrill is now saying that the problem with Dreaming is that it starts from the notion that “software is hard” and then explains why it’s hard by exploring what makes it unique — and he would have preferred a book that started from the notion that software is unique and then explored why its uniqueness makes it hard. At some point this becomes pretty abstruse, and I’ll leave it to those of you who want to compare what Cantrill says with what’s in the book to weigh our different perspectives.

I do want to correct his statement that I “picked a doomed project” from the start, as if that was my intention, in order to support a pessimistic view of software. At the time I began following Chandler, it had high hopes and a lot of enthusiastic support. I chose it because I cared about the problems it set out to solve, I thought the people involved were interesting, and I thought it had a reasonable chance of success. 20/20 hindsight leads Cantrill to dismiss Chandler as an ill-fated “garbage barge” from the start. But that’s hardly how it looked in 2002. It attracted people with considerable renown in the field, like Andy Hertzfeld and Lou Montulli, as well as other, equally smart and talented developers whose names are not as widely known. Nor, even at this late date, do I consider Chandler in any way to be definitively “doomed” — though certainly it has failed to live up to its initial dreams. There are too many examples of projects that took long slogs through dark years and then emerged to fill some vital need for anyone to smugly dismiss Chandler, even today.

Also, I need to say that my interest in the difficulty of software did not emerge as some ex post facto effort to justify the problems that OSAF and Chandler faced. In fact, as I thought I wrote pretty clearly, it emerged from my own experience in the field at Salon, where we had our own experience of a software disaster at the height of the dotcom boom.

Finally, in substantiating his criticism of what he calls my “tour de crank” of software thinkers, Cantrill still doesn’t make a lot of sense to me. In the two chapters near the end of the book that depart from Chandler to explore wider issues, I discuss the work and ideas of Edsger Dijsktra, Watts Humphrey, Frederick Brooks, Ward Cunningham, Kent Beck, Joel Spolsky, Jason Fried and Nick Carr (in Chapter 9), and Charles Simonyi, Alan Kay, Jaron Lanier, Richard Gabriel and Donald Knuth (in Chapter 10). (Marvin Minsky and Bill Joy, whom Cantrill mentions, are each cited only in passing.) Anyone might view some number of these figures skeptically — I do, too — but cranks, all?

Anyway, these are side issues. It’s later in Cantrill’s argument that we get to the heart of a real disagreement that’s worth digging into. Despite the evident pragmatism that’s on display in his Google talk, halfway through his post Cantrill reveals himself as a software idealist.

…software — unlike everything else that we build — can achieve a timeless and absolute perfection….software’s ability to achieve perfection is indeed striking, for it makes software more like math than like traditional engineering domains.

And once software has achieved this kind of perfection, he continues, it “sediments into the information infrastructure,” becoming a lower-level abstraction for the next generation to build on.

Ahh! Now we’re onto something substantial. Cantrill’s view that aspects of software can achieve this state of “perfection” is tantalizing but, to me, plain wrong. He makes a big point of insisting that he’s not talking simply about “software that’s very, very good” or “software that’s dependable” or even “software that’s nearly flawless”; he really means absolutely perfect. To me, this insistence — which is not at all incidental, it’s the core of his disagreement with me — is perplexing.

Certainly, the process by which some complex breakthrough becomes a new foundational abstraction layer in software is real and vital; it’s how the field advances. (Dreaming in Code, I think, pays ample attention to this process, and at least tries to make it accessible to everyday readers.) But are these advances matters of perfection? Can they be created and then left to run themselves “in perpetuity” (Cantrill’s phrase)?

On its own, an algorithm is indeed pure math, like Cantrill says. But working software instantiates algorithms in functional systems. And those systems are not static. (Nor are the hardware foundations they lie upon — and even though we’re addressing software, not hardware, it’s unrealistic to assume that you will never need to alter your software to account for hardware changes.) Things change all the time. This mutability of the environment is not incidental; it’s an unavoidable and essential aspect of any piece of software’s existence. (We sometimes jokingly refer to this as “software rot.“) Some of those changes break the software — if not over a matter of weeks or months, then over years and decades. It is then, I think, no longer accurate to call it “perfect” — unless you want to take the pedantic position that the software itself remains “perfect,” it’s the accursed world-around-the-software that’s broken!

This, of course, is a version of Joel Spolsky’s Law of Leaky Abstractions argument, which I present at length in Dreaming in Code: “perfect” abstractions that you can ignore are wonderful until something happens that makes it impossible to keep ignoring them. Such things happen with predictable regularity in the software world I know. I don’t know how you can discuss software as if this issue does not lie at its heart.

The strange thing about this disagreement is that, as far as I can tell, Cantrill is — like the engineers I know a lot better than I know him — a hands-on kind of guy. And DTrace, the project he’s known for, is by most accounts a highly useful tool for diagnosing the myriad imperfections of real-world software systems — for navigating those trips down the ladder of abstraction that software developers must keep making.

All of which leaves me scratching my head, wondering where the world is in which Cantrill has found software of “absolute perfection,” and which programs it is that have achieved such a pristine state, and how they — unlike all other programs in existence — escape the creep of software rot.
[tags]bryan cantrill, software, software development, dreaming in code[/tags]

Early in my online career — this goes back to around 1990 — I learned a basic principle about off-the-cuff criticism online: No flame flares in a vacuum. In other words, don’t be too glib with your put-downs — because before you know it, the person you’re putting down will find your comment and call you on it.

I recalled this today when I encountered (thanks to a mention on Sumana’s blog) a drive-by attack on Dreaming in Code. In the opening couple of minutes of a talk at Google this past summer, it seems, a Sun engineer named Bryan Cantrill declared, with some vociferousness, that my book — I quote — “sucks.” This judgment is now preserved in the perpetuity that is Google Video.

Now, Cantrill is one of the creators of DTrace, a popular, award-winning and innovative tool for managing Solaris, and my hat is instantly removed to anyone who bears responsibility for a successful piece of software. I am also not particularly shocked to hear that a smart programmer didn’t like my book; he’s neither the first nor the last in that group.

What’s just plain puzzling is exactly what Cantrill has to say in his handful of complaints about Dreaming in Code. Because every point he makes in explaining the basis for the book’s suckiness turns out to be a point that I have made at length in the book itself and in my talks this year about the book — including at Google, several months before Cantrill’s appearance there. Of course I’m not suggesting that he borrowed from me — he almost certainly hasn’t heard my presentation! But I am puzzled how he could so completely have missed my argument, and misrepresented my position, when it seems to be so close to his own.

As best I can make out, Cantrill believes that Dreaming in Code fails to acknowledge that software is uniquely different from other creative endeavors because (a) it’s not a physical entity; (b) we can’t see it; (c) it’s really an abstraction. These factors cause all the analogies that we draw to things like building bridges to break down. Cantrill describes himself as a “finisher” of books and I’ll take his word, but I’m flummoxed how anyone who has finished the book can knock it for failing to understand or express this view of software.

The critique gets sketchy from here on in; Cantrill draws some sort of analogy between Dreaming in Code and Apocalypse Now (a comparison I’ll gladly accept — it’s a reference I make in the book myself) and suggests that I got “hoodwinked” by “every long-known crank in software” (the lone “crank” cited is Alan Kay).

It’s true that the final section of the book surveys both the nuts-and-bolts methodologies that try to alleviate software’s practical difficulties and a whole gallery of software philosophers from both the mainstream and the fringes — people like Kay, Charles Simonyi, Donald Knuth and Jaron Lanier. If even discussing these people’s ideas constitutes “hoodwinking” I guess I’m guilty.

From here Cantrill wanders into his own case for software’s uniqueness, which as far as I can tell is nearly identical to the one I make in Dreaming in Code. “All the thinking around software engineering has come from the gentlemanly pursuit of civil engineering,” Cantrill says. “That’s not the way software is built.” Indeed.

So I’m not sure what the complaint is. Maybe analogies are so odious to Cantrill that he feels they should not even be discussed, even if the discussion is intended to expose their limitations. Maybe the notion of software’s uniqueness and its intractability to old-fashioned physical-world engineering principles seems so obvious to Cantrill that he is appalled anyone would even bother to explore it in a book. But there’s still an enormous amount of attention and money being applied to the effort to transform software development into a form of reliable engineering. I found thoughtful arguments on several different sides of the matter and thought it was worth the ink, although my own conclusion — that software is likely to remain “hard” and not become an easily predictable undertaking — is pretty clear.

Anyway: Go ahead and tell me my book sucks — I can take it! But don’t tell me that it sucks because it fails to acknowledge an argument that actually forms its very heart. Say that and, well, I’m just not going to be able to resist a retort.
[tags]dreaming in code, bryan cantrill, software engineering[/tags]