Wordyard

Hand-forged posts since 2002

Archives

About

Greatest hits

BASIC as mirror

by

Programming pioneer Edsger Dijkstra once said, “Teaching BASIC should be a criminal offense.” (At least it’s attributed to him — he had a lot of snarky things to say about a lot of other computer languages, so at least it’s in character.)

David Brin disagrees — in fact, Brin is appalled that BASIC, which used to be pre-installed on every personal computer available for purchase, is now a rarity. His article recounting his and his school-age son’s search for a simple BASIC tool is in Salon today.

Brin likes to draw — and rile up — a crowd, as he did years ago in arguing the case that “Star Trek” is philosophically superior to “Star Wars.” And he’s succeeded again.

What I’m finding most interesting in the 150+ letters his article has already generated, whether they share Brin’s views or disagree, is the sheer passion on the part of the programmers responding. I guess the topic combines programmers’ near-religious intensity on the topic of languages with the deep-seated connection all creators have to the tools of their youth.
[tags]programming languages, basic, david brin[/tags]

Proofs

by

Apologies for the week of blog silence here. I was otherwise engaged: the page proofs for Dreaming in Code landed on my doorstep right before the holiday weekend, so that was my Labor Day weekend, and the evenings since.

It is a strange — and, I have to admit, wonderful — feeling to see this project, which I first conceived in the fall of 2002, near completion. From vague notion to typeset pages — in only four years! (The official publication date is Jan. 16, 2007.) Next up: bound galleys… blurbs… then books!

Now my energy turns from the page to the Web, where I will be building out the book’s site, at present mostly a placeholder, with excerpts and more. Here on this blog, I’m also planning to start a new project around the book’s theme — the mysterious difficulties of creating software — that I’ll be announcing soon. I don’t mean to be mysterious; there are just some details to be worked out between me, my blog and Salon.

In the course of seeing my 100,000-or-so words transformed from a Word document (it’s what the publishers want!) to a typeset galley, I’ve learned a bit more about what goes into making the text of a book look good and fit right. (My editor and the Crown designers did a great job with Dreaming in Code.) So when I recently stumbled on this blog by a book designer — addressing the realms of typography, castoffs and such — I took note. Fascinating stuff if the extent of your knowledge of publishing design is, like mine, drawn primarily from the newspaper world and the computer desktop.

[tags]Dreaming in Code, books, publishing[/tags]

Kiko’s calendar auction and the old “incremental change” song

by

Kiko is an Ajax-style Web-based calendar service. (It’s also the title of a fantastic album by Los Lobos.) Kiko’s developers, only a few months after unveiling it, have put it up for sale on Ebay for $50,000. So far, despite wide linkage, no takers.

Robert Scoble says this presages a Web 2.0 shakeout: “There are simply too many companies chasing too few users…. Getting the cool kids to try your technology isn’t the same thing as having a long-term business proposition.”

Could be. With Google’s new calendar gobbling up mindshare in an already crowded space (haven’t you heard that “Google is the New Microsoft“?), Kiko didn’t seem to have much chance.

The problem is that, unlike photo-sharing or video-staring or link-listing or news-rating, activities that have provided grist for successful Web 2.0 mills, calendaring doesn’t easily lend itself to large-scale social interaction and wisdom-of-crowds behavior. Calendars are either personal or apply to small, well-defined workgroups or personal circles. The piece of calendaring that’s most amenable to wide Web networking — the listing and sharing of information about public events — is already being pursued by several ambitious companies (Eventful, Zvents, etc.).

But even if calendars aren’t going to fuel the next Web 2.0 wunder-company, we still need them. The future for calendar software, as Scott Mace keeps reminding us, is more about interoperability than about snazzy Ajax features. Making sophisticated calendar-sharing work, and multi-authoring possible, and import-export painless — these are the things that will matter in this category (as the folks working on Chandler whose work I followed for Dreaming in Code understand so well).

Meanwhile, Justin Kan, a Kiko founder, lists his own set of lessons from the experience. They include the following: “Build incrementally. We tried to build the ultimate AJAX calendar all at once. It took a long time. We could have done it piece by piece. Nuff said.”

But it’s not nuff said, it’s never said ’nuff, it needs to be said over and over until you’re blue in the face and all your coworkers hate you and think you’re a monomaniac who has gotten this word “incremental” implanted in his neurons like some sort of development-process idee fixe. It is an important but counter-intuitive insight. It’s not how businesspeople want things to be. It’s not how developers are used to thinking. So if you actually understand that an incremental process for building an ambitious program or Web site is the best approach, you will have to be insufferable about it.

My friend Josh Kornbluth (who recently recounted some ancient tales from our collaboration 20 years ago on a low-rent radio drama show in the Boston area) once wrote a song titled “Incremental Change.” It was a cappella, it lasted all of 25 seconds and its entire lyric consisted of the following:

I think incremental change is a good thing
I think incremental change is a good thing
Incremental change: good thing!

Software development was almost certainly not on his mind at the time of writing. But the sentiment holds across a surprisingly broad range of fields.

POSTSCRIPT: Paul Graham, whose Y Combinator funded Kiko, says the company spent so little money the failure’s no big deal: “This is not an expensive, acrimonious flameout like used to happen during the Bubble. They tried hard; they made something good; they just happened to get hit by a stray bullet.”
[tags]web 2.0, calendars, software development[/tags]

Standish’s CHAOS Report and the software crisis

by

Whenever there is an article about software failure, there is a quotation from the venerable CHAOS Report — a survey by a Massachusetts-based consultancy called the Standish Group, first conducted in 1994 and regularly updated since. The CHAOS Report presented dire statistics about the high failure rate of software projects: 31.1 percent of projects cancelled, 52.7 percent “challenged” (completed only way over budget and/or behind schedule), and only 16.2% deemed a success.

There aren’t a whole lot of other statistics out there on this topic, so the numbers from Standish get big play. I used them myself in my book proposal, and returned to the report as I researched the book, interested in finding out more about the methodology the researchers used — and also curious about what “CHAOS” actually stood for: Combinatorial Heuristic Algorithm for the Observation of Software? Combine Honnete Ober Advancer…?

Nope. As far as I could tell, CHAOS is an acronym for nothing at all. I tried to contact Standish for more information by telephone and e-mail but they never responded. It wasn’t essential to my work — there’s only a handful of sentences on the subject in Dreaming in Code — so I didn’t push hard. I thought, maybe this was the sort of consultancy that was only interested in the paying customers.

In the August issue of Communications of the ACM, Robert Glass has a column about Standish and the CHAOS Report that suggests my failure to get a response from this organization was hardly unique.

Several researchers, interested in pursuing the origins of this key data, have contacted Standish and asked for a description of their research process, a summary of their latest findings, and in general a scholarly discussion of the validity of the findings. They raise those issues because most research studies conducted by academic and industry researchers arrive at data largely inconsistent with the Standish findings….
Repeatedly, those researchers who have queried Standish have been rebuffed in their quest….

Glass is a widely known and respected authority on the software development process (I read, and can recommend, his book Facts and Fallacies of Software Engineering as part of my book research), and the Communications of the ACM is the centerpiece journal of the computing field’s main professional organization. So maybe the Standish group will respond to the plea with which Glass closes his column:

it is important to note that all attempts to contact Standish about this issue, to get to the heart of this critical matter, have been unsuccessful. Here, in this column, I would like to renew that line of inquiry. Standish, please tell us whether the data we have all been quoting for more than a decade really means what some have been saying it means. It is too important a topic to have such a high degree of uncertainty associated with it.

Indeed. The Standish numbers are precisely the sort of statistic that journalists in need of background “facts” — and scholars, too, for that matter — will quote in an endless loop of repetition, like Newsweek’s infamous stats showing that thirtysomething women were more likely to be killed by terrorists than to find husbands. The loop keeps repeating until someone provides a definitive debunking — and even then it doesn’t always stop.

So it’s ironic but hardly surprising to find the same magazine that contains Glass’s complaint also featuring a cover story on “The Changing Software Engineering Paradigm” that parrots the Standish numbers for the umpteenth time.
[tags]dreaming in code, software development, software failures, chaos report[/tags]

Outliners, trees and meshes

by

Thanks to Julian Bond for his response to last Thursday’s Outliners post, which pointed me to his essay from 2004 on outlining, “Outliners Considered Harmful.” (The title, I should point out for those of you who are not steeped in programming lore, is a nod to a tradition in that literature dating back to software pioneer Edsger Dijsktra’s 1968 paper “Goto Statement Considered Harmful.”)

When you use a tool that encourages you to think in terms of hierarchies, everything looks like a hierarchy. Unfortunately the world is much much messier than that. Almost everything is actually a mesh not a hierarchy. And when hierachies do exist in the data, it’s very likely that you will find 2 or more inherent hierarchies that are orthogonal and in most real world situations it’s more like 10…. I suspect that the move towards anarchic tagging in systems like del.cio.us and Flickr are driven by this tension between mesh and hierarchy as well though I haven’t seen it expressed like this. Tag driven systems look horribly uncontrolled to hierarchy people. But they actually reflect the real world much better than hierarchies….So in a nutshell, Outliners are harmful because they lead to hierarchy thinking. And hierarchy thinking is harmful because it leads to political hierarchies. And all of this is harmful because the world is actually mess(h)y and not structured into elegant trees.

I’m reasonably familiar with this argument. For one thing, it’s part of a discussion I closely followed while researching my book at OSAF. During some of the early work on Chandler, its designer, Mimi Yin, inspired by Christopher Alexander’s celebrated essay “A City Is Not A Tree,” tried to figure out a structure for personal-information management software that privileged “semi-lattices” (what Bond calls “meshes”) over hierarchies and trees. (Clay Shirky’s 2004 post on the Alexander essay is worth rereading.)

I’m also familiar with it because it is a theme that David Weinberger has been pursuing over the last several years — a pursuit that is culminating in the publication next year of his book, Everything is Miscellaneous, which, he tells us, he has recently completed, and which I’m eagerly awaiting.

The thing is, I’m not actually a particularly hierarchical thinker. My love of outlining is less a matter of obsession with rank and structure than an appreciation of flexibility. I don’t especially care that outlines are built around parent and child nodes neatly arrayed in tree structures; what matters is that outlines give me easy handles to move chunks of loosely structured information around, and they let me quickly zoom from a low-altitude view to a high-altitude overview and back.

All of which may explain why I still love Ecco Pro so much. For one thing, Ecco doesn’t force you to follow the outline structure rigidly; you can drag nodes pretty much anywhere you want. Even more important, each Ecco outline can also be fitted with “columns” — really, versatile tags or categories (they can be free text or checkbox or dropdown or date) that provide exactly the sort of cross-meshing or semi-latticing that Bond rightly reminds us we need. You get the best of two worlds — outlining structure and free-form mesh-i-ness. And it’s very easy to adapt to the David Allen “Getting Things Done” method for those who’ve been bitten by that bug.

I’ll stop raving now: Ecco is old, unsupported and probably has no future. Still, it’s rock-solid, and free, and it continues to serve me better than anything else I’ve found. As for the “Outliners make your brain too hierarchical” line, it might hold for some of the simpler outlining tools out there, but I really don’t think it applies to a program this versatile and fluid.
[tags]outliners, pims[/tags]

Outliners then and now

by

I am addicted to outlining as a means of organizing my work and life. (And no, the outliner in MSWord does not count, it’s a clumsy, kludgy horror that has probably turned off millions to the value of outlining.) I still use Ecco Pro — a long-orphaned Windows outliner — every day. (This old post has links to some of my writing on the topic.) I used Ecco to compile the research for my book, and I use it, GTD-style, to keep the spheres of my life moving in harmony.

Ecco is a fascinating hybrid of the pure outliner that Dave Winer pioneered in the 1980s and the free-form personal database exemplified by Mitch Kapor’s Lotus Agenda, which let you recategorize and invent new categories for your information on the fly. (Today’s tagging phenomenon is a latter-day version of the idea.) Chandler, the product whose development my book Dreaming in Code chronicles, started with the ambition of bringing some of these ideas into the present, though it has since evolved in different directions.

I was reminded of this complex software genealogy recently as I read a page that Winer recently linked to — a detailed chronicle, written in 1988, of how his once-popular outliners (ThinkTank and More) came to be developed. (I found it because Winer linked to it from another page about thinking about the Internet as an idea processor — which is also food for thought.)

I’ve never understood why outliners never found wider adoption. Is it just the curse of Word (once Microsoft “included” outlining in Word, however poorly, the market evaporated)? Is it that people associate outlining with boring work they had to do in high school composition class? Is it that the number of people who like to organize their thoughts in collapsible hierarchies is just not very high? But the alternative model of idea-organization tools, which provides you with more of a 2D or 3D space to place and link words and concepts (cf. The Brain and other “mind-mappers“) has never caught on in a big way, either. Maybe the vast majority of people are still too busy figuring out how to wrestle their computers into submission to concern themselves with trying to use them as (in Howard Rheingold’s phrase) “brain amplifiers.”

Many contemporary outliners (like Shadowplan) feel more like checklist organizers than tools for organizing large amounts of text. With the more sophisticated programs, one problem I have (I’m thinking here of tools like Zoot and InfoSelect) is that they are built like e-mail clients with separate panes — a pane on the left where you expand and collapse nodes, and then a pane on the right where you read the text associated with the node that’s highlighted at the left. This separates the “thing itself” from the “relationships between things.” That’s not the way my mind works: I want to see the things and their relationships — all at once!

In Ecco, as in More, you’ve got the full text of each node right in front of you, in place in the outline hierarchy. This allows you to use the tool — as I understand Dave Winer does — as a primary writing environment; it also allows you to dump huge amounts of information into the outline efficiently, move big pieces around easily, and swoop quickly from a top-level overview to the finer details.

Today Mac users can adopt OmniOutliner, which has a feature called “inline notes” that begins to move it toward the model I prefer. If I were using a Mac every day I’d also check out Eastgate’s Tinderbox, Circus Ponies Notebook and VoodooPad. Windows users can still get Ecco for free. In the new world of web-based apps, there’s not a lot of activity yet — though there is a rudimentary AJAX-based outliner called Sproutliner. 37 Signals, the “small is beautiful” web app company, has a lightweight listmaker called Tada List, along with another product that’s sort of a free-form personal info manager called Backpack. And then of course we come full circle back to Dave Winer, who has created the Web-based outline format OPML (the OPML editor is here) for constructing and sharing Web-based outlines.

I don’t know if outlining software will ever take off, but to me it feels like a natural way to use a computer. I will keep using Ecco until they invent a version of Windows that won’t run it, and I suspect I will outline until the day I die.

POSTSCRIPT: Doc Searls’ technography from Bloggercon IV is a good example of outlining in action. He wrote about it here.
[tags]outliners, pims[/tags]

Yahoo goes scrum

by

Buried at the tail end of yet another which-portal’s-on-top? feature in the Times is this interesting tidbit about software development practices at Yahoo:

Meanwhile, Yahoo says it is now trying to emulate Google’s faster method of creating products. Like most big companies, it used to develop software by first creating a comprehensive design that defined how features would be written and tested. Instead, it is now trying what is known as a scrum method, where it will plan, build and test parts of a product every 30 days.

“We may not know how everything fits together,” Mr. Patel said. But by creating partly completed products that can be shown to customers, “We can get insights from users and react to that over a three- or four-month period to put it all together,” he said.

Scrum is a species of agile software development in which the development team, among other things, holds quick daily meetings and delivers new bits of functioning software on very short schedules. It’s all about “moving the ball forward,” scrum expert Ken Schwaber says.

[tags]software, software development, yahoo, google, scrum[/tags]

Mashup Camp 2

by

Yesterday I spent the day at Mashup Camp 2. I missed the first one last winter, but what I read intrigued me enough to make a point of showing up when it came around again.

The two relevant things here, one having to do with mashups, the other with that word “camp,” which is really a proxy for the whole “unconference” movement of which this event is a high-profile example in the tech world. (Mashup Camp organizer David Berlind wrote about the first event’s experience with the format back in February.) Let’s start with that.

When I showed up at 9 a.m. down in Mountain View, at the Computer History Museum, the conference had no schedule — just an open grid on an eight-foot-long pad at the front of the meeting hall. An hour later, several dozen developers (and some “API providers,” a k a vendors or company reps) had introduced themselves, proposed sessions, posted the sessions on the grid, and presto, there it was, a conference schedule.

Mashup Camp instant schedule grid

There had been no arguments over process, no disputes, no grandstanding or boring throat-clearing. Part of that was the result of deft moderation by Kaliya Hamlin (she writes about the event here); part, no doubt, was the nature of the attendees — this was primarily an engineering conclave, after all. If we’d been talking about Iraq, something tells me the process might have been bumpier.

In the pop culture world, “mashup” means creating a new work by combining elements of two (or more) existing works. (Danger Mouse’s “Grey Album” — the Beatles’ White Album meets Jay-Z’s Black Album — is probably the highest-profile example in music to date.) In software, a mashup is a new program or service created by wiring up two or more existing programs or services.

Web-services mashups can be remarkably easy to hack together and provide immediately gratifying results — the canonical example was the Craigslist/GoogleMaps mashup that Paul Rademacher made last year, placing the Craigslist for-rent ads on Google’s map service. At Mashup Camp, developers got the opportunity to show off their projects during a “Speed Geeking” event (modeled on speed dating) at which visitors in groups of a half-dozen wandered from table to table to hear five-minute demos. Here’s a full list of the participating demo-ers.

I didn’t come away with the sense that any one of the projects I saw was going to change the universe. But put it all together and you got a window onto a simpler, faster, and perhaps smarter approach to software product development — one that trades in the virtue of from-the-ground-up consistency and thoroughness for the even more compelling virtue of “getting something working fast.” It’s software development as a Darwinian ocean in which large numbers of small projects are launched into the water. Only a handful will make it to land. But most of them required so little investment that the casualty rate is nothing to lose sleep over.

Fallows, PIMs and Chandler

by

James Fallows has been writing thoughtfully about computer software for longer than most of us have been using it. Years ago he wrote a definitive paean (long online here but apparently no longer) to Lotus Agenda, Mitch Kapor’s legendary personal information manager. (I say “a” rather than “the” because this program evoked such loyalty from smart writers it actually ended up with two definitive paeans; the other was by Jimmy Guterman.)

In the new issue of the Atlantic, Fallows writes about two latter-day PIMs — Microsoft’s OneNote and Chandler, the long-gestating project of Kapor’s Open Source Applications Foundation, the tale of which forms the central narrative of my book, Dreaming in Code. He interviewed me for the article; though most of our conversation wound up on the cutting-room floor, I did make it into one paragraph. I wish the article were online (there’s a stub here, but the full piece is only accessible to subscribers). But I couldn’t ask for a better venue for my first distant-early-warning book publicity. Here’s the relevant graph:

Despite substantial follow-up grants from foundations and universities, the team developing Chandler has so far released only a partly functional calendar application. Scott Rosenberg, of Salon magazine, became an “embedded journalist” on the Chandler project from 2003 to 2005 in order to investigate why good software is so hard to make. (His book about Chandler and complex software design, Dreaming in Code, will be published in November [now, January]). “It is taking a long time, but anyone who writes off Chandler is being short-sighted,” he told me. “They are on a quest.”

Fallows asked me whether I thought the book had turned out to be a comedy or a tragedy.

“Neither,” I replied, thinking furiously on my feet, my brain flashing back to my decade as a theater and movie critic. “It’s an epic!”
[tags]James Fallows, Dreaming in Code, Chandler[/tags]

Gates’ departure and a Windows Vista postmortem

by

This news of Bill Gates’ plans to step down in 2008 doesn’t really surprise me. From what I’ve been able to tell, Gates was engaged and excited during the early stages of Longhorn/Vista, back when Microsoft was promising revolutionary transformations to the file system and everything else. The following scaling-back and repeated delays of the project must have left him with an overwhelming sense of deja vu. When you’re the richest man in the world, who needs it?

So Gates will ride off into the sunset, and Microsoft will either find a way to reinvent itself, perhaps in the hands of Ray Ozzie, or gradually devolve into maintenance mode.

Given these events, this Microsoft blogger’s report on why the Vista delays happened is even more relevant. The posting was up for a while today, then the blogger, Philip Su, removed it — of his own accord, he says, not under pressure. The heated discussion remains on the page. Here are some extensive, relevant excerpts:

 

…The oft-cited, oft-watercooler-discussed dual phenomenon of Windows code complexity and Windows process burden seem to have dramatically affected its overall code velocity. But code can be simplified and re-architected (and is indeed being done so by a collection of veteran architects in Windows, none of whom, incidentally, look anything like Colonel Sanders). Process can be streamlined where inefficient, eliminated where unnecessary.

But that’s not where it ends. There are deeper causes of Windows’ propensity to slippage…

Deep in the bowels of Windows, there remains the whiff of a bygone culture of belittlement and aggression. Windows can be a scary place to tell the truth.

When a vice president in Windows asks you whether your team will ship on time, they might well have asked you whether they look fat in their new Armani suit. The answer to the question is deeply meaningful to them. It’s certainly true in some sense that they genuinely want to know. But in a very important other sense, in a sense that you’ll come to regret night after night if you get it wrong, there’s really only one answer you can give.

After months of hearing of how a certain influential team in Windows was going to cause the Vista release to slip, I, full of abstract self-righteous misgivings as a stockholder, had at last the chance to speak with two of the team’s key managers, asking them how they could be so, please-excuse-the-term, I-don’t-mean-its-value-laden-connotation, ignorant as to proper estimation of software schedules. Turns out they’re actually great project managers. They knew months in advance that the schedule would never work. So they told their VP. And he, possibly influenced by one too many instances where engineering re-routes power to the warp core, thus completing the heretofore impossible six-hour task in a mere three, summarily sent the managers back to “figure out how to make it work.” The managers re-estimated, nipped and tucked, liposuctioned, did everything short of a lobotomy — and still did not have a schedule that fit. The VP was not pleased. “You’re smart people. Find a way!” This went back and forth for weeks, whereupon the intrepid managers finally understood how to get past the dilemma. They simply stopped telling the truth. “Sure, everything fits. We cut and cut, and here we are. Vista by August or bust. You got it, boss.”

Every once in a while, Truth still pipes up in meetings. When this happens, more often than not, Truth is simply bent over an authoritative knee and soundly spanked into silence.

…Micromanagement, though not pervasive, is nevertheless evident. Senior vice presidents sometimes review UI designs of individual features, a nod to Steve Jobs that would in better days have betokened a true honor but for its randomizing effects. Give me a cathedral, give me a bazaar — really, either would be great. Just not this middle world in which some decisions are made freely while others are made by edict, with no apparent logic separating each from the other but the seeming curiosity of someone in charge.

…We shouldn’t forget despite all this that Windows Vista remains the largest concerted software project in human history. The types of software management issues being dealt with by Windows leaders are hard problems, problems that no other company has solved successfully. The solutions to these challenges are certainly not trivial.

An interesting question, however, is whether or not Windows Vista ever had a chance to ship on time to begin with. Is Vista merely uncontrolled? Or is it fundamentally uncontrollable? There is a critical difference.

It’s rumored that VPs in Windows were offered big bonuses contingent on shipping Vista by the much-publicized August 2006 date. Chris Jones even declared in writing that he wouldn’t take a bonus if Vista slips past August. If this is true, if folks like Brian Valentine held division-wide meetings where August 2006 was declared as the drop-dead ship date, if general managers were consistently told of the fiscal importance of hitting August, if everyone down to individual developers was told to sign on the dotted line to commit to the date, and to speak up if they had any doubts of hitting it — mind you, every last one of those things happened — and yet, and yet, the August date was slipped, one has to wonder whether it was merely illusory, given the collective failure of such unified human will, that Vista was ever controllable in the first place.