Wordyard

Hand-forged posts since 2002

Scott Rosenberg

Archives

New bridge, old book: the shape of software progress

by 3 Comments

New Bay Bridge east spanThe Bay Bridge’s new eastern span is about to open. When they started building it over a decade ago, I was beginning work on my book Dreaming in Code. As I began digging into the history of software development projects and their myriad delays and disasters, I kept encountering the same line: Managers and executives and team leaders and programmers all kept asking, “Why can’t we build software the way we build bridges?”

The notion, of course, was that somehow, we’d licked building bridges. We knew how to plan their design, how to organize their construction, how to bring them in safely and on time and within budget. Software, by contrast, was a mess. Its creators regularly resorted to phrases like “train wreck” and “death march.”

As I began my research, I could hear, rattling the windows of my Berkeley home, the deep clank of giant pylons being driven into the bed of San Francisco Bay — the first steps down the road that ends today with the opening of this gleaming new span. I wrote the tale of the new bridge into my text as an intriguing potential contrast to the abstract issues that beset programmers.

As it turned out, of course, this mammoth project proved an ironic case in any argument involving bridge-building and software. The bridge took way longer than planned; cost orders of magnitude more than expected; got hung up in bureaucratic delays, political infighting, and disputes among engineers and inspectors; and finally encountered an alarming last-minute “bug” in the form of snapped earthquake bolts.

So much for having bridges down. All that the Bay Bridge project had to teach software developers, really, was some simple lessons: Be humble. Ask questions. Plan for failure as well as success.

Discouraging as that example may be, I’m far more optimistic these days about the software world than I would ever have expected to become while working on Dreaming. Most software gets developed faster and in closer touch with users than ever before. We’ve turned “waterfall development” into a term of disparagement. No one wants to work that way: devising elaborate blueprints after exhaustive “requirements discovery” phases, then cranking out code according to schedules of unmeetable precision — all in isolation from actual users and their changing needs. In the best shops today, working code gets deployed regularly and efficiently, and there’s often a tight feedback loop for fixing errors and improving features.

My own recent experiences working closely with small teams of great developers, both with MediaBugs and now at Grist, have left me feeling more confident about our ability to wrestle code into useful forms while preserving our sanity. Software disasters are still going to happen, but I think collectively the industry has grown better at avoiding them or limiting their damage.

While I was chronicling the quixotic travails of OSAF’s Chandler team for my book, Microsoft was leading legions of programmers down a dead-end path named Longhorn — the ambitious, cursed souffle of an operating system upgrade that collapsed into the mess known as Windows Vista. At the time, this saga served to remind me that the kinds of delays and dilemmas the open-source coders at OSAF confronted were just as likely in the big corporate software world. Evidently, the pain still lingers: When Steve Ballmer announced his retirement recently, he cited “the loopedy-loo that we did that was sort of Longhorn to Vista” as his biggest regret.

But Longhorn might well have been the last of the old-school “death marches.” Partly that’s because we’ve learned from past mistakes; but partly, too, it’s because our computing environments continue to evolve.

Our digital lives now rest on a combination of small devices and vast platforms. The tech world is in the middle of one of the long pendulum swings between client and server, and right now the burden of software complexity is borne most heavily on the server side. The teeming hive-like cloud systems operated by Google, Facebook, Amazon and their ilk, housed in energy-sucking server farms and protected by redundancy and resilient thinking, are among the wonders of our world. Their software is run from datacenters, patched at will and constantly evolving. Such systems are beginning to feel almost biological in their characteristics and complexities.

Meanwhile, the data these services accumulate and the speed with which they can extract useful information from it leave us awe-struck. When we contemplate this kind of system, we can’t help beginning to think of it as a kind of crowdsourced artificial intelligence.

Things are different over on the device side. There, programmers are still coping with limited resources, struggling with issues like load speed and processor limits, and arguing over hoary arcana like memory management and garbage collection.

The developers at the cloud platform vendors are, for the most part, too smart and too independent-minded to sign up for death marches. Also, their companies’ successes have shielded them so far from the kind of desperate business pressures that can fuel reckless over-commitment and crazy gambles.

But the tech universe moves in cycles, not arcs. The client/server pendulum will swing back. Platform vendors will turn the screws on users to extract more investor return and comply with increasingly intrusive government orders. Meanwhile, the power and speed of those little handheld computers we have embraced will keep expanding. And the sort of programmers whose work I celebrated in Dreaming in Code will keep inventing new ways to unlock those devices’ power. It’s already beginning to happen. Personal clouds, anyone?

Just as the mainframe priesthood had to give way to the personal-computing rebels, and the walled-garden networks fell to the open Internet, the centralized, controlled platforms of today will be challenged by a new generation of innovators who prefer a more distributed, self-directed approach.

I don’t know exactly how it will play out, but I can’t wait to see!

Appearances elsewhere (NOOP.nl, Fray)

by 3 Comments

Item one: Jurgen Appelo was kind enough to include Dreaming in Code earlier this year in his Top 100 Best Software Engineering Books Ever. (Top 100 and best ever! Yow!) When he came back to me and asked me to do one of his “Five Easy Questions For…” interviews, how could I refuse?

Here is the result, in which I talk about, among other things, my early days as a freelancer, what inspires me to keep writing, and what is more interesting than software development.

Item two: Earlier this year, while I was deeply immersed in work on the new book, Derek Powazek asked if I’d contribute to Fray, his labor-of-love magazine — yes, a paper magazine, it’s a thing of beauty. The sensible thing to do would have been to beg off, and I was going to. But the topic of the issue was going to be “Geek: True Stories of People Taking Things Too Seriously,” and as I plowed through my book work I found my mind drifting back to my youth, and a time when I was a true geek, not for software or Tolkien or theater or politics or any of the other things I have geeked out on over the years, but for tropical fish — which was, for me, the ur-geekly pursuit.

So I wrote a little piece, titled Memories of a Fickle God, about those times. It’s online here, along with a bunch of far more captivating pieces.

But really, if you’re interested in this stuff, the thing to do is to buy the magazine (or subscribe), because it’s full of great stories and beautiful art. This is the future of the print magazine: once the profits have all migrated elsewhere, people will still publish on paper. But they’ll do it for their own damn reasons.

aquaristsmall

Interview with Lovink re: Dreaming in Code

by 5 Comments

And now, we take a brief break from the financial apocalypse for some personal notes.

I remain in deep writing mode here — can’t say there’s light at the end of the tunnel yet, but I can say the tunnel is reasonably comfortable!

But I wanted to point those interested to an in-depth interview I recently gave to Geert Lovink, the critic and author of works like “Zero Comments” and “Blogging: The Nihilist Impulse.” It’s all about Dreaming in Code and Chandler. Here’s a brief excerpt:

GL: Most IT books you can buy are propped-up business show cases that only talk about success. Dreaming in Code is so radically different in this respect. The project drags on, and at times, the text is amazing honest, up to the point of straight out European negativity. How did you manage to do this? You wrote the book in San Francisco, not in Berlin.

SR: I’ll take this as a compliment. I started my career as a theater critic. I prize honesty. I can’t imagine working on a book for 3-4 years if I didn’t set out to be honest. When I hear “how did you manage it?” it sort of sounds to me like, “how did you get away with it?” But in fact my publisher and editor were always behind the project. My book proposal was really clear about what kind of book it was going to be…

Some readers were disappointed that Dreaming in Code didn’t give them more bullet points about how to improve their development projects. I had hoped that it was clear from the first page that this just wasn’t going to be that kind of book. If I knew how to solve these problems I’d be busy solving them, not writing about them! But writing about them has value, nonetheless, I hope. Just serving witness to the incredibly difficult and uniquely problematic work that software developers do — that was my aim, in the end.

Chandler 1.0 ships

by

When I first began reporting on Chandler for Dreaming in Code at the very start of 2003, there was talk of shipping a 1.0 version within a year. Then, in following years, the project got so bogged down that at times it was hard to imagine it ever arriving at such a milestone.

Well, on Friday, the OSAF team released a 1.0 version of Chandler. At the moment I am too deep in the swamps of blog history circa 2001 to do full justice to this news, but must take note nonetheless.

Chandler, of course, is the personal-information-management application whose story sat at the center of my first book. I last checked in on the project at the start of this year, when OSAF and Kapor parted ways.

It’s been close to six years since Mitch Kapor first announced plans for Chandler, and the application today is quite different from what was envisioned then. But it does fulfill at least a portion of the ambitious agenda Kapor set: It’s fully cross-platform, and, from the user side, it takes a very flexible approach to data. The program was once positioned as a calendar with email and task capabilities, and it’s still got those features, but it’s now presented as a notebook program — it’s “The Note-To-Self Organizer.” You store information free-form and then can organize it according to now/later/done triaging, turn items into tasks and schedule them on the calendar, group data in multiple collections, and share it across the web via the Hub server. I’m looking forward to experimenting more with it.

The OSAF blog post announcement includes some more detail. And James Fallows has a good post up at the Atlantic.

Singing in Code

by 1 Comment

OK, this one is just for plain fun: it’s the first Wordyard playlist.

When I was planning my campaign of global domination for Dreaming in Code I had visions of a multimedia onslaught. I’d pull together video clips that epitomized the nightmare of software scheduling, from A Brief History of Time to Groundhog Day to Lawrence of Arabia (that quicksand scene, of course), and music that similarly reflected the themes.

Didn’t get too far…but I did compile a list of songs that might be the book’s soundtrack. (Tip of the hat to Largehearted Boy‘s custom of inviting authors to assemble playlists for their novels, and to Josh Kornbluth‘s loving selection of apropos tunes to precede his solo shows.)

(1) “Put Your Hand On The Computer,” They Might Be Giants — ‘Cause that’s how it always starts.

(2) “Bill Gates Must Die,” John Vanderslice — Certainly, most open source developers aren’t obsessive sociopaths like this song’s narrator. But they have always harbored a certain animosity toward the founder of Microsoft, and sometimes it gets a little personal. (Bonus rationale: This song once fried my motherboard.)

(3) “Source Tags and Codes,” And You Will Know Us By the Trail of Dead — “Spend half a life deciding what went wrong / Trying to find out what took you so long.”

(4) “Dot Dash,” Wire.

(5) “Systems Crash,” Guided By Voices.

(6) “I Want to Live on an Abstract Plain,” Frank Black.

(7) “Information Age,” Damon and Naomi.

(8) “Waiting for the Great Leap Forward,” Billy Bragg.

(9) “Raymond Chandler Evening,” Robyn Hitchcock — Chandler the software is named for the novelist. But the song’s last line (“And I’m lurking in the shadows / ‘Cause it hasn’t happened yet”) echoes my software epic’s in medias res ending, too!

(10) “Your Belgian Things,” the Mountain Goats — “The arteries are clogging in the mainframe / There’s too much information in the pipes.”

Those paperbacks are gone

by 2 Comments

I’m out of copies, so this concludes the free-paperbacks-for-bloggers program.

Two side notes:

A handful of the requests I received got trapped in the spam filter. I think the word FREE adds a lot of points. (This is something for Wired editor Chris Anderson to ponder as he pursues work on his new book about “Free” as a business model!)

Also, a surprisingly high percentage of the folks who requested the book failed to include their street addresses in their original email. Despite knowing that they were asking me to send them something via snailmail, their minds just blanked on that little prerequisitie.

I think many of us — certainly including me — do so much of our business on the Net these days that the pesky details of physical-world object transfer just slip right by us.

Chesterton quote archeology

by 4 Comments

That Orwell quote earlier this week that began “Our civilization is founded on coal” had a “pace Chesterton” at the start that puzzled me. A number of you wrote in with suggestions, including a pointer to a fascinating debate between Chesterton and Bernard Shaw about whether to nationalize the coal mines, moderated by Hilaire Belloc.

But I believe Mark Bernstein found the ur-instance of the Chesterton reference:

This is the true and exact account of the Great Cigar Fraud, and the moral of it is this–that civilisation is founded upon abstractions. The idea of debt is one which cannot be conveyed by physical motions at all, because it is an abstract idea.

So, what Orwell was really saying was: sorry, G.K., our civilization is not founded upon abstractions, it’s founded on the hard reality of coal mining. And thus Stroustrup’s reformulation — “our civilization is built on software” — takes us full circle, back to the many layers of abstraction that constitute our program code.

It all connects!

Judging books by the page

by 2 Comments

I confess I’m confused.

A good while back I read about the “page 69 test” — apparently descended to us from Marshall McLuhan. The idea is that you can open any book to page 69 and use that to determine whether you will like the book.

Well, OK. Page 69 of Dreaming in Code contains a description of Moore’s Law and concludes, “…there is no Moore’s Law for software. Chips may double in capacity every year or two; our brains don’t.” Whew. I think that’ll do the trick for at least some people.

Only next I read about a variation of this, called the “page 99 test,” and attributed to Ford Madox Ford: “0pen the book to page ninety-nine and read, and the quality of the whole will be revealed to you.” So, let’s see: on page 99 of my book you can read a story about how hard it is for developers to keep up with the tools available to them. In a visit to OSAF, a programmer named Anthony Baxter described his search for ways to speed up the processing of audio data in a Python application. Baxter was the release manager for the most recent version of Python, yet even he had forgotten that the programming language comes with a utility that exactly suited his needs. “The batteries were indeed included, as Python devotees liked to say. But with so many batteries, who could keep track of them all?”

OK. Fine. I’m willing to let my work be judged on this, too!

But now here comes the page 123 test! This one seems less about helping you decide whether to read a book and more about “bibliomancy,” or the art of making oracular use of arbitrarily selected passages of books. The page 123 test dictates that one “grab the nearest book, open to page 123, go down to the 5th sentence and type up the 3 following sentences.”

For Dreaming, this turns out to be a passage about the Chandler Project’s search for a development manager:

As the hunt dragged on, Lou Montulli and Aleks Totic suggested a name from their Netscape days. Michael Toy had been one of a band of employees at Silicon Graphics who left with its founder, Jim Clark, when Clark decided to start a new venture that would turn into Netscape. He had led the company’s programming team through several hyperspeed cycles of the browser wars in an era that redefined the norms for software development, establishing new benchmarks for fast releases of new versions.

I think I know what the multiplication of these memes is getting at. At this rate, authors are going to have to expect to be judged by every page they write. The nerve of that!

Books aren’t typically fractal — you can’t pull out lots of individual parts and allow each to stand for the whole. But each passage ought to count. In the end, every page and every sentence of a book ought to be able to present a good face for the larger entity it belongs to — like a diplomat abroad.

Code mining

by 3 Comments

I wrote Dreaming in Code because I believed that, as Bjarne Stroustrup says, “our civilization is built on software.” I noticed that creating software remains stubbornly difficult in certain ways, and, despite its centrality to our civilization, our understanding of that difficulty remains deficient. But I also wanted to create a journalistic record of the day-to-day experience of the software developer at the start of the 21st century — to tell a story about the act of programming itself.

I’m grateful that a good number of the book’s readers who’ve posted their thoughts feel that I achieved that goal. Others don’t think I did, and some days I agree with the criticism. Writing about the act of programming itself is as difficult as writing about any act of writing: the subject is an essentially interior process between the mind and the page (or screen), and it’s highly resistant to illumination.

Consider the difference when the topic of an essay is a rough physical act — like, say, digging coal out of the ground. I read a lot of George Orwell early in my career but I’d forgotten this passage, which Brad DeLong hoisted into the light of blog last year:

Our civilization, pace Chesterton, is founded on coal, more completely than one realizes until one stops to think about it. The machines that keep us alive, and the machines that make machines, are all directly or indirectly dependent upon coal. In the metabolism of the Western world the coal-miner is second in importance only to the man who ploughs the soil. He is a sort of caryatid upon whose shoulders nearly everything that is not grimy is supported. For this reason the actual process by which coal is extracted is well worth watching, if you get the chance and are willing to take the trouble.

When you go down a coal-mine it is important to try and get to the coal face when the ‘fillers’ are at work. This is not easy, because when the mine is working visitors are a nuisance and are not encouraged, but if you go at any other time, it is possible to come away with a totally wrong impression. On a Sunday, for instance, a mine seems almost peaceful. The time to go there is when the machines are roaring and the air is black with coal dust, and when you can actually see what the miners have to do. At those times the place is like hell, or at any rate like my own mental picture of hell. Most of the things one imagines in hell are if there — heat, noise, confusion, darkness, foul air, and, above all, unbearably cramped space. Everything except the fire, for there is no fire down there except the feeble beams of Davy lamps and electric torches which scarcely penetrate the clouds of coal dust.

It should go without saying that a wide gulf separates the strenuous and perilous experience of the “grimy” miners that Orwell depicted and the abstract, cerebral work of programming. The parallels are less obvious — but they jump out at me, too.

Both activities are essential to industry and highly profitable to those at the top of the economic pyramid they support. Both require the exploitation of long hours put in by young workers. In each, the treasures society values are struggled for in dim places and retrieved into the daylight after obscure labors to which their beneficiaries are oblivious.

Writing about a software project couldn’t have been more physically different from descending into a mine. But there were times, during my three years of research, when I felt like I was in an underground labyrinth, hunting for nuggets of insight in the dark.

It was, in any case, striking to find the “Our civilization is founded on…” construction that kicks off Dreaming in Code in Orwell’s penetrating lead. I’ve been trying to trace the Chesterton passage Orwell refers to as his antecedent, but so far no luck. Anyone have a clue?