Hand-forged posts since 2002
The Dreaming in Code 2007 World Tour (of the Greater Bay Area) is kicking off tonight (Thursday). At 7:30 p.m. I’ll be at the legendary Kepler’s Books in Menlo Park, talking about Dreaming in Code, reading a bit from it and — I hope — hearing from some of you about your experiences in the mire of software time.
Other events are listed here.
Apropos of my previous post on YouTube and MySpace, today I read this fascinating case-study from Baseline magazine about the saga of MySpace’s understandably overtaxed systems.
MySpace’s exploding popularity has basically forced its infrastructure through a continuous cycle of upgrades, refactorings and revampings. Its managers have never had the luxury of sitting back and calmly planning upgrades; they’ve had to perform their engine surgeries on a careening vehicle.
This is what Web 2.0 is like from the back end, and it ain’t pretty. Outside of the real masters of this stuff — the Googles and Yahoos that know how to deploy, manage and maintain vast online services — it’s a big mess. This is another little-understood dynamic of the Web 2.0 startup world: There are financial reasons a successful small service might want to be acquired, but there are even more pressing operational reasons. And the more success a service finds, the more likely it’s going to risk systems flameout.
It’s not at all clear from the Baseline piece that MySpace has yet achieved a level of stability that a more mature company might desire. MySpace, of course, was acquired not by a technology company but by a media outfit, so — unlike other popular companies that were acquired by Yahoo or Google — they’re still somewhat on their own.
The Baseline piece offers two other fascinating tidbits. In the first, a normal phenomenon for a successful site — massive surges of traffic — was interpreted as a bug by the Microsoft server platform MySpace uses:
Last summer, MySpace’s Windows 2003 servers shut down unexpectedly on multiple occasions. The culprit turned out to be a built-in feature of the operating system designed to prevent distributed denial of service attacks—a hacker tactic in which a Web site is subjected to so many connection requests from so many client computers that it crashes. MySpace is subject to those attacks just like many other top Web sites, but it defends against them at the network level rather than relying on this feature of Windows—which in this case was being triggered by hordes of legitimate connections from MySpace users.
“We were scratching our heads for about a month trying to figure out why our Windows 2003 servers kept shutting themselves off,” Benedetto says. Finally, with help from Microsoft, his team figured out how to tell the server to “ignore distributed denial of service; this is friendly fire.”
Second, it seems that MySpace didn’t actually originally intend to allow the level of customization that has made it so popular; its engineers just never got around to filtering out the user-customized formatting.
That feature was really “kind of a mistake,” says Duc Chau, one of the social networking site’s original developers. In other words, he neglected to write a routine that would strip Web coding tags from user postings– standard feature on most Web sites that allow user contributions.
The Web site’s managers belatedly debated whether to continue allowing users to post code “because it was making the page load slow, making some pages look ugly, and exposing security holes,” recalls Jason Feffer, former MySpace vice president of operations. “Ultimately we said, users come first, and this is what they want. We decided to allow the users to do what they wanted to do, and we would deal with the headaches.”
Here we have the state of Web development today: Your site’s massive success gets treated as a bug by your server; and the feature your users love best is something your programmers forgot to block.
[tags]baseline, myspace, web 2.0, software development[/tags]
Today, Dreaming in Code should be arriving in stores. Amazon has officially switched it from “available for preorder” to “in stock now.” And I’ve swapped out the “coming soon” banner on the book’s Web site for a “now available” sign.
For a writer whose career has arced from weekly newspapers to daily newspapers to round-the-clock Web sites, the transition to writing an old-fashioned book has been one big exercise in delayed gratification. By the time I started writing about the project on my blog, I’d already been working on it (researching and preparing a proposal) for almost two years. From that point to first draft was another year and half, and then more than one year further to get from draft to finished-book-in-your-hand. In this business, impatience doesn’t pay.
The extended timeline does provide many opportunities for reflection, and one of the things I’ve kept returning to is how utterly essential to my work the blogosphere has been. Of course I interviewed lots of people the old-fashioned way. But the fact that so many software developers now use the Web as an open notebook allowed me to explore the subject far more deeply and more widely than if I’d needed to track down and talk to each one of those programmers in person.
So thanks to every developer who’s posted thoughts on his or her work — you’ve made my work easier, and better. And thanks to all of you here who’ve followed along the escargot-paced progress of this project. During its multi-year course, writing here — and knowing that a bunch of great, smart people were reading and responding — helped keep me sane.
Now it gets fun. I’m not doing a world tour, a fact for which I and my family are grateful. But I’ll be making a number of public appearances here in the Bay Area, as well as some in-house events at some companies. Here’s the list (they’re also on Upcoming and Eventful):
Thursday, Jan. 18, 7:30 p.m.: Reading at Kepler’s Books, Menlo Park
Sunday, Jan. 21, 2 p.m.: Pleasanton Library (cosponsored by Towne Center Books), Pleasanton
Wednesday, Jan. 24, 12:30 p.m.: Reading at Stacey’s Books, San Francisco
I’m also doing presentations at Yahoo, Microsoft (Redmond) and Google over the next couple of weeks. If you work at one of those places and want more info, just let me know.
We’ll see what I learn from the people at those events, who undoubtedly will know more than I do, and I’ll try to bring some of it back to the Berkeley Cyber Salon on Feb. 25, where I’ll be hosting a panel on the themes of Dreaming in Code.
A book should stand on its own, reviews should stand on their own, and in general, it little profiteth a writer to reply directly to criticism. But at the end of his review of Dreaming in Code in the Journal (which is now available for free at this link), Paul Boutin asked me a direct question. So it doesn’t seem out of line to answer.
The author reluctantly condenses what he’s learned into Rosenberg’s Law: “Software is easy to make, except when you want it to do something new.” Cute, but MySpace and YouTube went from half-baked ideas to billion-dollar businesses while Mr. Rosenberg was writing his book. Is he saying they were hard, or that they do nothing new?
There’s no question that creating software using the Web as a platform provides multiple advantages, and has helped lots of companies, including those Boutin names, roll out products fast and improve them quickly. Web apps can be upgraded and patched without troubling the user and they come with a tight built-in user-feedback loop, so they lend themselves perfectly to an incremental improvement process, which is the smartest way to avoid the software-disaster swamp. They also solve cross-platform issues and provide sharing and collaborative features as an organic part of their environment.
This change in the software landscape is a running theme in Dreaming in Code, and one that the Chandler developers were acutely conscious of. But in today’s Cambrian explosion of Web-based applications, I can’t really think of each service as a discrete piece of software. The teeming startups of Web 2.0 are really each adding a feature or two — photo-sharing! backup! social bookmarking! word-processing! etc. — to the vast application that is the Web itself. (That’s why they make such fine acquisition-bait for the larger Web-platform companies who are competing with arsenals of such features.)
I do not intend any denigration of the huge accomplishments of the creators of both MySpace and YouTube, and I have no idea how hard or easy their development efforts were. But I think of these companies’ achievement as more in the realm of community-building than software invention. Of course they had to develop some software along the way. But their software platforms — for enabling the hang-out-together experience (at MySpace) or the sharing of short videos (on YouTube) — were not the centerpiece of their achievement. Each company refined and elaborated functions already being performed by other services. Each of them triumphed not by inventing novel extensions to the world of stuff you could do on the Web but by making things people were already doing online much easier, particularly for non-geeks.
As for their creation of “billion-dollar businesses”: remember that these are businesses that haven’t seen anything like billion-dollar revenues; they were valued by acquirers at half a billion dollars (in MySpace’s purchase by Fox) and $1.5 billion (in YouTube’s acquisition by Google)– and much of these prices was paid in stock, not cash.
Still, Boutin’s point is that both companies produced significant value quickly, and I’m not going to argue with that. But in each case what the acquisitors valued was less the code than the customers. You can be idealistic and call that “community,” or be cynical and call it “eyeballs”; either way, it’s what Google and Fox were buying.
In any case, my tongue-in-cheek Rosenberg’s Law made no claim to address the relative ease or difficulty of making money in the software business. I was trying to say something about the correlation between the ambition to innovate and the likelihood of ending up with your wheels in a ditch. MySpace wasn’t the first online hangout, and YouTube wasn’t the first video-sharing service. Instead, each of these companies went to school on their predecessors over the last decade of social media on the Web. And good for them! I imagine that what they did was “hard” in its own way — just not in the specific way that Dreaming in Code explores.
Dreaming in Code isn’t in stores till Tuesday, but there’s already been quite a bit of coverage.
Today saw Paul Boutin’s review in the Wall Street Journal (subscriber-only content, I’m afraid). I’m very happy the Journal chose to assign my book to a writer with Boutin’s experience and expertise in the software world. Also pleased with the review’s length and prominence. And I loved the illustration — a Sisyphus shouldering a boulder of code.
One thing Boutin noted made me smile as I read it: “His goal seems to be to teach non-programming managers not how to fix late projects, but how to accept them.” I can just see myself hanging out the guru shingle and peddling the Zen of project management!
In truth, my goal wasn’t to teach anyone anything. I neither promise nor deliver bullet-points of how-to advice. For me, the best non-fiction provides readers with an initiation into the complexity and fascination of a world that they barely even knew existed. I’m happiest when I hear from a reader who feels that I accomplished at least some of that.
But certainly I came to the conclusion, based on the evidence, that late software projects are — maybe not forever, but probably in our lifetimes — a fact of life. They are like quarreling among children or rebelliousness in adolescents. These are things we can mitigate, certainly, and learn to cope with better, or even sometimes turn to our advantage. But we are foolish if we think we can “fix” them or eliminate them. Sanity dictates some sort of accommodation with this kind of reality.
Boutin also concluded the review with a question to me about the principle I jokingly dub “Rosenberg’s Law” in the book. I actually have a lot to say about in response. But I must leave that for a post of its own!
Scott Berkun, who unlike me really does have a lot to teach people about how to avoid the pitfalls of bad project management, has also written up a measured and well-reasoned response to the book. And Rick Kleffel has some entertaining thoughts on the topic even before reading the book:
One of the most interesting aspects of my interview with Vernor Vinge was the point where we talked about what might suggest that we were or were not going to experience the Singularity in a manner described by most science fiction writers. Vinge suggested that the failure or success of large-scale software development projects would be a fine way to measure our progress towards the Infocalypse. Judging by the events described in ‘Dreaming in Code’, well, it appears that the Singularity Is Not Quite So Near as one might hope, if one were to hope It Is Near.
I’ve got a little link backlog. Let’s do something about it!
Rosen parses the motives and suggests that the warblog crowd would have done their cause a favor by being more critical of the Bush administration’s reality-evasion from the start:
For Bush supporters who soldier on, the choices resemble what the go-getters from Enron faced: confront the bad accounting that’s gone on for years or adopt even more desperate measures to conceal losses and keep your hand alive. But if the AP had fabricated a source and relied on that source 60 times, maybe the tables could be turned again and the reckoning put off….
If you really wanted Bush to succeed in Iraq, and you noticed that he could never be wrong or accept that bad news bearers could be right, this was a warning sign that the warbloggers themselves, as friends of the president’s project, should have taken the lead in discussing. Why didn’t they?
The children of Agnew have been fully on his side, soldiers in his struggle, happy warriors with Bush because they believe in their red state bones the press is biased against them. Like him they also disbelieve the bad news on principle, and then find someone more loyal to look into it.
This passage rung my Dreaming in Code bell:
The blog seemed like an awfully modest thing to me. But Meyerrose insists that the future of spying will be revolutionized as much by these small-bore projects as by billion-dollar high-tech systems. Indeed, he says that overly ambitious projects often result in expensive disasters, the way the F.B.I.’s $170 million attempt to overhaul its case-handling software died in 2005 after the software became so complex that the F.B.I. despaired of ever fixing the bugs and shelved it. In contrast, the blog software took only a day or two to get running. “We need to think big, start small and scale fast,” Meyerrose said.
One of the big problems the agencies have, even with their closed networks, is persuading intelligence officers to share information. On the one hand, their desire to protect sources is understandable; on the other, the information doesn’t do the U.S. any good unless it gets circulated to people who can assess its significance.
Is this the sort of information that is safe to share widely in an online network? Many in the intelligence agencies suspect not. Indeed, they often refuse to input sensitive intel into their own private, secure databases; they do not trust even their own colleagues, inside their own agencies, to keep their secrets safe. When the F.B.I. unveiled an automated case-support system in 1995, agents were supposed to begin entering all information from their continuing cases into it, so that other F.B.I. agents could benefit from the collected pool of tips. But many agents didn’t. They worried that a hard-won source might be accidentally exposed by an F.B.I. agent halfway across the country. Worse, what would happen if a hacker or criminal found access to the system?
[tags]journalism, fundamentalism, intelligence, open source spying[/tags]
On the Times op-ed page, Anthony Cordesman offers a devastating reality check on the Bush speech — made all the more withering for its even-handed calm.
Too bad that, in the print version, the type is so tiny; and in the online version, the critique is literally hidden from view until you click. This material should be highlighted, not buried.
The president’s speech offered the administration’s first grudging admission, after four years, that things aren’t as they should be in Iraq. But the phrasing was classic CEO buck-avoidance:
“Where mistakes have been made, the responsibility rests with me.”
Consider the different emotional impact of applying a simple Strunk and White transformation to the statement:
“We made mistakes, and I’m responsible for them.”
[tags]president bush, iraq, new york times, language, usage[/tags]
As of yesterday the Bush administration has definitively moved from the phase of “let’s pretend to explore all our options honestly, given how badly things have gone” to the phase of “let’s do everything we can to change the game with reckless expansion of the war that we now realize we’ve pretty much lost, so we don’t have to admit that we’ve lost it.”
This phase is deeply and painfully familiar to those of us who remember the arc of Vietnam. Richard Nixon and his henchman Henry Kissinger arrived at it in 1970, when they decided to invade Cambodia. It was a slap in the face to Americans who’d elected Nixon because he promised he had a secret plan to end the war; instead of peace, they got escalation. Protests erupted across the United States; during them, unarmed American students were fired upon and died at Kent State.
Today’s talk of a “surge” now looks like only a small part of a larger picture that involves expanding American operations in the Middle East with moves against Syria and Iran. As William Arkin puts it in the Washington Post, “Clearly the Vice President’s office and the hardliners scored a major victory.” Or, as Josh Marshall puts it, “The Veep’s office and the nutjobs are still running the show.”
With Kissinger whispering in the president’s ear today, I guess we shouldn’t be surprised at this nightmarish historical loop. The theory, then and now, was that the problems inside the country (South Vietnam or Iraq) stemmed from the influx of supplies and troublemakers from neighboring countries; cut those supply lines and the insurgency will dry up. It was a disastrous failure in Southeast Asia, and there isn’t the slightest indication that it will be any different in Iraq.
Things look like they’ll get a lot worse before they get any better. And this administration seems to be positively eager to wash its hands in more blood.
Lulu, the innovator in online self-publishing of printed materials, has an amusing little tool up at their site: you plug in a title and it tells you that title’s likelihood of becoming a bestseller. Is it an algorithm? Is it magic? Or is it just a silly online trick? I don’t know, but here’s what I learned about my prospects of bestseller-dom — far better than I’d expect for a book about software development and programming!
While we’re goofing around, here is a very brief clip recorded by celeb videoblogger Amanda Congdon. After she left the Rocketboom gig but before she graduated to the major leagues, she stopped in to hang out with Ed Cone, who had my book on his desk because he was preparing to interview me. Ms. Congdon picked up the book, and here’s what she had to say. (The full-length original clip that contains this excerpt is here. YouTube embedding seems funky in some browsers so: here’s the direct link to the clip on the YouTube site.)
[tags]dreaming in code, amanda congdon, lulu, ed cone[/tags]
Tonight President Bush will tell us about his plan for a “New Way Forward” in Iraq.
This White House is not well-versed in history, so I don’t suppose the people who fashioned the slogan thought much about its similarity to the verbiage the totalitarian leaders of the 20th century hung over their policies. 
But the choice is in keeping with Bush administration iconographic bombast of the past: check out this photo from November, 2005, when Bush was touting one of the “New Way Forward”‘s many predecessor blueprints for success in Iraq.
Whether you call it a “New Way Forward” or a “Great Leap Forward,” the idea that sending 20,000 more troops to Iraq is going to transform the reality of U.S. defeat reflects the self-delusion of a dictatorial mind. Bush’s approach to Iraq is as out of touch with reality as the command-economy follies of Joseph Stalin or Mao Zedong: the resemblance lies in the determination to force a theory down reality’s throat no matter what the cost in human suffering or damage to national interest.
The particular peculiarity of Bush’s fiasco is that he is wrecking the American military, and prolonging a doomed fight, on behalf of the abstract notion of Iraq’s in potentio democracy — while flagrantly and petulantly ignoring the thunderous outcome of America’s own democratic process last November, which delivered a clear verdict on Bush’s war. (And that verdict was not “Escalate now!”)
No matter. Our Maximum Leader knows better than his generals and better than the people. Tonight he will address us from the royal balcony. A few toadies will applaud, but the rest of us will be counting the days till we can throw him out of office.
[tags]president bush, iraq, new way forward, totalitarianism[/tags]