This is the thirteenth edition of Code Reads, a series of discussions of some of the central essays, documents and texts in the history of software. You can go straight to the comments and post something if you like. Here’s the full Code Reads archive.

This month’s paper, Daniel Berry’s “The Inevitable Pain of Software Development, Including of Extreme Programming, Caused by Requirements Volatility,” is a sort of update and latter-day restatement of Frederick Brooks’s classic “No Silver Bullet” argument — that the traits of software development work that make it difficult are inherent in the enterprise and extremely unlikely to be vanquished by some breakthrough innovation.

Berry is, as he admits, not the first to locate the source of software’s essential difficulty in “requirements volatility” — unpredictable fluctuations in the list of things that the software being built is expected to be able to do (including variations in user behavior scenarios, data types and all the other factors that a working piece of software must take into account). Read any development manual, listen in on any software team’s gripe session and you will hear curses directed at “changing requirements.”

Every new approach to improving the software development process includes a proposed method for taming this beast. These methods all fail, Berry maintains, leaving software development just as much of a “painful” exercise as it was before their application.

In each case, Berry locates this failure in some aspect of or practice dictated by a particular method that programmers find to be too much of a pain to actually perform.

Every time a new method that is intended to be a silver bullet is introduced, it does make many part of the accidents easier. However, as soon as the method needs to deal with the essence or something affecting or affected by the essence, suddenly one part of the method becomes painful, distasteful, and difficult, so much so that this part of the method gets postponed, avoided and skipped….

Each method, if followed religiously, works… However, each method has a catch, a fatal flaw, at least one step that is a real pain to do, that people put off. People put off this painful step in their haste to get the software done and shipped out or to do more interesting things, like write more new code.

So that, for instance, the method of “requirements engineering” (exhaustively “anticipate all possible requirements and contingencies” before coding) offers many benefits, but “people seem to find haggling over requirements a royal pain.” Also, it demands that “people discover requirements by clairvoyance rather than by prototyping.”

Similarly, Extreme Programming (XP) depends on programmers writing test cases first. That’s a step that in itself seems to be painful for many developers. When requirements change, XP calls for frequent refactoring of existing code. “Refactoring itself is painful,” Berry notes. “Furthermore, it may mean throwing out perfectly good code whose only fault is that it no longer matches the architecture, something that is very painful to the authors of the code that is changed. Consequently, in the rush to get the next release out on time or early, refactoring is postponed and postponed, frequently to the point that it gets harder and harder.”

Berry goes right down the list and confirms that the pain he diagnoses is a condition universal in the field.

The situation with software engineering methods is not unlike that stubborn chest of drawers in the old slapstick movies; a shlimazel pushes in one drawer and out pops another one, usually right smack dab on the poor shlimazel’s knees or shins. If you find a new method that eliminates an old method’s pain, the new method will be found to have its own source of pain.

Berry’s paper concludes with an alternative version of the principle that in Dreaming in Code I dubbed, tongue-in-cheek, Rosenberg’s Law:

To the extent that we can know a domain so well that production of software for it becomes almost rote, as for compiler production these days, we can go the engineering route for that domain, to make building software for it as systematic as building a bridge or a building. However, for any new problem, where the excitement of innovation is, there is no hope of avoiding relentless change as we learn about the domain, the need for artistry, and the pain.

Berry writes about the creation of software as much from the vantage of psychology as from that of engineering, and that gives his observations a dimension of bracing realism. In “The Inevitable Pain of Software Development” I found a willingness to examine the actual behavior of working programmers that’s rare in the software-methodology literature.

Too many authors are all too eager to pronounce what developers should do without considering the odds that any particular developer actually will do these things. Berry is a realist, and he keeps asking us to consider the cascade of consequences that flows from each method’s weak spots.

His case against “pain” seems not to be a naive attitude of trying to take a process that’s fundamentally difficult and somehow conjure the hardness right out of it. Instead, he asks us to note carefully the location of the “pain points” in any particular approach to software creation — because, given human nature, these are its most likely points of failure.

For as long as I can remember, people have gotten Salon and Slate confused. Maybe it was that they are somewhat similar sites — at least if you compare both to, say, EBay or Flickr — and both have five-letter names beginning with “S.” I don’t know why. But here it is, 12 years after I joined Salon’s merry startup crew and several months after I finally left the place, and I’m still finding myself referred to as “Slate’s Scott Rosenberg” (this by tech blogger Robert Scoble a little while back, since corrected) or “Slate co-founder Scott Rosenberg” (this by Cyberjournalist.net, published by the Online News Association, which, you know, really ought to know better).

I’m always grateful for the links, but before Google starts to associate my name too closely with that of a publication I’ve never been connected to, I think it’s time to stop this train. So, once and for all, let me provide a simple disambiguation guide.

• Salon is a fine publication that began publishing in November 1995. Slate is another fine publication that started about eight months later. Both were pioneers of the “webzine” format.
• Salon has always been an independent company. Slate was funded first by Microsoft and is now owned by the Washington Post.
• Salon was edited for many years by David Talbot and is now edited by Joan Walsh. Slate was edited for many years by Michael Kinsley and is now edited by Jacob Weisberg.
• Salon has plenty of great commentary but prides itself on its independent investigative reporting. Slate occasionally breaks stories but seems more editorially centered on digest-style features like “Today’s Papers” and explanatory commentary.

I wrote a huge number of articles for Salon over the years and also edited many more and helped run the publication and manage its site for many years. I have never, that I can recall, written for Slate.

There. I feel better now.

News organizations spend an extraordinary amount of time and effort deciding what “leads” — what goes on the front page; what goes in the newscast at the top of the hour; what’s important. This is how professional news organizations deploy the minds and time of some of their best-paid and most experienced employees: They sit down at daily meetings and argue this stuff out; sometimes they agonize over it.

In the era of scarce column-inches and broadcast time this made a lot of sense. But that era is fading. With the Web reshuffling how the most avid users of news get their information, editors’ roles are changing — not vanishing, but definitely being challenged.

These thoughts are occasioned by Dave Winer’s new experiments remixing the New York Times. A while back he offered us the Times River — a simple reverse-chronological list of “head-and-deck” links from the newspaper’s RSS feed that is perfect for scanning on mobile devices or just checking in to see what the latest Times stories are. In his latest rethinking of the flow of Times headlines, Winer has built an outline-style interface to the same set of headlines, built around the Times’ own keywords.

These pages are notable for their simplicity. There are no distracting ads, no complex navigational tools, no typographical elegance or design flourishes. It’s just the text and you. A part of me looks at this and thinks, “How crude.” Another part of me looks at it and sees the same spare utility as the original Google home page — and wonders if, a handful of years from now, I’m going to prefer keeping up with my Times this way over continuing to kill trees with my lifelong (but now imperiled) newspaper consumption habit.

Years ago, during the dotcom mania, as Salon’s home page got more and more festooned with stuff that Salon was playing around with to try to increase revenue, a software developer did something similar with our news flow — he “screen-scraped” our headlines and presented them in an ultra-simple list form. (His script still appears to be running but it no longer works properly — Salon’s home page has been redesigned a bunch of times since then.) This was a kind of proto-Salon River. Use of it never spread beyond a tiny handful of geeks. If it had — if hordes of Salon users essentially defected and said they preferred that version of our home page to our own — it would have presented us with a business dilemma.

But I think the real resistance to this new vision for news delivery will be less on the business end (business tends to extract some kind of value anywhere large numbers of people can be congregated) than in the newsroom itself. Because the whole “river of news” approach, like the “newest posts on top” design of all blogs, takes a big bite out of the editor’s job. The reader who looks at Times River and says “this is how I want my news” is a reader who is saying to the Times editors, “Don’t waste all that time figuring out what to tell me you think is important.”

As Winer put it, “They [editors] have a very powerful internal gravity driven by a philosophy that their job is to arrange our thinking.”

I think that there are still plenty of readers who like what editorial judgment adds to the arrangement of the news. Of course, they don’t always agree with it, and many like to argue with it. But they want their quick scans of the news to be ordered by something besides chronology, so they choose a publication to make a deal with, saying, in effect, “I’m giving you my attention and you tell me what you think is important. If I disagree often enough I’ll move on, but in the meantime, tell me what you think matters.”

The real question over the next decade or so will be, how many of those readers are there? Is it the vast majority — which is what most editors believe? Or is it a shrinking tribe of news consumers who grew up under the old dispensation?

Although most professional editors will immediately dismiss the scenario, I think it’s quite possible that the “editors’ cut” of the news will dwindle in importance until we hit some threshold where the majority of users decide they don’t want their thinking “arranged” for them.

At that point, the “river” will roll right across the front page. And some editors may need to find other outlets for their talents.

Thoughts occasioned by the confirmation hearings for Michael Mukasey to become the next U.S. attorney general:

Apparently there have been some interesting changes in the whole notion of the constitutional balance of powers since I studied such matters. As most of us learned at some point in our schooling, there are three branches of government established in the U.S. constitution. Congress passes the laws, as defined by Article I. the president executes the laws and handles a bunch of other stuff as defined by Article II. And the supreme court interprets the laws, as defined by Article III. Yes, I’m aware that the whole judicial review thing evolved over time and wasn’t grounded that explicitly in the constitution’s language. On the other hand, it’s served us pretty well for over 200 years, and it has been a keystone of the checks-and-balances system that has proven so resilient over those centuries.

Under the Bush administration we have seen two fundamental assaults on this system. One, embodied in the idea of “signing statements” that the president makes when he signs congressional legislation, proposes that the president is himself equal to the supreme court in his power to review the constitutionality of legislation. According to this notion, the chief executive has the unilateral authority to say, “I don’t think this or that part of this law is constitutional, so I will reserve the right not to enforce or obey it.” He’s not saying, “I think this is an unconstitutional law, so I’m going to challenge it before the supreme court.” He’s saying, “I think this is an unconstitutional law, so I’m going to ignore it.”

The second assault centers on the notion of the “unitary executive.” This theory proposes that the entire executive branch is a sort of “off limits” zone for congress. To the extent that a congressional law or rule constrains the president’s authority over the executive branch in some way, he is free to ignore it, because it’s unconstitutional — and, right, he gets to ignore laws he believes are unconstitutional.

Put these two notions together and you have, I think it’s fair to say, a whole new game in the federal government town. Forget checks and balances, or “government by laws and not men.” Say hello to a new world in which the unitary executive claims supremacy over both the congress (whose laws he can ignore at will and whose powers cannot reach into the executive branch) and the supreme court (whose role as reviewer of the constitutionality of legislation the president is now quite able to assume himself).

Now, it’s true that, as we say here on the Internets, I am not a lawyer. But I’m a citizen. And I have to report that these new ideas about the constitution make me a little concerned for the future of our political system.

I know that we have a vice president who got, let’s just say, peeved that the congress reined in a criminal president back when he was a young man, and who has spent the rest of his life itching to redress that old grievance. But this isn’t a partisan matter. An autocratic view of the chief executive — which is what Bush’s lawyers have propounded, and Munkasey, for all his superior forthrightness compared with the henchman who preceded him, endorsed in his testimony today — is a time-bomb for both parties. Once precedents for unchecked authority are set, who is to say that a Democratic president might not avail himself (or herself) of them?

“Checks and balances” is a big fat cliche, but it’s also a foundation that has supported two centuries and more of American political stability. Long after the pathetic corruptions and petty inhumanities of the Bush administration have receded from view, we’re still going to be trying to patch together a constitution that the Bush/Cheney legal establishment has shredded.

I’ve been laying low this week completing a draft of a new book proposal. More on that as we get closer to the finish line. This is the first year I’ve not attended the Web 2.0 conference, but, you know, I need to focus — and I think I wasn’t that eager to hear Rupert Murdoch, anyway.

In the meantime, I’m happy to report that I have successfully managed to get Ecco Pro running on a Mac via Parallels. I actually achieved this goal a decade ago using Virtual PC, but boy was it slow! The Parallels set up, by contrast, is snappy and, so far, foolproof. Thanks to all of you who advised me on this dilemma. Very exciting. (The “coherence” mode of Parallels is remarkable — its puts the Windows taskbar and WinXP program windows on an equal footing on the Mac screen with the OSX stuff, turning your display into a sort of operating-system hermaphrodite.)

As I close in on my next book-project goal, I would also like to draw your attention to this quotation from William Gibson (in a Washington Post interview from last month), musing on the persistence of the book:

It’s the oldest and the first mass medium. And it’s the one that requires the most training to access. Novels, particularly, require serious cultural training. But it’s still the same thing — I make black marks on a white surface and someone else in another location looks at them and interprets them and sees a spaceship or whatever. It’s magic. It’s a magical thing. It’s very old magic, but it’s very thorough. The book is very well worked out, somewhat in the way that the wheel is very well worked out.

There’s an interesting dustup in the journalism world about the Q&A column in the Sunday New York Times magazine. A New York Press story about these interviews by Deborah Solomon included complaints from two of her subjects — one of them This American Life’s Ira Glass, himself no interviewing naif — that she misrepresented them and inserted questions in her own voice that she hadn’t actually asked them. Then on Sunday Times “public editor” Clark Hoyt devoted a whole column to the matter.

It always seemed hugely obvious to me that Solomon’s terse, one-page interviews were boiled-down and heavily edited. But I look at these things as an editor with some experience. What this controversy really reveals is the gulf between the reverence most newspaper reporters have for quotation marks and the relatively cavalier stance assumed by many magazine writers and editors. (Yes, of course these are gross generalizations, and the world has plenty of careless newspaper reporters and careful magazine journalists. But the patterns do exist, in my experience.) So it’s no wonder that the flashpoint for confusion here should be in the weekly magazine published by a daily newspaper. Times mag editor Gerald Marzorati told Hoyt, “This is an entertainment, not a newsmaker interview on ‘Meet the Press.’” But it’s also part of the New York Times, and that still carries a set of expectations about the reliability of everything between quote marks.

I got my start in journalism in newspapering, and what I learned was that anything between direct quotation marks ought to be a verbatim quote from your subject. If you took words out, you marked it with an ellipsis. If you were paraphrasing or otherwise changing words, you had to take the quote marks off — you then had an indirect quote.

When I started freelancing and experiencing the wide variety of editing standards at different publications I was appalled to discover that some significant portion of my editors were “fixing up” quotes in various ways. When I complained, they dismissed my objections. It seemed that they believed we had license to improve the statements of the subject for the benefit of the reader.

As with so many aspects of journalism, the rules here vary far more than just from publication to publication: there’s essentially a different set of rules for each journalist you meet.

Over the course of my career I came to the following set of practices: In news coverage of any kind, I stick to the verbatim quotation-mark reverence of my training. In my book, anything between quote marks represented words somebody said. In lengthy Q&A interviews where I’ve taped an interview, and where the purpose of the piece is to talk with a writer or artist about his or her work, I will take the liberty of tightening rambling answers and sharpening both questions and answers. I’ve done a lot of these and never heard an objection. If you’re helping interviewees explain their work or expose their ideas, they’re usually grateful for you to do a little editing, as long as it doesn’t alter the substance of their statements. But if you’re challenging them, it’s best to stick to the tape.

It looks like Solomon got into trouble because she frequently adopts a confrontational stance. That’s part of the appeal of her column, and there’s nothing wrong with it. But if you’re practicing “gotcha” journalism you can’t take liberties with the transcript. It’s inevitable that you’ll get called on it. And in today’s media environment, you’ll get called fast.

Yesterday afternoon I hopped over to Emeryville to hear Joel Spolsky talk. He’s on the road promoting the new, 6.0 version of Fog Creek Software’s bug-tracking product. I’d paid little attention to the evolution of this product — Salon’s team long ago chose the open-source Trac, OSAF used Bugzilla, and when I first looked over FogBugz ages ago it looked like a perfectly serviceable Windows-based bug-tracking tool, no more.

Well, in the intervening time, the thing has gone totally Web-based and AJAX-ified, and it’s pretty cool just on those terms. It’s also grown a wiki and become more of a full-product-lifecycle project management tool, with integration for stuff like customer service ticket management.

Still, what’s most interesting about the new FogBugz is what Spolsky and his team are calling “Evidence Based Scheduling” (or — because everything must have an acronym — EBS). Now, anyone who’s read Dreaming in Code knows that I devote considerable verbiage to the perennial problem software teams face in trying to estimate their schedules. This is in many ways the nub of the software problem, the gnarly irreducible core of the difficulty of making software.

With EBS, FogBugz keeps track of each individual developer’s estimates (i.e., guesses) for how long particular tasks are going to take, then compares those estimates with the actual time the task took to complete. Over time it develops a sense of how reliable a particular developer is, and how to compensate for that developer’s biases (i.e., “Ramona consistently guesses accurately except that things always take her 20 percent longer than she guesses”).

With this information in place — and yes, that’s right, to use this system the developers have to keep track of how much time they spend on each task — the software can turn around and provide managers with a graph of ship-date likelihoods. You can’t say for sure, “The product will ship by March 31,” but you could say, “We have a 70 percent likelihood of shipping y March 31,” and then you can fiddle with variables (like “Let’s only fix priority one bugs”) and test out different outcomes.

Spolsky explained how FogBugz uses a Monte Carlo simulation algorithm to calculate these charts. (He provided a cogent explanation that my brain has now partially scrambled, but I think it’s like running a large number of random test cases on the data to generate a probability curve.) In any case, while I’m sure many managers will be interested in the prospect of a reliable software-project estimation tool, what I find intriguing is the chance that any reasonably wide deployment of FogBugz might yield some really valuable field data on software schedules.

The sad truth is that there’s very little good data out there. As far as I understand it, the CHAOS report is all self-reported (i.e., CTOs filling out surveys). To the extent that users of FogBugz are working from the hosted service rather than on their own installations of the software, the product will gradually produce a fascinating data set on programmer productivity. If that’s the case, I hope Spolsky and his company will make the data available to researchers. Of course, you’d want all the individual info to be anonymized and so on.

As I said, all of this depends on developers actually inputting how they spend their time. They’ll resist, of course — time sheets are for lawyers! Spolsky said Fog Creek has tried to reduce the pain in several ways: The software makes it easy to enter the info, you don’t worry about short interruptions and “bio-breaks,” i.e., bathroom runs (hadn’t heard that term before!), you just try to track tasks at the hourly or daily level, and you chunk all big tasks down to two-day or smaller size pieces. Still, I imagine that if evidence-based scheduling doesn’t catch on, this will be its point of failure. Otherwise, it sounds pretty useful.

UPDATE: Rafe Colburn is starting to use FogBugz 6.0 and has more comments…

From the earliest days of the Web to the present, there’s been a fundamental split between people who get the value of “human-readable URLs” and people who don’t. A human-readable URL is a Web address that tells you a lot of useful information about the page it represents. For instance, Salon URLs always tell you the date an article was posted, the section of the site the article appeared in, and a few words describing the subject matter of the article. By comparison, the typical URL at, say, CNET, looks like this: http://reviews.cnet.com/4520-10895_7-6782817-1.html. It is, essentially, human-unreadable.

In the old days, writers and editors who actually knew and used HTML always appreciated a good human-readable URL; and typically, for the ugly gibberish URLs, we had to thank (some) software architects and (some) publication managers who’d never hand-coded a link themselves. At Salon, we editors knew we’d be typing (and proofing) a zillion of those URLs ourselves; we insisted on something we could work with. (Our developers “got it” too.)

The cause of human-readable URLs got a great shot in the arm when sites began trying to optimize themselves for Google, because Google gives a little extra weight to text hints in URLs. So a lot of sites (like the New York Times) that had a history of human-unfriendly page addresses began to do better.

Today, though, we’re taking a step backwards, or at least sideways, in the cause of human readability, thanks to the growing popularity of the “tinyurl.”

When the tinyurl first crossed my radar I understood it to be a convenient way to tame unmanageably long Web addresses. (The Tinyurl site focuses in particular on how long Web addresses break in email messages.)

That’s all fine. But the tinyurl giveth and the tinyurl taketh away. When you encode a Web address as a tinyurl you’re hiding its target. Normally, when I read an article on the Web that has a link, I’ll hover my cursor over the link to see where it points. Even on a site with human-unfriendly URLs like CNETs, at least I can see that the link points to CNET.

With a tinyurl, I know nothing about the link except what the author chose to say about it. I can’t tell if it’s a reference to an article I’ve already read. If I want to find out, I have no choice but to click.

My sense is that tinyurls have grown in popularity with the rise of Twitter (where the strict character limit of messages means you don’t want to fill up a whole message with an URL), as well as the growing use of mobile devices for Web-posting activities. These are perfectly understandable reasons. But still, each time I see a tinyurl I think, there goes another tiny piece of the Web’s transparency.

Some new music I’ve been enjoying from three artists/bands whose work I’ve been following since the mid or late ’80s:

The Mekons, “Natural”: Their last collection of new material, “OOOH,” was an exploration of religion and ritual; this new batch takes a turn to the pastoral. Rough-hewn even in this laid-back mode; mysteriously allusive as always; and worth heavy rotation as ever.

They Might Be Giants, “The Else”: I never fully warmed to their previous outing, “The Spine,” but “The Else” strikes me as a return to form. I’m enjoying it and — even though this is ostensibly a “grownup” album and not one of the band’s “children’s music” works — so are my seven-year-olds. Standout tracks: the patter-song “Bee of the Bird of the Moth” (is it an ode about genetic engineering? I don’t know, but I’ll remember the “Sleep of Reason Corporation”!) and “Contre Coup” (a song about phrenology, concussions and love), which introduced me to the obscure word “limerent.”

Black Francis, “Bluefinger”: Frank Black reverts to his old Pixies moniker for this new album, inspired (as the folks in the FrankBlack.net forum figured out and the official site confirms) by the life saga of Dutch glam rocker Herman Brood. Hard for me to see how all 11 tracks fit that template. But it’s easy for me to love the entire album, which marks a return to energetic form after Black’s previous duo of interesting but somewhat enervated discs recorded in Nashville. Standout songs: Beastie-Boys-style rave up “Threshold Apprehension,” in which Black shrieks and yowls like he hasn’t since Pixie days; “Lolita,” which sounds like one of the great numbers from the days of Frank Black and the Catholics (reminiscent of the Jonathan Richman tribute “The Man Who Was Too Loud”); the singalong “She Took All of the Money”; and “Angels Come to Comfort You,” which rises to the catchiest, sunniest chorus of death you can imagine.

I missed TMBG on their current tour — they swung through the Bay Area while I was out of town last week — but it seems that both the Mekons and Frank Black/Black Francis are performing over the next week at the Cafe du Nord. I expect to be there.