Software glitch leads to Dow conundrum

I was sitting in a long news meeting this morning, laptop in front of me, checking every now and then to see how bad a drubbing the stock market was taking. One minute around noon, West Coast time, I saw that the Dow was down around 250; a few minutes later, somehow, it was down 500. I thought, “Whoa, was there another terrorist attack? Did Alan Greenspan say something? What happened?”

It turns out that what happened was some as yet undefined software problem. As this AP story describes it, the New York Stock Exchange’s systems were falling steadily farther behind all day — in other words, the actual drop in the market was already worse than it was being reported when we thought the Dow was down 250. When the market’s managers realized what was going on, they flipped a backup into place, and suddenly, the backlog cleared — leading to that huge plunge at 3 pm Eastern time.

What’s interesting to me if you look at that chart is, once the drop became known to the market — once the backup system was in place and accurately reporting the deeper plummet — the market actually bounced back to where it thought it had been, even though that wasn’t really where it was. I’m not enough of a stock geek to fully understand this, but it’s fascinating, on some level of paradoxical reasoning.

Whoever said markets were perfect information systems?

UPDATE: Based on Wednesday AM coverage it sounds like the problem was specifically with Dow Jones’ systems, not the general stock exchange systems.
[tags]stock market, dow, software, bugs[/tags]


 

Berkeley Cybersalon

This coming Sunday, March 4, come on down to the Hillside Club, where, from 5 to 7 p.m., at the Berkeley Cybersalon, we’ll be talking about why software is (so often) so hard.

For this event, practically in my backyard and as part of what must be the longest-running technology salon around (I started going over a decade ago), I didn’t want to just give another spiel about Dreaming in Code.

Instead, it’s a panel discussion, with Eric Allman of Sendmail (program and company), Chad Dickerson of Yahoo’s Developer Network, and Lisa Dusseault of CommerceNet (and formerly of the Open Source Applications Foundation).

It was originally scheduled for yesterday, 2/25, but we realized that counter-programming against Oscar Night was foolhardy — even for an event as geekish as this.


 

Stealth fighter trips over dateline

Back when my job as Salon managing editor involved overseeing our daily production, I noticed that, every spring and fall, almost without fail, our publishing system would experience a glitch of some kind on the weekend that the clocks got moved forward or back — nothing serious, mind you, but enough to throw a wrench in the works of our site updates. It wasn’t a single bug, but some sequence of related bugs, so we’d fix one and then six months later something else would happen. Eventually we got in the habit of just making sure that one of the developers kept a close eye on things when that weekend rolled around. It was prudent.

I thought of that as I read these accounts that are filtering out about the F-22 Raptors that, the speculation is, lost their bearings when they crossed the International Date Line. (Further speculation is that this was somehow connected to a software patch/upgrade related to the coming change in the date of Daylight Savings Time onset, but that’s harder to source.) The planes, en route to Japan, limped back to Hawaii instead

The F-22 costs $125 million or so and its operating system is written in 1.5 million lines of Ada code. It appears that, for all its “stealth” prowess and advanced weaponry, its soft underbelly may lie in the realm of the abstract.

It seems that this is one of the unexpected consequences of living in a world operated by software: new danger zones lie where human abstractions — borders, measurements, languages — change or conflict or fail to behave as expected. Clocks and calendars and maps are no longer just assists for human understanding; they are symbols at the heart of systems upon whose performance lives depend. I suppose this started with the first railway schedule, but with the dateline-addled F-22 it has entered a whole new realm of disconcert.
[tags]f-22, bugs, software[/tags]


 

Dreaming on the Well

The Well is where I first experienced the addictive power of online conversation, almost two decades ago. So I couldn’t be happier that the Well is now hosting an ongoing Q&A with me about Dreaming in Code. Since this is taking place in the Well’s Inkwell forum it’s readable to all (not just Well members), and open to questions from non-members, too (via e-mail — see the discussion for details).

Check it out — Christian Crumlish is hosting the interview, and we’ve already talked about how Dreaming in Code is (or isn’t) like Moby-Dick, and ways in which the Chandler project is (and isn’t) like the Iraq war.


 

Beginning to see the light

I don’t read the East Bay Express as much as I used to since it lost its old-style Berkeley individuality and got swallowed up by the big alt-weekly chain that is now known as Village Voice Media. But I stumbled on this very funny interview in it today, in which a hapless music writer quizzes Lou Reed about his soundtrack for a new Tai Chi DVD.

I was in the audience last fall at the Web 2.0 conference when Reed’s iceberg-like self-possession collided with the tanker of the Web industry elite’s smug self-regard — a fiasco set up through the offices of then-AOL honcho Jonathan Miller, who explained that he and Reed met because they study with the same Tai Chi master. So Reed’s martial-arts enthusiasm didn’t come as the surprise to me that it seems to have for the Express writer.

What do you say to the people who can’t reconcile your classic Velvet Underground druggy image with this healthy New Age one?

“That was forty years ago!” he implores. “This is 2006! 2007! My God! I can’t worry about things like that. If I did, I wouldn’t do anything! I can’t live in 1967 for people. That’s crazy. I have a broader palette. Everything I do, I’ve always tried to do the best that I could as honestly as I could from wherever space I was viewing things at the time. I can’t satisfy everyone, and I’m not trying to.”

[tags]Lou Reed, martial arts, tai chi, dvds, east bay express[/tags]


 

Teraflop software?

Of the many “laws” I encountered in the course of writing Dreaming in Code, I think Wirth’s law (by the software pioneer Niklaus Wirth) is my favorite: Software gets slower faster than hardware gets faster.

Here is a contemporary instance. All right, it’s not exactly parallel; but it’s an example of the very common situation we encounter as hardware improves exponentially while software improves on only a linear basis.

This is from John Markoff’s recent piece about Intel’s demo of a prototype of a new chip-making technique that packs 80 processor cores on a single chip (the “Teraflop Chip”):

The shift toward systems with hundreds or even thousands of computing cores is both an opportunity and a potential crisis, computer scientists said, because no one has proved how to program such chips for many applications.

“If we can figure out how to program thousands of cores on a chip, the future looks rosy,” said David A. Patterson, a University of California, Berkeley computer scientist who is a co-author of one of the standard textbooks on microprocessor design. “If we can’t figure it out, then things look dark.”

Mr. Patterson is one of a group of Berkeley computer scientists who recently issued a challenge to the chip industry, demanding that companies like Intel begin designing processors with thousands of cores per chip.

In a white paper published last December, the scientists said that without a software breakthrough to take advantage of hundreds of cores, the industry, which is now pursuing a more incremental approach of increasing the number of cores on a computer chip, is likely to hit a wall of diminishing returns — where adding more cores does not offer a significant increase in performance.

I wrote about this “multicore competency” issue a couple of years ago. Looks like it’s not going away.

UPDATE: Corrected to fix a (happily) mistaken suggestion that Wirth had passed away.


 

Damien Cave in Baghdad

I worked with Damien Cave for years at Salon, where he did great work for our technology section. Several years ago he decamped for New York and ended up at the New York Times. He’s now reporting from Baghdad.

I’ve been catching up on reading some old papers that I neglected during the frenzy of my book launch. This morning I read his two-week-old piece “‘Man Down’: When One Bullet Alters Everything.” It’s a remarkable bit of eyewitness reporting from Haifa Street in central Baghdad, just outside the Green Zone, where Cave accompanied an American platoon on a sweep. It’s about the difficult choices facing U.S. forces trying to coordinate with Iraqis who are ostensibly leading the mission. It’s about the terrors and horrors facing Iraqi residents of the torn city. But mostly it’s about the choices and emotions encountered by the young American soldiers when one of their sergeants is struck down by a sniper. It is entirely sympathetic to the embattled Americans at the same time it illuminates how futile their effort is.

Perhaps the next time President Bush calls a press conference, the White House correspondents could collectively agree to stop wasting their time asking questions of a leader who will not give truthful answers — and instead, each read a sentence from this article, telling the president a story about what his mistakes have wrought.