Apropos of my previous post on YouTube and MySpace, today I read this fascinating case-study from Baseline magazine about the saga of MySpace’s understandably overtaxed systems.
MySpace’s exploding popularity has basically forced its infrastructure through a continuous cycle of upgrades, refactorings and revampings. Its managers have never had the luxury of sitting back and calmly planning upgrades; they’ve had to perform their engine surgeries on a careening vehicle.
This is what Web 2.0 is like from the back end, and it ain’t pretty. Outside of the real masters of this stuff — the Googles and Yahoos that know how to deploy, manage and maintain vast online services — it’s a big mess. This is another little-understood dynamic of the Web 2.0 startup world: There are financial reasons a successful small service might want to be acquired, but there are even more pressing operational reasons. And the more success a service finds, the more likely it’s going to risk systems flameout.
It’s not at all clear from the Baseline piece that MySpace has yet achieved a level of stability that a more mature company might desire. MySpace, of course, was acquired not by a technology company but by a media outfit, so — unlike other popular companies that were acquired by Yahoo or Google — they’re still somewhat on their own.
The Baseline piece offers two other fascinating tidbits. In the first, a normal phenomenon for a successful site — massive surges of traffic — was interpreted as a bug by the Microsoft server platform MySpace uses:
Last summer, MySpace’s Windows 2003 servers shut down unexpectedly on multiple occasions. The culprit turned out to be a built-in feature of the operating system designed to prevent distributed denial of service attacksâ€”a hacker tactic in which a Web site is subjected to so many connection requests from so many client computers that it crashes. MySpace is subject to those attacks just like many other top Web sites, but it defends against them at the network level rather than relying on this feature of Windowsâ€”which in this case was being triggered by hordes of legitimate connections from MySpace users.
“We were scratching our heads for about a month trying to figure out why our Windows 2003 servers kept shutting themselves off,” Benedetto says. Finally, with help from Microsoft, his team figured out how to tell the server to “ignore distributed denial of service; this is friendly fire.”
Second, it seems that MySpace didn’t actually originally intend to allow the level of customization that has made it so popular; its engineers just never got around to filtering out the user-customized formatting.
That feature was really “kind of a mistake,” says Duc Chau, one of the social networking site’s original developers. In other words, he neglected to write a routine that would strip Web coding tags from user postings– standard feature on most Web sites that allow user contributions.
The Web site’s managers belatedly debated whether to continue allowing users to post code “because it was making the page load slow, making some pages look ugly, and exposing security holes,” recalls Jason Feffer, former MySpace vice president of operations. “Ultimately we said, users come first, and this is what they want. We decided to allow the users to do what they wanted to do, and we would deal with the headaches.”
Here we have the state of Web development today: Your site’s massive success gets treated as a bug by your server; and the feature your users love best is something your programmers forgot to block.
[tags]baseline, myspace, web 2.0, software development[/tags]
There are no revisions for this post.