This is the thirteenth edition of Code Reads, a series of discussions of some of the central essays, documents and texts in the history of software. You can go straight to the comments and post something if you like. Here’s the full Code Reads archive.
This month’s paper, Daniel Berry’s “The Inevitable Pain of Software Development, Including of Extreme Programming, Caused by Requirements Volatility,” is a sort of update and latter-day restatement of Frederick Brooks’s classic “No Silver Bullet” argument — that the traits of software development work that make it difficult are inherent in the enterprise and extremely unlikely to be vanquished by some breakthrough innovation.
Berry is, as he admits, not the first to locate the source of software’s essential difficulty in “requirements volatility” — unpredictable fluctuations in the list of things that the software being built is expected to be able to do (including variations in user behavior scenarios, data types and all the other factors that a working piece of software must take into account). Read any development manual, listen in on any software team’s gripe session and you will hear curses directed at “changing requirements.”
Every new approach to improving the software development process includes a proposed method for taming this beast. These methods all fail, Berry maintains, leaving software development just as much of a “painful” exercise as it was before their application.
In each case, Berry locates this failure in some aspect of or practice dictated by a particular method that programmers find to be too much of a pain to actually perform.
Every time a new method that is intended to be a silver bullet is introduced, it does make many part of the accidents easier. However, as soon as the method needs to deal with the essence or something affecting or affected by the essence, suddenly one part of the method becomes painful, distasteful, and difficult, so much so that this part of the method gets postponed, avoided and skipped….
Each method, if followed religiously, works… However, each method has a catch, a fatal flaw, at least one step that is a real pain to do, that people put off. People put off this painful step in their haste to get the software done and shipped out or to do more interesting things, like write more new code.
So that, for instance, the method of “requirements engineering” (exhaustively “anticipate all possible requirements and contingencies” before coding) offers many benefits, but “people seem to find haggling over requirements a royal pain.” Also, it demands that “people discover requirements by clairvoyance rather than by prototyping.”
Similarly, Extreme Programming (XP) depends on programmers writing test cases first. That’s a step that in itself seems to be painful for many developers. When requirements change, XP calls for frequent refactoring of existing code. “Refactoring itself is painful,” Berry notes. “Furthermore, it may mean throwing out perfectly good code whose only fault is that it no longer matches the architecture, something that is very painful to the authors of the code that is changed. Consequently, in the rush to get the next release out on time or early, refactoring is postponed and postponed, frequently to the point that it gets harder and harder.”
Berry goes right down the list and confirms that the pain he diagnoses is a condition universal in the field.
The situation with software engineering methods is not unlike that stubborn chest of drawers in the old slapstick movies; a shlimazel pushes in one drawer and out pops another one, usually right smack dab on the poor shlimazel’s knees or shins. If you find a new method that eliminates an old method’s pain, the new method will be found to have its own source of pain.
Berry’s paper concludes with an alternative version of the principle that in Dreaming in Code I dubbed, tongue-in-cheek, Rosenberg’s Law:
To the extent that we can know a domain so well that production of software for it becomes almost rote, as for compiler production these days, we can go the engineering route for that domain, to make building software for it as systematic as building a bridge or a building. However, for any new problem, where the excitement of innovation is, there is no hope of avoiding relentless change as we learn about the domain, the need for artistry, and the pain.
Berry writes about the creation of software as much from the vantage of psychology as from that of engineering, and that gives his observations a dimension of bracing realism. In “The Inevitable Pain of Software Development” I found a willingness to examine the actual behavior of working programmers that’s rare in the software-methodology literature.
Too many authors are all too eager to pronounce what developers should do without considering the odds that any particular developer actually will do these things. Berry is a realist, and he keeps asking us to consider the cascade of consequences that flows from each method’s weak spots.
His case against “pain” seems not to be a naive attitude of trying to take a process that’s fundamentally difficult and somehow conjure the hardness right out of it. Instead, he asks us to note carefully the location of the “pain points” in any particular approach to software creation — because, given human nature, these are its most likely points of failure.
[tags]code reads, software development, software methodologies, daniel berry[/tags]
Post Revisions:
There are no revisions for this post.
Maybe part of the answer is to make the painful parts more fun? Thanks to improvements in tools, writing tests first and refactoring are much less painful than they used to be, at least in some environments.
I recently wrote an article debunking the No Silver Bullet paper. It’s long so be warned.
http://my.opera.com/Vorlath/blog/2007/10/28/no-silver-bullet-debunked
More important than that is how Berry misquoted the NSB paper. My reply talks about that here.
http://my.opera.com/Vorlath/blog/2007/11/01/code-reads-13
Abstract: My biggest complaint is that Berry doesn’t realise that most of what he says applies to all fields. Like what Scott quoted above:
“However, for any new problem, where the excitement of innovation is, there is no hope of avoiding relentless change as we learn about the domain, the need for artistry, and the pain.”
These things are not what makes software different. They apply everywhere. His paper is like this all over the place. Sorry, but he completely misses the boat on this one.
Jack W. Reeves’ essay, ‘What Is Software Design?’ suggests that all programming is a design activity. Design is about creating a solution, and trading off between conflicting goals and constraints. Understanding those trade-offs, and what they mean, takes vision and clarity, and creating something new takes courage. A software methodology is supposed to ease the job of making software, to remove the pains from the process, but I don’t know if you can remove the pain from design and creation.
“There cannot be any significant change in programming until someone figures out how to deal, with a lot less pain, with the relentless change of requirements and all of its ripple effects. Perhaps, we have to accept that development is an art and that no amount of systematization will make it less so.”
We always ask that question: is software engineering, or art? It’s practical, like engineering, but requires creativity and flexibility, like art. Maybe design is mid-way between engineering and art, and software is all design. Maybe we live with the pains.
Thanks for the early Dijkstra Code Reads…they were my introduction to his writings. I’d like to suggest ‘What is Software Design?’, I think it’d make a great Code Reads.
http://www.developerdotstar.com/mag/articles/reeves_design_main.html
My only real problem with this paper is that it doesn’t provide enough distinctions between the kinds of pain points. Some pain points exist because you are trying to do something fundamentally impossible (trying to capture all of the requirements in advance). Other pain points are more because of human nature (letting smaller refactorings go until you have to do a large painful one). I’m sure the readers here could come up with their own classifications. Not all pain points are created equal and I’d rather have the some types of pain over other types of pain.
I wasn’t too impressed with the paper. To the extent it is saying that no way of programming is perfect, big whoop; this doesn’t distinguish programming from anything else in the world. Pointing out the flaws in different methods is not so banal, but to the extent that it leaves the impression that all flaws are equal (which is the sense I get from the Conclusion), I disagree. (I also agree with John Dougan’s comment that it’s useful to, for example, distinguish pain points that arise from fundamental impossibilities than pain points that arise from other areas.)
The one bit in the paper that’s not so vacuous is the claim that the pain points all come from requirements issues. This is interesting, and perhaps computer science curricula should spend more time talking about that and less time talking about, say, algorithms, but it’s also not news: the methodologies mentioned in the paper generally try to attack that issue head-on, after all.