I had a root oops at work the other day. Nothing that can't be recovered; it was an unintended consequence of a feature of pbuilder that our build process uses to get at a bunch of "static" data that gets incorporated into the product. It can thus all be recovered, if nothing else from the finished builds. What was particularly annoying was that it occurred about fifteen minutes before I was planning to leave work and not come back for a week of badly needed decompression. (and turkey. Mmmm, turkey.)
The failure mode was an interesting example of what happens when you add shiny new semantics to a well established system. pbuilder is a program that builds debian packages with their specified build dependencies in a chroot environment. At various points it will clean this up. The new feature of pbuilder (which is enormously useful to us, because it means we can minimize the number of times we copy a few gigabytes of data at build time) that eventually bit us was a flag to use a linux bind mount to cause our datasets to "appear" in the build environment. Of course, when the time came to run the cleanup utility on a build that had been interrupted and thus not unmounted the directory structure, the important stuff got removed along with the trash.
Now, of course, the people who implemented this feature thought of this problem, and did something slightly clever involving the -xdev option to find to attempt to prevent the removal from wandering into bind-mounted territory. However, this appears to work by checking the st_dev member returned by the stat system call, which gives you the device number of the filesystem that a file is on... except of course, that we're building on the same filesystem that the datasets are mounted from, and the numbers are preserved across bind mounts.
oops.
Part of the reason I feel the need to write about this, interestingly enough, was my reaction to the cleanup, which is to say, reconstructing enough of this stuff so our builds could go through. I had a powerful urge to fix what I (however innocently) broke, and an equally powerful urge to lie in bed and stare at the ceiling for a week. Result: It took me two days to get around to doing two hours (or so) worth of work.
It was thinking about this which led me to the current name of this 'blog. What really struck me though, was the pure irony of the situation: my procrastination held off for a couple of days my ability to nothing without guilt! The work I was holding off wasn't particularly onerous, and even afforded a couple of opportunities to be slightly clever (since I was dealing with debian packages, all the relevant files had already been md5sum'd, and I could take advantage of that.)
I'm sure there's some deep insight into the nature of procrastination and it's relationship to actually getting things done, but I haven't gotten around to figuring it yet. When I do, you'll be the first to know.
