I had a root oops at work the other day. Nothing that can't be
recovered; it was an unintended consequence of a feature of
pbuilder that our build process uses to get at a bunch
of "static" data that gets incorporated into the product. It can
thus all be recovered, if nothing else from the finished builds.
What was particularly annoying was that it occurred about fifteen
minutes before I was planning to leave work and not come back for a
week of badly needed decompression. (and
turkey. Mmmm, turkey.)
The failure mode was an interesting example of what happens when you
add shiny new semantics to a well established system.
pbuilder
is a program that builds debian packages with their specified build
dependencies in a
chroot
environment. At various points it will clean this up. The new
feature of pbuilder (which is enormously useful to us,
because it means we can minimize the number of times we copy a few
gigabytes of data at build time) that eventually bit us was a flag to
use a linux bind mount to cause our datasets to "appear" in the build
environment. Of course, when the time came to run the cleanup
utility on a build that had been interrupted and thus not unmounted
the directory structure, the important stuff got removed along with
the trash.
Now, of course, the people who
implemented this feature
thought of this problem, and did something slightly clever
involving the -xdev option to find to
attempt to prevent the removal from wandering into bind-mounted
territory. However, this appears to work by checking the
st_dev member returned by the stat system
call, which gives you the device number of the filesystem that a file
is on... except of course, that we're building on the same filesystem
that the datasets are mounted from, and the numbers are preserved
across bind mounts.
oops.
Part of the reason I feel the need to write about this, interestingly
enough, was my reaction to the cleanup, which is to say,
reconstructing enough of this stuff so our builds could go through.
I had a powerful urge to fix what I (however innocently) broke, and
an equally powerful urge to lie in bed and stare at the ceiling for a
week. Result: It took me two days to get around to doing two hours
(or so) worth of work.
It was thinking about this which led me to the current name of this
'blog. What really struck me though, was the pure irony of the
situation: my procrastination held off for a couple of days my
ability to nothing without guilt! The work I was holding off wasn't
particularly onerous, and even afforded a couple of opportunities to
be slightly clever (since I was dealing with debian packages, all the
relevant files had already been md5sum'd, and I could
take advantage of that.)
I'm sure there's some deep insight into the nature of
procrastination
and it's relationship to
actually getting things done,
but I haven't gotten around to figuring it yet. When I do,
you'll be the first to know.