Retrospect .sparseimage Warning

Retrospect has a newish setting, which is turned on by default: “Don’t back up FileVault sparseimages.”

Beware this setting if you manually create/use sparse disk images. You probably want to turn it off.

Backgrounder:

FileVault doesn’t play well with Retrospect, which works by placing your home folder on an encrypted sparse disk image.

This puts Retrospect in a bind. First, incremental backup basically Goes Away for home folders stashed in a disk image — the image is just a large opaque bag of bytes to Retrospect. Any little change (probably even just mounting the disk image) is enough to flag the entire file as “changed”, increasing backup resource consumption (space+time).

Worse, Retrospect can’t guarantee a correct backup of the file behind a mounted writable disk image. Remember, a disk image is just a bag of blocks, which may or may not at any given time represent a consistent file system. Journalling helps limit damage on a structural basis, but it does squat for file data itself. (Futurist rant: we’re way overdue to drop the file system tyranny and embrace transactional persistent object soups.)

Geek aside: Interestingly, some versions of Windows have a low-level checkpointing feature, where you can “freeze” a file system in time and perform a backup of this guaranteed-consistent image. This “freezing” is nonblocking for the rest of the system — it’s essentially a dynamic read-only sandboxing. It’s something I desperately wish Mac OS X supported.

So it’s understandable why Dantz EMC added code explicitly to notice .sparseimage files and, by default, exclude them from backup. However, if also means you manually create and/or use sparse image files yourself, those aren’t backed up by default.

I noticed this option when it first appeared, but I didn’t realize how Retrospect’s code to detect FileVault files would also match my .sparseimage files. I incorrectly assumed that FileVault images would have their own extension or a unique type/creator code. I was very much mistaken.

(I don’t use FileVault at all. It scares the hell of out me.)

I only noticed now when I somehow accidently deleted a sparse image file. I have a lot of Other People’s Code on my portable drive, so I have a responsibility to keep it secure. I find encrypted sparse disk images are great for this task, so long as you’re careful not to store their passwords in your keychain (but that’s another rant).

Accidental deletion is usually no biggie since I backup nightly. Hmm, the file isn’t there in the latest snapshot. I go back another night. And another. And another. That sinking feeling starts kicking in. I finally go back to my very first snapshot, and it’s missing there. Horrors. At first I couldn’t figure out what was happening — every other single file seemed to be there. Then the FileVault explanation hit me like a ton of bricks.

Again, I managed to scrape by without any data loss. The key was paranoia: a few months back I used SuperDuper to clone the drive and put it on the shelf. It was old, but the only stuff that was seriously outdated on the disk image was code that also exists in a cvs repo. A quick cvs -q update -dP brought me back up to date.

Two morals to this story:

1. Test your backup. I got lazy. and haven’t tried a full restore in a long while, diffing against the original disk. It’s a big pain, but it would have caught this rogue FileVault filter much sooner.

2. Use incremental and snapshot based systems. Ideally from different vendors. The “bug” in Retrospect didn’t exist in SuperDuper, and it saved my bacon.

Finally, I wouldn’t really classify Retrospect’s matching of nonFileVault sparse images as a bug, it’s more of an unfortunate interaction/documentation problem.

Tuesday, June 20, 2006
01:20 AM