Tag: ibid

The current version of my backup tool ibid is 52. Since the last time I wrote about it, I’ve added a new option, –check-names, for when you can’t rely on device/inode pairs (e.g., when moving your files onto a new filesystem); and fixed some important bugs, including ones that created unnecessary duplicate backups of files and unnecessary “update” records.

Download ibid here.

The last I wrote about my backup tool, ibid, was three years ago (here; earlier post here), but I’ve continued making refinements to it. Then it was at version 24; now it’s at version 47 (download it here). Here are the changes since then, minus the uninteresting ones:

25. Add –maxfiles.
32. Don’t use Storable for the complete record structure; apparently a stringified form gets constructed in memory before it’s written to the file, which is disastrous for very large records. Use a custom streaming serialization method instead. Also, detect and reject unknown record versions.
35. Another major rewrite. This one does away with the old runtime data structures based on big, inefficient Perl hashes, replacing them with strings containing compact “pack”ed values. This change yields enormous runtime memory savings, which matters after a few hundred sessions and many tens of thousands of files have accumulated in your fileset record. Also fixed a few documentation bugs and eliminated some dead code.
36. Rename options for greater consistency: –limit (-l) is now –maxbytes (-b); –files (-f) is now –maxfiles (-f).
37. Report when one or the other limit (bytes or files) is reached.
38. Add –prune-sessions option.
41. Add new –single-file-size-limit option; renamed –maxbytes and –maxfiles to –session-size-limit and –session-files-limit, respectively. Switched from &foo() function-calling syntax to foo().
42. Add a new history-entry type: zero-length (“empty”) files. These are recorded in the session record but not copied to the archive, to save overhead.
43. Include <dev:ino> in –dump output when –verbose is supplied.
44. Support a device-map file, $HOME/.ibid/.devmap, for tracking a filesystem when its device number changes.
45. Document the .devmap file; add –trim-report; support “xz” compression of session files; support optional callbacks in foreach_name_history().
46. Add another level of depth to the “target” path for each new power of 10 in the session number. So session 7311 is rooted at TARGET/FILESET/7000/300/7311, and session 29582 is rooted at TARGET/FILESET/20000/9000/500/29582. Path elements that would start with a 0 are omitted; e.g., session 4006 is rooted at TARGET/FILESET/4000/4006, not at TARGET/FILESET/4000/000/4006.
47. Document –trim-report.

There’s still no home page for ibid, but at least now all ibid-related posts on my blog are grouped under the tag ibid.

Ibid (“incremental backups to infinite disk”) is a tool I’ve written for backing up files. When I first posted it (read about it here, download it here) it was at release number 18. Now it’s at release 24. Here are the changes between then and now.

19. If the same pathname is encountered a second time in one session, don’t treat it as a hardlink to the first time.
20. Restructure the ~/.ibid directory and add an extra level to target pathnames when many sessions begin to accumulate.
21. If the old session file was compressed, compress the new session file the same way.
22. For determining whether a file is a mere rename of one already archived, consider its size in addition to its device, inode, and modtime.
23. When running with –ensure, flag (in the session record) the files that have been verified as present in the archive, so that future sessions won’t have to reverify the same files.
24. Remove an extraneous newline from “ibid -D” output.

Still to do: a proper home page for ibid.

You back up the files on your computer regularly, right? You don’t? You’re just asking for trouble. But you already knew that, didn’t you? You’d do periodic backups if you could, but there just are no good backup tools, right? They’re too complicated to use? You don’t have enough CD-R’s to hold all your files? Or you already have too many and are drowning in the clutter?

Well now you can save the lame excuses for why you don’t floss. I’ve written ibid, a simple tool for performing incremental backups on Linux and Unix-like systems — possibly also Mac OS X, I haven’t checked yet. And who knows, maybe it can even be made to run under Windows.

For a long time I used a makeshift system that backed up files to blank CDs or DVDs, but it was slow and difficult to use and it required my constant attendance (for swapping discs in and out) and very often the disc burner would crap out at the last minute after half an hour of burning and produce an unreadable disc. I didn’t trust the archives that I created with this system; it was annoying to have to render botched discs unreadable with scissors all the time (for privacy reasons); and the more backups I did, the more crowded with discs my safe-deposit box got.

Then I discovered Jungledisk, a commercial “infinite filesystem” service that’s easy to use and has good privacy and security features, not to mention reasonable fees. In conjunction with davfs it can be made to act just like a mounted filesystem.

Ibid doesn’t care if you’re using Jungledisk or some other destination service or media. It simply copies the files you ask for to the target directory tree. The backed-up copies are just plain files residing under their original names in a plain filesystem, so retrieving old data is simple.

Here are ibid’s other main features:

• Written as a single Perl file, no complicated build/install process;
• Limit sessions by size;
• Exclude files and directories by pattern (using Perl “regular expressions”);
• Files with multiple “hard links” stored only once — link names stored separately;
• Files re-backed up when they change;
• Files not re-backed up if they’re merely renamed (the new name is stored separately);
• Can run unattended, and even at automated intervals (e.g. via cron).

You can download it here. At some point soon perhaps I will create a proper website for it, at which time I will update this blog post with the new information; meanwhile, this is the definitive home of ibid (and I will keep the copy at the above link up to date with bugfixes and other changes).

Here is the documentation for ibid.

Continue reading “Ibid: Incremental backups to infinite disk”