Durable data

As evidenced by these notebooks, I am a packrat. I hoard information and try to organize it: I keep email going back to 2007. I have an (infrequently updated) journal going back to 2006. I have something like 1,100 PDFs of scientific papers squirreled away, the detritus of past research projects and idle curiosity.

But I am also aware that digital data tends to rot. Not only do hard drives fail and files get corrupted (or just lost in the shuffle between devices), but formats and services disappear. I have old Pages documents in its proprietary file format, which basically no other software supports; my extensive Mathematica notebooks will become useless if I lose my student license; many social media, photo sharing, and “cloud” websites will likely fold sooner or later when the venture capital runs out; and half my old programming projects are probably incompatible with the latest version of their languages or libraries.

I’m also worried by the iOS ecosystem, where the “file” metaphor is nearly gone and apps don’t store their data in any readily accessible place. There are a zillion notebook and journal apps – how many of those will still be functioning ten years and ten iOS upgrades from now? If they disappear, how do we get our data out?

Plain text tools

Plain text formats are the ultimate in durability: even if the fancy tools no longer exist, ordinary ASCII and UTF-8 text files should be readable for decades, and you can always extract the structured data one way or another if it’s sufficiently important.

Ledger, hledger, and friends: tools for double-entry accounting with simple plain text files. I use hledger to track my personal spending and savings.
Org mode for Emacs: the plain-text kitchen sink. I use it for simple text markup (like these notebooks), journaling, task tracking and scheduling, and lecture notes, and it can do literate programming, calendaring, and entire websites if you so desire.
My email is in a Maildir, which is about as straightforward as it can get – just emails in files. I organize it with notmuch, which makes it all instantaneously searchable and taggable. You might ask “Why not delete it?” I ask “Why bother?”

References on durability

Backblaze does good annual reports on hard drive failure rates. The annualized failure rate is about 2%; an older post suggests about 20% of drives fail within four years, because the mortality increase spikes after three years.
Vines, T. H. et al. (2014). The availability of research data declines rapidly with article age. Current Biology, 24(1), 94–97. doi:10.1016/j.cub.2013.11.014
Research data (at least in one niche) rapidly disappears as computers die or get replaced, scientists move, and old floppy disks become unreadable. “For papers where the authors gave the status of their data, the odds of a data set being extant fell by 17% per year.”
Hart, E. M. et al. (2016). Ten Simple Rules for Digital Data Storage. PLOS Computational Biology, 12(10), e1005097. doi:10.1371/journal.pcbi.1005097

Open questions

Suppose I want to save old photo albums – not just the JPEGs but dates, locations, and maybe comments or names of those in the photo. I can do this in various photo apps, but is there a standard file format for storing all that metadata? Could I use it to archive Facebook photo albums? Or should I just have custom printed books for everything worth keeping?
Some research papers could be illustrated with interactive figures. How can I program one that’ll be useful to a grad student twenty years from now? Ten years ago I’d have used Flash, which is dying; ten years before that I’d have used Java, which is nearly impossible to get working in browsers now. Will d3 be compatible with browsers in twenty years?