🎉 Celebrating 25 Years of GameDev.net! 🎉

Not many can claim 25 years on the Internet! Join us in celebrating this milestone. Learn more about our history, and thank you for being a part of our community!

Good ways to back up?

Started by
3 comments, last by swiftcoder 14 years, 10 months ago
I have about 32GB of digital photo's, and growing. I don't really like backups on DVD anymore. It's "only" about 8 DVD's to back up these photo's, but still, it's messy, takes a lot of room, ... Code is easy to backup, just put it somewhere online, like gmail, some SVN service, ... But photo's tend to take up a lot of room. There are so many possibilities (external HD's, solid state disks, USB sticks, SD cards, tape, optical, ...) that I don't really know what's best. Doesn't there exist anything where you can put your gigabytes of data on and be sure that it'll still work in 20 years? Photos should survive very long. Something cool like micro inscriptions, data cubes, ... What do you use? Any tips?
Advertisement
For home use, an external hard drive is probably your best bet. Its not going to protect you if your place burns down, but it will guard you against your computer frying (I have no idea if fire-proof safes are good enough to protect a hard drive in the event of a fire).

If you need a what-if-my-house-burns-down level of protection, then its best to find someplace to offload your data. I had a friend who was trying to write something that would encrypt and back data up to something like Amazon's S3, so that might be something you could look into if you need that level of protection.

The only sure-fire way of protecting your data is to back up to tape (or some other media), truck it to some off-site storage facility, and then regularly take your backups and restore them just to confirm that its still working. But at that point you're putting in full-time-sysadmin levels of effort.

Personally, I have a Linux filesharing server that I regularly back up to an external disk. This is good enough for most day-to-day scenarios, and I have very little data that would be devastating to lose to a fire or natural disaster (about the closest things are tax returns and immigration paperwork).
Quote: Original post by Lode
Doesn't there exist anything where you can put your gigabytes of data on and be sure that it'll still work in 20 years? Photos should survive very long. Something cool like micro inscriptions, data cubes, ...
The problem isn't longevity of the media - any type of disk lasts pretty much forever given a suitable, climate-controlled storage environment.

Unfortunately, while your media will survive just fine, chances are you won't be able to retrieve the data from it in 20 years... Twenty years ago, the format of choice was the 5 1/4 floppy disk (i.e. the big ones that you could actually bend) - do you know where to get ahold of a 5 1/4 floppy drive for your modern PC? I sure don't [wink]

With that in mind, magnetic tape is undoubtedly the best bet for long term storage. It is cheap, durable, and has been used for data storage for nearly 60 years. More importantly, the technology needed to read it is very low-tech, so even if tape drives completely disappear in the future, building/salvaging the necessary equipment should be trivial.

Furthermore, tape degrades gracefully - if the images are stored in a raw bitmap format, you can resurrect most of an image even if the tape is partially destroyed (think vintage film reels). And you do have to store the images as plain bitmaps - who knows if anyone will remember the details of JPEG compression in 20 years?

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

Quote: Original post by swiftcoder
Quote: Original post by Lode
Doesn't there exist anything where you can put your gigabytes of data on and be sure that it'll still work in 20 years? Photos should survive very long. Something cool like micro inscriptions, data cubes, ...
The problem isn't longevity of the media - any type of disk lasts pretty much forever given a suitable, climate-controlled storage environment.

Unfortunately, while your media will survive just fine, chances are you won't be able to retrieve the data from it in 20 years... Twenty years ago, the format of choice was the 5 1/4 floppy disk (i.e. the big ones that you could actually bend) - do you know where to get ahold of a 5 1/4 floppy drive for your modern PC? I sure don't [wink]

With that in mind, magnetic tape is undoubtedly the best bet for long term storage. It is cheap, durable, and has been used for data storage for nearly 60 years. More importantly, the technology needed to read it is very low-tech, so even if tape drives completely disappear in the future, building/salvaging the necessary equipment should be trivial.

Furthermore, tape degrades gracefully - if the images are stored in a raw bitmap format, you can resurrect most of an image even if the tape is partially destroyed (think vintage film reels). And you do have to store the images as plain bitmaps - who knows if anyone will remember the details of JPEG compression in 20 years?


Ahh, 20 years isn't THAT long... My first backup CD-ROMs are from 2000, which is almost 10 years ago. Everyone knew the details of JPEG compression just as good back then as now. And 20 years is just twice that time :)
Quote: Original post by Lode
Ahh, 20 years isn't THAT long... My first backup CD-ROMs are from 2000, which is almost 10 years ago. Everyone knew the details of JPEG compression just as good back then as now. And 20 years is just twice that time :)
Recall that computer development has a tendency to occur exponentially, so I would be careful about applying linear rules [wink]

20 years ago:
  • optical drives were not yet in use
  • 3.5 inch floppy disks weren't yet common
  • SCSI and IDE were both brand new, and not that widespread
  • FAT16 was brand new, but long filenames weren't developed yet
  • JPEG compression was under development, but not yet standardised


If you had backed-up images 20 years ago, on 5 1/4 inch floppy disks, formatted in FAT12, with pre-standardisation JPEG compression, we can still read them - but only barely. Chances are, you would have to do considerable scrounging around to find the relevant 5 1/4" diskette drive, driver for said drive in FAT12, and software to decode the older JPEG standard.

Now consider if you had stored bitmaps on a DAT tape as a tar archive: no problem. Tape drives are still widely available, tar archives are in wide use, and bitmaps can be read by any program under the sun.

My point is just that you need to consider the longevity of every component in your backup scheme - file formats, disk formats, the media itself, the relevant media drive, etc.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

Changing media formats or filesystems is not a problem as you must make copies on a new medium every now and then anyway, if you care about your data. Most media, in particular the optical ones, live not nearly as long as the expectancy/advertising goes. In any case, even if they may live that long, they're not as reliable as they should be.

When CD came out, they said "100 years", which was later corrected to "roughly 20 years". I own CDs from 1984 which are entirely good as of today, but some CDs I bought in 2005 have developed audible defects since then.
CDROM was attributed "at least 20 years" when it came out shortly thereafter, and this was soon changed to "probably up to 20 years". In reality, you get dropouts anywhere between 2 and 20 years.
Similarly, DVD+R was said to last "100-200 years" while DVD-R was said to last "25 or more years", and as a matter of fact, my DVD-Rs rarely survive longer than 3 years before they begin to show some problems. They're not unreadable at that time, but the point is they do have errors, and you just can't trust a medium that has errors.
You first notice that when it takes much longer than usual for the disc to pop up a drive icon or showing a directory. If your drive is seeking around for 15 seconds, the odds are good that the medium contains enough randomly appearing errors which exceed the error correction's capabilities (4000 bits of data), so the drive is re-reading a couple of times, until it finally catches something.
About every 10th disc goes right into the trash can, too. I'm verifying all discs on another drive immediately after burning and verifying it in the burner just to be sure, and about every 10th disc fails.

The same goes for (probably) all other storage media such as hard disks. While my oldest hard disk is 19 years (lives in a 12 year old computer) and has no problem whatsoever, this doesn't mean that it won't refuse to spin up tomorrow. Hard disks do that kind of thing in an entirely unexpected manner, and in particular modern drives seem to fail a lot more often than in the good old days.

The only possible exception to this are flash drives, which I honestly don't know about and lack long-time experience (don't we all...). As I understand it, the electrons trapped in the floating gate have no easy way of getting out, unless you expose the flash memory to gamma rays, place it in a MRT, or do any similarly silly thing. Thus, in my understanding, while the number of overwrites is clearly limited on flash, there is not necessarily a limit on its lifetime as such.
So... who knows... a flash drive that is kept in a metal box might live for many decades.

This topic is closed to new replies.

Advertisement