The sinister secret in every hard disk
These are not my words...
"The job of any filesystem boils down to this: when asked to read (data), it should return the same data that was previously written ... Incredibly, most filesystems fail this test. They depend on the underlying hardware to detect and report errors. If a disk simply returns bad data, the average filesystem won't even detect it."
Those are the words of Jeff Bonwick, a senior engineer at Sun Microsystems (now part of Oracle Corporation).
While Bonwick's original blog is quite technical, he identifies the fundamental problem in his opening lines: the filesystem software (part of the operating system) assumes the disk hardware provides a 100% safety net for your data. The disks don't do this: they assume the filesystem is responsible.
Why have I chosen to call this publicly documented fact a `sinister secret'? Well, it is sinister because it can kill your data (whether it be your wedding video, or your business payroll system). I use the word secret because you probably didn't know about it - many professionals in the IT industry are even unaware of it or don't bother telling you about it.
You expect that what you put in is what comes back
Most people have the expectation that what they save in a computer is what will come back tomorrow. After all, if you save something, isn't it permanent? Haven't you saved thousands of documents and they always come back with exactly the same words (maybe even the same typos) that you keyed in yesterday? The fact that it is working today has made you complacent, feeling it will work just the same tomorrow.
In fact, there is no such guarantee
Guarantees typically cover outright failures (where the motor in the disk burns out and your screen goes blue).
The technical standards for hard disks do not provide any means to prove the integrity of data stored on the disk.
Whether you consider it right or wrong, your hard disk manufacturer will tell you that it is the responsibility of your software.
So surely my software warns me if a file is corrupt?
Some modern applications have a way of keeping a `checksum' in each file, and warning you if the file has been corrupted since you last looked at it.
This safety check is just that: it doesn't help you recover the data. For example, you have a file containing your grandma's bank account number and you want to send her some cash. The file has suffered a minor corruption - one of the digits changed from 3 to 4 (see the graphic for an action replay).
- In the worst case scenario, you use the wrong account number, sending the cash to the wrong account.
- If you are lucky, your software will detect that the file is corrupt and inform you. It will not fix the file - you will have to go to your paper records to find that account number.
- In the optimum system (which we will get on to shortly), the file will be corrected for you, perfectly, or if it really can't be receoved, you will be guaranteed to get an error message telling you not to send any money because the file is corrupt.
I mirrored my hard disks, surely I am safe?
Many PC makers now sell two hard disks with the suggestion that you can mirror them and guarantee the safety of your data.
In fact, mirroring, as a standalone solution, only gives a false sense of security.
If two hard disks are mirrored, and one fails, you are still able to access the copy of the data on the other disk. In most cases, your computer will even keep running off just one disk while you replace the other - this lack of downtime is very attractive for a 24x7 business.
However, there is no guarantee that the data stored on either of the mirrored disks is 100% free of any minor corruption. Mirroring systems only detect the catastrophic failure of a disk, not the small degradations that creep in over an extended period of time.
If I back up to tape, I can get my data back safely?
When you backup to tape, the backup software reads the data from your hard disk. If your hard disk is not reading the data with 100% accuracy, then your tape will contain some bad data too.
Is there anything that can be done?
Hard disks will not change: however, change has arrived, in the modern operating systems Linux, FreeBSD and Solaris.
Those operating systems are all available for free. If you paid for Windows, you might be disappointed to find that the Windows filesystem (NTFS) does not offer integrated checksums to protect your data.
All three of these open operating systems now offer the ability to make a `checksum' on all of your files. This is done behind the scenes, for every file, from every application. This is the type of watertight guarantee that you would have asked for if you knew it was necessary to ask.
On it's own, this `checksum' technology can catch any minor data corruption and prevent you from using the bad data.
The icing on the cake: combine this with a mirroring strategy (two hard disks) to seamlessly guarantee that if the copy of a file on one disk becomes slightly corrupt:
- the system will detect the corruption, automatically, when you request the file
- the system will know which of the disks has a good copy, and which one is bad, because only one will match the checksum
- the system will use the copy of the file from the disk that is not corrupt, so the application software continues to run normally
- you get a helpful warning that one of your disks is starting to wear out - and you order a replacement
What to ask for
On Linux, the solution is called the btrfs filesystem.
On FreeBSD and Solaris, you need the ZFS filesystem.
On all of these systems, the filesystem should be combined with the mirroring technology to ensure that it can (hopefully) find a good copy of your file whenever corruption is detected.
The solution is even free
In fact, all three operating systems (Linux, FreeBSD and Solaris) can be downloaded free of charge:
You simply need to download the disk image, burn it to a CD or DVD, and boot your computer with the CD/DVD to install it.