Storage controllers for small Linux NFS networks

While contemplating the disk capacity upgrade for my Microserver at home, I've also been thinking about adding a proper storage controller.

Currently I just use the built-in controller in the Microserver. It is an AMD SB820M SATA controller. It is a bottleneck for the SSD IOPS.

On the disks, I prefer to use software RAID (such as md or BtrFs) and not become dependent on the metadata format of any specific RAID controller. The RAID controllers don't offer the checksumming feature that is available in BtrFs and ZFS.

The use case is NFS for a small number of workstations. NFS synchronous writes block the client while the server ensures data really goes onto the disk. This creates a performance bottleneck. It is actually slower than if clients are writing directly to their local disks through the local OS caches.

SSDs on an NFS server offer some benefit because they can complete write operations more quickly and the NFS server can then tell the client the operation is complete. The more performant solution (albeit with a slight risk of data corruption) is to use a storage controller with a non-volatile (battery-backed or flash-protected) write cache.

Many RAID controllers have non-volatile write caches. Some online discussions of BtrFs and ZFS have suggested staying away from full RAID controllers though, amongst other things, to avoid the complexities of RAID controllers adding their metadata to the drives.

This brings me to the first challenge though: are there suitable storage controllers that have a non-volatile write cache but without having other RAID features?

Or a second possibility: out of the various RAID controllers that are available, do any provide first-class JBOD support?

Observations

I looked at specs and documentation for various RAID controllers and identified some of the following challenges:

• HP Smart Array: many discussions suggest they don't support genuine JBOD disks (without metadata) and users have had to use workarounds like creating single-disk RAID0 volumes.
• Adaptec: the 5445Z controller appears like a good physical fit for the Microserver and supports up to 250,000 IOPS. However, according to the manual, it supports JBOD but won't allow booting from JBOD disks. It is necessary to configure the disks as Simple Volumes (which involves writing some metadata) to have them bootable.
• LSI: it appears that they also had a problem with booting from JBOD but it was fixed in a firmware update (bug reference: SCGCQ00424928). The 9280-4i4e card fits the Microserver and offers 150,000 IOPS and 512MB battery-backed cache. The newer 9271-4i also appears to fit, offering 465,000 IOPS and 1GB flash-backed cache. However, some discussions also suggest that MegaRAID doesn't offer real JBOD although those comments are from 2010 so maybe the newer card/newer firmware is suitable.

Next steps

Are there other options to look at, for example, alternatives to NFS?

If I just add in a non-RAID HBA to enable faster IO to the SSDs will this be enough to make a noticeable difference on the small number of NFS clients I'm using?

Or is it inevitable that I will have to go with one of the solutions that involves putting a vendor's volume metadata onto JBOD volumes? If I do go that way, which of the vendors' metadata formats are most likely to be recognized by free software utilities in the future if I ever connect the disk to a generic non-RAID HBA?

Thanks to all those people who provided comments about choosing drives for this type of NAS usage.

Related reading

• A discussion of ZFS, NFS and write caches but without mention of stable/battery-backed write cache.
• Cache flushes, ZFS and NFS
• Some deprecated comments about write barriers
• A forum discussion about HBAs with non-volatile write cache
• A blog about using LSI controllers with disks in RAID0 for ZFS / FreeBSD