Building a file server for home (and small VFX usage)


I have been building a server for a bit of time now.  It’s a FreeBSD build from ZFSGuru. I have 16 2tb drives on the array in a RaidZ2 (Raid 6) formation.  There are 2 vdevs of RaidZ which means I have 8 drives x 2.  Each vdev has RaidZ2.  This results in better iops and overall performance.

Here is a photo of the rig :

The system is running ZFS as the filesystem.
You can read about ZFS here:
and watch a video here :

Currently I have been doing tests on performance to the system with compressed file systems.  ZFS is awesome in that you can have differing levels of compression for each “file system” (aka a folder).  Very handy. My goal is to have a “live” work area and an “archive” work area.  This means the archive data can be compressed and the live area will be either minimally compressed or not at all.  If I don’t need to get to the data all the time then I can certainly have it compressed right?  Seems to make sense to me.

I have been doing tests with a ~220gb job that has all kinds of files in it (mb, ma, tga, tif, exr, nk, jpg, rar, zip, mov) which is pretty representative of the type of data that lives in most of our job structures.

Here are some results of writing to the array.:

Compression : None Compression : LZJB Compression : GZIP1 Compression : GZIP6
Speed : 45660075 Bytes/sec. Speed : 41307227 Bytes/sec. Speed : 25659846 Bytes/sec.
Speed : 2612.690 MegaBytes/min. Speed : 2363.618 MegaBytes/min. Speed : 1468.268 MegaBytes/min. Speed : < 1000 MegaBytes/min.
Compression ratio 1.34 (55.42GiB reclaimed) Compression ratio 1.60 (82.20GiB reclaimed) Compression ratio 1.66 (87.12GiB reclaimed)
Uncompressed size 218.42G Uncompressed size 219.2G Uncompressed size 219.12G
Compressed size 163G Compressed size 137G Compressed size 132G

The big issue here is that the processor usage goes up with each level of compression (gzip6 being pretty heavy).  GZIP in fact caused a slow down on me copying the data.  Doing a simple ‘top’ from an ssh session results in a 50-80% processor utilization on the Xeon 5160 processor in the box I have.  I could add another proc and I bet the speed would increase quite a bit.  Power usage would also go up about 70 watts!

GZip1 is potentially very good for the archival space however.  It will take a long time to put stuff on there, but when I do, it’s going to save a ton of space.  I imagine several terrabytes of space will be saved when I am done copying all the data.  for the little bit of extra space saved (6%) the speed difference is not worth it.

Looking at another share and copying large files (2-8gb) in size yields this:

Speed : 81554689 Bytes/sec.
Speed : 4666.596 MegaBytes/min.

Why is this the case?  It’s large sustained transfers.  This will showcase the setup quite a bit more.

That’s much more like it.  I may need to do more Samba tuning.  If I find a proper Samba/FreeBSD expert I will try tuning this some more.

I am not using and ZIL cache or L2ARC cache either.  It’s raw drives.  I would imagine that with a ZIL log device (SSD) caching it would speed some things up.