Step 1:
I make sure that I have any desired changes in /boot/loader.conf, /etc/sysctl.conf, and /etc/rc.conf before creating any ZFS volumes.
/boot/loader.conf changes:
# disable the auto-generated Disk IDs for disks kern.geom.label.disk_ident.enable="0" # disable the auto-generated GPT UUIDs for disks kern.geom.label.gptid.enable="0"
/etc/sysctl.conf changes:
# set 4K ashift vfs.zfs.min_auto_ashift=12
/etc/rc.conf changes:
# enable ZFS file system zfs_enable="YES"
Step 2:
For my most recent setup, I used a Dell MD1200 w/ 12 HDDs. For this example, I will call it "JBOD01".
When inserted, the drives were identified as da0 through da11.
I initialized each as GPT:
gpart create -s gpt /dev/da0 gpart create -s gpt /dev/da1 gpart create -s gpt /dev/da2 ...
I then created a ZFS partition, 1MB-aligned, on each disk and labeled it:
gpart add -t freebsd-zfs -a 1M -l jbod01-00 /dev/da0 gpart add -t freebsd-zfs -a 1M -l jbod01-01 /dev/da1 gpart add -t freebsd-zfs -a 1M -l jbod01-02 /dev/da2 ...
The output of "gpart show" looked similar to this:
=> 40 19532873648 da0 GPT (9.1T)
40 2008 - free - (1.0M)
2048 19532869632 1 freebsd-zfs (9.1T)
19532871680 2008 - free - (1.0M)
=> 40 19532873648 da1 GPT (9.1T)
40 2008 - free - (1.0M)
2048 19532869632 1 freebsd-zfs (9.1T)
19532871680 2008 - free - (1.0M)
=> 40 19532873648 da2 GPT (9.1T)
40 2008 - free - (1.0M)
2048 19532869632 1 freebsd-zfs (9.1T)
19532871680 2008 - free - (1.0M)
...
Step 3:
I then created a "RAID60" equivalent (two raidz2 vdevs, striped) with the following command (all one line). I alternated every other disk in each vdev.
zpool create tank raidz2 gpt/jbod01-01 gpt/jbod01-03 gpt/jbod01-05 gpt/jbod01-07 gpt/jbod01-09 gpt/jbod01-11 raidz2 gpt/jbod01-00 gpt/jbod01-02 gpt/jbod01-04 gpt/jbod01-06 gpt/jbod01-08 gpt/jbod01-10
I verified that I the new tank was 4K-aligned:
zdb | egrep 'ashift| name'
I set some ZFS options used for our setup:
zfs set compression=lz4 tank zfs set snapdir=visible tank zfs set aclmode=passthrough tank zfs set aclinherit=passthrough tank zfs set atime=off tank
Performance has so far been quite nice. Running a scrub goes faster than I expected.
pool: tank
state: ONLINE
scan: scrub in progress since Tue Sep 4 12:53:44 2018
576G scanned out of 8.24T at 1.23G/s, 1h46m to go
0 repaired, 6.83% done
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gpt/jbod01-01 ONLINE 0 0 0
gpt/jbod01-03 ONLINE 0 0 0
gpt/jbod01-05 ONLINE 0 0 0
gpt/jbod01-07 ONLINE 0 0 0
gpt/jbod01-09 ONLINE 0 0 0
gpt/jbod01-11 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gpt/jbod01-00 ONLINE 0 0 0
gpt/jbod01-02 ONLINE 0 0 0
gpt/jbod01-04 ONLINE 0 0 0
gpt/jbod01-06 ONLINE 0 0 0
gpt/jbod01-08 ONLINE 0 0 0
gpt/jbod01-10 ONLINE 0 0 0