I did some exploration over the past few months on storage for my home lab (This post is not about number crunching different setups).
I already have a FreeNAS system doing mass storage (8T), mostly media. I added an additional pool of 700G of Flash. I wanted to compare VMware (ESXi) performance between NFS and iSCSI (I already have some ideas of what performance is like for VMware and Fiber Channel from work). After a little testing, and reading, I concluded that between VMware NFS doing extra checks to ensure data integrity, and ZFS doing extra checks to ensure data integrity, things would not speed up enough to be useful. I was stuck around 1-200Mbit.
iSCSI, is it worth it?
I next turned to iSCSI. As a rule, I avoid iSCSI because the idea of running a UNIX block device over a best-effort protocol is not a recipe for success. On FreeNAS with ZFS, I can export a zvol of block space over iSCSI and let VMware manage it completely. In theory, this would be faster. It turns out that in this case at least, iSCSI was significantly faster. I’ve seen speeds up to 850Mbit between ESXi and FreeNAS. The FreeNAS has 2x 1G links configured for iSCSI, and the ESXi box has 2x 1G links configured for iSCSI (I did this because I actually have 2 ESXi boxes, and I didn’t want them to bottleneck on a single 1G link to FreeNAS).
If you aren’t familiar with the iSCSI docs for both VMware , and FreeNAS , you should do a little reading.
JunOS bits
Here are the Juniper config bits (based off JunOS 13.2):
1 louisk@switch0.cmhome> show configuration interfaces ge-0/0/42 | display set
2 set interfaces ge-0/0/42 description "FreeNAS Storage 1"
3 set interfaces ge-0/0/42 mtu 9216
4 set interfaces ge-0/0/42 unit 0 family ethernet-switching
5 {master:0}
6 louisk@switch0.cmhome> show configuration interfaces ge-0/0/43 | display set
7 set interfaces ge-0/0/43 description "FreeNAS Storage 1"
8 set interfaces ge-0/0/43 mtu 9216
9 set interfaces ge-0/0/43 unit 0 family ethernet-switching
10 {master:0}
11 louisk@switch0.cmhome> show vlans storage1
12 Name Tag Interfaces
13 storage1 12
14 ge-0/0/42.0*
15 {master:0}
16 louisk@switch0.cmhome> show vlans storage2
17 Name Tag Interfaces
18 storage2 13
19 ge-0/0/43.0*
20 {master:0}
21 louisk@switch0.cmhome>
FreeNAS screenshots
Here are the screenshots showing the config of both the FreeNAS and ESXi (through vCenter VCSA):
vCenter screenshots
I’ve got 2 vmkernel interfaces for iSCSI configured, on different networks, using jumbo frames. This gives me multi-pathing, for additional throughput, in addition to making it fault-tolerant. Coupled with using SSDs as the backing store gives me reasonable VM performance, even when I need to do things like source code compiles.
Comments