VSAN 6.0 includes a large number of new enhancements and capabilities that I am sure many of you are excited to try out in your lab. One of the challenges with running VSAN in a home lab environment (non-Nested ESXi) is trying to find a platform that is both functional and cost effective. Some of the most popular platforms that I have seen customers use for running VSAN in their home labs are the Intel NUC and the Apple Mac Mini. Putting aside the memory constraints in these platforms, the number of internal disk slots for a disk drive is usually limited to two. This would give you just enough to meet the minimal requirement for VSAN by having at least a single SSD and MD.

If you wanted to scale up and add additional drives for either capacity purposes or testing out a new configurations, you are pretty much out of luck, right? Well, not necessary. During the development of VSAN 6.0, I came across a cool little nugget from one of the VSAN Engineers where USB-based disks could be claimed by VSAN which could be quite helpful for testing in a lab environment, especially using the hardware platforms that I mentioned earlier.

For a VSAN home lab, using cheap consumer USB-based disks which you can purchase several TB's for less than a hundred dollars or so and along with USB 3.0 connectivity is a pretty cost effective way to enhance hardware platforms like the Apple Mac Mini and Intel NUCs.

Disclaimer: This is not officially supported by VMware and should not be used in Production or evaluation of VSAN, especially when it comes to performance or expected behavior as this is now how the product works. Please use supported hardware found on the VMware VSAN HCL for official testing or evaluations.

Below are the instructions on how to enable USB-based disks to be claimable by VSAN.

Step 1 - Disable the USB Arbitrator service so that USB devices can been seen by the ESXi host by running the following two commands in the ESXi Shell:

/etc/init.d/usbarbitrator stop
chkconfig usbarbitrator off

vsan-usb-disk-1
Step 2 - Enable the following ESXi Advanced Setting (/VSAN/AllowUsbDisks) to allow USB disks to be claimed by VSAN by running the following command in the ESXi Shell:

esxcli system settings advanced set -o /VSAN/AllowUsbDisks -i 1

vsan-usb-disk-2
Step 3 - Connect your USB-based disks to your ESXi host (this can actually be done prior) and you can verify that they are seen by running the following command in the ESXi Shell:

vdq -q

vsan-usb-disk-3
Step 4 - If you are bootstrapping vCenter Server onto the VSAN Datastore, then you can create a VSAN Cluster by running "esxcli vsan cluster new" and then contribute the storage by adding the SSD device and the respective USB-based disks using the information from the previous step in the ESXi Shell:

esxcli vsan storage add -s t10.ATA_____Corsair_Force_GT________________________12136500000013420576 -d mpx.vmhba32:C0:T0:L0 -d mpx.vmhba33:C0:T0:L0 -d mpx.vmhba34:C0:T0:L0 -d mpx.vmhba40:C0:T0:L0

vsan-usb-disk-4
If we take a look a the VSAN configurations in the vSphere Web Client, we can see that we now have 4 USB-based disks contributing storage to the VSAN Disk Group. In this particular configuration, I was using my Mac Mini which has 4 x USB 3.0 devices that are connected and providing the "MD" disks and one of the internal drives that has an SSD. Ideally, you would probably want to boot ESXi from a USB device and then claim one of the internal drives along with 3 other USB devices for the most optimal configuration.

vsan-usb-disk-5
As a bonus, there is one other nugget that I discovered while testing out the USB-based disks for VSAN 6.0 which is another hidden option to support iSCSI based disks with VSAN. You will need to enable the option called /VSAN/AllowISCSIDisks using the same method as enabling USB-based disk option. This is not something I have personally tested, so YMMV but I suspect it will allow VSAN to claim an iSCSI device that has been connected to an ESXi host and allow it to contribute to a VSAN Disk Group as another way of providing additional capacity to VSAN with platforms that have restricted number of disk slots. Remember, neither of these solutions should be used beyond home labs and they are not officially supported by VMware, so do not bother trying to do anything fancy or running performance tests, you are just going to let your self down and not see the full potential of VSAN ๐Ÿ™‚

19 thoughts on “Home Labs made easier with VSAN 6.0 + USB Disks

  1. Incredible article! I know this isn’t really about speeds, as you clearly state. But, I still can’t resist asking, can you confirm that the hypervisor/VSAN is actually able to leverage USB 3.0 speeds in this scenario?

    If yes, well, wow! With UASP adapters and enclosures these days (for reduced protocol overhead and even better throughputs), the opportunities are pretty wonderful, for the unsupported home lab. If merely USB 2.0 speeds for now, well, then .

    • vSphere 5.5 was the first release to support USB 3.0 IIRC, so this should already be possible. I’ve not done any specific test, so it’s something you’ll want to try out ๐Ÿ™‚

  2. “Ideally, you would probably want to boot ESXi from a USB device”

    I would argue that ideally while I do want to boot ESXi from a USB device, it would be a smallish (~8 GB or so) memory card of some kind, something that would complement ordinary storage, not compete for it. Once you get logging, etc. off the boot device, ESXi doesn’t need to write. This used to be known as an ’embedded’ install. For all I know, it still is.

  3. I’m trying to do a demo VSAN entirely on a NUC. Have a host ESX and three nested ESX … all would be well except I can’t seem to tag the USB attached flash as flash so I can use it in a disk group.

    It seems that the USB driver just doesn’t support Flash (when I click on the option, I get “Cannot change host configuration”).

    Thoughts?

  4. I just installed ESXi 6 on Mac Mini 2012 Late model, everything works well except USB 3.0, the usb 3.0 passthrough devices (harddisk or nic) is functional… however speed is limited to usb2.0 spec. Run “lsusb” from ssh, shown the Host found 2.0 root hub only. Do you know any chance to activate USB3.0 support on Mac Mini ESXi? Thanks alot!

    #> lsusb
    Bus 002 Device 011: ID 0bda:8153 Realtek Semiconductor Corp.
    Bus 002 Device 010: ID 0bda:8153 Realtek Semiconductor Corp.
    Bus 002 Device 009: ID 05ac:828a Apple, Inc.
    Bus 002 Device 008: ID 05ac:820b Apple, Inc. Bluetooth HID Mouse
    Bus 002 Device 007: ID 05ac:820a Apple, Inc. Bluetooth HID Keyboard
    Bus 002 Device 006: ID 05ac:8242 Apple, Inc. Built-in IR Receiver
    Bus 002 Device 005: ID 0a5c:4500 Broadcom Corp. BCM2046B1 USB 2.0 Hub (part of BCM2046 Bluetooth)
    Bus 002 Device 004: ID 0424:2512 Standard Microsystems Corp. USB 2.0 Hub
    Bus 002 Device 003: ID 0781:5583 SanDisk Corp.
    Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

    #> lsusb -t

    Bus# 2
    -Dev# 1 Vendor 0x1d6b Product 0x0002
    -Dev# 2 Vendor 0x8087 Product 0x0024
    |-Dev# 11 Vendor 0x0bda Product 0x8153
    |-Dev# 10 Vendor 0x0bda Product 0x8153
    |-Dev# 3 Vendor 0x0781 Product 0x5583
    -Dev# 4 Vendor 0x0424 Product 0x2512
    |-Dev# 5 Vendor 0x0a5c Product 0x4500
    | |-Dev# 7 Vendor 0x05ac Product 0x820a
    | |-Dev# 8 Vendor 0x05ac Product 0x820b
    |
    -Dev# 9 Vendor 0x05ac Product 0x828a
    -Dev# 6 Vendor 0x05ac Product 0x8242
    Bus# 1
    -Dev# 1 Vendor 0x1d6b Product 0x0002
    `-Dev# 2 Vendor 0x8087 Product 0x0024

  5. Virtual SAN with USB Disks is awsome for my Intel NUC based Homelab ๐Ÿ™‚

    But according to USB3, same problem here. I see an USB 3 hub, but all devices (USB 3 SSD / HDD) are always USB 2.

    Bus 001 Device 005: ID 174c:1153 ASMedia Technology Inc.
    Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
    Bus 001 Device 003: ID 0930:6545 Toshiba Corp. Kingston DataTraveler 102 Flash Drive / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick
    Bus 001 Device 002: ID 8087:8001 Intel Corp.
    Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

    Any ideas how to get USB3 to work?

    • I’ve found the problem. In the UEFI on my system the default mode for xHCI was set to “Smart Auto” which seems to be an issue with ESXi. After setting xHCI to “Enabled”, the 2nd 2.0 root hub is gone, and finally my devices are detected as USB 3:

      ~ # lsusb
      Bus 002 Device 002: ID 174c:1153 ASMedia Technology Inc. <—
      Bus 001 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver
      Bus 001 Device 002: ID 0930:6545 Toshiba Corp. Kingston DataTraveler 102 Flash Drive / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick
      Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
      Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  6. Does the USB sticks still have to be in FAT16 format in the VSAN 6.0 to be recognized ? Thanks for your great articles.

  7. Hi, my Samsung Portable SSD T1 is not recognized as a flash device by ESX.
    [root@esxi-2:~] esxcli storage core device list -d mpx.vmhba33:C0:T0:L0
    mpx.vmhba33:C0:T0:L0
    Display Name: Local USB Direct-Access (mpx.vmhba33:C0:T0:L0)
    Has Settable Display Name: false
    Size: 238475
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0
    Vendor: Samsung
    Model: Portable SSD T1
    Revision: 0
    SCSI Level: 2
    Is Pseudo: false
    Status: on
    Is RDM Capable: false
    Is Local: true
    Is Removable: true
    Is SSD: false <—–
    Is VVOL PE: false
    Is Offline: false

    This is annoying as hell to setup a proper VSAN. Any ideas how to trick ESX in thinking this actual USB SSD is a SSD device?

  8. I’m considering different options for building a lab and stumbled upon this post. With this setup, am I reading correctly that it would only require a single host?

    • Well, VSAN requires a minimum of 3 ESXi hosts for proper functionality, that’s how its designed. You *can* run it on a single node, but if your drives fail, you would lose data. Something to keep in mind ๐Ÿ™‚

      With VSAN 6.1, you can run VSAN on 2 ESXi hosts w/3rd being a Virtual ESXi VM which can run else where (ideally not on the same VSAN Cluster) as it is your quorum node.

  9. William, any idea if I can use a Time Capsule as extended datastore for esxi? and can I use the internal switch to connect to my new NUC?

Thanks for the comment!