Today I found an interesting article on my Twitter timeline regarding some issues when trying to install and run Nested ESXi on top of a VSAN datastore. Shortly after the ESXi installation begins, the following error message is observed:

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320

This of course peaked my interest given the topic and I would have expected this to just work and thought this might have been some miss-configuration. I decided to try this out in my lab and to my surprise, I encountered the exact same problem.

Here is a quick screenshot of error message:

I pinged a couple of folks from the VSAN development team to see if this was a known issue and if so, why was it occurring? After a couple of email exchanges, it turns the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. Even though VMFS-5 no longer uses SCSI-2 reservations, the underlying LVM (Logical Volume Manager) driver for VMFS still requires it. Since VSAN does not make use of SCSI-2 reservations, it did not make sense to support it and hence the issue. Having said that, since Nested Virtualization is heavily used at VMware, the VSAN development team has come up with a nifty solution as they too hit this problem early on during the development of VSAN. Big thanks to Christian Dickmann (Tech Lead on the VSAN Engineering team) for providing this little tidbit.

Disclaimer: Nested Virtualization is not officially supported by VMware nor are the configuration changes described below, please use at your own risk.

To get around this problem, the VSAN team added in an advanced ESXi setting that would "fake" SCSI Reservations and this needs to be configured for the ESXi hosts providing up the VSAN datastore.

Run the following ESXCLI command (either locally on ESXi Shell or remotely)

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

A system reboot is not required and this change can be done live on the ESXi host prior to starting the ESXi installation. Once this is done, you will now be able to proceed with installing Nested ESXi on top of a VSAN datastore. Here is a screenshot of my Nested ESXi VM running on top of Nested ESXi VSAN datastore 🙂

Hopefully this workaround will be useful for anyone running VSAN and would like to fully make use of this storage by running Nested ESXi for development or testing.

17 thoughts on “How to run Nested ESXi on top of a VSAN datastore?

  1. Hi ,William!
    Thank you for your great article.

    The command “esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1” should be run on the ESXi Host(The really one),right?
    I have a question with it. If I run the command,does it make any negative consequence for the VSAN?I mean it sounds like to hack the ESXi, so I worry about that.

    Thank you very much.(Sorry for my English.)

  2. Hi, I’m getting the same 1048320 error during the /usr/sbin/vmkfstools stage, but I’m not trying to do a nested config, I’m doing a bare metal install on a Cisco C240 with RAID6 4Tb drive config (36Tb total).
    We were able to install no problem on 2 other C240’s (only difference was they had 2Tb drives – 18Tb total)
    Had an open VMware case for a while and not much to show – latest suggestion is to make the 36Tb volume smaller as a test.
    Any other ideas?

    • Fletcher. Did you ever come up with a solution? I am having the same problem you were having.

      thanks brother

  3. Just a note that this command needs to be run on every host in the Vsan cluster. Thought this would be fixed by vsan 6, but it’s not 🙁

  4. I have a question about whether this will cause any side-effects on VSAN hosts which also run real LUNs.. Will faking reservations carry over to my real block storage which require it? Is the host smart enough to which one to fake it on and which one not to?

  5. I guess it doesn’t seem to have any negative effect, as I enabled it and my LUNs are all still working. They do have hardware accelerated ATS locking enabled (don’t use scsi-2 or 3 locking) so not sure if that is relevant or not, and at any rate the command from this article does have the path VSAN in it, as if it only changes it for VSAN..

    esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

    I wonder why they don’t just default it to enabled if it doesn’t have any bad effects?

    Oh well, I will leave it enabled then. THANKS

Thanks for the comment!