If you own or have recently purchased Crucial NVMe SSD such as CT1000P1SSD8 (1TB M.2 NVMe SSD) or CT500P1SSD8 (500GB M.2 NVMe SSD), please be aware that these devices may no be recognized by ESXi after upgrading to the latest release. Thanks to Pete Lindley, (OCTO for End-User Computing), who reached out last week regarding the observation as well as a workaround for the problem. This was also quite timely as I recently purchased a Crucial M.2 NVMe SSD and would have also ran into this problem.
It turns out these Crucial devices were working fine while running on ESXi 6.5 Update 2 but was no longer recognized in latest release of ESXi 6.7 Update 2. It is unclear whether support for these SSDs were removed intentionally or unintentionally, but in either case, these devices are not officially on VMware's Hardware Compatibility List (HCL).
UPDATE (05/23/19) - After speaking with a few folks who took a closer look, the issue is due to the fact that we added support for NVMe 1.3 spec in latest ESXi 6.7 Update 2 release, but because these are "consumer" devices, they did not conform to the latest specification and hence the driver is unable to claim the device. This is another good reminder when using components not on VMware HCL, this is always a risk from a home lab perspective. In general, I know Samsung and Intel NVMe SSD usually works quite well without issues but always good to do some research. I think Engineering is looking to see if there are other workarounds for the future, but for now, you can use the workaround below.
The easy workaround that Pete found was to simply replace the NVMe driver from ESXi 6.7 Update 2 (1.2.2.27-1vmw.670.2.48.13006603) with one found in ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253). To so do, simply copy nvme.v00 to /bootbank from either an existing ESXi 6.5 Update 2 system or directly from the ISO. Please note, any future updates or patches to the ESXi host will most likely override the updated driver.
For my 1TB HP EX920 I had to downgrade to 6.7 GA driver for NVMe to get 6.7u1 to see my drive. Haven’t checked 67u2, but I recond it will be the same. Maybe 67GA is the same driver as in 65u2?
It was discussed here: https://www.reddit.com/r/vmware/comments/a80r3y/issue_with_hp_nvme_drive_in_esxi/
In my experience, I had the same issue with 6.7U1 and 6.7U2 – I’m assuming its using the same driver.
6.7U1 was when they updated the driver. The 6.7GA driver seems to work fine. I really wish this was resolved, makes homelab stuff kind of a pain.
Thanks for this! I got my 1TB back!
Does somone have the issue with samsung evo nvme?
Turning off “Intel Rapid Storage Technology” worked for me to get a corsair MP510 detected and working
How did you do this? I cannot get this to work.
Hello mate, thank you for this. It worked for me! I’m running ESXi on an Optiplex 7060 Micro. It managed to detect Crucial NVMe SSD CT1000P1SSD8. I now have 1 NVME and 1 SATA — booting VMware ESXi, 6.7.0, 15160138 on a 32GB USB Thumb Drive! I you giving me the idea appreciate it.
To those wondering where can this be found — it’s in the BIOS — I’ve selected AHCI under SATA operation (it was previously set to RAID ON).
Works well now.
Cheers!
I did replace nvme.v00 with the the version from 6.7.0 GA.
Now ESXi is no longer booting 🙁
Did you get this working on ESXi 6.7 Update 2? I really don’t need to rebuild ESXi again.
were you able to resolve? I cant discover my datastores
I also wonder if the samsung evo 970 nvme has the same problem. I just received my adata SX8200 pro and after rolling back the driver version it works fine..but I wonder if I should go for the Samsung after all, if it wouldn’t have these driver issues.
Just stumbled across this article. I literally went to best buy after reading this and purchased a Samsung 970 EVO PLUS 500GB to test on a homelab system (ASROCK X470D4U Motherboard w/ AMD 2700x CPU) running a fresh install of ESXi 6.7 U2. ESXi had no issues detecting the drive, creating datastores, and VMs etc.
replacing the nvme.00 in the /bootbank per the steps provided will PSOD. You wont be able to boot anymore with error:
Failed to verify signatures of the following vib(s): [nvme]. All tardisks validated.
I ran into this issue with my own HP Homelab setup. I documented it here:
https://www.ovdamn.com/2019/07/not-all-nvm-drives-are-created-equal.html
It’s actually funny that you mentioned it, I tweeted at William Lam about this. He updated this post to show that the issue is that his Crucial drive didn’t support NVMe 1.3, however the HP EX920 is listed as supporting NVMe 1.3. I also have the 1TB version. My understanding is that the EX920 gets pretty hot, I’ve never seen mine go offline, but I don’t do a whole ton with it.
Running 6.5U3: Samsung 970 EVO = Yes + Intel 660p 1Tb = No. After cloning 6.5U3 image profile using Web Client/Auto deploy, deselect nvme 1.2.2.28 and select nvme 1.2.1.34, then generate install ISO. Shame, 660p officially supports NVMe 1.3 too
Thank you! That helped a lot! The only image available for download was the HPE one:
https://my.vmware.com/group/vmware/details?downloadGroup=OEM-ESXI65U2-HPE&productId=614
The last firmware version of the Crucial P1 seems to fix the problem (P3CR013).
Now I can see the disk in my ESXi 6.7u3
Thanks Benoit, much easier than injecting a deprecated driver. Helped me a lot!
The workaround Pete found is good! I was going nuts trying to inject the driver into the ISO.
/bootbank is /tmp ?
inject drivers into ISO steps
download ESXi Customizer Powershell script
https://www.v-front.de/p/esxi-customizer-ps.html#download
Download NVMe 1.2.1.34-1vmw.650.2.50.8294253 from website below; direct download VIB link also below
https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/nvme/VMW_bootbank_nvme_1.2.1.34-1vmw.650.2.50.8294253.vib
https://esxi-patches.v-front.de/ESXi-6.5.0.html
Create folder C:\VM and copy ESXi650-201908001.zip and copy ESXi customizer script into it
Create folder C:\VM\pkg (this is for you VIB files
Open Powershell as Admin
Goto C:\VM and you execute the following command
.\ESXi-Customizer-PS-v2.6.0.ps1 -zip .\ESXi670-201912001.zip -pkgDir c:\VM\pkg -v67
Now download Rufus, select ISO and write to USB, boot off USB
Happy to find this!! Thank you!! Just installing it outside the ISO solved my issue. Hopefully it’ll make its way back into 6.7!!
After updating NVMe back to ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253) driver, when starting to transfer data onto hard drive it goes inaccessible and 0 bytes
Anyone else had any luck fixing this issue
Have a Samsung 970 Pro 1TB M.2
Hi, anyone know of a 2TB NVMe that is consumer grade and doesn’t suffer from this issue with ESXi 6.7 U1 and later? Preferably not Samsung. Thanks.
I did some testing with a few M.2 SSDs I had on-hand to try to help other with this. Installing ESXi 6.7 U3b on an Intel NUC10i7FNH, I had the following results:
Samsung EVO 970 Plus (500GB)
Unknown firmware version (whatever it shipped with)
Detected, fully compatible
Corsair MP600 (1TB)
Firmware EGFM11.3 (Most recent as of 2020-03-14)
Note my test setup only supports PCIe 3.0, no idea if PCIe 4.0 impacts compatibility
Detected, fully compatible
ADATA XPG SX8200 Pro (2TB)
Firmware 42A4SANA (Most recent as of 2020-03-14)
NOT detected, NOT compatible
I’m disappointed that the ADATA didn’t work since the 2TB is only $269 USD and it’s a really good price for performance SSD. I specifically bought it to use in this ESXi build, but I’d rather return it for a compatible one. I did install the ADATA SSD in a Windows PC and it works fine there, just not in the latest ESXi update.
I’d love to hear reports of other compatible/incompatible SSD models!