If you own or have recently purchased Crucial NVMe SSD such as CT1000P1SSD8 (1TB M.2 NVMe SSD) or CT500P1SSD8 (500GB M.2 NVMe SSD), please be aware that these devices may no be recognized by ESXi after upgrading to the latest release. Thanks to Pete Lindley, (OCTO for End-User Computing), who reached out last week regarding the observation as well as a workaround for the problem. This was also quite timely as I recently purchased a Crucial M.2 NVMe SSD and would have also ran into this problem.
It turns out these Crucial devices were working fine while running on ESXi 6.5 Update 2 but was no longer recognized in latest release of ESXi 6.7 Update 2. It is unclear whether support for these SSDs were removed intentionally or unintentionally, but in either case, these devices are not officially on VMware's Hardware Compatibility List (HCL).
UPDATE (06/13/20) - Thanks to reader Dave, it looks like this trick also works with ESXi 7.0 but the filename has changed. Simply copy nvme.v00 VIB from the ESXi 6.5 Update 2 and replace it on ESXi 7.0 system (either live under /bootbank or part of the installer) but rename the file to nvme_pci.v00 which is the new filename for NVMe driver.
UPDATE (05/23/19) - After speaking with a few folks who took a closer look, the issue is due to the fact that we added support for NVMe 1.3 spec in latest ESXi 6.7 Update 2 release, but because these are "consumer" devices, they did not conform to the latest specification and hence the driver is unable to claim the device. This is another good reminder when using components not on VMware HCL, this is always a risk from a home lab perspective. In general, I know Samsung and Intel NVMe SSD usually works quite well without issues but always good to do some research. I think Engineering is looking to see if there are other workarounds for the future, but for now, you can use the workaround below.
The easy workaround that Pete found was to simply replace the NVMe driver from ESXi 6.7 Update 2 (1.2.2.27-1vmw.670.2.48.13006603) with one found in ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253). To so do, simply copy nvme.v00 to /bootbank from either an existing ESXi 6.5 Update 2 system or directly from the ISO. Please note, any future updates or patches to the ESXi host will most likely override the updated driver.
For my 1TB HP EX920 I had to downgrade to 6.7 GA driver for NVMe to get 6.7u1 to see my drive. Haven’t checked 67u2, but I recond it will be the same. Maybe 67GA is the same driver as in 65u2?
It was discussed here: https://www.reddit.com/r/vmware/comments/a80r3y/issue_with_hp_nvme_drive_in_esxi/
In my experience, I had the same issue with 6.7U1 and 6.7U2 – I’m assuming its using the same driver.
6.7U1 was when they updated the driver. The 6.7GA driver seems to work fine. I really wish this was resolved, makes homelab stuff kind of a pain.
Thanks for this! I got my 1TB back!
Does somone have the issue with samsung evo nvme?
Turning off “Intel Rapid Storage Technology” worked for me to get a corsair MP510 detected and working
How did you do this? I cannot get this to work.
Hello mate, thank you for this. It worked for me! I’m running ESXi on an Optiplex 7060 Micro. It managed to detect Crucial NVMe SSD CT1000P1SSD8. I now have 1 NVME and 1 SATA — booting VMware ESXi, 6.7.0, 15160138 on a 32GB USB Thumb Drive! I you giving me the idea appreciate it.
To those wondering where can this be found — it’s in the BIOS — I’ve selected AHCI under SATA operation (it was previously set to RAID ON).
Works well now.
Cheers!
I did replace nvme.v00 with the the version from 6.7.0 GA.
Now ESXi is no longer booting 🙁
Did you get this working on ESXi 6.7 Update 2? I really don’t need to rebuild ESXi again.
were you able to resolve? I cant discover my datastores
I also wonder if the samsung evo 970 nvme has the same problem. I just received my adata SX8200 pro and after rolling back the driver version it works fine..but I wonder if I should go for the Samsung after all, if it wouldn’t have these driver issues.
Just stumbled across this article. I literally went to best buy after reading this and purchased a Samsung 970 EVO PLUS 500GB to test on a homelab system (ASROCK X470D4U Motherboard w/ AMD 2700x CPU) running a fresh install of ESXi 6.7 U2. ESXi had no issues detecting the drive, creating datastores, and VMs etc.
replacing the nvme.00 in the /bootbank per the steps provided will PSOD. You wont be able to boot anymore with error:
Failed to verify signatures of the following vib(s): [nvme]. All tardisks validated.
I ran into this issue with my own HP Homelab setup. I documented it here:
https://www.ovdamn.com/2019/07/not-all-nvm-drives-are-created-equal.html
It’s actually funny that you mentioned it, I tweeted at William Lam about this. He updated this post to show that the issue is that his Crucial drive didn’t support NVMe 1.3, however the HP EX920 is listed as supporting NVMe 1.3. I also have the 1TB version. My understanding is that the EX920 gets pretty hot, I’ve never seen mine go offline, but I don’t do a whole ton with it.
Running 6.5U3: Samsung 970 EVO = Yes + Intel 660p 1Tb = No. After cloning 6.5U3 image profile using Web Client/Auto deploy, deselect nvme 1.2.2.28 and select nvme 1.2.1.34, then generate install ISO. Shame, 660p officially supports NVMe 1.3 too
Thank you! That helped a lot! The only image available for download was the HPE one:
https://my.vmware.com/group/vmware/details?downloadGroup=OEM-ESXI65U2-HPE&productId=614
The last firmware version of the Crucial P1 seems to fix the problem (P3CR013).
Now I can see the disk in my ESXi 6.7u3
Thanks Benoit, much easier than injecting a deprecated driver. Helped me a lot!
The workaround Pete found is good! I was going nuts trying to inject the driver into the ISO.
/bootbank is /tmp ?
inject drivers into ISO steps
download ESXi Customizer Powershell script
https://www.v-front.de/p/esxi-customizer-ps.html#download
Download NVMe 1.2.1.34-1vmw.650.2.50.8294253 from website below; direct download VIB link also below
https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/nvme/VMW_bootbank_nvme_1.2.1.34-1vmw.650.2.50.8294253.vib
https://esxi-patches.v-front.de/ESXi-6.5.0.html
Create folder C:\VM and copy ESXi650-201908001.zip and copy ESXi customizer script into it
Create folder C:\VM\pkg (this is for you VIB files
Open Powershell as Admin
Goto C:\VM and you execute the following command
.\ESXi-Customizer-PS-v2.6.0.ps1 -zip .\ESXi670-201912001.zip -pkgDir c:\VM\pkg -v67
Now download Rufus, select ISO and write to USB, boot off USB
Happy to find this!! Thank you!! Just installing it outside the ISO solved my issue. Hopefully it’ll make its way back into 6.7!!
After updating NVMe back to ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253) driver, when starting to transfer data onto hard drive it goes inaccessible and 0 bytes
Anyone else had any luck fixing this issue
Have a Samsung 970 Pro 1TB M.2
Hi, anyone know of a 2TB NVMe that is consumer grade and doesn’t suffer from this issue with ESXi 6.7 U1 and later? Preferably not Samsung. Thanks.
I did some testing with a few M.2 SSDs I had on-hand to try to help other with this. Installing ESXi 6.7 U3b on an Intel NUC10i7FNH, I had the following results:
Samsung EVO 970 Plus (500GB)
Unknown firmware version (whatever it shipped with)
Detected, fully compatible
Corsair MP600 (1TB)
Firmware EGFM11.3 (Most recent as of 2020-03-14)
Note my test setup only supports PCIe 3.0, no idea if PCIe 4.0 impacts compatibility
Detected, fully compatible
ADATA XPG SX8200 Pro (2TB)
Firmware 42A4SANA (Most recent as of 2020-03-14)
NOT detected, NOT compatible
I’m disappointed that the ADATA didn’t work since the 2TB is only $269 USD and it’s a really good price for performance SSD. I specifically bought it to use in this ESXi build, but I’d rather return it for a compatible one. I did install the ADATA SSD in a Windows PC and it works fine there, just not in the latest ESXi update.
I’d love to hear reports of other compatible/incompatible SSD models!
I just bought the SX8200 pro on the island.
esxi did not see 6.5 u3 and 6.7 u3.
nvme drive saw that i installed old version. but there is a problem.
I am testing hdd speed in windows or other operating system
I see 3500mb / sec write and 3400mb / sec read values.
but i installed windows vm in esxi
I did the same test.
1500mb / s read gives 1300mb / s write value.
The adata xpg s40 product did not have such a resistance problem or speed issue.
I used windows vm in esxi while using xpg s40
I saw 3500mb / s write and 3400mb / s read values.
So when the driver sees the old version, he sees the SX8200 pro on the island, but the performance is halved. 🙁
note: the servers I’ve tested: dell r720 and r730 and esxi 6.5u3- 6.7u3
Has anyone tried this with ESXi 7.0? I have an SX8200 installed and working with the drive rollback on 6.7U3, and am holding off on 7.0 for now because of this.
Having the same issue with XPG8200 Pro and ESXi 7. Rolled back to 6.7 U3 with older nvme drivers. I hope if there is a way to fix this :/
thanks for testing that, have the same drive. hope there will be a fix soon.
To echo Michael, thanks for trying! 🙂 To confirm, you’re saying the 6.5U2 driver referenced in this fix doesn’t work on 7?
yeah exactly. I tried to install older nvme drivers at ESXi 7 as I did for 6.7 U3, but once ESXi 7 rebooted, I also noticed my nics couldn’t be detected. I will give it a try again this weekend with “esxcli software profile update” command William mentioned at his latest post. I’m using E200-8D by the way.
Good to know – I have the same motherboard series (X10SDV)
Still no news with 7.0 and SX8200? I’m facing same isssue.
Download ESXi 6.5 U2 Driver Rollup from my.vmware.com and copy nvme.v00 to \bootbank\nvme_pci.v00 (7.0) or \bootbank\nvme.v00 (6.7). I used SCP to do this. My SX8200 works!
I just made a request (for Adata SX8200Pro) via https://www.adata.com/en/support/online/# to add NGUID and EUI64 NVMe Identifier, just like https://www.crucial.com/support/ssd-support/p1-support
Please also make the same request to Adata. I doubt they will add.. unless lot of people use ESXi.
Did that – thanks for the hint. Though I couldn’t find the sx8200 as a SSD drive on the support page.
I need help, I have installed Esxi 7.0 on NUC10i7FNH installed with CORSAIR MP510 1.9TB PCIe NVMe Performance M.2 SSD, when i go in storage adapters it is showing me unknown and performance is not as it should have to be. Please can you suggest what drivers i can use to overcome this issue ?
Did you figure this out? I find my NVME drives in my NUC10 (with ESXi 6.7 and 7.0) are being claimed by AHCI and not working well at all.
Hi,
I try to install EVO 970 plus.
in ESXi 6.7u2 don’t see disks but I can see the controller(BOSS-S1), I tried the steps of old driver, but ESXi don’t see them in anyway.
Any idea?
Hi,
I have a Crucial MX 300 MVNe 275 Gb drive, and it is recognized by ESXi 7.0 (displayed under Storage as a local ATA Disk). It used to hold a Windows 10 installation in the same NUC7i3BNH. I have cleared partition table in ESXi. When I attempt to create a new data store, it fails with these log entries in wmkwarnong.log:
2020-06-29T17:07:10.197Z cpu2:526150 opID=33ff0c)WARNING: Vol3:
1410: Crucial 275 GB/5efa1fbe-aee2b6c2-0645-f44d306972ed: Error initializing JB SFD: I/O error
2020-06-29T17:07:10.197Z cpu2:526150 opID=33ff0c)WARNING: Vol3:
1651: Error creating system files: I/O error
I have tried the switch of 6.5U2 driver to \bootbank\nvme_pci.v00 and restarted, however it made no difference.
I have no reason to believe the drive is faulty as it booted Win10 just fine within the past couple of weeks.
Any idea what would cause this behavior?
Had to come back and share – updated a few more of my ESXi 6.7 instances to 7 and ran into my storage drivers not showing up again. Come to find out – the nvme_pci.v00 wasn’t listed in the /bootbank/boot.cfg file. (Could have been removed at some point?) I still had to reapply the old 6.5 driver (6.7 wouldn’t work…) – but seemed to help out. I do think I’m getting snagged by the consumer NVME drives that I’m using – as other drives work without the workaround. Of course – sent me into a bit of a panic hoping that I could scrounge up a fix!
1. Upload the following files to esxi “/tmp” directory
VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib
(Files can be download from http://221.126.249.9/nvme.7z)
2. SSH run the following commands to install 2 files:
esxcli software vib install -v /tmp/VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
esxcli software vib install -v /tmp/VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib
3. Reboot > Done !! Enjoy 🙂
Screen Dump:
————————————————————————–
login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
The time and date of this login have been sent to the system logs.
WARNING:
All commands run on the ESXi shell are logged and may be included in
support bundles. Do not provide passwords directly on the command line.
Most tools can prompt for secrets or accept them from standard input.
VMware offers supported, powerful system administration tools. Please
see http://www.vmware.com/go/sysadmintools for details.
The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[[email protected]:~] esxcli software vib install -v /tmp/VMware_bootbank_vmware-es
x-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.81 69922
VIBs Removed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2.48.130 06603
VIBs Skipped:
[[email protected]:~] esxcli software vib install -v /tmp/VMW_bootbank_nvme_1.2.1.3
4-1vmw.670.0.0.8169922.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922
VIBs Removed: VMW_bootbank_nvme_1.2.2.28-1vmw.670.3.73.14320388
VIBs Skipped:
[[email protected]:~]
Has anyone been able to try PNY XLR8 CS3030?