If you own or have recently purchased Crucial NVMe SSD such as CT1000P1SSD8 (1TB M.2 NVMe SSD) or CT500P1SSD8 (500GB M.2 NVMe SSD), please be aware that these devices may no be recognized by ESXi after upgrading to the latest release. Thanks to Pete Lindley, (OCTO for End-User Computing), who reached out last week regarding the observation as well as a workaround for the problem. This was also quite timely as I recently purchased a Crucial M.2 NVMe SSD and would have also ran into this problem.
It turns out these Crucial devices were working fine while running on ESXi 6.5 Update 2 but was no longer recognized in latest release of ESXi 6.7 Update 2. It is unclear whether support for these SSDs were removed intentionally or unintentionally, but in either case, these devices are not officially on VMware's Hardware Compatibility List (HCL).
UPDATE (07/29/20) - Over the past few months, I have had a number of folks share feedback that using the trick mentioned below for ESXi 7.0, they have had success of ESXi detecting their NVMe SSD. I wanted to share some of the model and/or vendors that folks have reported success with. I will keep this list updated, so feel free to leave a comment below.
- OWC Aura Pro X2 2TB NVMe
- ADATA XPG
- Sabrent
UPDATE (06/13/20) - Thanks to reader Dave, it looks like this trick also works with ESXi 7.0 but the filename has changed. Simply copy nvme.v00 VIB from the ESXi 6.5 Update 2 and replace it on ESXi 7.0 system (either live under /bootbank or part of the installer) but rename the file to nvme_pci.v00 which is the new filename for NVMe driver.
UPDATE (05/23/19) - After speaking with a few folks who took a closer look, the issue is due to the fact that we added support for NVMe 1.3 spec in latest ESXi 6.7 Update 2 release, but because these are "consumer" devices, they did not conform to the latest specification and hence the driver is unable to claim the device. This is another good reminder when using components not on VMware HCL, this is always a risk from a home lab perspective. In general, I know Samsung and Intel NVMe SSD usually works quite well without issues but always good to do some research. I think Engineering is looking to see if there are other workarounds for the future, but for now, you can use the workaround below.
The easy workaround that Pete found was to simply replace the NVMe driver from ESXi 6.7 Update 2 (1.2.2.27-1vmw.670.2.48.13006603) with one found in ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253). To so do, simply copy nvme.v00 to /bootbank from either an existing ESXi 6.5 Update 2 system or directly from the ISO. Please note, any future updates or patches to the ESXi host will most likely override the updated driver.
For my 1TB HP EX920 I had to downgrade to 6.7 GA driver for NVMe to get 6.7u1 to see my drive. Haven’t checked 67u2, but I recond it will be the same. Maybe 67GA is the same driver as in 65u2?
It was discussed here: https://www.reddit.com/r/vmware/comments/a80r3y/issue_with_hp_nvme_drive_in_esxi/
In my experience, I had the same issue with 6.7U1 and 6.7U2 – I’m assuming its using the same driver.
6.7U1 was when they updated the driver. The 6.7GA driver seems to work fine. I really wish this was resolved, makes homelab stuff kind of a pain.
Thanks for this! I got my 1TB back!
Does somone have the issue with samsung evo nvme?
Turning off “Intel Rapid Storage Technology” worked for me to get a corsair MP510 detected and working
How did you do this? I cannot get this to work.
Hello mate, thank you for this. It worked for me! I’m running ESXi on an Optiplex 7060 Micro. It managed to detect Crucial NVMe SSD CT1000P1SSD8. I now have 1 NVME and 1 SATA — booting VMware ESXi, 6.7.0, 15160138 on a 32GB USB Thumb Drive! I you giving me the idea appreciate it.
To those wondering where can this be found — it’s in the BIOS — I’ve selected AHCI under SATA operation (it was previously set to RAID ON).
Works well now.
Cheers!
I did replace nvme.v00 with the the version from 6.7.0 GA.
Now ESXi is no longer booting 🙁
Did you get this working on ESXi 6.7 Update 2? I really don’t need to rebuild ESXi again.
were you able to resolve? I cant discover my datastores
I also wonder if the samsung evo 970 nvme has the same problem. I just received my adata SX8200 pro and after rolling back the driver version it works fine..but I wonder if I should go for the Samsung after all, if it wouldn’t have these driver issues.
Just stumbled across this article. I literally went to best buy after reading this and purchased a Samsung 970 EVO PLUS 500GB to test on a homelab system (ASROCK X470D4U Motherboard w/ AMD 2700x CPU) running a fresh install of ESXi 6.7 U2. ESXi had no issues detecting the drive, creating datastores, and VMs etc.
replacing the nvme.00 in the /bootbank per the steps provided will PSOD. You wont be able to boot anymore with error:
Failed to verify signatures of the following vib(s): [nvme]. All tardisks validated.
I ran into this issue with my own HP Homelab setup. I documented it here:
https://www.ovdamn.com/2019/07/not-all-nvm-drives-are-created-equal.html
It’s actually funny that you mentioned it, I tweeted at William Lam about this. He updated this post to show that the issue is that his Crucial drive didn’t support NVMe 1.3, however the HP EX920 is listed as supporting NVMe 1.3. I also have the 1TB version. My understanding is that the EX920 gets pretty hot, I’ve never seen mine go offline, but I don’t do a whole ton with it.
Running 6.5U3: Samsung 970 EVO = Yes + Intel 660p 1Tb = No. After cloning 6.5U3 image profile using Web Client/Auto deploy, deselect nvme 1.2.2.28 and select nvme 1.2.1.34, then generate install ISO. Shame, 660p officially supports NVMe 1.3 too
Thank you! That helped a lot! The only image available for download was the HPE one:
https://my.vmware.com/group/vmware/details?downloadGroup=OEM-ESXI65U2-HPE&productId=614
The last firmware version of the Crucial P1 seems to fix the problem (P3CR013).
Now I can see the disk in my ESXi 6.7u3
Thanks Benoit, much easier than injecting a deprecated driver. Helped me a lot!
The workaround Pete found is good! I was going nuts trying to inject the driver into the ISO.
/bootbank is /tmp ?
inject drivers into ISO steps
download ESXi Customizer Powershell script
https://www.v-front.de/p/esxi-customizer-ps.html#download
Download NVMe 1.2.1.34-1vmw.650.2.50.8294253 from website below; direct download VIB link also below
https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/nvme/VMW_bootbank_nvme_1.2.1.34-1vmw.650.2.50.8294253.vib
https://esxi-patches.v-front.de/ESXi-6.5.0.html
Create folder C:\VM and copy ESXi650-201908001.zip and copy ESXi customizer script into it
Create folder C:\VM\pkg (this is for you VIB files
Open Powershell as Admin
Goto C:\VM and you execute the following command
.\ESXi-Customizer-PS-v2.6.0.ps1 -zip .\ESXi670-201912001.zip -pkgDir c:\VM\pkg -v67
Now download Rufus, select ISO and write to USB, boot off USB
Happy to find this!! Thank you!! Just installing it outside the ISO solved my issue. Hopefully it’ll make its way back into 6.7!!
The zip file should be ESXi670-201912001.zip going into the C:\VM folder
Also, add -nsc to the command line to stop errors from doing a signature check which stops ISO from being written.
.\ESXi-Customizer-PS-v2.6.0.ps1 -zip .\ESXi670-201912001.zip -pkgDir c:\VM\pkg -v67 -nsc
Also if you have any other package .zip or .vib files, just add them to the Pkg folder and it will add them also.
After updating NVMe back to ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253) driver, when starting to transfer data onto hard drive it goes inaccessible and 0 bytes
Anyone else had any luck fixing this issue
Have a Samsung 970 Pro 1TB M.2
Hi, anyone know of a 2TB NVMe that is consumer grade and doesn’t suffer from this issue with ESXi 6.7 U1 and later? Preferably not Samsung. Thanks.
I did some testing with a few M.2 SSDs I had on-hand to try to help other with this. Installing ESXi 6.7 U3b on an Intel NUC10i7FNH, I had the following results:
Samsung EVO 970 Plus (500GB)
Unknown firmware version (whatever it shipped with)
Detected, fully compatible
Corsair MP600 (1TB)
Firmware EGFM11.3 (Most recent as of 2020-03-14)
Note my test setup only supports PCIe 3.0, no idea if PCIe 4.0 impacts compatibility
Detected, fully compatible
ADATA XPG SX8200 Pro (2TB)
Firmware 42A4SANA (Most recent as of 2020-03-14)
NOT detected, NOT compatible
I’m disappointed that the ADATA didn’t work since the 2TB is only $269 USD and it’s a really good price for performance SSD. I specifically bought it to use in this ESXi build, but I’d rather return it for a compatible one. I did install the ADATA SSD in a Windows PC and it works fine there, just not in the latest ESXi update.
I’d love to hear reports of other compatible/incompatible SSD models!
I just bought the SX8200 pro on the island.
esxi did not see 6.5 u3 and 6.7 u3.
nvme drive saw that i installed old version. but there is a problem.
I am testing hdd speed in windows or other operating system
I see 3500mb / sec write and 3400mb / sec read values.
but i installed windows vm in esxi
I did the same test.
1500mb / s read gives 1300mb / s write value.
The adata xpg s40 product did not have such a resistance problem or speed issue.
I used windows vm in esxi while using xpg s40
I saw 3500mb / s write and 3400mb / s read values.
So when the driver sees the old version, he sees the SX8200 pro on the island, but the performance is halved. 🙁
note: the servers I’ve tested: dell r720 and r730 and esxi 6.5u3- 6.7u3
Has anyone tried this with ESXi 7.0? I have an SX8200 installed and working with the drive rollback on 6.7U3, and am holding off on 7.0 for now because of this.
Having the same issue with XPG8200 Pro and ESXi 7. Rolled back to 6.7 U3 with older nvme drivers. I hope if there is a way to fix this :/
thanks for testing that, have the same drive. hope there will be a fix soon.
To echo Michael, thanks for trying! 🙂 To confirm, you’re saying the 6.5U2 driver referenced in this fix doesn’t work on 7?
yeah exactly. I tried to install older nvme drivers at ESXi 7 as I did for 6.7 U3, but once ESXi 7 rebooted, I also noticed my nics couldn’t be detected. I will give it a try again this weekend with “esxcli software profile update” command William mentioned at his latest post. I’m using E200-8D by the way.
Good to know – I have the same motherboard series (X10SDV)
Still no news with 7.0 and SX8200? I’m facing same isssue.
Download ESXi 6.5 U2 Driver Rollup from my.vmware.com and copy nvme.v00 to \bootbank\nvme_pci.v00 (7.0) or \bootbank\nvme.v00 (6.7). I used SCP to do this. My SX8200 works!
I just made a request (for Adata SX8200Pro) via https://www.adata.com/en/support/online/# to add NGUID and EUI64 NVMe Identifier, just like https://www.crucial.com/support/ssd-support/p1-support
Please also make the same request to Adata. I doubt they will add.. unless lot of people use ESXi.
Did that – thanks for the hint. Though I couldn’t find the sx8200 as a SSD drive on the support page.
I need help, I have installed Esxi 7.0 on NUC10i7FNH installed with CORSAIR MP510 1.9TB PCIe NVMe Performance M.2 SSD, when i go in storage adapters it is showing me unknown and performance is not as it should have to be. Please can you suggest what drivers i can use to overcome this issue ?
Did you figure this out? I find my NVME drives in my NUC10 (with ESXi 6.7 and 7.0) are being claimed by AHCI and not working well at all.
Hi,
I try to install EVO 970 plus.
in ESXi 6.7u2 don’t see disks but I can see the controller(BOSS-S1), I tried the steps of old driver, but ESXi don’t see them in anyway.
Any idea?
Hi,
I have a Crucial MX 300 MVNe 275 Gb drive, and it is recognized by ESXi 7.0 (displayed under Storage as a local ATA Disk). It used to hold a Windows 10 installation in the same NUC7i3BNH. I have cleared partition table in ESXi. When I attempt to create a new data store, it fails with these log entries in wmkwarnong.log:
2020-06-29T17:07:10.197Z cpu2:526150 opID=33ff0c)WARNING: Vol3:
1410: Crucial 275 GB/5efa1fbe-aee2b6c2-0645-f44d306972ed: Error initializing JB SFD: I/O error
2020-06-29T17:07:10.197Z cpu2:526150 opID=33ff0c)WARNING: Vol3:
1651: Error creating system files: I/O error
I have tried the switch of 6.5U2 driver to \bootbank\nvme_pci.v00 and restarted, however it made no difference.
I have no reason to believe the drive is faulty as it booted Win10 just fine within the past couple of weeks.
Any idea what would cause this behavior?
Had to come back and share – updated a few more of my ESXi 6.7 instances to 7 and ran into my storage drivers not showing up again. Come to find out – the nvme_pci.v00 wasn’t listed in the /bootbank/boot.cfg file. (Could have been removed at some point?) I still had to reapply the old 6.5 driver (6.7 wouldn’t work…) – but seemed to help out. I do think I’m getting snagged by the consumer NVME drives that I’m using – as other drives work without the workaround. Of course – sent me into a bit of a panic hoping that I could scrounge up a fix!
1. Upload the following files to esxi “/tmp” directory
VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib
(Files can be download from http://221.126.249.9/nvme.7z)
2. SSH run the following commands to install 2 files:
esxcli software vib install -v /tmp/VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
esxcli software vib install -v /tmp/VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib
3. Reboot > Done !! Enjoy 🙂
Screen Dump:
————————————————————————–
login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
The time and date of this login have been sent to the system logs.
WARNING:
All commands run on the ESXi shell are logged and may be included in
support bundles. Do not provide passwords directly on the command line.
Most tools can prompt for secrets or accept them from standard input.
VMware offers supported, powerful system administration tools. Please
see http://www.vmware.com/go/sysadmintools for details.
The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[[email protected]:~] esxcli software vib install -v /tmp/VMware_bootbank_vmware-es
x-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.81 69922
VIBs Removed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2.48.130 06603
VIBs Skipped:
[[email protected]:~] esxcli software vib install -v /tmp/VMW_bootbank_nvme_1.2.1.3
4-1vmw.670.0.0.8169922.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922
VIBs Removed: VMW_bootbank_nvme_1.2.2.28-1vmw.670.3.73.14320388
VIBs Skipped:
[[email protected]:~]
Has anyone been able to try PNY XLR8 CS3030?
Seeking advice.
I am using ADATA XPG, and trying to patch my Lab-ESXi and run into this problem. (again)
Sadly it’s my boot drive, so the patching rolls back after 2nd reboot.. (somewhat magical that it even boots up to me)
Any one has similar encounter can post a solution?
Since you’re using a device that is NOT detected automatically as boot, you could try re-authoring the patch image using Image Builder (UI or CLI) to incorporate the 6.5 NVMe VIB. If not, you may need to consider installing ESXi on USB device
Thanks for replying William,
Saw this 1 hours late, i ended up installing on another drive, i will definitely look into authoring the patch image in future attempts.
I gave up, installed a old intel ssd to replace the NVME as boot drive.
I also tried ESXi 7.0 from USB earlier and it’s much easier to replace nvme_pci.v00 (renamed from nvme.v00).
For anyone who comes after me who is also new, here’s what i did
1. SCP to the host (you need to enable SSH on host 1st. I used WinSCP)
2. browse to bootbank and replace nvme.v00
@Webster IP:
I followed your method. That let me boot (as opposed to PSOD when I simply replaced the NVME file) but then I regressed back to “No network adapter found” (previously I created a USB stick with Intel 219LM (Jacksonville) drivers for the 10th gen Intel CPU motherboards) — I’m guessing that last part did it:
Step 1:
[[email protected]:~] esxcli software vib install -v /tmp/VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922
VIBs Removed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.37-1vmw.700.1.0.15843807
VIBs Skipped:
Step 2:
[[email protected]:~] esxcli software vib install -v /tmp/VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922
VIBs Removed: VMware_bootbank_native-misc-drivers_7.0.0-1.25.16324942
VIBs Skipped:
Does anyone have any advice for how to fix this? Yes, I can reimage the USB stick with the ISO I made, which then gives me back the NIC but I lose the NVME drivers required for my 8200 Pro. If I use this process to put the NVME drivers in the right place, I lose my ability to use my NIC. 🙁
Lenovo M80q. 10th gen Intel i7-10700T with 219LM (Jacksonville) NIC. 🙁
I tried ESXi Customizer v2.81 to do the following:
PS C:\esx> .\esxi-customizer281.ps1 .\ESXi7\ESXi-7.0b-16324942-standard.zip -pkgdir c:\esx\pkg -v70
This is ESXi-Customizer-PS Version 2.8.1 (visit https://ESXi-Customizer-PS.v-front.de for more information!)
(Call with -help for instructions)
Logging to C:\Users\X\AppData\Local\Temp\ESXi-Customizer-PS-1992.log …
Running with PowerShell version 5.1 and VMware PowerCLI version .. build
Adding base Offline bundle .\ESXi7\ESXi-7.0b-16324942-standard.zip … [OK]
Getting Imageprofiles, please wait … [OK]
Using Imageprofile ESXi-7.0b-16324942-standard …
(Dated 06/02/2020 17:26:43, AcceptanceLevel: PartnerSupported,
The general availability release of VMware ESXi Server 7.0b brings whole new levels of virtualization performance to datacenters and enterprises.)
Loading Offline bundles and VIB files from c:\esx\pkg …
Loading C:\esx\pkg\VMW_bootbank_ne1000-intelnuc_0.8.4-3vmw.670.0.0.8169922.vib … [OK]
Add VIB ne1000-intelnuc 0.8.4-3vmw.670.0.0.8169922 [OK, added]
Loading C:\esx\pkg\VMW_bootbank_nvme_1.2.1.34-1vmw.650.2.50.8294253.vib … [OK]
Add VIB nvme 1.2.1.34-1vmw.650.2.50.8294253 [OK, added]
Exporting the Imageprofile to ‘C:\esx\ESXi-7.0b-16324942-standard-customized.iso’. Please be patient …
…burning that tells me the NIC drivers cannot be found. However, if I then do exactly the same sequence, just without the NVME VIB, the NIC works great – but then, of course, the NVME VIB doesn’t install, and I cannot see my SX8200 Pro.
I tried just replacing the file per the 6/13/2020 post at the top of this thread, but the machine PSOD’d at bootup – could not verify drivers, exactly what Wissam Mahmassani wrote.
The pkg directory has two files:
-a—- 7/28/2020 9:22 AM 184184 VMW_bootbank_ne1000-intelnuc_0.8.4-3vmw.670.0.0.8169922.vib
-a—- 9/12/2020 10:40 AM 87418 VMW_bootbank_nvme_1.2.1.34-1vmw.650.2.50.8294253.vib
Help!? The stated goal: Working Lenovo M80Q Tiny, which has Intel 10th gen CPU with 219LM (Jacksonville) NIC on it, and an XPG (ADATA?) SX8200 Pro in it.
But wait, there’s more!
Running with PowerShell version 5.1 and VMware PowerCLI version .. build
Adding base Offline bundle .\ESXi7\ESXi-7.0b-16324942-standard.zip … [OK]
Getting Imageprofiles, please wait … [OK]
Using Imageprofile ESXi-7.0b-16324942-standard …
(Dated 06/02/2020 17:26:43, AcceptanceLevel: PartnerSupported,
The general availability release of VMware ESXi Server 7.0b brings whole new levels of virtualization performance to datacenters and enterprises.)
Loading Offline bundles and VIB files from C:\esx\pkg_nic_change\ …
Loading C:\esx\pkg_nic_change\VMW_bootbank_ne1000_0.8.4-3vmw.670.3.99.32543355.vib … [OK]
Add VIB ne1000 0.8.4-3vmw.670.3.99.32543355 [OK, replaced 0.8.4-10vmw.700.1.0.15843807]
Loading C:\esx\pkg_nic_change\VMW_bootbank_nvme_1.2.1.34-1vmw.650.2.50.8294253.vib … [OK]
Add VIB nvme 1.2.1.34-1vmw.650.2.50.8294253 [OK, added]
Exporting the Imageprofile to ‘C:\esx\ESXi-7.0b-16324942-standard-customized.iso’. Please be patient …
…. same result – no NIC found.
The change this time: I used the ne1000 843 VIB (ne1000 0.8.4-3vmw.670.3.99.32543355) rather than the newer VIB (ne1000-intelnuc 0.8.4-3vmw.670.0.0.8169922). The feedback from the ESXI-Customizer tool changed a bit ([OK, added] was the old feedback when the VIB was added; this time the feedback was [OK, replaced 0.8.4-10vmw.700.1.0.15843807]) but the end result was the same – NIC not found.
🙁
I face exactly the same problem with ADATA XPG SX8200 Pro and Intel I219-V Onboard NIC. If I add ne1000 VIB, the NIC is recognized by ESXi. However, If I add ne1000 and nvme VIBs, ESXi don’t recognize NIC.
I had struggled for several hours, and I gave up ESXi 7.0. Now, it works well on ESXi 6.7.