Supermicro kits such as the E200-8D is a very popular platform amongst the VMware community and with powerful Xeon-based CPUs and support for up to 128GB of memory, it is perfect for running a killer vSphere/vSAN setup!
Earlier this Fall, Supermicro released a "big daddy" version to the E200-8D, dubbed E300-9D and specifically, I want to focus on the 8-Core model (SYS-E300-9D-8CNTP) as this system actually listed on the VMware HCL for ESXi! The E300-9D can support up to half a terabyte of memory and with the 8-Core model, you have access to 16 threads. The E200-8D is also a supported platform by VMware, you can find the VMware HCL listing here.
I was very fortunate to get my hands on a loaner E300-9D (8-Core) unit, thanks to Eric and his team at MITXPC, a local bay area shop specializing in embedded solutions. In fact, they even provided a nice vGhetto promo discount code for my readers awhile back, so definitely check it out if you are in the market for a new lab. As an aside, when doing a quick search online, they also seem to be the only ones actually selling the E300-9D (8-Core) system which you can find here and in general, they seem to be priced fairly competitively. This is not an endorsement for MITXPC, but recommend folks to compare all prices when shopping online, especially as today is Black Friday in the US and Cyber Monday is just a few days away.
When I first heard about the E300-9D platform, I was excited for the platform as it listed support for up to two M.2 devices, which would match what the Intel NUC Hades Canyon supported. The other reason is that there are actually a number of supported M.2 NVMe devices listed on VMware's HCL which you can find here. This is actually greats news for those in the market for a small form factor systems that is officially supported by VMware, especially useful for ROBO or Edge-based deployment where space and cost is usually a constraint.
However, upon closer inspection, it looks like the system only supports a single M.2 (M-Key) which is the traditional M.2 SSD devices you would purchase for a platform like the Intel NUC (lower left hand corner of the picture). Support for the other M.2 device is actually using the B-Key form factor (immediately to the right of the M-Key in the picture) and this would be SATA based rather than PCIe.
Having said that, thanks to David Chung who was the first to share the news, you can easily add an M.2 to PCIe add-on card enabling a secondary M.2 NVMe (M-Key) device for just $13 bucks. Here is the Amazon listing, in case folks are interested. This add-on card fits nicely in both the E200-8D and E300-9D (pictured above) and this will give you another M.2 NVMe using PCIe!
With this, you can add 2nd #NVMe drive on #Supermicro E200-8D or any other low profile 1U server. You can run all-NVMe high performance #vSAN #HomeLab in a very small form factor. Picked up few from Amazon and it's running great in my #SDDCinaBox #vmware #HCI #SDDC pic.twitter.com/9pUAC3VK37
— David Chung (@dchung615) September 28, 2018
Several folks have also shared that Supermicro makes a PCIe add-on card that supports dual M.2 that can also be used for these 1U low profile systems, which is nice from a support standpoint with just having a single vendor for the system itself.
Once ESXi is installed, both of my NVMe M.2 devices were automatically detected as shown in the output below. I also used a SATADOM for installing ESXi (pictured upper right hand corner) which not only gives me a higher endurance device, but it also provides a place for me to store vSAN tracefiles and coredumps.
The networking on the E300-9D has also been significantly upgraded compared to the E200-8D which comes with 1 x 1GbE for the BMC, 2 x 1GbE and 2 x 10GbE (RJ45 or SFP+). The E300-9D includes the standard 1 x GbE for BMC, but now includes 4 x 1GbE, 2 x 10GbE (RJ45) and 2 x 10GbE (SFP+), this should be plenty of ports for anyone wanting to do some serious NSX networking!
- 4 x 1GbE is based on Intel I350 and uses the igbn driver
- Both 2 x 10GbE (RJ45) and 2 x 10GbE (SFP+) is based on Intel X722 and uses the i40en driver
All network adapters were also automatically detected by ESXi, no additional drivers or tricks are required.
Compared to the E200-8D, it is definitely a bit more louder when the system boots up but then idles out with a normal hum. Paul over at TinkerTry did a review of the 4-Core system which includes noise analysis which you can check out here. Zack Widing who also works for VMware has been spending quite a bit of time working with the E200-8D, he recently shared on Twitter that you can purchase Noctura silent fans that work nicely with both E200/E300 systems.
Compared to its little brother E200-8D, the E300-9D is definitely bigger but trying to get a sense of how large the unit is not always an easy thing if you do not have it in front of you. I thought it would be nice to compare the new E300-9D to a number of other systems the community is already familiar with.
Top to bottom:
- Intel NUC 6th Gen
- Intel NUC Hades Canyon
- Apple Mac Mini 5,3
- Supermicro E200-8D
- Supermicro E300-9D
- 1 x Supermicro E300-9D (SYS-E300-9D-8CN8TP) which MITXPC sells here. (vGhetto Discount Code)
- 1 x M.2 to PCIe Add-On Card
- Any supported LRDIMM memory (I had 128GB from a previous setup using Samsung 32GB DIMM)
- Any supported M.2 NVMe storage (I had 2 x WD which I had pulled from my Hades Canyon setup)
The E300-9D is definitely a powerful platform that packs a ton of resources in a relatively small amount of space. Depending on your goals, it may be a nice home lab setup (but I suspect it might be out of most folks budget) but where a system like this really shines would be for the Enterprise, especially for ROBO/Edge-based deployments. The other huge benefit is that this system is on VMware's HCL and is fully supported including networking and storage! If you are in the market for a small form factor system that packs a punch, definitely take a look at this system and who knows, this might even make a nice vSAN Ready Node if you ask me 😀
UPDATE (01/14/19) - Using the Supermicro AOC, you can get two additional M.2 devices, including Intel's latest M.2 4801x Optane devices. For more details, please take a look at this blog post here.
The E300-9D also comes with two U.2 connectors (located at the very back of the board pictured above) which in theory can then be connected to supported SSD devices for use with vSAN. However, looking at the space within the chassis, it might be difficult to get two standard devices inside, at least with no proper mounting which maybe a concern for Production usage. I know some folks were discussing the potential of mounting the devices outside via an external enclosure but I think that is probably more for home labs usage than having something that can just be shipped out to remote locations and just simply work. In any case, we have reached out to a partner to see if we can get access to supported SSD devices and using the U.2 slots. I will update this blog post if I make any progress in this area.
Thanks to our friends over at Intel, I was able to get my hands on a couple of the Intel Optane P4800X Series drives (2.5"). You will need to install the Intel NVMe driver, which you can find the latest version for ESXi 6.7 Update 1 here.
— William Lam (@lamw) December 1, 2018
Although I was able to get vSAN up and running using the Optane drive in the E300-9D, as you can see from picture below, there is simply no room to fit both the drive + cable (which also requires an additional power cable), yet alone trying to close up the case.
Oh yeaaaa, vSAN running on Intel Optane!
Good news is the #E300-9D can use either U.2 or M.2 cable (uses up M.2 slot)
Bad news, there’s no room in E300-9D to fit all these cables, drive + close case 🙁 pic.twitter.com/SaVHnq03ZB
— William Lam (@lamw) December 1, 2018
For both E200-8D and E300-9D, the recommendation is definitely to stick with the PCIe AOC riser card and using two standard M.2 (M-Key) which you can verify using the VMware HCL.