Customizing the ESXi DCUI to show number of VMs

Last week there was a question that was posted internally asking if it was possible to customize the ESXi DCUI screen to include the number of Virtual Machines? Although there is nothing out of the box, you can in fact add add almost anything to the DCUI screen by modifying the /etc/vmware/welcome configuration file which I had blogged about several years back on adding a splash of color to the ESXi DCUI. There was even a recent VMware Fling that provides a VIB that applies a variety of DoD STIG implementations, one of which was to update the DCUI screen with some specific text.

However, instead of having to manually edit the file directly on the ESXi host, we also provide an API in the way of an ESXi Advanced Setting called Annotations.WelcomeMessage which can then be updated remotely using anyone of the vSphere SDK/CLIs that you are familiar with.

Here is an example PowerCLI snippet to connect to an ESXi host (you can also do this by connecting to vCenter Server) and extracting all Virtual Machines residing on the host and then updating the DCUI screen with the total number of VMs as well as the names of each VM. Obviously, if you have more than 10 or so VMs, it may not make much sense to actually list them as it will just run off the screen, but this just gives you an example of some of the things you can do leveraging the vSphere API or any other data that you might have at your disposal.

Note: If you wish to only filter out powered on Virtual Machines, you can add the following: -Filter @{"Runtime.PowerState" = "PoweredOn"} to the end of the Get-View command

Here is a screenshot quick screenshot of what this would look like on the DCUI screen of your ESXi host:

displaying_vms_in_esxi_dcui_1
Note: You can actually run the DCUI over SSH, which can be useful for testing purposes without having to directly login to the console remotely. Take a look at this blog post here for more details.

One final thing to be aware of when editing the welcome message on your ESXi host is that it will also be displayed in the ESXi Embedded Host Client login page as shown in the screenshot below. Just something to be aware of in case you plan to make any sensitive information available as this can be seen without needing to login to the ESXi host (just like the DCUI interface).

displaying_vms_in_esxi_dcui_2

ESXi on the new Intel NUC Skull Canyon

Earlier this week I found out the new Intel NUC "Skull Canyon" (NUC6i7KYK) has been released and have been shipping for a couple of weeks now. Although this platform is mainly targeted at gaming enthusiast, there have also been a lot of anticipation from the VMware community on leveraging the NUC for a vSphere based home lab. Similiar to the 6th Gen Intel NUC system which is a great platform to run vSphere as well as VSAN, the new NUC includes a several new enhancements beyond the new aesthetics. In addition to the Core i7 CPU, it also includes a dual M.2 slots (no SATA support), Thunderbolt 3 and most importantly, an Intel Iris Pro GPU. I will get to why this is important ...
intel_nuc_skull_canyon_1
UPDATE (05/23/16) - Shortly after sharing this article internally, Jason Joy, a VMware employee shared the great news that he has figured out how to get ESXi to properly boot and install. Jason found that by disabling unnecessary hardware devices like the Consumer IR/etc in the BIOS, it allowed the ESXi installer to properly boot up. Jason was going to dig a bit further to see if he can identify the minimal list of devices that needed to be disabled to boot ESXi. In the meantime, community blogger Erik Bussink has shared the list of settings he has applied to his Skull Canyon to successfully boot and install latest ESXi 6.0 Update 2 based on the feedback from Jason. Huge thanks to Jason for quickly identifying the workaround and sharing it with the VMware community and thanks to Erik for publishing his list. For all those that were considering the new Intel NUC Skull Canyon for a vSphere-based home lab, you can now get your ordering on! 😀

Below is an except from his blog post Intel NUC Skull Canyon (NUC6I7KYK) and ESXi 6.0 on the settings he has disabled:

BIOS\Devices\USB

  • disabled - USB Legacy (Default: On)
  • disabled - Portable Device Charging Mode (Default: Charging Only)
  • not change - USB Ports (Port 01-08 enabled)

BIOS\Devices\SATA

  • disabled - Chipset SATA (Default AHCI & SMART Enabled)
  • M.2 Slot 1 NVMe SSD: Samsung MZVPV256HDGL-00000
  • M.2 Slot 2 NVMe SSD: Samsung MZVPV512HDGL-00000
  • disabled - HDD Activity LED (Default: On)
  • disabled - M.2 PCIe SSD LEG (Default: On)

BIOS\Devices\Video

  • IGD Minimum Memory - 64GB (Default)
  • IGD Aperture Size - 256 (Default)
  • IGD Primary Video Port - Auto (Default)

BIOS\Devices\Onboard Devices

  • disabled - Audio (Default: On)
  • LAN (Default)
  • Thunderbolt Controller (Default)
  • disabled - WLAN (Default: On)
  • disabled - Bluetooth (Default: On)
  • Near Field Communication - Disabled (Default is Disabled)
  • SD Card - Read/Write (Default was Read)
  • Legacy Device Configuration
  • disabled - Enhanced Consumer IR (Default: On)
  • disabled - High Precision Event Timers (Default: On)
  • disabled - Num Lock (Default: On)

BIOS\PCI

  • M.2 Slot 1 - Enabled
  • M.2 Slot 2 - Enabled
  • M.2 Slot 1 NVMe SSD: Samsung MZVPV256HDGL-00000
  • M.2 Slot 2 NVMe SSD: Samsung MZVPV512HDGL-00000

Cooling

  • CPU Fan HEader
  • Fan Control Mode : Cool (I toyed with Full fan, but it does make a lot of noise)

Performance\Processor

  • disabled Real-Time Performance Tuning (Default: On)

Power

  • Select Max Performance Enabled (Default: Balanced Enabled)
  • Secondary Power Settings
  • disabled - Intel Ready Mode Technology (Default: On)
  • disabled - Power Sense (Default: On)
  • After Power Failure: Power On (Default was stay off)

Over the weekend, I had received several emails from folks including Olli from the nucblog.net (highly recommend a follow if you do not), Florian from virten.net (another awesome blog which I follow & recommend) and few others who have gotten their hands on the "Skull Canyon" system. They had all tried to install the latest release of ESXi 6.0 Update 2 including earlier versions but all ran into a problem while booting up the ESXi installer.

The following error message was encountered:

Error loading /tools.t00
Compressed MD5: 39916ab4eb3b835daec309b235fcbc3b
Decompressed MD5: 000000000000000000000000000000
Fatal error: 10 (Out of resources)

intel_nuc_skull_canyon_2
Raymond Huh was the first individual who had reach out to me regarding this issue and then shortly after, I started to get the same confirmations from others as well. Raymond's suspicion was that this was related to the amount of Memory-Mapped I/O resources being consumed by the Intel Iris Pro GPU and does not leave enough resources for the ESXi installer to boot up. Even a quick Google search on this particular error message leads to several solutions here and here where the recommendation was to either disable or reduce the amount of memory for MMIO within the system BIOS.

Unfortunately, it does not look like the Intel NUC BIOS provides any options of disabling or modifying the MMIO settings after Raymond had looked which including tweaking some of the video settings. He currently has a support case filed with Intel to see if there is another option. In the mean time, I had also reached out to some folks internally to see if they had any thoughts and they too came to the same conclusion that without being able to modify or disable MMIO, there is not much more that can be done. There may be a chance that I might be able to get access to a unit from another VMware employee and perhaps we can see if there is any workaround from our side, but there are no guarantees, especially as this is not an officially supported platform for ESXi. I want to thank Raymond, Olli & Florian for going through the early testing and sharing their findings thus far. I know many folks are anxiously waiting and I know they really appreciate it!

For now, if you are considering purchasing or have purchased the latest Intel NUC Skull Canyon with the intention to run ESXi, I would recommend holding off or not opening up the system. I will provide any new updates as they become available. I am still hopeful  that we will find a solution for the VMware community, so crossing fingers.

Support your Virtualization Bloggers by voting for Top vBlog 2016

It is that time of the year again, Eric Siebert who runs the popular vSphere-land.com website has just opened up the voting polls for the Top 25 Virtualization Blogs of 2016. There are over 300+ bloggers this year and it is a very impressive list! Here is your chance to show your support for your favorite bloggers by casting a vote which only takes a few minutes. Before voting, be sure to check out Eric's blog post on the criteria's you should consider when voting such as Longevity, Length, Frequency & Quality.

Lastly, I want to thank Eric for all of his hard work for putting this together year after year. I know he spends an enormous amount of time and energy to make this happen and make sure to support Eric and his sponsors by visiting their sites as this would not be possible without them. Happy voting!

20150211_203831-small

Vote now!

Generating vCenter Server & Platform Services Controller deployment topology diagrams

A really useful capability that vCenter Server used to provide was a feature called vCenter Maps. I say "used to" because this feature was only available when using the vSphere C# Client and was not available in the vSphere Web Client. vCenter Maps provided a visual representation of your vCenter Server inventory along with the different relationships between your Virtual Machines, Hosts, Networks and Datastores. There were a variety of use cases for this feature but it was especially useful when it came to troubleshooting storage or networking connectivity. An administrator could quickly identify if they had an ESXi host that was not connected to the right datastore for example with just a few clicks.

vcenter_server_and_platform_services_controller_topology_diagram_3
Although much of this information can be obtained either manually or programmatically using the vSphere API, the consumption of this data can sometimes be more effective when it is visualized.

I was recently reminded of the vCenter Maps feature as I have seen an increase in discussions around the different vSphere 6.0 deployment topology options. This is an area where I think we could have leveraged visualizations to provide a better user experience to help our customers understand what they have deployed as it relates to install, upgrade and expansion of their vSphere environment. Today, this information is spread across a variety interfaces ranging from the vSphere Web Client (here and here) as well as across different CLIs (here and here) and there is nothing that aggregates all of this dispart information into an easy to consume manner. Collecting this information can also be challenging as you scale up the number of environments you are managing or dealing with complex deployments that can also span multiple sites.

Would it not be cool if you could easily extract and visualize your vSphere 6.0 deployment topology? 🙂

Well, this was a little side project I recently took up. I have created a small python script called extract_vsphere_deployment_topology.py that can run on either a Windows Platform Services Controller (PSC) or a vCenter Server Appliance (VCSA) PSC and from that system extract the current vSphere deployment topology which includes details about the individual vCenter Servers, SSO Sites as well as the PSC replication agreements. The result of the script is outputted in the DOT format, a popular graph description language which can then be used to generate a diagram like the example shown below.vcenter_server_and_platform_services_controller_topology_diagram_0Requirements:

  • vSphere 6.0 environment
  • Access to either a Windows or VCSA PSC as a System Administrator
  • SSO Administrator credentials

Step 1 - Download the extract_vsphere_deployment_topology.py python script to either your Windows vCenter Server PSC or vCenter Server Appliance (VCSA) PSC.

Step 2 - To run on a vCenter Server Appliance (VCSA) PSC, you will need to first set the script to an executable by running the following command:

chmod +x extract_vsphere_deployment_topology.py

To run on a vCenter Server for Windows PSC, you will need to first update your environmental PATH variable to include the python interpreter. Follow the directions here if you have never done this before and add C:\Program Files\VMware\vCenter Server\python

Step 3 - The script requires that you provide an SSO Administrator username and password. You can specify everything in the command-line or you omit the password in which you would then be prompted to enter.

To run the script on a VCSA PSC, run the following command specifying your credentials:

./extract_vsphere_deployment_topology.py  -u administrator@vghetto.local -p VMware1!

To run the script on Windows VC PSC, run the following command specifying your credentials:

python C:\Users\primp\Desktop\extract_vsphere_deployment_topology.py  -u administrator@vsphere.local -p VMware1!

Here is an example output from one of my environments.

Step 4 - Save the output from the script and then open a browser that has internet access to the following URL: http://www.webgraphviz.com Paste the output and then click on the "Generate Graph" which will generate a visual diagram of your vSphere deployment. Hopefully it is pretty straight forward to understand and I have also colorized the nodes to represent the different functionality such as Blue for a vCenter Server and Green for Platform Services Controller.

vcenter_server_and_platform_services_controller_topology_diagram_4
In addition, if you have deployed an Embedded vCenter Server which is replicating with an External PSC (which is considered a deprecated topology and will not be supported in the future), you will notice the node is colored Orange instead as seen in the example below.

vcenter_server_and_platform_services_controller_topology_diagram_1
This is pretty cool if you ask me! 😀 Just imagine the possibilities if you could use such an interface to also manage operations across a given vSphere deployment when it comes to install, upgrade and expansion of your existing environment. What do you think, would this be useful?

I have done a limited amount of testing across Windows and the VCSA using a couple of deployment scenarios. It is very possible that I could have missed something and if you are running into issues, it would be good to provide some details about your topology to help me further troubleshoot. I have not done any type of testing using load balancers, so it is very likely that the diagram may not be accurate for these scenarios but I would love to hear from folks if you have tried running the script in such environments.

Test driving VMware Photon Controller Part 3c: Deploying Docker Swarm

In this final article, we will now take a look at deploying a Docker Swarm Cluster running on top of Photon Controller.

test-driving-photon-controller-docker-swarm-cluster
A minimal deployment for a Docker Swarm Cluster consists of 3 Virtual Machines: 1 Masters, 1 etcd, 1 Slave. If you only have 16GB of memory on your ESXi host, then you will need override the default VM Flavor used which is outlined in Step 1. If you have more than 16GB of memory, then you can skip Step 1 and move directly to Step 2.

Deploying Docker Swarm Cluster

Step 1 -If you have not already created a new cluster-tiny-vm VM Flavor from the previous article that consists of 1vCPU/1GB memory, please run the following command:

./photon -n flavor create --name cluster-tiny-vm --kind "vm" --cost "vm 1 COUNT,vm.flavor.cluster-other-vm 1 COUNT,vm.cpu 1 COUNT,vm.memory 1 GB,vm.cost 1 COUNT"

Step 2 - Download the Swarm VMDK from here

Step 3 -We will now upload our Swarm image and make a note of the ID that is generated after the upload completes by running the following command:

./photon -n image create photon-swarm-vm-disk1.vmdk -n photon-swarm-vm.vmdk -i EAGER

Step 4 - Next, we will also need the ID of our Photon Controller Instance deployment as it will be required in the next step by running the following command:

./photon deployment list

Step 5 - We will now enable the Docker Swarm Cluster Orchestration on our Photon Controller instance by running the following command and specifying the ID of your deployment as well as the ID of the Swarm image from the previous two steps:

./photon -n deployment enable-cluster-type cc49d7f7-b6c4-43dd-b8f3-fe17e6648d0f -k SWARM -i 13ae437d-3fd1-48a3-9d14-287b9259cbad

test-driving-photon-controller-docker-swarm-0
Step 6 -We are now ready to spin up our Docker Swarm Cluster by simply running the following command and substituting the network information from your environment. We are going to only deploying a single Swarm Slave (if you have additional resources you can spin up more or you can always re-size the cluster after it has been deployed). Do not forget to override the default VM Flavor used by specifying -v option and providing the name of our VM Flavor which we had created earlier called cluster-tiny-vm. You can just hit enter when prompted for the two zookeeper IP Addresses.

./photon cluster create -n swarm-cluster -k SWARM --dns 192.168.1.1 --gateway 192.168.1.1 --netmask 255.255.255.0 --etcd1 192.168.1.45 -s 1 -v cluster-tiny-vm

test-driving-photon-controller-docker-swarm-1
Step 7 - The process can take a few minutes and you should see a message like the one shown above which prompts you to run the cluster show command to get more details about the state of the cluster.

./photon cluster show 276b6934-6eb5-42fd-9fb1-031e311b3c45

test-driving-photon-controller-docker-swarm-2
At this point, you have successfully deployed a Docker Swarm Cluster running on Photon Controller. What you will be looking for in this screen is the IP Address of the Master VM which we will need in the next section if you plan to explore Docker Swarm a bit more.

Exploring Docker Swarm

To interact with your newly deployed Docker Swarm Cluster, you will need to ensure that you have a Docker client that matches the Docker version running the Docker Swarm Cluster which is currently today 1.20. The easiest way is to deploy PhotonOS 1.0 TP2 using either an ISO or OVA.

To verify that you have the correct Docker client version, you can just run the following command:

docker version

test-driving-photon-controller-docker-swarm-5
Once you have verified that your Docker Client matches the version, we will go ahead and set the DOCKER_HOST variable to point to the IP Address of our Master VM which you can find above in Step 7. When you have identified the IP Address, go ahead and run the following command to set variable:

export DOCKER_HOST=tcp://192.168.1.105:8333

We can run the following command to list the Docker Containers running for our Docker Swarm Cluster:

docker ps -a

test-driving-photon-controller-docker-swarm-3
Lets go ahead and download a Docker Container which we can then use to run on our Docker Swarm Cluster. We will download the VMware PhotonOS Docker Container by running the following command:

docker pull vmware/photon

Once the Docker Container has been downloaded, we can then run it by specifying the following command:

docker run --rm -it vmware/photon

test-driving-photon-controller-docker-swarm-6
For those familiar with Docker, you can see how easily it is to interact with the Docker interface that you are familiar with. Underneath the hood, Photon Controller is automatically provisioning the necessary infrastructure needed to run your applications. This concludes our series in test driving VMware's Photon Controller. If you have made it this far, I hope you have enjoyed the series and if you have any feedback or feature enhancements on Photon Controller, be sure to file an issue on the Photon Controller Github page.