virtuallyGhetto

ESXi

How to Enable Nested vFT (virtual Fault Tolerance) in vSphere 5

by 5 Comments

The ability to enable virtual Fault Tolerance in nested virtual machines running in vESX(i) is not a new feature in vSphere 5, vFT has been an unsupported feature since vSphere 4 and was initially identified by Simon Gallagher. The process is exactly the same in vSphere 5 in which three virtual machine configuration options need to be configured for the virtual machine to be enabled with FT, not the vESXi VM.

replay.supported = "true"
replay.allowFT = "true"
replay.allowBTOnly = "true"

During the beta of vSphere 5, I did enable vFT but on an offline virtual machine to conserve on unnecessary compute resources. Today there was a question on the beta community around configuring vFT for vSphere 5 and I wanted to quickly validate the configurations still hold true. I ran into a interesting error when trying to enable vFT, the power on process for the secondary virtual machine failed with the following error:

This was not an error I had seen before in vSphere 4 and looking at the vmkernel and vmware.log files, I noticed the following:

2011-07-31T17:31:39.314Z| vcpu-0| [vob.vmotion.stream.keepalive.read.fail] vMotion migration [ac1e0050:1312133702562144] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout
2011-07-31T17:31:39.314Z| vcpu-0| [msg.checkpoint.precopyfailure] Migration to host <> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).
2011-07-31T17:31:39.324Z| vcpu-0| Migrate: secondary failure during migration: error Connection closed by remote host, possibly due to timeout.

I tried changing the advanced option on the vESX(i) host to increase the vMotion timeout but continued to hit the same error. I decided to look more into the first error message "failed to read stream keepalive" and found an advanced ESX(i) setting called /Migrate/VMotionStreamDisable, this advanced option has been available since ESX(i) 4.x.

I decided to disable vMotion Stream and to my surprised, it allowed FT to power on the secondary virtual machine and no longer ran into that error.

Note: You may or may not run into this error message and the configuration may not be necessary. If you enable vFT on an offline VM, you should not have any issues as long as you meet the minimum Fault Tolerance requirements.

You can configure the advanced ESXi option using either esxcli or legacy esxcfg-advcfg commands:

esxcli system settings advanced set -o /Migrate/VMotionStreamDisable -i 0
esxcfg-advcfg -s 0 /Migrate/VMotionStreamDisable

It is important to understand that even though one can setup a vESX(i) hosts and test and play with some of the advanced functionality such as vMotion and FT that the actual behavior is unpredictable as these configurations are unsupported by VMware. This of course is also great feature for home labs and studying for VMware certifications such as VCP and VCAP-DCA, but that should be the extent of leveraging these unsupported configurations.

How to Add a Splash of Remote Color to ESXi Shell

by 6 Comments

This morning I noticed a very interesting retweet by fellow vExpert Wil van Antwerpen from another vExpert: Richard Cardona (You may know him as rcardona2k on the VMTN Community Forums) about a neat little trick with the use of remote ESXi Shell (previous known as remote TSM).

For those of you who login remotely via SSH to the ESXi Shell (previously known as unsupported mode and Tech Support Mode) know that you can run the DCUI utility remotely by just typing "dcui". The remote DCUI works just like it does using the direct console, with the exception of displaying the famous yellow and black screen that we are familiar with.

Richard came upon a neat little trick by setting the terminal type to "linux" from the default "xterm" that the yellow and black can be enabled when using the remote DCUI.

Before launching DCUI utility, you will need to run the following command on the ESXi Shell:

export TERM=linux

Next you will just type "dcui" and hit enter

Here is an example of running remote DCUI in color on ESXi 5

Here is an example of running remote DCUI in color on ESXi 4.1

Note: As you can see this is not a new trick in vSphere 5, but has been there since 4.x days but one big change with vSphere 5 is the full resolution of DCUI which many have complained about in the past.

If you are interested in other ways of customizing the DCUI, take a look at this blog post How to add a splash of color to ESXi DCUI Welcome Screen

Don't forget to play some cool soundtrack music when using the DCUI 😉

How to Format and Create VMFS5 Volume using the CLI in ESXi 5

by 38 Comments

VMware always recommends formatting and creating a new VMFS volume using the vSphere Client as it automatically aligns your VMFS volume. However, if you do not have access to the vSphere Client or you wanted to format additional VMFS volumes via a kickstart, you can do so using the CLI and the partedUtil under /sbin.

~ # /sbin/partedUtil
Not enough arguments
Usage:
Get Partitions : get
Set Partitions : set ["partNum startSector endSector type attr"]*
Delete Partition : delete Resize Partition : resize
Get Partitions : getptbl
Set Partitions : setptbl

With ESXi 5, an MBR (Master Boot Record) partition table is no longer used and has been replaced with a GPT (GUID Partition Table) partition table. There is also only one block size of 1MB versus the 2,4 and 8 that was available in ESX(i) 4.x

We can view the partitions of a device by using the "getptbl" option and ensure we don't have an existing VMFS volume:

~ # /sbin/partedUtil "getptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0"
gpt
652 255 63 10485760

Next we will need to create a partition by using the "setptbl" option:

/sbin/partedUtil "setptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0" "gpt" "1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0"

The "setptbl" accepts 3 arguments:

  • diskName
  • label
  • partitionNumber startSector endSector type/GUID attribute

The diskName in this example is the full path to the device which is /vmfs/devices/disks/mpx.vmhba1:C0:T2:L0

The label will be gpt

The last argument is actually a string comprised of 5 individual parameters:

  • partitionNumber - Pretty straight forward
  • startSector - This will always be 2048 for 1MB alignment for VMFS5
  • endSector - This will need to be calculated based on size of your device
  • type/GUID - This is the GUID key for a particular partition type, for VMFS it will always be AA31E02A400F11DB9590000C2911D1B8

To view all GUID types, you can use the "showGuids" option:

~ # /sbin/partedUtil showGuids
Partition Type       GUID
vmfs                 AA31E02A400F11DB9590000C2911D1B8
vmkDiagnostic        9D27538040AD11DBBF97000C2911D1B8
VMware Reserved      9198EFFC31C011DB8F78000C2911D1B8
Basic Data           EBD0A0A2B9E5443387C068B6B72699C7
Linux Swap           0657FD6DA4AB43C484E50933C84B4F4F
Linux Lvm            E6D6D379F50744C2A23C238F2A3DF928
Linux Raid           A19D880F05FC4D3BA006743F0F84911E
Efi System           C12A7328F81F11D2BA4B00A0C93EC93B
Microsoft Reserved   E3C9E3160B5C4DB8817DF92DF00215AE
Unused Entry         00000000000000000000000000000000

Once you have the 3 arguments specified, we can now create the partition:

~ # /sbin/partedUtil "setptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0" "gpt" "1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0"
gpt
0 0 0 0
1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0

UPDATE (01/15) - Here is a quick shell snippet that you can use to automatically calculate End Sector as well as creating the VMFS5 volume:

partedUtil mklabel ${DEVICE} msdos
END_SECTOR=$(eval expr $(partedUtil getptbl ${DEVICE} | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1)
/sbin/partedUtil "setptbl" "${DEVICE}" "gpt" "1 2048 ${END_SECTOR} AA31E02A400F11DB9590000C2911D1B8 0"
/sbin/vmkfstools -C vmfs5 -b 1m -S $(hostname -s)-local-datastore ${DEVICE}:1

Note: You can also use the above to create a VMFS-based datastore running on a USB device, however that is not officially supported by VMware and performance with USB-based device will vary depending on the hardware and the speed of the USB connection. 

We can verify by running the "getptbl" option on the device that we formatted:

~ # /sbin/partedUtil "getptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0"
gpt
652 255 63 10485760
1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Finally we will now create the VMFS volume using our favorite vmkfstools, the syntax is the same as previous release of ESX(i):

~ # /sbin/vmkfstools -C vmfs5 -b 1m -S himalaya-SSD-storage-3 /vmfs/devices/disks/mpx.vmhba1:C0:T2:L0:1
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs5 file system on "mpx.vmhba1:C0:T2:L0:1" with blockSize 1048576 and volume label "himalaya-SSD-storage-3".
Successfully created new volume: 4dfdb7b0-8c0dcdb5-e574-0050568f0111

Now you can refresh the vSphere Client or run vim-cmd hostsvc/datastore/refresh to view the new datastore that was created.

How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5

by 65 Comments

With the release of vSphere 5, one of the most sought out feature from VMware is the ability to run nested 64bit and Hyper-V guest virtual machines in a virtual ESXi instance. Previous to this, only 32bit virtual machines were supported as the VT-x/AMD-V Hardware Virtualization CPU instructions could not be virtualized and presented to the virtual ESX(i) guest. This feature is quite useful for home and lab setups in testing new features or studying for VMware certifications and running multiple vESX(i) instances.

You will still be required to have a 64bit capable system and CPU and you will need to be running ESXi 5.0, this will not work for ESX(i) 4.x or older.

The above diagram depicts the various levels of inception where pESXi is your physical ESXi 5.0 hosts. We then create a vESXi 5.0 host which will contain the necessary Hardware Virtualization CPU instructions to support a 64bit nested guest OS which I've created as another ESXi host called vvESXi.
Note: You will not be able to run a 4th level nested 64bit VM (I have tried by further passing the HV instructions in the nested guest) and it will just boot up and spin your CPUs for hours.
This feature by default is disabled in ESXi 5.0, to enable this virtualized HV (Hardware Virtualization) you will need to add the following string vhv.allow = "TRUE"  to /etc/vmware/config of your Physical ESXi 5.0 host
Once the configuration change has been made, the feature goes into effect right away. A reboot of the system is not necessary. To verify, you should now be able to power on a 64bit guest OS and see that the HV instructions bits are being passed into the guestOS which will then allow you to run a nested 64bit guestOS. You can also verify by looking in the vmware.log file of the virtual machine and grep for the string "monitorControl.vhv" and if you see the following message, then Virtualized HV is not enabled.
In the past to run a virtual ESX(i) instance, a few advanced .vmx configuration entries were needed as documented here. With ESXi 5.0, if you are using virtual hardware version 8, then you do not need to make any additional changes. If you are using hardware version 4 or 7, then you will need to add a few changes to the VM's configuration file.
Creating vESXi 5.0 Instance using Hardware Version 8:
1. To create a virtual ESXi 5.0 instance, start off by just creating a standard RHEL5/6 64bit VM using the vSphere Client
2. Once the VM has been created, edit the settings of the VM and change over to the "Options" and now have the ability to select a new guestOS type: VMware ESXi 5.x or VMware ESXi 4.x under the "Other" section.

Note: I'm not sure why these two additional guestOS type is not available from the default creation menu, but are available after the initial VM shell is created.

3. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance as denoted from the picture below

Creating vESXi 5.0 Instance using Hardware Version 4/7:
1. To create a virtual ESXi 5.0 instance, start off by just creating a standard RHEL5/6 64bit VM using the vSphere Client

2. Now you will need to add the following advanced .vmx parameter:  monitor.virtual_exec = "hardware" which can be done through the vSphere Client and/or editing the .vmx parameter manually.

3. Next you will need to add some cpuid bits, depending if you are running an Intel or AMD CPU, the respective entries are required:

Intel Hosts:

cpuid.1.ecx = "----:----:----:----:----:----:--h-:----"

AMD Hosts:

cpuid.80000001.ecx.amd = "----:----:----:----:----:----:----:-h--"
cpuid.8000000a.eax.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.ebx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.edx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"

4. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance

By using a VM that is hardware version 8, you can easily automate the creation of vESXi 5.0 instance by changing the guestOS string in the .vmx parameter to "vmkernel" and the above configurations other than "vhv" string needed for either an Intel or AMD system are automatically configured.

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.  

Creating a vHyper-V  Instance on physical ESXi 5.0:
1. To create a virtual Hyper-V instance, start off by creating a Windows 2008 Server R2 64bit VM using the vSphere Client

2. If you are using Hardware Version 7, you will need to follow the instructions in "Creating vESXi 5.0 Instance using Hardware Version 4/7" to add the additional parameters to the VM. If you are using Hardware Version 8, you just need to change the guestOS type to VMware ESXi 5.0

3. You need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE

4. You are now ready to install Hyper-V in a virtual machine and you can also spin up nested 64bit guestOSes in this virtual Hyper-V instance.

As you can see, now you can even run Hyper-Crap, err I mean Hyper-V as a virtualized guest under ESXi 5.0. I did not get a chance to try out Xen, but I'm sure with the ability to virtualize the Hardware Virtualization instructions, you should be able to run other types of hypervisors for testing purposes.

This is a really awesome feature but note that this is not officially supported by VMware, use at your own risk.

Semi-Interactive automated ESXi installation

by 18 Comments

There was a recent thread in the VMTN community forums where a user wanted to provide user input prior to an automated ESXi installation. This may sound odd at first, especially when we are talking about an unattended installation, the last thing you want is any type of user interaction. The reason for this requirement was to maintain a generic kickstart configuration file and based on the site specific input (IP Address, Netmask, Gateway, Hostname and DNS Servers), the ESXi installation would be able to deploy and configure itself the same across multiple locations.

UPDATE (01/10/19) - For ESXi 6.5 or greater, please take a look at this blog post for an updated solution

UPDATE (10/28/15) - Please take a look at this blog post on how to prompt for user input during an interactive or scripted installation of ESXi.

The other reason for this requirement is that all hosts in the user's environment must be configured with a static IP Address, this is not an uncommon requirement for many production environments to not have DHCP enabled networks. I can only assume the initial network the host is being built is either a private build network or booting off of local media such as a USB or CD-ROM device.

Whether you are booting off of local media or via PXE over the network, you have the ability to specify some boot parameters which includes things like IP Address, Netmask, Gateway and DNS Servers. These "bootstrap" options are fully supported and documented in the ESXi Installation Section. You can also specify advanced VMkernel boot parameters which may not be officially supported by VMware, but you can take a look here for more details.The concept of specifying these boot parameters is nothing new and has been supported since the early days of classic ESX and other popular UNIX/Linux distros for PXE boot/installations.

When you boot the ESXi installer, you may have seen a screen similar to the following in which you have a few seconds to hit the "tab" key to edit the boot options.

Once you hit the "tab" key, you will be able to see what the default boot options are and if you are PXE booting, you will also see some IP information appended towards the end of the string. This is where you can append or update additional parameters and later read in by your kickstart script.

Here is an example of an ESXi installation being PXE booted over the network and I have added 4 supported boot parameters and 2 custom ones.

If you take a look at the screenshot, the "+++" IP information towards the end is what was given by the DHCP server but I am interested in specifying a different IP Address for the ESXi installer to boot from. I provided new entries for ip, netmask, gateway and nameserver. You can also see that I introduced two new variables called hostname and dc, these will be used to specify the hostname of the ESXi host and also the name of the datacenter which will be used later in the kickstart script to rename a datastore.

Note: If you redefine the IP information, you do not need to change the IP information found after the "+++", these will just be over-written with the new IP information.

When specifying these boot parameters, you need to make sure it is after the "vmkboot.gz" but before "--- vmkernel.gz". To help make this clear, I have colorized the section that was appended to the default options.

vmkboot.gz dc=ghettoDC hostname=vesxi41-2.primp-industries.com ip=172.30.0.200 netmask=255.255.255.0 gateway=172.30.0.1 nameserver=172.30.0.100 ks=http://172.30.0.108/esxi41u1/ks.cfg --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz

Note: You can create any custom variables, the key in using the custom variables will be parsing from the boot command line using vsish which will be defined in your kickstart script. This will work also work for local media, but you MUST use a kickstart to perform the actual installation or inject the kickstart script into a custom ISO.

After you have provided all the input, you will then boot the installer and the the following kickstart configuration file is used to parse the boot options using vsish.

The script does the following:

  1. Reads the boot options using vsish and temporarily stores the output to /tmp/cmdline for later use
  2. Extracts all the relevant boot parameters in /tmp/cmdline and stores them in variables to be used in the script.
  3. Creates the "network" stanza for static IP assignment of the ESXi host which includes: Hostname, IP Address, Netmask, Gateway and Nameserver
  4. Writing out the custom variable "dc" into /var/log/esxi_install.log because this will be persisted through the reboot and can be later read in for any %firstboot operations.
  5. Upon the reboot, %firstboot will execute and read from /var/log/esxi_install.log to extract the "dc" variable in which it will use to rename the local datastore

Note: The above is just an example of what you can do with custom parameters. You can easily add as many as you need for site specific configurations and then use those input in your post configure your ESXi host. Also note that you do not need to make sure of ip,netmask,gateway,dns parameters to use custom variables, these are all optional.

You can add as many custom entries as you would like, but this can easily get error prone due to the amount of typing required. One thing you can do to minimize the amount of typos is to pre-specify the custom variables in your PXE/TFTP configuration file.

To do so, your PXE/TFTP configuration file would look something like this:

This not only reduces the amount of typing but also lets the user know what variables must be defined in order for the installation to proceed.

I totally understand where the user is coming from and I have had this requirement in the past but I think having a static DHCP entry and maintaining a simple configuration file can easily solve this problem. This not only makes the deployment process hands-off as it should be in the first place but also removing the human factor out of the equation. Can you imagine deploying 100-300 ESXi hosts in a day? I sure can not if I had to manually type out all those addresses by hand.

How to add a splash of color to ESXi DCUI Welcome Screen

by 24 Comments

Earlier this year I created a simple vSphere SDK for Perl script that allows you to update ESXi's DCUI (Direct Console User Interface) banner with multiple lines of text. I originally thought you could not customize the text color or the background color, though recently I found out that was not the case. While doing some testing on ESXi 4.1, I noticed two files (support,welcome) located under /etc/vmware and looking at the contents of support, it made realize we might be able to change the colors.

Here is the contents of /etc/vmware/support, notice the special formatting of the variables including color tags:

I decided to use one of my favorite UNIX utility, "strings" to take at the dcui binary that is located under /sbin in the Busybox Console (Tech Support Mode) and discovered you can control both the font color and background color. There are also special variables that can be used to display information about the ESXi host such as the product version or IP Address.

Here are the supported colors:

white
black
dark-grey
light-grey
yellow

Here are the special variables:

assettag
BIOSversion
BMCversion
CIM_Chassis
CPLDversion
esxproduct
esxversion
hostname
ip
license
memory
OMC_MCFirmwareIdentity
OMC_SMASHFirmwareIdentity
OtherIdentifyingInfo
PLSAversion
serial-number
servicetag
ssl-thumbprint
supportperiod
supportstart
VersionString
VMware_HHRCSoftwareIdentity

There are two ways of updating the DCUI welcome banner: using local or remote esxcfg-advcfg or manually editing /etc/vmware/welcome file.

Here is an example of using vCLI's esxcfg-advcfg:

Here is what that looks like on the DCUI:

As you can see, this is not easy if you want to update multiple lines. You would need to add a lot more spaces to force newlines, but this becomes tedious and pretty much unreadable. The second method is edit the welcome file that is located in the Busybox Console, which requires you to enable ESXi's Tech Support Mode. I wrote a quick Perl script called generateDCUIScreen.pl which accepts an input file and allows a user to customize the output and the script generates the "welcome" file which is uploaded to your ESXi host.

Here is an example of the input file:

The script will parse the input file which will contain definitions for:

  • bgcolor and color as described above
  • special variables as described above (must use braces for variables to be translated)
  • custom text
  • "=space=" string which is unique to my script which generates the newlines

The script requires that you have Perl, but you do not need to have vSphere SDK for Perl. For ease of use, I executed the script using vMA.

Here is an example execution using the input file from above:

You will need to scp the new"welcome" file to your ESXi host under /etc/vmware which is empty by default. For the changes to take effect, you will need to run the following command at the console:

kill $(ps | grep dcui | awk '{print $1}')

This will kill dcui utility and watchdog process will spawn a new instance causing the change to take effect Note: A reboot will also do the job, but be sure to run /sbin/auto-backup.sh before doing so, that way the change will be backed up.

Here is what DCUI screen looks like:

As you can see, you can control variety of pre-defined variables including hostname and IP Address and custom text for your organization. This is useful for those that do not want to expose all the information that available on the default DCUI screen, which may be a security concern for some organizations. A few things to note, I was not able to fill the entire screen like the default DCUI banner and the "welcome" file is character sensitive and you need to use tabs or white spaces to force the background to get filled. There is also a limit in the number of characters per row before it wraps to the next line.

I am sure there is someone out there that will create some interesting ASCII art, but here is my 5min free hand attempt at it 😉

This can easily be integrated into a scripted install using the new ESXi 4.1 kickstart feature.