virtuallyGhetto

kickstart

Using ESXi Kickstart %firstboot with Secure Boot

by 4 Comments

If you install ESXi via a Kickstart script and make use of the %firstboot option to execute commands on the first boot of the ESXi host after installation, you should be aware of its incompatibility with the Secure Boot feature. If you install ESXi where Secure Boot is enabled, the Kickstart will install ESXi normally only execute up to the %post section. However, it will not execute the %firstboot scripts and if you look at the /var/log/kickstart.log after the host boots, you should see the following message:

INFO UEFI Secure Boot Enabled, skipping execution of /var/lib/vmware/firstboot/001.firstboot_001

If you have Secure Boot enabled, %firstboot is not supported. The reason for this is Secure Boot mandates only known tardisks which can hold executable scripts, and a kickstart script is an unknown source so it can not run when Secure Boot is enabled. If you wish to continue using %firstboot scripts, the only option is to disable Secure Boot and then re-enable it after the installation. A preferred alternative is to convert your %firstboot logic into an external script which can then be applied using the vSphere API (recommended method) and this way you can still customize your ESXi host after the initial installations. I have already filed an internal documentation bug to add a note regarding Secure Boot and %firstboot, hopefully that will roll out with the net documentation refresh.

Quick Tip – What hashing algorithm is supported for ESXi Kickstart password?

by Leave a Comment

I had a question the other day asking whether the encrypted password which can be specified within an ESXi Kickstart file (denoted by the --isencrypted flag) can use a different hashing algorithm other than MD5? The answer is absolutely yes. In fact, MD5 as a default hashing algorithm has NOT been used for a number of releases, probably dating back to classic ESX (you know, the version that had the Service Console).

For all recent releases of ESXi including 5.5 to 6.7, the default hashing algorithm has been SHA512 for quite some time now. Below are two ways in which you can check which default hashing algorithm is currently being used:

Option 1 - SSH to ESXi host and take a look at /etc/pam.d/passwd


Option 2 - SSH to ESXi host and take a look at /etc/shadow and look at the field prior to the salt.

As a reference:

  • $1$ - MD5
  • $5$ - SHA256
  • $6$ - SHA512

How to Netboot install ESXi onto Apple Mac Hardware?

by 9 Comments

The ability to perform an ESXi Scripted Installation over the network has been a basic capability for non-Apple hardware customers since the initial release of classic ESX. However, for customers who run ESXi on Apple Mac Hardware (first introduced in vSphere 5.0), being able to remotely boot and install ESXi over the network has not been possible and customers could only dream of this capability which many of us have probably taken for granted.

Unlike traditional scripted network installations which commonly uses Preboot eXecution Environment (PXE), Apple Mac Hardware actually uses its own developed Boot Service Discover Protocol (BSDP) which ESXi and other OSses do not support. In addition, there are very few DHCP servers that even support BSDP (at least this may have been true 4 years ago when I had initially inquired about this topic). It was expected that if you were going to Netboot (equivalent of PXE/Kickstart in the Apple world) a server that you would be running a Mac OS X system. Even if you had set this up, a Netboot installation was wildly different from a traditional PXE installation and it would be pretty difficult to near impossible to get it working with an ESXi image. With no real viable solution over the years, it was believed that a Netboot installation of ESXi onto Mac Hardware just may not be possible.

tl;dr - If you are interested in the background to the eventual solution, continue reading. If not and you just want the goods, jump down a bit further. Though, I do think it is pretty interesting and worth getting the full context 🙂

[Read more...] about How to Netboot install ESXi onto Apple Mac Hardware?

How to prompt for user input during an interactive or scripted install of ESXi?

by 24 Comments

A question that continues to pop up from time to time is whether it is possible to prompt for user input during an interactive or scripted installation of ESXi? This is actually something I have written about before using PXE boot options as a workaround to provide for a semi-interactive automated installation of ESXi. The most recent request for this was not actually from a customer but rather someone internally working at VMware. The individual noted that he had already read my blog and a few other references but was still hopeful for a solution. I remember when I had looked into this problem several years back, I was not able to find anything and the inquiries to VMware (which I was still a customer at the time) came up empty.

After replying back to the individual with the information that I had, it actually got me thinking which is usually not good 😉 Having just recently finished building a new Kickstart environment to test UEFI PXE boot for ESXi 6.0, I figure I might as well take another look at this topic again. I wanted to see if there was something that could be done with one of the /dev/ttyl (teletype) interfaces while the ESXi Installation was running. I did a couple of Google searches and to my surprise, there was someone on the VMTN Community forum who had already solved this problem and posted a solution almost 1 year before my article, not sure how I could have missed that?!

In ESXi, there are only two TTY interfaces: TTY1 (/dev/tty1) and TTY2 (/dev/tty2) and during the installation of ESXi, /dev/tty2 is used by the interactive installer and /dev/tty1 is used by the ESXi Shell login for troubleshooting/debugging purposes. If you try to overlay another menu system on top of the existing one, what you will find is that although it is possible to prompt for user input, the user keystrokes would not be properly being sent due to having two menus as mentioned by VMTN Community user mossko. The clever solution that was identified by cacheman was to simply disable the console login and free up TTY1 which could then be used to present a custom menu and prompt for user input. This was accomplished by creating a custom TGZ file called "extras.tgz" (you can name it anything you want) which would contain a modified copy of the /etc/inittab file which has the following line commented out:

#tty1::respawn:/bin/initterm.sh tty1 /bin/techsupport.sh

You would then append this TGZ to the ESXi's boot.cfg file which would then overwrite the existing inittab and prevent ESXi Shell from running on TTY1. I thought this was very cool way of solving this problem, nice work and thanks for sharing cacheman!

I wanted to verify that this solution still works with the latest release of ESXi 6.0 Update 1 and luckily it still does and here is a sample menu prompt that I created which prompts users for the networking settings for the ESXi host which will then be used by the ESXi Kickstart script. This is just one example of user prompts that can be created, this is only limited by your imagination and the amount of input you wish to put your users through 😉 If you are interested in the ASCII art, go to the very bottom for the full details.

prompt-user-input-during-interactive-or-scripted-install-of-esxi
Here are the exact instructions along with the sample menu that was used in the screenshot above.

Step 1 - Create a temporary directory which will be used to store our modified inittab file by running the following command:

mkdir -p temp/etc

Step 2 - Change into the "temp" directory

Step 3 - Copy the /etc/inittab file into the etc directory and you should have the following: temp/etc/inittab

Step 3 - Edit the etc/inittab file and comment out the following line:

#tty1::respawn:/bin/initterm.sh tty1 /bin/techsupport.sh

Step 4 - Next, we need to create a TGZ file of the etc directory by running the following command:

tar -czvf extras.tgz etc

Step 5 - Copy the extras.tgz to root of your ESXi installation if you are PXE booting or if you are re-authoring a new ESXi ISO

Step 6 - Edit the ESXi's boot.cfg configuration file and append the following after imgpayld.tgz:

--- extras.tgz

Step 7 - Finally, you will add your custom menu prompt into the %pre section of an ESXi Kickstart file. Below is an example Kickstart that I used to build the menu shown in the screenshot above.

Lines 10-11 - Redirects the output to /dev/ttyl and changes to tty1 virtual console.

Lines 12-16 - Define a couple of empty string variables that will be used to store user input.

Lines 18-28 - Prompt user input and loop until all variables are not empty.

Lines 30 - Using the user input, generate the networking configuration for the ESXi Kickstart.

Line 32-38 - Display the user input to the user and sleep for 5 seconds before switching back to the ESXi installer.

Line 39 - Change to /dev/tty2 virtual console which will show the regular ESXi installer menu.

Line 43-44 - Modify the /etc/inittab to re-enable ESXi Shell for /dev/tty1. For the changes to go into effect, you will need reboot and you can either do this when you need the capability (which is not ideal) or you can just automatically issue a second reboot in the %firstboot section of the Kickstart, so the ESXi Shell is available when you need it.

For those interested in how I added the ASCII art, I used this site here to generate the text. Once you are happy with the text and how it is displayed, which may require several rounds of testing, you will then want to save it to a file. In my setup I just called it vg-ascii.cfg and this should be placed on either your HTTP Server where the Kickstart is being pulled or embedded into the ESXi ISO if you are doing local installs only. You will then need to add the following lines to the "pre" section of the Kickstart right before the loading of the menu prompt. In this example, I am downloading the ASCII file from my web server which is also serving my kickstart and then clearing the screen before iterating through the file to display the text.

wget http://192.168.1.180/esxi60u1/vg-ascii.cfg -O /tmp/vg-ascii.cfg
clear
IFS=$'\n';for i in $(cat /tmp/vg-ascii.cfg);do echo $i;done

UEFI PXE boot is possible in ESXi 6.0

by 20 Comments

A couple of days ago I received an interesting question from fellow colleague Paudie O'Riordan, who works over in our Storage and Availability Business Unit at VMware. He was helping a customer who was interested in PXE booting/installing ESXi using UEFI which is short for Unified Extensible Firmware Interface. Historically, we only had support for PXE booting/installing ESXi using the BIOS firmware. You also could boot an ESXi ISO using UEFI, but we did not have support for UEFI when it came to booting/installing ESXi over the network using PXE and other variants such as iPXE/gPXE.

For those of you who may not know, UEFI is meant to eventually replace the legacy BIOS firmware. There are many benefits with using UEFI over BIOS, a recent article that does a good job of explaining the differences can be found here. In doing some research and pinging a few of our ESXi experts internally, I found that UEFI PXE boot support is actually possible with ESXi 6.0. Not only is it possible to PXE boot/install ESXi 6.x using UEFI, but the changes in the EFI boot image are also backwards compatible, which means you could potentially PXE boot/install an older release of ESXi.

Note: Auto Deploy still requires legacy BIOS firmware, UEFI is not currently supported today. This is something we will be addressing in the future, so stay tuned.

Not having worked with ESXi and UEFI before, I thought this would be a great opportunity for me to give this a try in my homelab which would also allow me to document the process in case others were interested. For my PXE server, I am using CentOS 6.7 Minimal (64-Bit) which runs both the DHCP and TFTP services but you can use any distro that you are comfortable with.

Step 1 - Download and install CentOS 6.7 Minimal (64-Bit)

Step 2 - Login to the CentOS system via terminal and perform the following commands which will update the system and install the DHCP and TFTP services:

yum -y update
yum -y install dhcp tftp-server

Step 3 - Download and upload an ESXi 6.x ISO to the CentOS system. In example here, I am using latest ESXi 6.0 Update 1 image (VMware-VMvisor-Installer-6.0.0.update01-3029758.x86_64.iso).

Step 4 - Extract the contents of the ESXi ISO to the TFTP directory by running the following commands:

mount -o loop VMware-VMvisor-Installer-6.0.0.update01-3029758.x86_64.iso /mnt/
cp -rf /mnt/ /var/lib/tftpboot/esxi60u1
umount /mnt/
rm VMware-VMvisor-Installer-6.0.0.update01-3029758.x86_64.iso

Step 5 - Copy the custom ESXi bootx64.efi bootloader image to the root of the extracted ESXi directory by running the following command:

cp /var/lib/tftpboot/esxi60u1/efi/boot/bootx64.efi /var/lib/tftpboot/esxi60u1/mboot.efi

Step 6 - Next, we need to edit our DHCP configuration file /etc/dhcp/dhcpd.conf to point our hosts to the mboot.efi image. Below is an example configuration and you will need to replace it with the network configuration of your environment. If you are running the TFTP server on another system, you will need to change the next-server property to the address of that system else you will just specify the same IP Address as the DHCP server.

Step 7 - Next, we will need to edit our TFTP configuration file /etc/xinetd.d/tftp to enable the TFTP service by modifying the following line from yes to no:

disable = no

Step 8 - By default, the ESXi's boot.cfg configuration file refers to all packages under / path. We will need to remove that reference and can easily do so by running the following command:

sed -i 's/\///g' /var/lib/tftpboot/esxi60u1/boot.cfg

Step 9 - Finally, we need to restart both the TFTP (under xinetd) and DHCP services. For testing purposes, I have also disabled firewall for ipv4/ipv6, of course in a real production environment you will probably want to only open the ports required for TFTP/DHCP.

/etc/init.d/xinetd restart
/etc/init.d/dhcpd restart
/etc/init.d/iptables stop
/etc/init.d/ip6tables stop

We can now boot up either a physical host that is configured to use UEFI firmware OR we can also easily test using Nested ESXi. The only change we need to make to our ESXi VM is by setting the firmware mode from BIOS to EFI which can be done using the vSphere Web/C# Client as shown in the two screenshots below:

uefi-pxe-boot-esxi-6.0-0 uefi-pxe-boot-esxi-6.0-1
If everything was successfully configured, we should now see our system PXE boot into ESXi installer using UEFI as seen in the screenshot below.

uefi-pxe-boot-esxi-6.0-2
If you run into any issues, I would recommend checking system logs on your PXE server (/var/log/messages) to see if there are any errors. You can also troubleshoot by manually using tftp client and connecting to your TFTP Server to ensure you are able to pull down the files such as the boot.cfg by running the following command:

tftp [PXE-SERVER]
get esxi60u1/boot.cfg

For additional resources on scripted installation of ESXi also referred to as Kickstart, be sure to take a look here. I also would like to give a big shoutout and thanks to Tim Mann, one of the Engineers responsible for adding UEFI support into ESXi and for answering some of my questions while I was setting up my environment.

How to create custom ESXi boot menu to support multiple Kickstart files?

by 20 Comments

I recently received a question from one of my readers who was looking to migrate from ESXi 4.1 to newer version and one of the challenges they faced was around their ESXi scripted installs, better known as ESXi Kickstart. Previously, they had relied on using a custom syslinux boot menu to be able to select a specific Kickstart configuration file that resided locally on a bootable ESXi Image (USB, ISO or CDROM) as a PXE/DHCP environment was not allowed in their environment. There was a small change to how ESXi boot files were reference between ESXi 4.x and ESXi 5.x/6.x and a new boot.cfg configuration is now used which I had written about here with respect to scripted installs when ESXi 5.0 was first released.

Luckily, even with these changes one can still use a custom menu with ESXi 5.x/6.x and be able to select a specific Kickstart configurations based on user input. Here is a screenshot example of a custom ESXi Image that I built providing three different install options that could be selected which would map to three different Kickstart configurations which can be either local to the boot media or can also be retrieved remotely.

bootable-esxi-image-with-multiple-kickstart-option
The first thing you should be aware of if you plan to boot the custom ESXi Image from local media such as USB, CDROM or ISO is that the path to the Kickstart file must be in all UPPER CASE which is mentioned in this VMware KB 1026373. The next caveat that I found in my testing is that if you plan to store the local Kickstart files inside of a directory within the ESXi Image, the name of the directory can not be too long. I would recommend using "ks" as "kickstart" apparently was too long.

After you have extracted the contents of an ESXi ISO which you have downloaded, you will want to create a root directory called "ks" which will contain the different Kickstart configuration files. Here is an example of what structure look like:

ks
├── ks1.cfg
├── ks2.cfg
└── ks3.cfg

Next, you will need to edit the isolinux.cfg file which comes by default within the ESXi ISO. This is where you will add the different Kickstart options that a user will be able to select from. In this first example, we will look at referencing the Kickstart files locally on the media which can be either USB or CDROM and you will need to ensure you specify the right boot option as shown here in the VMware documentation. The path to the Kickstart file needs to be appended to the line that contains boot.cfg reference and you must ensure you include "+++" at the end of that line.

Here is an example of referencing a Kickstart file that lives on a USB device under this path /ks/ks.cfg:

APPEND -c boot.cfg ks=usb:/KS/KS.CFG +++

Here is an example of my isolinux.cfg for the boot menu that I have shown above which provides three different options mapping to three different Kickstart configuration files:

As I mentioned earlier, the Kickstart configuration file can either be retrieved locally or it can also be retireved remotely using one of the following supported protocols: http, https, ftp & nfs as shown here in the VMware documentation.

Here is an example of isolinux.cfg for a boot menu which references both a local kickstart as well as one that remotely lives on a web server:

For additional ESXi Kickstart resources and example, be sure to check out my pages here.