There are some minor changes with kickstarting ESXi 5.0 but the majority of your existing ESXi 4.1 kickstart configurations can be re-used with a few modifications. One of my goals during the vSphere 5.0 beta was to automate as much of the configurations of an ESXi host as possible within the kickstart process. I also converted as many of the legacy esxcfg-* commands as I could over to the enhanced esxcli namespaces as the esxcfg-* commands will eventually be deprecated in favor of esxcli. Hopefully the tips & tricks and the example kickstart configuration file will be useful in aiding the transition to ESXi 5.0.

As usual, before diving in and creating an ESXi 5.0 kickstart configuration, make sure you spend some time going over the documentation provided by VMware, specifically the ESXi Installable and vCenter Server Setup Guide. If you would like to get a sense for what a ESXi 5.0 kickstart could look like, please jump to the bottom of this post to get a complete working example displaying the various types of configurations.

Tip #1

If you want to have your ESXi pxelinux configuration boot up to a kickstart configuration file, you have two methods of specifying this with ESXi 5.0.

The first is using the defualt installer method which specifies a new boot.cfg configuration file to contain all the boot parameters.

Here is an example of what the pxelinux default configuration file would look like:

Here is an example of what the boot.cfg looks like:

As you can see the default boot.cfg that is included in the ESXi 5.0 installer in the same directory as all the modules needed to boot up ESXi.

The second method is specifying the kickstart configuration file in the actual pxeboot file versus in a seperate boot.cfg file. This is similar to the old method of doing things, but you will have to also include all the entries that are in the boot.cfg if you decide to go down this route.

Here is an example of specifying ks.cfg in pxelinux default file:

Notice the "pxebooting ks=" stanza used to specify the ks.cfg configuration file and list of modules in boot.cfg. If you need to append additional parameters such as "IPAPPEND 1", you will need to add "+++" (three plus characters) at the very end of the module list separated with a new line for your additional entries. If you forget the "+++" symbols, you will not be able to successfully boot up the installer and an error will be thrown.

The default method of relying on the boot.cfg is the recommended approach. You can still append custom kernel parameters such as "IPAPPEND 1" which will still be in the pxelinux file, but your kickstart entry will now be part of the boot.cfg configuration file. Below is an example of the above configuration but leveraging the additional boot.cfg configuration file.

Here is an example of what the pxelinux file should look like:

As you can see, we still need to specify the "+++" to include additional parameters, but the pxelinux file is much cleaner now.

Here is an example of what the boot.cfg should look like:

As you can see, all we needed to do is append the following line "kernelopt=ks=http://172.30.0.108/esxi5/ks.cfg" which specifies the source to the kickstart configuration file.

Tip #2 

This is not really a new tip, but definitely make use of the "dryrun" mode in the ks.cfg and reviewing the esxi_install.log and hostd.log for any errors. The logs will only persist upon the first reboot, if you specify a secondary reboot for whatever reason, these logs will be lost. If you would like to automate the copying of the installation logs to help debug/troubleshoot, you can copy them to a local VMFS. Here is a quick snippet that can be used (This assumes your local VMFS will have the name with the following format $(hostname -s)-local-storage-1):

cp /var/log/hostd.log "/vmfs/volumes/$(hostname -s)-local-storage-1/firstboot-hostd.log"
cp /var/log/esxi_install.log "/vmfs/volumes/$(hostname -s)-local-storage-1/firstboot-esxi_install.log"

Tip #3

VMware has once again and hopefully for the last time, changed what was known as Busybox Console, Unsupported Mode, Tech Support Mode to ESXi Shell. In doing so, the old vim-cmd to enable both local (TSM) and remote (SSH) to ESXi Shell needs to be updated.

To enable local ESXi Shell (previously known as Tech Support Mode) you will need to run the following commands to enable and start ESXi Shell.

vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

To enable remote ESXi Shell (previously known as TSM SSH) you will need to run the following commands to enable and start remote ESXi Shell for SSH support.

vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

Note: During the beta, there was actually an intermediate name for this which was known as ESX Shell but through our feedback, we had recommended ESXi Shell so that users do not confuse old classic ESX with ESXi and this may or may not change from the actual vim-cmd's

Tip #4

The --level XX that would specify when a %firstboot script would execute is no longer supported and deprecated in ESXi 5.0. By default all %firstboot script will automatically execute after all default ESXi start up scripts have finished running. The location of the %firstboot scripts are in /etc/rc.local.d/001.fireboot_001

Tip #5

You should be able leverage the new changes in esxcli to configure majority of your ESXi 5.0 hosts, the are various examples below in the example kickstart. There are still a few things that are currently not included in esxcli such as configuring NTP, host licensing, ESXi Shell, SSH, and a few others that rely on vim-cmd or some minor hacks. VMware recommends you start to get familiar with esxcli, as the old esxcfg-* will eventually be deprecated and removed in future vSphere releases and be completely replaced with esxcli.

Tip #6

You now have the ability to configure multiple syslog hosts with ESXi 5.0 but also the ability to control individual loggers such as for vmkernel, hostd, vpxa, fdm, etc. Here is an example of changing the default syslog rotation from 10 to 20 and specifying two syslog hosts:

esxcli system syslog config set --default-rotate 20 --loghost syslog1.primp-industries.com:514,syslog2.primp-industries.com:1514

Note: The new syslog facility supports tcp,udp and tcps but there is a syntax bug on how to specify normal tcp, udp and or tcps.

By default, TCP is assumed and you can specify that using the following formats:

syslog1.primp-industries.com:514
tcp://syslog1.primp-industries.com:514

To specify UDP, you will need to use the following syntax:

udp://syslog1.primp-industries.com:514

To specify TCPS, you will need to use the following syntax:

ssl://syslog1.primp-industries.com:514

To view the individual syslog types, you can run the following command:

esxcli system syslog config logger list

To update the individual syslog types, you can run the following command:

esxcli system syslog config logger set --id=fdm --rotate=20 --size=2048

Tip #7

In ESXi 4.1, to enable the SSH security banner, you had to make some minor hacks. With ESXi 5, there is not an official sshd_config and you can configure the security banner by editing /etc/issue. You can also edit the motd under /etc/motd.

Tip #8

To create custom firewall rules, take a look at the blog post here

Tip #9

Here is post on how to create custom bootable ESXi ISO along with specifying static IP Address using kernel boot options.

Tip #10

Here is a post on how you can automate host cache configuration.   

Tip #11

If you are going to SSH out of an ESXi host at any point (%post, %firstboot) ensure that you enable sshClient via the ESXi firewall else you will get a connection denied. To enable the ESXi firewall use the following command: 

esxcli network firewall ruleset set --ruleset-id sshClient --enabled yes

Note: If you're doing this during %post section, hostd is not running and you will need to use "localcli" to enable the firewall which by-passes hostd.

Tip #12
Here is a post on how to Disable LUN During ESXi Installation.

 

Tip #13 

Here is a post on Removing Previous Local Datastore Label for Reinstall in ESXi 5

 

Tip #14 

Here is a post on How to Deploy an OVF/OVA in the ESXi Shell

 

Tip #15

How to persist custom firewall rules in ESXi 5, take a look at these two articles here and here

Tip #16

Here is a post on Configuring ESXi Power Management Policy Using the CLI

 

Tip #17

Here's an article about creating custom VIB for ESXi 5.0

Here is a complete working example of an ESXi 5.0 kickstart that can help you convert your existing ESX(i) 4.x to ESXi 5.x

54 thoughts on “Automating ESXi 5 Kickstart Tips & Tricks

  1. @Dominic,

    Thanks! I was hoping I cover majority of the cases & conversion to esxcli to help others convert their existing 4.x ks to 5.x

  2. Hi, I get an error :

    Configuration error while parsing boot.cfg
    Fatal error :34 (Unexpected EOF)

    TFTP is passing the default boot.cfg file that comes with the installer. The installer works fine.

    Do you have an idea what can be the problem.

  3. @Rocky,

    Sounds like you may have some syntax errors in your boot.cfg. Did you create the boot.cfg on a Linux system or Windows, you may have some hidden Windows carriage return which is causing bad parse as mentioned by the error message

  4. Hi William,

    Actually i too had the same doubt, i have files created on both linux and windows but same error on both scenarios. File content :

    bootstate=0
    title=Loading ESXi installer
    kernel=/tboot.b00
    kernelopt=ks=http://192.168.0.19/ISO/esxi5/ks.cfg
    modules=modules=/b.b00 --- /useropts.gz --- /k.b00 --- /a.b00 --- /ata-pata.v00 --- /ata-pata.v01 --- /ata-pata.v02 --- /ata-pata.v03 --- /ata-pata.v04 --- /ata-pata.v05 --- /ata-pata.v06 --- /ata-pata.v07 --- /block-cc.v00 --- /ehci-ehc.v00 --- /s.v00 --- /weaselin.i00 --- /ima-qla4.v00 --- /ipmi-ipm.v00 --- /ipmi-ipm.v01 --- /ipmi-ipm.v02 --- /misc-cni.v00 --- /misc-dri.v00 --- /net-be2n.v00 --- /net-bnx2.v00 --- /net-bnx2.v01 --- /net-cnic.v00 --- /net-e100.v00 --- /net-e100.v01 --- /net-enic.v00 --- /net-forc.v00 --- /net-igb.v00 --- /net-ixgb.v00 --- /net-nx-n.v00 --- /net-r816.v00 --- /net-r816.v01 --- /net-s2io.v00 --- /net-sky2.v00 --- /net-tg3.v00 --- /ohci-usb.v00 --- /sata-ahc.v00 --- /sata-ata.v00 --- /sata-sat.v00 --- /sata-sat.v01 --- /sata-sat.v02 --- /sata-sat.v03 --- /scsi-aac.v00 --- /scsi-adp.v00 --- /scsi-aic.v00 --- /scsi-bnx.v00 --- /scsi-fni.v00 --- /scsi-hps.v00 --- /scsi-ips.v00 --- /scsi-lpf.v00 --- /scsi-meg.v00 --- /scsi-meg.v01 --- /scsi-meg.v02 --- /scsi-mpt.v00 --- /scsi-mpt.v01 --- /scsi-mpt.v02 --- /scsi-qla.v00 --- /scsi-qla.v01 --- /uhci-usb.v00 --- /tools.t00 --- /imgdb.tgz --- /imgpayld.tgz
    build=
    updated=0

  5. I also have a beta version of vSphere 5, build 381646. I tried putting the modules line in the pxelinux.cfg/default file as well, but the pxelinux.o fails to parse the configuration and present the menu. I’m stuck and really need to have this functionality. Any assistance would be greatly appreciated.

  6. @Duncan,

    Make sure you’re using syslinux 3.86, that is the only supported version for ESXi 5. Also is your boot.cfg in the same directory as the modules?

  7. @William that was totally the problem! Many thanks!! Your blog is great! I’ll be at VMworld this year I’ll have to buy you a drink!

  8. Will, I’m guessing you use DHCP to get an IP to the vmk0, we don’t have DHCP running on our server vlan nor do we have a kickstart server so I was using kernel options to pass info to the ks script. In ESXi 4.1 for example, I modified the isolinux.cfg so I could hit Tab and be presented with the following line:

    append vmkboot.gz ks=cdrom:/KS.CFG HOST= IP= VMKIP= MASK= GATE= NAME1= NAME2= DOMAIN= ENV= --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz

    variables should be obvious (ENV is to specify prod, dev, or lab which kicked off different KS.cfg commands for different environments.) With the kernel variables going to the boot.cfg file, and no longer visible from the initial boot command line, how would i modify the “APPEND -c boot.cfg” line to get these variables into my KS script?

  9. I was able to use cat, grep, awk, and sed on the esxi_install.log to fill the variables that i placed on the command line since ‘weasel’ ignores them

  10. Hi,

    Anyone knows how to pass multiple kernel options in the boot.cfg file? If I try to use multiple kernelopt= lines, then it will take into account only the last definition. I’d imagine a separator has to be used“ but I haven’t been able to find any examples anywhere.

    Thanks!

  11. Just figured it out“ using space as a delimiter did the trick. So something like:
    kernelopt=ks=http://server/path/to/ks/file BOOTIF=aa:bb:cc:dd:ee:ff

  12. @Roderick,

    HTTP protocol is still supported in vSphere 5. I would highly recommend you take a look at the vSphere 5 documentation before getting started

  13. Well, the vSphere docs say it is possible to get an automated installation to work via HTTP, but it is very difficult.

    I already had a PXE server. I made adjustments to the DHCP server for gPXE boot.

    I had to download a gpxe image from http://rom-o-matic.net. I added an embedded script:

    #!gpxe
    echo Roderick ICT Consultancy
    echo Performing DHCP on first network interface

    dhcp net0
    kernel -n mboot.c32 http://10.0.2.14:8080/vSphere/ESXi_5.0/MBOOT.C32
    imgargs mboot.c32 -c http://10.0.2.14:8080/vSphere/ESXi_5.0/BOOT.CFG
    boot mboot.c32

    This script runs automaticaly so you don’t have to enter all these commands after booting the server.

    The VMware installer starts loading, but it stops. It says it needs the BOOTIF parameter containing the MAC address of the server. So I replaced one line with this one:

    imgargs mboot.c32 -c http://10.0.2.14:8080/vSphere/ESXi_5.0/BOOT.CFG -c BOOTIF=01:${net0/mac}

    A new error message appears:
    configuration error while parsin /BOOTIF=00:50:56 “ etc.
    Fatal error: 15 (not found)

    There is info about this BOOTIF option in the vSphere docs but I have to find the location and format for this option.

    I have the feeling installation via HTTP is made difficult on purpose by VMware to promote their own distribution product. It used to be very simple.

    I continue searching for the solution.

  14. Using this tutorial and the one at the URL below I was trying to figure out how to PXE boot using a kickstart by specifying it in the pxelinux.cfg boot file.

    The way it is described here did not work for me. I changed it to the way my pxelinux.cfg (default) file is below and it worked.

    DEFAULT ESX_5.0_install

    LABEL ESX_5.0_install
    KERNEL /esxi/5.0/mboot.c32
    APPEND -c /esxi/5.0/boot.cfg pxebooting ks=http://172.22.0.2/kickstart/01-00-25-90-53-9c-fe.ks

    You do not need all of the modules in a separate APPEND in this file. Just edit your boot.cfg to add prefix= and remove the / before all of the modules.

    http://www.vcritical.com/2011/07/vmware-esxi-5-interactive-pxe-installation-improvements/

  15. I made a comment on your hostsvc.xml posting in regards to 4.1. I figured out an easy way around connecting to MOB and doing the python stuff that works like a champ that may help some of you out.

    I have a hostsvc.xml file with both vmk’s as out on the same webserver as my kickstart file.

    I pull it down with wget to /tmp, copy /tmp/hostsvc.xml

    restard hostd then refresh networking (in that specific order) and those magical tick boxes are checked.

  16. What exactly does this portion do?
    # enable management interface
    # Still need to use python/MOB trick

    I just test installed several times and the management interface came up every time without having this portion in the kickstart.

    As for gpxe, I’ve got a working environment in conjunction with cobbler. In short, it should look something like:

    #!gpxe
    kernel -n mboot.c32 http://SRVNAME/mboot.c32
    imgargs mboot.c32 -c http://SRVNAME/BOOT.CFG
    boot mboot.c32

    where BOOT.CFG looks something like:
    title=Loading ESXi installer
    kernel=http://SRVNAME/ESXi/5.0/tboot.b00
    kernelopt=ks=http://SRVNAME/cblr/svc/op/ks/system/esxsrv01 BOOTIF=ff:ff:ff:ff:ff:ff
    modules=“
    where of course BOOTIF contains the server’s mac address and modules= contains a list of URLs for modules to be loaded.

  17. @xman,

    It should automatically enable the firewall for NFS traffic, but you should test to confirm.

    @LaClair,

    Yes this is another alternative which is listed I believe under ESXi 4.x kickstart post which can be utilized in ESXi 5 as you’ve tested

    @Andrei,

    An earlier release of ESXi 5, mgmt traffic was not enabled by default, this may have changed when it GA’ed.

  18. If someone is facing a problem when booting from pxe and the screen is blank, check your pxelinux.0 version..

    PXELINUX 3.10 2005-08-24 - gives only blank black screen after booting from pxe.
    PXELINUX 3.30 2006-09-18 - shows the graphics, but fails to load all the installation files.. guess there is somekinda limitation on amounth of files that can be downloaded
    PXELINUX 3.50 2007-06-09 and newer - work just perfectly

    Took me day to find out.

    • Hi Kimmo,

      I ran into the same problem. Thanks for sharing the info (PXELINUX versions). I initially stared with whaever pxelinux version that comes with centos5.8. Ran into the same issue as yours. Then I downloaded the recommended version - pxelinux 3.86 and it worked fine!

  19. What am I missing? I cannot get the %pre section of the above script to obtain the IP information from an ESX 4.1 host. Is it because there are very limited commands available at the time when this portion of the script is executed? All of the parameters to the network command wind up being blank!

    The syntax of these commands look more like ESXi 5 commands. Is that the case?

    Any help is appreciated.

    Thanks,
    Jim

  20. @William, I am doing a fresh install via an ESX 5.0 ISO image with a ks.cfg file I added with MagicISO. I also modified boot.cfg to point to the local ks.cfg i.e., ks=cdrom:/KS.CFG

    The script executes but none of the network commands’ parameters are available, which tells me the %pre section did not do its job in my case.

    Thanks.

  21. Addon.
    If you want to enable cut and paste between the VM console and the computer running the VI Client, you van add this code.

    # enable Cut & Paste between VI Client Guest and VM Console
    # http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026437
    echo “isolation.tools.copy.disable=\”FALSE\”” >> /etc/vmware/config
    echo “isolation.tools.paste.disable=\”FALSE\”” >> /etc/vmware/config

    A. Mikkelsen
    http://www.amikkelsen.com

  22. @William“.how would I use, or is the another way, to create an AutoDeploy image that would use this command ‘esxcli storage core device setconfig -d --perennially-reserved=true’ to prevent slow boot times on ESXi hosts running MS Failover Clusters. I have boot times of 40 mins on some host due to this. Since it is a stateless deploy I am looking for a way to set during the load of ESXi and before it gets to the point where it hangs trying to claim the RDM disks that are part of the cluster.
    See http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1016106&sliceId=1&docTypeID=DT_KB_1_1&dialogID=218399559&stateId=0%200%20218397548

    • @William, I can understand the idea for doing this in an install“.but the piece I am missing is with an Offline Bundle using AutoDeploy, how do I call a script to run when ESXi is starting up(not installing)“thx

    • Not sure I follow your question, Auto Deploy is stateless, so if you’re able to get the suggestion working, then you can technically embed anything that’ll run upon startup. Again, this probably would not be officially supported by VMware, but you probably could get it to work.

  23. Yes“.that is my question, how do I actually embed those commands to run at startup. That is the piece I am missing. What file would those commands go in and how do I call that script to run as part of the startup with AutoDeploy? I assume those commands could just be the esxcli commands. I am not doing this in production, just testing in the dev environment.

  24. @William..
    Yes“.that is my question, how do I actually embed those commands to run at startup. That is the piece I am missing. What file would those commands go in and how do I call that script to run as part of the startup with AutoDeploy? I assume those commands could just be the esxcli commands. I am not doing this in production, just testing in the dev environment.

  25. Hi,

    i used also a Kickstart for ESXI5, But i have a Problem after booting up. Network connectivity is not working.

    Because i have to manually de/configure the vmnic0.

    i tried to

    esxcli network vswitch standard uplink remove -u vmnic0 -v vSwitch0
    esxcli network vswitch standard remove -v vSwitch0

    to delete it through the kickstart. but it’s still in place.

    Anybody knows how to delete the default switch of the installer with the vmnic?

    thx

    Max

    • Try this“

      #remove temp install vSwitch
      esxcli network vswitch standard uplink remove --uplink-name=vmnic0 --vswitch-name=vSwitch0

      esxcli network vswitch standard portgroup remove --portgroup-name=”temp” --vswitch-name=vSwitch0

      esxcli network vswitch standard remove --vswitch-name=vSwitch0

  26. William,
    there is a typo in your NTP configuration:
    restrict default kod nomodify notrap noquerynopeer
    is supposed to be
    restrict default kod nomodify notrap noquery nopeer

    Any advice on how to update it on 120+ servers?

    Thanks,
    Mike

  27. Is it possible to point the boot.cfg file to an nfs export for the installation? I am trying with the following configuration and I am told the file cannot be found.

    title=Loading ESXi installer
    prefix=nfs:10.211.4.212:/srv/data/install/VMware/ESXi/5.1/
    kernel=tboot.b00
    kernelopt=runweasel
    modules=b.b00 --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- s.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- weaselin.t00 --- esx_dvfi.v00 --- xlibs.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- misc_cni.v00 --- misc_dri.v00 --- net_be2n.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_nx_n.v00 --- net_r816.v00 --- net_r816.v01 --- net_s2io.v00 --- net_sky2.v00 --- net_tg3.v00 --- net_vmxn.v00 --- ohci_usb.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_lpf.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- scsi_qla.v01 --- scsi_rst.v00 --- uhci_usb.v00 --- tools.t00 --- xorg.v00 --- imgdb.tgz --- imgpayld.tgz
    build=
    updated=0

    • No, you would specify the kickstart file which could be on an NFS server which would then tell where to install from which can also be hosted on an NFS server.

  28. I am trying to find a way to prompt for the hostname at the start of the install. Is this possible? We’ll be provisioning about 70 servers and then shipping them all over the world and I want the provisioning team to just type in the hostname so that I can then use that to do more host-specific config later on like using the hp utility to configure the iLO and embedding the server name in the datastore name.

  29. Well done as always! I have everything working very well.

    How can I review the logs from the kickstart script to see if anything failed? All of my %firstboot commands worked except for the wget command to copy the authorized_keys. The wget command works fine if I manually run it later but does not during the kickstart.

    • If you have not rebooted after the %firstboot, then they’re stored in /var/log/esxi_install.log and probably check out /var/log/syslog.log for details of the script executing

  30. Hi William,

    I am trying to do non-interactive installation of 5.1.
    It is downloading the kernel from tftp server but it is going to interactive install and it is not using the kickstart file.
    I checked the boot options by pressing shift+O while loading the installer and it has the ks file path.
    I have tried all possible things I can but I did not succeed.

    Could you please help me here?

  31. Will,

    I modified this excellent script to do a fresh install and not upgrade ESXi, which works like magic. Is there anyway one can add the server to vcenter from within the script to join an existing cluster?

    In other words are there any command line that one can run from the ESX prompt to join a cluster on vcenter?

Thanks for the comment!