If you are interested in advertising with us, please contact us at admin[at]virtuallyghetto[dot]com
Monday, February 28, 2011
New sponsor: Opvizor
virtuallyGhetto would like to welcome Opvizor as our latest blog sponsor. Opvizor is a SaaS based application that detects issues that may affect your virtualized IT environment before they cause instability and impact performance. The analysis is done by uploading your host or virtual machine logs to Opvizor, in which they can be processed by their intelligent rule-set or shared amongst your peers and trusted third party vendors. You can easily get started with their free entry level version by registering at opvizor.com.
If you are interested in advertising with us, please contact us at admin[at]virtuallyghetto[dot]com
If you are interested in advertising with us, please contact us at admin[at]virtuallyghetto[dot]com
Potential ESX(i) 4.1 Update 1 upgrade caveat
If you are planing on upgrading to the recent release of ESX(i) 4.1 Update 1 from ESX(i) 4.1, you may want to verify that you will not be impacted by a previous security/password bug found in ESX(i)4.1. The security bug that was identified with ESX(i) 4.1 was the encryption algorithm which changed from default MD5 as previous releases of ESX(i) to legacy DES.
This bug has since been resolved and a VMware KB article (KB1024500) was released with a temporary fix by adding the keyword "md5" to system-auth PAM configuration entry. This fix also required the user to update the root password afterwards, as the previous password was encrypted using DES.
The potential caveat is if you did not apply the fix as mentioned from the above VMware KB article prior to upgrade, after the ESX(i) 4.1 Update 1 upgrade, you will continue to run into the same problem. The fix is to reset your root password after the upgrade of ESX(i) 4.1 Update 1, this will ensure that the new password will be encrypted using the MD5 algorithm. Though the fix is simple, it can be tedious and manual for users who do not regularly rotate their root passwords or have an automated password management system.
To check whether this will impact your upgrade, login to ESXi Tech Support Mode or classic ESX Service Console and check whether the root password was encrypted using MD5 or DES. To do so, you will cat out the contents of /etc/shadow
If the root password was encrypted using MD5 algorithm, you should see root hash start with "$1$"
If the root password was encrypted using DES algorithm, the root hash will not start with "$1$"
If it is the latter, you will need to either apply the fix and update the root password before upgrading or reset the password after your upgrade. To change the root password, you need to login to either the Serivce Console for ESX or Tech Support Mode for ESXi and run the passwd utility to change your password.
It is probably quicker to rebuild than to login to each host an update the root password, especially if you have an automated kickstart environment. This will ensure that that all hosts will be consistent and no manual fixes will be required. IMHO, this is something that VMware should have clearly pointed out in their release notes as not everyone may be aware of the VMware KB article and implemented the fix prior to upgrade.
This bug has since been resolved and a VMware KB article (KB1024500) was released with a temporary fix by adding the keyword "md5" to system-auth PAM configuration entry. This fix also required the user to update the root password afterwards, as the previous password was encrypted using DES.
The potential caveat is if you did not apply the fix as mentioned from the above VMware KB article prior to upgrade, after the ESX(i) 4.1 Update 1 upgrade, you will continue to run into the same problem. The fix is to reset your root password after the upgrade of ESX(i) 4.1 Update 1, this will ensure that the new password will be encrypted using the MD5 algorithm. Though the fix is simple, it can be tedious and manual for users who do not regularly rotate their root passwords or have an automated password management system.
To check whether this will impact your upgrade, login to ESXi Tech Support Mode or classic ESX Service Console and check whether the root password was encrypted using MD5 or DES. To do so, you will cat out the contents of /etc/shadow
If the root password was encrypted using MD5 algorithm, you should see root hash start with "$1$"
Wednesday, February 23, 2011
Automating Active Directory User Management in ESXi Kickstart
In the previous post we looked at Automating Active Directory Domain Join in ESXi Kickstart. We are now going to look at adding domain users without having to manually go through vSphere Client or external scripts after an ESXi host has been provisioned. We are going to be leveraging vim-cmd to accomplish this during the kickstart installation. Before doing so, you will need to know the the available roles on a default ESXi host and the syntax for a given permission.
To see the available roles, you can run the following command on an already provisioned ESXi host:
vim-cmd vimsvc/auth/roles | less
The default roles on an ESXi host are:
vim-cmd vimsvc/auth/permissions
These entries will match what you see in the vSphere Client and dictate who has login access to the ESXi host.
To add a new permission, we will be using the "vim-cmd vimsvc/auth/entity_permission_add" and it requires five parameters.
Note: We need to use the double slash "\" to escape the initial slash when running the query. Also make note of the domain name as it may or may not match your full domain name.
We are now ready to craft a simple script that will add domain users as part of the ESXi kickstart process. The following snippet should be placed in the %firstboot section of your kickstart and after your Active Directory domain join code. Make sure you replace the DOMAIN_NAME variable along with the usernames. In the example I have two separate for loops to handle ReadOnly and Admin users, you do not need both if you are only adding one type of users.
The script basically performs a simple 60sec sleep to ensure the domain join process has completed before continuing. If you do not place a sleep, the subsequent code will fail to execute. The next step is to validate the user by doing a simple lookup using "id" command and upon successful look up of the user, we add the appropriate permissions.
vim-cmd vimsvc/auth/role_add
Note: The syntax for the privileges parameter lists only five, but it actually accepts as many as you need with the custom role
To see the available roles, you can run the following command on an already provisioned ESXi host:
vim-cmd vimsvc/auth/roles | less
- NoAccess
- ReadOnly
- Admin
vim-cmd vimsvc/auth/permissions
To add a new permission, we will be using the "vim-cmd vimsvc/auth/entity_permission_add" and it requires five parameters.
- First - Entity (This can be found by looking at the output from permissions)
- Second - Username
- Third - Boolean on whether this is a group or not (should be false)
- Fourth - The role to be applied to the user
- Fifth - Boolean on whether to propagate this permission
We are now ready to craft a simple script that will add domain users as part of the ESXi kickstart process. The following snippet should be placed in the %firstboot section of your kickstart and after your Active Directory domain join code. Make sure you replace the DOMAIN_NAME variable along with the usernames. In the example I have two separate for loops to handle ReadOnly and Admin users, you do not need both if you are only adding one type of users.
The script basically performs a simple 60sec sleep to ensure the domain join process has completed before continuing. If you do not place a sleep, the subsequent code will fail to execute. The next step is to validate the user by doing a simple lookup using "id" command and upon successful look up of the user, we add the appropriate permissions.
Note: We only have two add these two entities: "vim.Folder:ha-folder-root" and "vim.ComputeResource:ha-compute-res" to properly add a permission.
If everything was successful, after your ESXi installation you now should have your host joined to your Active Directory and a list of domain users who now have permission to login to the ESXi host. You can verify by using the vSphere Client and taking a look at the Permissions tab.
If you would like to create custom roles on your ESXi host, you can use the following command:
If everything was successful, after your ESXi installation you now should have your host joined to your Active Directory and a list of domain users who now have permission to login to the ESXi host. You can verify by using the vSphere Client and taking a look at the Permissions tab.
vim-cmd vimsvc/auth/role_add
Note: The syntax for the privileges parameter lists only five, but it actually accepts as many as you need with the custom role
Tuesday, February 22, 2011
Automating Active Directory Domain Join in ESX(i) Kickstart
I recently received an email asking if it was possible to automate the joining of an ESXi 4.1 host to Active Directory within a kickstart installation. The simple answer is yes; you can easily do so using the same trick found in tip #7 in Automating ESXi 4.1 Kickstart Tips & Tricks post which connects locally to the vSphere MOB using python.
Having said that, there is a small caveat to this solution in which the credentials used to join the ESX(i) host must be exposed in clear-text, this may or may not be acceptable. A potential way of getting this to work is to create a dedicated windows service account with limited privileges to add only ESXi hosts to your domain.
The reason for the use of clear-text password is the vSphere API method that is used to join an ESX(i) host to AD domain, joinDomain(), only supports plain text.
Ideally, the password parameter should accept either an encrypted hash or some type of certificate which can be validated by Active Directory server. The actual solution is pretty straight forward, by crafting the appropriate call to vSphere MOB, we are able to generate a python script during the %firstboot section of ESX(i) kickstart installation which will execute upon the initial boot up of the host.
Update (03/26/11): Thanks to VMTN user klich who has found a solution to embed the password of the Active Directory user in a base64 encoding so that it is not visible in plain text for anyone who has access to the kickstart configuration file. To create the encoded hash, you will need access to either a ESX(i) or UNIX/Linux system which has python interpreter installed.
You will need to run the following command and substitute the password you wish to encode.
python -c "import base64; print base64.b64encode('MySuperSecurePasswordYo')"
The output will be your encoded hash:
Note: Make sure your ESX(i) hostname is configured with a FQDN (Fully Qualified Domain Name), else you will get an error when trying to join to AD domain.
The following snippet should be added to your %firstboot section of your kickstart. Remember to replace the variables: domainname, ad_username and encodedpassword with your configurations and make sure to leave the password variable blank.
ESX(i) 4.1 Update 1
After your ESXi host has completed installation, you can login to the ESX(i) host using the vSphere Client and you should see a recent successful task on the domain join process.
You can also verify by clicking on Configurations->Authentication Services and verify that your ESX(i) host is now part of the Active Directory domain.
In the next post, I will go over the details of adding domain users within the ESX(i) kickstart.
As a side note, it is quite unfortunate that ESXi only supports Microsoft Active Directory and does not integrate with other well known directory services such as NIS/NIS+, Kerberos, OpenLDAP, eDirectory, etc; this was actually possible with classic ESX. VMware has continued to focus on Windows-centric solutions and neglect the UNIX/Linux community by not supporting OS agnostic management tools that integrate with their vSphere platform. I hope this will change in the future for true interoperability.
Having said that, there is a small caveat to this solution in which the credentials used to join the ESX(i) host must be exposed in clear-text, this may or may not be acceptable. A potential way of getting this to work is to create a dedicated windows service account with limited privileges to add only ESXi hosts to your domain.
The reason for the use of clear-text password is the vSphere API method that is used to join an ESX(i) host to AD domain, joinDomain(), only supports plain text.
Update (03/26/11): Thanks to VMTN user klich who has found a solution to embed the password of the Active Directory user in a base64 encoding so that it is not visible in plain text for anyone who has access to the kickstart configuration file. To create the encoded hash, you will need access to either a ESX(i) or UNIX/Linux system which has python interpreter installed.
You will need to run the following command and substitute the password you wish to encode.
python -c "import base64; print base64.b64encode('MySuperSecurePasswordYo')"
The output will be your encoded hash:
Note: Make sure your ESX(i) hostname is configured with a FQDN (Fully Qualified Domain Name), else you will get an error when trying to join to AD domain.
The following snippet should be added to your %firstboot section of your kickstart. Remember to replace the variables: domainname, ad_username and encodedpassword with your configurations and make sure to leave the password variable blank.
ESX(i) 4.1
If you are using ESX(i) 4.1 Update 1 or if you run into the "urllib2.HTTPError: HTTP Error 403: Forbidden: Possible Cross-Site Request Forgery" you will need to use the following snippet below which includes the session cookie as part of the request to the vSphere MOB due to a recent CSRF patch from VMware identified by VMTN user klitch.
As a side note, it is quite unfortunate that ESXi only supports Microsoft Active Directory and does not integrate with other well known directory services such as NIS/NIS+, Kerberos, OpenLDAP, eDirectory, etc; this was actually possible with classic ESX. VMware has continued to focus on Windows-centric solutions and neglect the UNIX/Linux community by not supporting OS agnostic management tools that integrate with their vSphere platform. I hope this will change in the future for true interoperability.
Sunday, February 20, 2011
How to compile a statically linked rsync binary for ESXi
Running rsync on ESXi is not a new topic, there are a number of users in the community who have made this work. One well-known user, Dave Mishchenko, who runs vm-help and recently authored VMware ESXi: Planning, Implementation, and Security, provides an rsync binary that can be downloaded to run on ESXi. Not that I did not trust Dave and the source of his rsync binary, but I wanted to compile my own and understand the process Dave took to get to a compatible rsync for ESXi. In talking to Dave, I found out the binary was actually provided by a VMTN user, named glim who had noted some of the details in this VMTN thread.
The user basically created a statically linked versus a dynamically linked version of rsync which runs without relying on libraries that may or may not be present on ESXi. My initial attempts failed using both a 32/64bit CentOS 5.5 and latest version of rsync 3.0.7 and 3.0.3. The compilation of rsync was successful, but upon executing on ESXi, it immediately had a segmentation fault. I started to think there might be some dependencies in libraries that were used to compile rsync that I was missing. I was also in conversation with Alain Spineux, creator of MKSBackup regarding this same topic and he had found a way to compile rsync based on the article Hacking ESXi by Stjepan Groš.
Stejepan found that using CentOS 3.9, one could compile both statically and dynamically linked binaries without any modifications to ESXi. Though Stejepan did not provide the steps to compile rsync, it is relatively simple to do so. I also found that you do not need to use older 3.0.3 release of rsync, you can use the latest version which is 3.0.7 as of writing this article.
Before we get started, let me remind you of the disclaimer:
*** THIS IS PROBABLY NOT SUPPORTED BY VMWARE, USE AT YOUR OWN RISK ***
yum -y install gcc
tar -zxvf rsync-3.0.7.tar.gz
./configure
make CFLAGS="-static" EXEEXT="-static"
strip rsync-static
You now have a self-contain rsync binary labeled rsync-static found in the current working directory
Next up, you will SCP the rsync-static binary to your ESX(i) host and set the appropriate permissions to execute. I decided to temporarily store it in /tmp, but if you want the binary to persist through a reboot, you may want to store it in either a local or remote VMFS/NFS volume.
I quickly verified that rsync was in functional, by rsyncing /etc directory in ESXi over to another host. I use the following command:
/tmp/rsync-static --compress --progress --stats -e dbclient -Rr /etc build-centos:/upload
Once the file transfer has completed, you should see the /etc directory replicated on the destination host.
You may have noticed in the command line we are specifying "dbclient", by default rsync will go over SSH but ESXi does not actually have the traditional ssh client that we are all used to. It actually uses dropbear which is tiny SSH server and client which is frequently used in embedded systems such as Busybox.
Since /bin/ssh path does not exists, rsync will fail unless you specify the dbclient to the -e parameter.
Alain also found that retrieving peer name may fail and the rsync code currently does not properly handle this failure and just exits. By looking at the clientname.c source code which is part of rsync, Alain noted that even the developers left a comment that states the error could probably be handled without exiting. The fix is to just comment out the rsynerr line which does not force the program to exit when hitting this problem and this can be found on line 171.
Once you have made the changes, you just need to follow the instructions above to compile your static rsync binary. This is not necessary required, but if you run into this issue, this is a quick hack that can help.
I would like to thank Alain for sharing this tidbit of information
The user basically created a statically linked versus a dynamically linked version of rsync which runs without relying on libraries that may or may not be present on ESXi. My initial attempts failed using both a 32/64bit CentOS 5.5 and latest version of rsync 3.0.7 and 3.0.3. The compilation of rsync was successful, but upon executing on ESXi, it immediately had a segmentation fault. I started to think there might be some dependencies in libraries that were used to compile rsync that I was missing. I was also in conversation with Alain Spineux, creator of MKSBackup regarding this same topic and he had found a way to compile rsync based on the article Hacking ESXi by Stjepan Groš.
Stejepan found that using CentOS 3.9, one could compile both statically and dynamically linked binaries without any modifications to ESXi. Though Stejepan did not provide the steps to compile rsync, it is relatively simple to do so. I also found that you do not need to use older 3.0.3 release of rsync, you can use the latest version which is 3.0.7 as of writing this article.
Before we get started, let me remind you of the disclaimer:
*** THIS IS PROBABLY NOT SUPPORTED BY VMWARE, USE AT YOUR OWN RISK ***
Step 1
Download and install CentOS 3.9 i386 - You can find a list of download sites hereStep 2
With a default installation of CentOS 3.9, you just need to get GCC installed yum -y install gcc
Step 3
Next you will need to download the latest version of rsync which is 3.0.7 here and SCP it to your CentOS systemStep 4
Next you will now extract the contents of the rsync-3.0.7.tar.gztar -zxvf rsync-3.0.7.tar.gz
Step 5
Now you will change into the rsync-3.0.7 directory and run the configure command./configure
Step 6
Now you will run the make command which will compile rsync, but you will also be passing in additional flags to create a statically linked binarymake CFLAGS="-static" EXEEXT="-static"
Step 7
Finally, you will need to run the strip utility which discards all symbols from object filesstrip rsync-static
You now have a self-contain rsync binary labeled rsync-static found in the current working directory
/tmp/rsync-static --compress --progress --stats -e dbclient -Rr /etc build-centos:/upload
You may have noticed in the command line we are specifying "dbclient", by default rsync will go over SSH but ESXi does not actually have the traditional ssh client that we are all used to. It actually uses dropbear which is tiny SSH server and client which is frequently used in embedded systems such as Busybox.
Alain also found that retrieving peer name may fail and the rsync code currently does not properly handle this failure and just exits. By looking at the clientname.c source code which is part of rsync, Alain noted that even the developers left a comment that states the error could probably be handled without exiting. The fix is to just comment out the rsynerr line which does not force the program to exit when hitting this problem and this can be found on line 171.
I would like to thank Alain for sharing this tidbit of information
Tuesday, February 15, 2011
ESXi Lockdown mode does not play nice with vMA
Today on the VMTN community forums, a user identified an interesting side effect when using vMA's vilogger and enabling lockdown mode on an ESXi host. What the user found was the vilogger daemon stopped collecting logs when lockdown mode was enabled for an ESXi host. At first, I thought lockdown mode should have no affect on vilogger, as it only disables the "root" account from accessing ESXi host other than from the DCUI (Directo Console User Interface).
I replicated the setup in my lab using an ESXi host that was being managed by vMA via vi-fastpass (fpauth) and enabled vilogger for this host. I verified log collection was functional before enabling lockdown mode on the ESXi host, right away vilogger stopped collecting logs when lockdown mode was enabled. When using the "vilogger list" command, the status of the ESXi host goes from "collecting" to "No Permission". I found this to be quite odd and verified what the user was describing in his environment.
Next was to take a look at the vilogger logs which is stored in /var/log/vmware/vma/vilogd.log and I found the same "No Permission" error.
I decided to login to the ESXi host and tailed the hostd logs to see what was going on when lockdown mode was being enabled. What I found was pretty surprising to me, there was a task that removed permissions from the vi-adminXX user account, I was pretty sure at this point, the culprit was related to lockdown mode.
I decided to take a look at VMware's documentation to see what the behavior of Lockdown Mode was and the following snippet taken from vSphere's online documentation explains it all:
The text highlighted in red is the key to the issue the user is facing and specifically the very last section where it states:
This has a few implications for users considering Lockdown Mode:
I replicated the setup in my lab using an ESXi host that was being managed by vMA via vi-fastpass (fpauth) and enabled vilogger for this host. I verified log collection was functional before enabling lockdown mode on the ESXi host, right away vilogger stopped collecting logs when lockdown mode was enabled. When using the "vilogger list" command, the status of the ESXi host goes from "collecting" to "No Permission". I found this to be quite odd and verified what the user was describing in his environment.
I decided to take a look at VMware's documentation to see what the behavior of Lockdown Mode was and the following snippet taken from vSphere's online documentation explains it all:
you cannot run vCLI commands from an administration server, from a script, or from vMA against the hostThis meant that not only the root account was locked out, but all other accounts found on the ESXi host whether they are custom from your environment or from auxiliary systems such as VMware vMA, would be completely disabled. What is even more interesting, even read-only accounts would no longer function, they too had to go to vCenter to be re-proxied to specific ESXi host.
This has a few implications for users considering Lockdown Mode:
- All scripts including resxtop and user authentication must go through vCenter. If vCenter went down, you have no remote way to access your ESXi host. This also meant that you could not remotely start up vCenter if it was hosted in a virtual machine but rather from DCUI after enabling Local Tech Support Mode
- The use of vMA's vilogger is completely useless when Lockdown Mode is enabled for ESXi host. Users may want to consider setting up a traditional syslog server and have the logs forwarded from the ESXi host
How to compile Busybox for ESXi ... kind of Part 2
Continuing from part1 of How to compile Busybox for ESXi ... kind of Part 1 we found some challenges in compiling VMware's Busybox version found in ESXi OSS Source Code. In this article, we will go through process of compiling the latest version of Busybox which is currently 1.18.3 to run on ESXi.
Again, before getting started, a word of caution:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!
The build environment that I used is running the latest version of CentOS 5.5 64bit which can be downloaded here.
yum install -y gcc flex bison texinfo ncurses-devel libselinux-devel.x86_64 pam-devel.x86_64
tar -xvjpf busybox-1.18.3.tar.bz2
make help
You can customize the build of Busybox by enabling and disable specific applets to be compiled. If you want to build Busybox with no applets (not really useful), you can run the following command:
make allnoconfig
make
Once the compilation is complete, you will now have a Busybox binary in which you can run but it does nothing useful:
You can also perform the exact opposite by enabling all options which can be called using the "allnoconfig" or "defconfig", you can run the following command:
make defconfig
make
Once the compilation is complete, you now have Busybox binary which has all the available applets by default:
Enabling all applets, the Busybox binary still comes out to be less than 900k. If you wanted to create a custom Busybox with specific applets, you can manually edit the .config file which is where the features are either enabled or disabled. This can get very tedious, luckily there is a make option which allows for an interactive menu on selecting the applets you would like to include in the build.
make menuconfig
From here, you will be able to configure the Busybox settings and the various applet types by their functions such as networking or mail utilities. One feature that is actually disabled in the VMware's version of Busybox is the support for large files, if you tried to tar up a file that was larger than 2GB and then extract the file, you will notice you get an error regarding the file size. The reason is VMware decided to not enable this feature in their Busybox build.
make
If you would like to customize the version information found when just running busybox command without any arguments, you can edit the Makefile and append your custom text to EXTRAVERSION variable.
Now you just need to re-run the "make" command and your new busybox binary will include the additional version information.
Now before you jump and start creating your own Busybox binary, I must throw in another caveat that re-iterates the title of this post. Just because you have Busybox built with all these applets, it does not mean it will run 100% on ESXi, the reason being is there are specific dependencies that the non-VMware Busybox applets may rely on which are just not available with ESXi. This is one of the reasons why VMware did not just enable all applets from Busybox to begin with, your mileage will vary depending on the particular command you have enabled.
Here is an example of my custom Busybox applet "crontab" using the "-l" flag which lists any active cron entries for the particular user:
It runs just fine on ESXi as you can see, but let's try using the "-e" flag which if you are familiar with using crontab you will know that is the option to edit your cron entries:
Ooops, what's this? It looks like this particular operation of crontab relies on vfork which is not implemented. I have seen other similar errors with other applets relying on particular dependencies in the environment that just does not exist with ESXi. I have not tested every single applet and some may work 100% while others may be partially functional, you will need to test and verify for yourself.
If you feel adventurous, go ahead and download Busybox and start playing with it. I would highly recommend that you test this in your vSphere development environment before trying this on a production host. I would also recommend that you do NOT replace the default Busybox binary found on ESX(i) as you can run into some serious issues. Since the Busybox binary is self-contained, you can store it in /tmp or somewhere more permanent such as a local or remote VMFS datastore and renaming the file will also remove any confusion.
If you find other utilities that you feel that should be included in the default Busybox applet and do not want to resort to something like Poor man's traceroute for ESXi, be sure to submit your feedback to either your VMware rep or file a feature request.
If you are interesting to learning more about Busybox, check out their FAQ page and this page for more details.
Again, another warning:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!
Again, before getting started, a word of caution:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!
The build environment that I used is running the latest version of CentOS 5.5 64bit which can be downloaded here.
Step 1
Download busybox-1.18.3.tar.bz2 from the Busybox's website and SCP it to your build system. Step 2
You will also need to install the following packages, you can do so using yum if you are using CentOS or RHEL and have a proper repository configured. You can use the following:yum install -y gcc flex bison texinfo ncurses-devel libselinux-devel.x86_64 pam-devel.x86_64
Step 3
Extract the contents of busybox-1.18.3.tar.bz2 using the following command:tar -xvjpf busybox-1.18.3.tar.bz2
Step 4
You will now change into the busybox-1.18.3 directory and from here you have a few ways of building Busybox. To get a list of build options, use the following command:make help
You can customize the build of Busybox by enabling and disable specific applets to be compiled. If you want to build Busybox with no applets (not really useful), you can run the following command:
make allnoconfig
make
Once the compilation is complete, you will now have a Busybox binary in which you can run but it does nothing useful:
make defconfig
make
Once the compilation is complete, you now have Busybox binary which has all the available applets by default:
Step 5
To enable the interactive menu, you will run the following command:make menuconfig
Step 6
Once you are done selecting or de-selecting all your Busybox settings and applets, you will need to save your configuration which is stored in the .config.
Step 7
Once you have exited from the interactive Busybox menu, you are now ready to build your custom version of Busybox, you will run the following command:make
Step 8
If everything went well, you now should have a busybox binary in your current working directory. Here is an example of my custom Busybox image:
Here is an example of my custom Busybox applet "crontab" using the "-l" flag which lists any active cron entries for the particular user:
If you feel adventurous, go ahead and download Busybox and start playing with it. I would highly recommend that you test this in your vSphere development environment before trying this on a production host. I would also recommend that you do NOT replace the default Busybox binary found on ESX(i) as you can run into some serious issues. Since the Busybox binary is self-contained, you can store it in /tmp or somewhere more permanent such as a local or remote VMFS datastore and renaming the file will also remove any confusion.
If you find other utilities that you feel that should be included in the default Busybox applet and do not want to resort to something like Poor man's traceroute for ESXi, be sure to submit your feedback to either your VMware rep or file a feature request.
If you are interesting to learning more about Busybox, check out their FAQ page and this page for more details.
Again, another warning:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!
Sunday, February 13, 2011
How to compile Busybox for ESXi ... kind of Part 1
I had initially researched this topic a few months back and decided not to write about it since it did not yield the results I was expecting. Some recent twitter conversations with Carter Shanklin and Didier Pironet this morning motivated me to dig up this content and share it with the community.
Before I get started, a word of caution:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!
As you all know, ESXi does not have a Service Console like the classic ESX, but it does have a limited console based on Busybox, specifically v1.9.1 which is still the current version even with the latest release vSphere 4.1 Update 1. Busybox is heavlily customized by VMware and you will find a limited set of POSIX-like utilities, it is no where feature complete to your traditional POSIX environment and there is good reason for this.
If you have ever downloaded vSphere Hypervisor (formally known as ESXi), you may have noticed an "Open Source" tab at the top. This page contains some of the open source licensing information that VMware has used and incorporated into their products, most of these are just text files listing the various packages. One file that is of interest is the "OSS Source Code for ESX, ESXi and ESXi Embedded 4" which actually includes some of the open source RPMs and packages used to build Busybox for both classic ESX and ESXi (Yes, classic ESX also includes a busybox binary).
One interesting thing to note, is the date of this package, it has not been updated for almost two years. I am not sure if this is done intentionally, but I doubt that would be the case. It is also odd that VMware is still using an old release of Busybox (1.9.1), the latest release is at 1.18.3 and there have been quite a few updates between 1.9.1 and the current release. There has been speculation and rumors in the past that VMware choose a particular version of Busybox to get around the licensing of Busybox with ESXi. I do not know if this is true, but I do know, there have been quite a few bug fixes since incarnation of ESXi and you may hit some of those if you venture into Busybox console also known as Tech Support Mode (formally known as Unsupported Console)
The build environment that I used is running the latest version of CentOS 5.5 64bit which can be downloaded here.
yum install -y gcc flex bison texinfo ncurses-devel libselinux-devel.x86_64 pam-devel.x86_64
mkdir -p /usr/src/redhat/SOURCES
Next, you will install the vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm by using the following command:
rpm -ivh vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm
Note: Don't worry about the warnings regarding the user/groups that do not exists, these are most likely a default from VMware's environment that expects the mts user and group.
You will now extract the contents of the Busybox archive by using the following command:
tar -xvjpf busybox-1.9.1.tar.bz2
Next you will copy the vmware.patch into the busybox-1.9.1 directory by using the "cp" command and then change busybox-1.9.1 directory. If you do a "ls" you should see the following files:
patch --dry-run -p1 < vmware.patch
To apply the patch, you will run the following command:
patch -p1 < vmware.patch
After the patch has been applied, there are two interesting files that will be created in addition to the source code updates:
Basically you will be executing the following commands:
VMWARE_TARGET=visor
source vmware-env.sh
cp ../config-vmware-visor .config
make oldconfig
busybox-build
Now if you executed the above, you will find that you quickly run into some build errors.
This is where the "kind of" part of the blog post comes in. If you recall, we sourced the vmware-env.sh script just prior to building, this is a script that VMware wrote to quickly setup the build environment for compiling Busybox for either ESX or ESXi.
For this particular build and release, they are using specific versions of GCC, Binutils and PAM. You might ask why not substitute these with the versions found on system? I have actually gone through that, but ran into other compliation issues which maybe due to fact that GCC and other components are much newer and not compatible with older versions of Busybox. I have also tried to match the build environment by using buildroot but continue to hit other compilation issues. The only way I can see building this particular patched version of Busybox is to get a build environment that matches what VMware uses.
In Part2, I will go into details on building your own Busybox binary for ESXi which I promise will be much more fruitful than this. Stay tuned ...
Update: It looks like there is a slightly more recent version of ESX(i) OSS released on 07/13/2010, for whatever reason, it was filed under VMware View section. The source RPM for ESXi still looks like the same one released back in 2009, but there some other packages that differ from the one found on vSphere Hypervisor page.
Before I get started, a word of caution:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!
As you all know, ESXi does not have a Service Console like the classic ESX, but it does have a limited console based on Busybox, specifically v1.9.1 which is still the current version even with the latest release vSphere 4.1 Update 1. Busybox is heavlily customized by VMware and you will find a limited set of POSIX-like utilities, it is no where feature complete to your traditional POSIX environment and there is good reason for this.
If you have ever downloaded vSphere Hypervisor (formally known as ESXi), you may have noticed an "Open Source" tab at the top. This page contains some of the open source licensing information that VMware has used and incorporated into their products, most of these are just text files listing the various packages. One file that is of interest is the "OSS Source Code for ESX, ESXi and ESXi Embedded 4" which actually includes some of the open source RPMs and packages used to build Busybox for both classic ESX and ESXi (Yes, classic ESX also includes a busybox binary).
The build environment that I used is running the latest version of CentOS 5.5 64bit which can be downloaded here.
Step 1
Download VMware-esx-public-source-4.0-162945.tar.gz (565MB) from from VMware's website and SCP it to your build systemStep 2
You will also need to install the following packages, you can do so using yum if you are using CentOS or RHEL and have a proper repository configured. You can use the following:yum install -y gcc flex bison texinfo ncurses-devel libselinux-devel.x86_64 pam-devel.x86_64
Step3
Create a temporarily directory and move the tar.gz file into the directory and then extract the contents using the following command:tar -zxvf VMware-esx-public-source-4.0-162945.tar.gzYou now should have a bunch of *.rpm files that have been extracted. There are two interesting RPM packages that we are interested that have "busybox" in the name, one of classic ESX (vmware-esx-busybox-1.9.1-1.624.vmw.src.rpm) and one for ESXi (vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm)
Step 4
You will need to create the following directory before installing the ESXi RPM package. Use the following command:mkdir -p /usr/src/redhat/SOURCES
Next, you will install the vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm by using the following command:
rpm -ivh vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm
Step5
Now you will change into the /usr/src/redhat/SOURCES directory and you should see three files:- busybox-1.9.1.tar.bz2 (Busybox v1.9.1)
- config-vmware-visor (Busybox config file from VMware)
- vmware.patch (VMware patch for building Busybox)
tar -xvjpf busybox-1.9.1.tar.bz2
Next you will copy the vmware.patch into the busybox-1.9.1 directory by using the "cp" command and then change busybox-1.9.1 directory. If you do a "ls" you should see the following files:
Step 6
We are now going to apply the patch provided by VMware which will update some of the C source code found in default Busybox 1.9.1 along with setting up a build environment. We go through a dryrun process to see what files will be patched, use the following command:patch --dry-run -p1 < vmware.patch
patch -p1 < vmware.patch
- README.vmware (Pretty well documented set of changes + how to build Busybox)
- vmware-env.sh (environment script that is sourced to build Busybox)
Step7
Now we are ready to build Busbox, the exact instructions are specified towards the end of the README.vmware file 
VMWARE_TARGET=visor
source vmware-env.sh
cp ../config-vmware-visor .config
make oldconfig
busybox-build
Now if you executed the above, you will find that you quickly run into some build errors.
In Part2, I will go into details on building your own Busybox binary for ESXi which I promise will be much more fruitful than this. Stay tuned ...
Update: It looks like there is a slightly more recent version of ESX(i) OSS released on 07/13/2010, for whatever reason, it was filed under VMware View section. The source RPM for ESXi still looks like the same one released back in 2009, but there some other packages that differ from the one found on vSphere Hypervisor page.
Tuesday, February 8, 2011
Another way to enable management traffic on ESXi
Here is another way in you can enable the management traffic type on a VMkernel interface in ESXi without having to resort to using the vSphere API, this especially useful when automating a kickstart installation.
When you enable a specific vmkX interface (esxcfg-vmknic -l) to allow for the management traffic type, there is an entry that is made in the /etc/vmware/hostd/hostsvc.xml file. The specific interface is denoted by a unique nic id which starts off at 0000 and is incremented by one for additional VMkernel interfaces that are added.
If you add a second VMkernel interface called vmk1 and you wanted to also enable it for management traffic, the file would look like the following:
For the changes to take effect, you will need to restart hostd agent by running /etc/init.d/hostd restart. You will also need refresh the network sub-system by using vim-cmd hostsvc/net/refresh which will refresh the vSphere Client view else you will have to do it manually before you can see the update.
This is probably overkill, but I decided to write a simple script in which you can pass the VMkernel interface name and the script will update hostsvc.xml file with the proper nic id/etc. Here is an example for enabling management traffic for vmk1:
Download: enableMgmt.sh
If you would like to integrate this into your ESXi kickstart, you can easily do so based on the number of VMkernel interfaces you will be creating during the installation. You can add the following into your %post section which uses a here document to overwrite the existing hostsvc.xml with the expected VMkernel interfaces that should have the management traffic type enabled.
This is a cleaner alternative than using python and connecting to the vSphere API locally on an ESXi host which is described in my blog article Automating ESXi 4.1 Kickstart Tips & Tricks Tip #7
When you enable a specific vmkX interface (esxcfg-vmknic -l) to allow for the management traffic type, there is an entry that is made in the /etc/vmware/hostd/hostsvc.xml file. The specific interface is denoted by a unique nic id which starts off at 0000 and is incremented by one for additional VMkernel interfaces that are added.
If you would like to integrate this into your ESXi kickstart, you can easily do so based on the number of VMkernel interfaces you will be creating during the installation. You can add the following into your %post section which uses a here document to overwrite the existing hostsvc.xml with the expected VMkernel interfaces that should have the management traffic type enabled.
HOSTSVC_FILE=/etc/vmware/hostd/hostsvc.xml
cat > ${HOSTSVC_FILE} << __CREATE_HOST_SVC__
<configroot>
<mangementvnics>
<nic id="0000">vmk0</nic>
<nic id="0001">vmk1</nic>
<nic id="0002">vmk2</nic>
</mangementvnics>
<mode>normal</mode>
<service>
<tsm>on</tsm>
<tsm-ssh>on</tsm-ssh>
<ntpd>on</ntpd>
</service>
</configroot>
__CREATE_HOST_SVC__
This is a cleaner alternative than using python and connecting to the vSphere API locally on an ESXi host which is described in my blog article Automating ESXi 4.1 Kickstart Tips & Tricks Tip #7
Thursday, February 3, 2011
How to backup/restore vMA's config + vi-fastpass DB
I recently wrote an article on the process of cloning vMA which provides a way of backing up vMA. Due to the ease of deploying vMA as a virtual appliance (using OVF), there aren't too many reasons you would need to backup this virtual machine. If you lost the system for whatever reason, you can easily re-deploy with just a few clicks.
However, having said that, if you extensively make use of vi-fastpass fpauth and manage a lot of targets whether they are ESX(i) or vCenter hosts, you need to understand it is not simply just re-deploying another vMA host. When you add a target to vMA's vi-fastpass, two accounts are provisioned on the host "vi-adminXX" and "vi-userXX", these accounts are associated with an encrypted cipher located on vMA which allows for "fastpass" access to the host without having to re-type the password to the host each time. If you were to re-deploy a new vMA host and add the targets again, your host will not only contain the old entries but now a new set of accounts for your new vMA host. This can be an issue as you start to have stale accounts on your ESX(i) or vCenter host.
To prevent this issue, you can actually backup both vMA's configurations which is primarily stored in a sqlite database and the vi-fastpass credential store. In the following example, I have two ESXi hosts being managed by my primary vMA and I also have a standby vMA DR host in which I will backup the files to.
First, you want to make sure you do not have any active vifptarget sessions, this is not a requirement but it can ensure you do not copy over any vMA "session cache" files to your DR site. You can check by doing a long listing in /home/vi-admin/.vmware and looking for the directory vmasessioncache which will contain any active cached sessions if you recently initialized a fastpass target.
Note: Again this is not really necessary and you can exclude the vmasessioncache directory as part of your backup
You will first need to "dump" the existing vMA database into a file and provide a name of your choice, you will need to run the following command:
You now should login to your vMA DR host and you should see only two files in the home directory of the vi-admin user: vMA.conf and vMA.db.backup (.vmware directory is a hidden directory in /home/vi-admin)
From here, you will restore vMA.conf and you will need to run the following command:
At this point, we can verify the database contents by just running the ".dump" command:
Now, we're not done yet, we need to run one additional tool that will perform some VMware "black magic" which will allow this new vMA to access all your ESX(i) and/or vCenter targets just as you had it before on your primary vMA.
You will need to create a file that provides some dynamic shared libraries for the tool we are going to execute. Create a file under /etc/ld.so.conf.d/vmware-vma using "sudo" and paste the following two lines:
Now you will need to run the following to read in the configuration:
Once you see the successful message on completing the migration of your credential store, you have now fully restored your original vMA configuration. Here we perform a list of the active servers that was once accessible on primary vMA and initializing a target and verifying that it does in fact work.
One thing to note, if you still have access to your primary vMA, both your vMA's are now in an active/active state, with caveat that your primary vMA is the only one allowed to make any changes. What I mean by this, is when you initially add a host to vMA's fastpass, it not only creates two accounts, but it also associates it's system's UUID as part of the unique identifier which is stored on the host with the key VMAID
This means that if you deleted the target off your vMA DR host, it does not actually remove this entry on your ESX(i) and/or vCenter host. Only the primary vMA which has the matching UUID is able to remove the entry all together when you perform a "vifp removeserver" operation.You can see the system UUID by using the dmidecode utility.
We can also see this within the vMA database, when viewing this on the vMA DR host, you will actually see both entries of the primary and the DR vMA UUIDs because we restored the database with the original vMA's config.
If you need your vMA DR host to be able to modify entries or rotate passwords, you will need to shutdown vMA and update it's bios.uuid within the .vmx entry. You must use the "original" vMA's UUID which you should see from the database by running the following command:
However, having said that, if you extensively make use of vi-fastpass fpauth and manage a lot of targets whether they are ESX(i) or vCenter hosts, you need to understand it is not simply just re-deploying another vMA host. When you add a target to vMA's vi-fastpass, two accounts are provisioned on the host "vi-adminXX" and "vi-userXX", these accounts are associated with an encrypted cipher located on vMA which allows for "fastpass" access to the host without having to re-type the password to the host each time. If you were to re-deploy a new vMA host and add the targets again, your host will not only contain the old entries but now a new set of accounts for your new vMA host. This can be an issue as you start to have stale accounts on your ESX(i) or vCenter host.
To prevent this issue, you can actually backup both vMA's configurations which is primarily stored in a sqlite database and the vi-fastpass credential store. In the following example, I have two ESXi hosts being managed by my primary vMA and I also have a standby vMA DR host in which I will backup the files to.
You will first need to "dump" the existing vMA database into a file and provide a name of your choice, you will need to run the following command:
sqlite3 /var/opt/vmware/vMA/vMA.db .dump > vMA.db.backupNext, you will need to "scp" the following files to your vMA DR:
- vi-fastpass encrypted credential store file
- /home/vi-admin/.vmware
- vMA's configuration dealing with vi-fastpass targets + vi-logger
- /home/vi-admin/vMA.db.backup
- vMA's default logging configs + paths
- /etc/vmware/vMA/vMA.conf
sudo mv vMA.conf /etc/vmware/vMA/vMA.confNext, you will restore vMA.db and you will need to run the following command:
sudo sqlite3 /var/opt/vmware/vMA/vMA.db < vMA.db.backup
sqlite3 /var/opt/vmware/vMA/vMA.db .dump
You will need to create a file that provides some dynamic shared libraries for the tool we are going to execute. Create a file under /etc/ld.so.conf.d/vmware-vma using "sudo" and paste the following two lines:
- /opt/vmware/vma/lib
- /opt/vmware/vma/lib64
sudo ldconfig -f /etc/ld.so.conf.d/vmware-vmaNow you are ready to run the "migratecredstore" utility which is located under /opt/vmware/vma/bin which will perform the "black magic" and make sure you use sudo.
sqlite3 /var/opt/vmware/vMA/vMA.db "select * from management_info;"You will also need to delete the "new" UUID to ensure that only one exists which should be the "original" UUID, you can so by running the following command substituting your UUID:
sudo sqlite3 /var/opt/vmware/vMA/vMA.db "delete from management_info where myUUID='422E4042-63EE-86D1-D22A-79B6ABCA8D68';"At this point, your vMA DR is now your primary and your old vMA is no longer needed.
Subscribe to:
Posts (Atom)