virtuallyGhetto

esxcli Part2 – Automating esxcli using vMA

by 3 Comments

In the previous article, esxcli Part1, we discussed what esxcli is and what it can be used for. In this article, we will show you how you can automate and perform a common esxcli operation against a set of ESX and/or ESXi hosts using vMA. As mentioned in the previous article, esxcli needs to be executed against a specific host, it is not vCenter aware and cannot connect to vCenter to perform an operation against a host.

One of the pre-requisites before starting is that all ESX and ESXi hosts need to be managed by vMA and accessible using vi-fastpass authentication. Please refer to the vMA documentation on configuring this on vMA.

Download:esxcli-automation.pl

The following script is based off of useVIFastpassOnvMAToRunPerlScriptWithoutClearTextPassword.pl. The modified script utilizes the vi-fastpass library to properly access a set of target(s) and performs the specified esxcli operation without having to provide credentials to each and every host.

The following example will demonstrate iSCSI multipath configuration on two hosts using the method described by Duncan Epping in this article.

1) Create a file containing the list host to perform the operations against:

[[email protected] ]$ cat hosts
esxi4-1.primp-industries.com
esxi4-3.primp-industries.com

2. Run the script with the specific esxcli parameters as you would normally by passing it into the --cmd_option argument (double quote the arguments):

Binding iSCSI interface to vmk0

[[email protected] ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic add -n vmk0 -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
Executing script on "esxi4-3.primp-industries.com" ...
Binding iSCSI interface to vmk1

[[email protected] ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic add -n vmk1 -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
Executing script on "esxi4-3.primp-industries.com" ...

3. Verify iSCSI multipathing has been properly configured:

[[email protected] ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic list -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
vmk0
pNic name: vmnic0
ipv4 address: 172.30.0.183
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:69:7a
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 167051261
packets sent: 232132
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
vmk1
pNic name: vmnic1
ipv4 address: 172.30.0.184
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:68:6c
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 145924
packets sent: 54502
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
Executing script on "esxi4-3.primp-industries.com" ...
vmk0
pNic name: vmnic0
ipv4 address: 172.30.0.233
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:33:95
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 138535
packets sent: 8026
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
vmk1
pNic name: vmnic1
ipv4 address: 172.30.0.234
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:27:65
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 145924
packets sent: 112
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A

Now you will be able to automate any of the supported esxcli operations against multiple hosts!

In Part3, we will take a look at automating esxcli operations in a Windows environment using Powershell.

esxcli Part1 – What is esxcli?

by 8 Comments

esxcli is a new CLI (commandline interface) framework in vSphere that provides a modular architecture for various components called namespaces running in the VMkernel. Some of these namespaces are nmp (Native Multipathing Plugin) for the new VMware Pluggable Storage Architecture, corestorage for claim rules used for masking certain devices to a host, and swiscsi for managing iSCSI interface.

esxcli can be executed within the classic ESX Service Console, the unsupported Busybox console in ESXi or using the vCLI's remote version of esxcli. There are currently 3 namespaces (nmp,corestorage and swiscsi) with the current release of vSphere and we may see others introduced in future releases of vSphere. One important thing to note is that because these modules run within the host, using the vCLI's version of esxcli, you will need to authenticate to the host first to see what modules will be available for access. Currently, esxcli is not vCenter aware, meaning you must connect to a specific ESX or ESXi host when performing an operation.

Here is an example of esxcli executed without connecting to the host first:

Here is an example of esxcli being executed after connecting to the host:

When invoking the esxcli command, you may also notice an esxcli.log is generated. If the command is successfully executed, the log will generally be empty, but if there was an error you may want to take a look at esxcli.log if the command does not provide any output to the screen.

Here is an example of using an auth configuration file and because of the case sensitivity of esxcli, the VI_PROTOCOL entry is failing with HTTPS vs https:

[[email protected] ~]$ cat esxcli.log

[root CRITICAL] Exception:Unsupported protocol
[root CRITICAL] Traceback (most recent call last):
File "esxcli.py", line 387, in _GetStub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 239, in stub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 299, in Login
Exception: Unsupported protocol

There is not a whole lot of information available to the public about esxcli. From what I understand after talking to a few VMware engineers, esxcli has an API, but it is currently not exposed to the public for consumption. Not only is there an API, but 3rd party providers or users can potentially create their own modules and install it using the VIB format also known as vSphere Installation Bundle.

Some well known packages that are currently being distributed in the VIB format today are: Cisco Nexus 1000V VEM, HP Insight Manager Agent, EMC PowerPathV/E, Xsigo ESX IB drivers and VMware ESX/ESXi/vMA updates, to name a few. Hopefully in the future, VMware will expose the esxcli API functionality to the developer ecosystem.

Here are a few blog posts with detail examples on using esxcli with the various namespaces:

Stay tuned for Part2 and Part3 where we will look at automating esxcli operations using both the vSphere SDK for Perl and Windows PowerShell!

Is the release vSphere 4.0 Update 2 imminent?

by Leave a Comment

There's already been some chatter in the community regarding vSphere 4.0 Update 2, including a VMware KB article:

http://kb.vmware.com/kb/1021023
http://www.yellow-bricks.com/2010/03/30/whats-the-point-of-setting-iops1/
http://www.vladan.fr/vsphere-4-update-2-is-next/
http://virtualization.info/en/news/2010/03/vsphere-40-update-2-to-add-new-ha.html
http://www.yellow-bricks.com/2010/03/29/cool-new-ha-feature-coming-up-to-prevent-a-split-brain-situation/

While recently downloading a copy of ESX 4.0u1 for testing, I noticed the download URL was the following:

http://www.vmware.com/downloads/download.do?downloadGroup=ESX40U1

The link will generally take you to landing page to login if you have not ready logged in:

I thought I try to plug-in "2" and see what happens, to my surprise it actually displayed some interesting text:

http://www.vmware.com/downloads/download.do?downloadGroup=ESX40U2

When you login, it will throw an error that the download group does not exists.

However, if you tried another invalid entry say "ESX40U3", you'll get an error all together such as the following even before logging in:

http://www.vmware.com/downloads/download.do?downloadGroup=ESX40U3

So does this mean vSphere 4.0 Update 2 about to be released? Your guess is as good as mine, we will just have to wait and see.

Filed Under: Uncategorized Tagged With:

How to remove stale targets from vMA

by Leave a Comment

If you have used vMA's vi-fastpass authentication, you will know how easy it is to setup using vifp utility which supports both ESX/ESXi and vCenter targets.

Here's an example of adding ESXi target:

[[email protected] ~]$ sudo vifp addserver esxi3-1.primp-industries.com
*protected email*'s password:

Here's an example of the listing of the available fastpass targets:

[[email protected] ~]$ sudo vifp listservers
esxi3-1.primp-industries.com ESXi

During this process, two accounts (vi-userXX & vi-adminXX) are created on the target host with a password that vMA management creates and caches it locally in an obfuscated but not encrypted form. This will allow you to initialize a fastpass target using vifpinit utility and execute commands against the target host without having to manually type in the credentials.

The fastpass targets are stored in 2 configuration files on vMA:

1) The obfuscated cached credentials is stored in /home/vi-admin/.vmware/credstore/vicredentials.xml

If you cat out the contents, it will look something like this:

2) A More detailed configuration for each of the targets along is stored in /etc/vmware/viconfig/viconfig.xml

If you cat out the contents, it will look something like this:

What happens when you rebuild your host, or the system is no longer available because it has been decommissioned or being used for another purpose? vMA will still think it's managing the host and the fastpass credentials will no longer function as the account is no longer valid the host. If you try to remove the old target, you will see the following error:

[[email protected] ~]$ sudo vifp removeserver esxi3-1.primp-industries.com
*protected email*'s password:

Error: Failed to connect. Please make sure the server is up and is of supported version.

The reason this occurs is that vMA is unable to login to the host and remove the two accounts that were initially created and fails to remove the target. What you will need to do is actually pass in an additional parameter to vifp command "--force" which will forcefully remove the target from vMA management. This command actually does not require the user to enter the correct password to the host even if it is still reachable by vMA. By specifying this flag and providing some input when prompted for the password, vMA will purge the target from it is system.

[[email protected] ~]$ sudo vifp removeserver esxi3-1.primp-industries.com --force
*protected email*'s password:

After a target is removed from vMA, it is also removed from the two above files. You do not manually tweak either of these configuration files or it may lead to issues on your vMA host.

Best practice for decommissioning a host that has been added to vMA's management is the following:

  1. Disable vilogger if you've enabled it for the host
  2. Remove target from vMA management
  3. Verify the host is no longer being managed by vMA
  4. Decomission host

Increase Syslog count in ESXi using Busybox

by 2 Comments

There was an interesting question on the ESXi forum today using syslog and logging to a local VMFS volume and potentially filling up the datastore. By default, the log files are 2MB each before they are rotated out with a total of 9 copies all together. This ensures that the logs will not grow beyond 18MB and potentially fill up your datastore.

Within the unsupported console of ESXi, there is a little tool called "Busybox" which implements a set of familiar command line utilities called applets. Some of these applets are uptime, crond, chroot, md5sum, etc... If you login to the unsupported console and just type "busybox" on the console, you will see the following:

If you look carefully, you will see that one of the applets is "syslogd", which is the syslog daemon running in ESXi. To access a specific applet, you will just type busybox and then the name of the applet along with -h which forces the applet to provide a set of available options. If we do this for syslogd, you will see the following:

The default configurations should ensure that your logs don't go crazy and fill up the volume, but the downside is that your log history will not be kept forever. That is why it is a best practice to setup a remote syslog server to send all your logs for further processing and auditing if necessary.

However, if you wanted to change this defaut, you can. As you can see from the options above, you can configure not only the size (max of 2MB) before rotating out, but also the number of logs to keep (max of 99). While you cannot increase the size of the individual logs, you can specify a larger number of logs to keep. If you want to make the changes live while the system is running, you'll need to perform the following:

1. Find the current pid of syslogd

~ # ps | grep syslog
4313 4313 busybox syslogd

2. Kill the syslog process

~ # kill $(cat /var/run/syslogd.pid)

3. Let's say you want to keep 50 copies instead of 9

~ # busybox syslogd -b 50

Note:

  • The input to -s is in bytes with range (0-2097151)
  • The input to -b has range (1-99)

It's important to note that this change will not persist through a reboot. I have not been able to figure out where this is set; it could just be hardcoded in the binary by default. A way around this is to re-define the configuration upon boot up by adding an entry to /etc/rc.local which will kill the current running syslogd and then start up with the new parameters as shown above.

Add the following lines to the end of /etc/rc.local:

kill $(cat /var/run/syslogd.pid)
busybox syslogd -b 50

Now for this change to fully persist, you need to do one more thing, you'll need to run /sbin/auto-backup.sh which will force a local backup of the ESXi configuration files which includes /etc/rc.local so that it'll survive through the next reboot. Now you'll be able to store additional ESXi logs for a longer period of time if you choose to log locally.

The vExpert 2010 Crew

by Leave a Comment

The wait is finally over! The highly anticipated e-mail from John Troyer announcing the second group of individuals for the vExpert 2010 award went out today. I'm very honored to be accepted into the vExpert 2010 program and look forward to sharing it with a great group of individuals! I want to thank John for organizing this program and for all the hard work he has put in to recognizing everyone. Congratulations to the vExpert 2010 crew! Let's make 2010 a great year!

Here are some links/blog posts to the other vExpert's for 2010:

Arnim van Lieshout
Tom Howarth & Wil van Antwerpen
Brian Knudtson
Jase McCarty
Maish Saidel-Keesing
Barry Coombs
Joe Kelly
Mike Laverick
Paul Davey
Aaron Delp
Erik Scholten
Justin Emerson
Didier Pironet
Carlo Costanzo
David Marshall
Scott Lowe
Mark Vaughn
Ed Saipetch

Additional Links:
http://www.vmware.com/communities/vexpert/
http://www.yellow-bricks.com/2009/10/01/vexperts-2010/

Here is a vExpert 2010 twitter list put together by @maishsk that you can follow:
http://twitter.com/maishsk/vmware-vexpert-2010