virtuallyGhetto

My VCAP4-DCA beta experience

06/21/2010 by William Lam 5 Comments

Over the weekend I participated in the VMware VCAP4-DCA beta exam, which is set to release later this year. My exam consisted of 41 live labs and I completed the exam in a little under 4.5 hours. Due to the NDA restriction, I will not be able to go into the specifics of the exam but I can share with you my experience while taking the exam.

The biggest issue I had while taking the exam was the user experience, which for me personally sucked! After 30 minutes or so into the exam, I noticed the display started to overlay multiple images on top of each other and not refreshing properly. Not only did I have interlaced images, certain parts of the screen were blank white and required multiple minimizations to force a redraw of the screen, which did not always work. This was very frustrating to deal with throughout the exam and at one point I had to call the proctor in since I was not able to figure out what I was clicking on as it actually was selecting something else. I ended up moving forward with the exam since I was already half way into the exam and there was nothing more the testing facility could have done. I did have the staff create an incident report which will be submitted to VMware for feedback on the display and connectivity issue.

After the exam ended, I spoke to the proctors and they had mentioned they had a similar issue earlier in the week with another candidate taking a VMware certification. They did not say if it was for VCAP-DCA but did say it was not an issue with Pearson but with the actual labs at VMware where these environments were hosted. If a user paid $400 for this exam and you received the type of experience I had, you would not be a happy camper. VMware still has some work to do in this area before the exam is ready to roll out later this year.

Other than a few minor issues/typos with some of the questions and pristineness of the environment, I was very impressed overall with the exam. I thought that the questions and tasks were very fair, some definitely took much longer to complete than others, so be cautious of your time. I thought I would be able to go back and check on some of the questions I had marked to come back to but ended up running out of time at the end. Luckily, I was able to provide comments on every question, though not in the detail as I would have liked. Looking back at the VCAP-DCA blueprint, I think the certification team did a pretty good job of providing an outline of topics to study for. Though it can be daunting at first glimpse, it definitely will be your best source of reference material. Unfortunately most of the beta invitees had less than 2 weeks if not less to prepare for this exam.

I also concur with Chris Dearden's observations with the blueprint, another beta participant of the VCAP-DCA exam. I felt that the exam was geared more towards consultants and/or outsourcing backgrounds than just VMware administrators. The topics on the blueprint ranged from the core vSphere components such as vCenter, ESX and ESXi but also additional auxiliary components that may not be used in all vSphere environments (e.g. Orchestrator, vSheild Zones, Heartbeat, etc). I have to agree with Chris' comment, it is probably rare that you will find all of VMware's enterprise products implemented in a single production vSphere environment other than maybe a test or development lab. For me, this was the first time I have implemented some of these systems in my lab while studying for the exam.This was definitely a challenge with unfamiliar technology and a limited amount of time to to prepare for the exam.

Speaking of a lab, I would highty recommend that you setup a test lab as mentioned by Chris. You will probably need a minimum of two hosts to test and setup the various configuration laid out in the blueprint. I was fortunate enough to have access to two physical hosts and deployed 4 vESX(i) (virtual ESX & ESXi) hosts while studying for the exam.

In terms of advice, I would say to make sure you go through the blueprint thoroughly and make sure you can perform all the skills and abilities listed in each of the objectives. It is also very important to be able to perform the tasks using both the CLI and the GUI, making sure you are comfortable with both.

All in all, I was pretty happy with the exam minus the user experience. I will probably be a wreck until the results are announced (4-6 weeks maybe?) and hopefully I will pass. In the meantime, I will be able to get back to some of the scripts that I have put on hold while studying for this exam!

Good luck to anyone that may be taking the exam on the last day of the beta program which ends on Monday June 21st, 2010. I think the VMware community will be very happy with the new VCAP-DCA certification when it is released!

Update 10/14 - I just received an email that I passed VCAP4-DCA beta exam! I'm glad the wait finally over

Update 12/15 - I just received an email on my VCAP4-DCA number, I am VCAP4-DCA #6

esxcli Part3 – Automating esxcli using PowerShell

06/18/2010 by William Lam Leave a Comment

In this final three part esxcli series, I will show you how to automate esxcli operations in a Windows environment using PowerShell. There was a recent question on the VMTN forums regarding this very topic. As a simple solution to the problem, a community member provided a snippet of PowerShell code. Of course the snippet of PowerShell code originated from the great PowerCLI Master - Luc Dekens.

I have slightly modified the original version of the script as I am not a fan of hard coding passwords within a script. This version of the script requires plink which is a command-line interface to the free Windows SSH PuTTY back end. The script performs an SSH connection to either an ESX or ESXi host to run esxcli command locally, with the specified parameters as you would if you were on the Service Console, or the unsupported Busybox console.

Copy the snippet of code below and create a new script called esxcli-automation1.ps1 and replace all the variables matching your environment including the $cmd which specifies the exact esxcli command you are running on the remote host.

esxcli-automation1.ps1

$User = "root"
$Computer = "esxi4-1.primp-industries.com"

#prompt user for password
$passwordInput = Read-Host -AsSecureString:$true "Please enter password for $Computer"

#convert secure password to normal string
$Ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($passwordInput)
$Password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($Ptr)
[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($Ptr)

$plink = "C:\plink.exe"
$esxcli = "esxcli"
$plinkoptions = " -v -batch -pw '$Password'"
$cmd = 'nmp device list'
$remoteCommand = $esxcli + " " + '"' + $cmd + '"'
$command = $plink + " " + $plinkoptions + " " + $User + "@" + $computer + " " + $remoteCommand
Invoke-Expression -command $command

$User = "root"

$Computer = "esxi4-1.primp-industries.com"

#prompt user for password

$passwordInput = Read-Host -AsSecureString:$true "Please enter password for $Computer"

#convert secure password to normal string

$Ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($passwordInput)

$Password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($Ptr)

[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($Ptr)

$plink = "C:\plink.exe"

$esxcli = "esxcli"

$plinkoptions = " -v -batch -pw '$Password'"

$cmd = 'nmp device list'

$remoteCommand = $esxcli + " " + '"' + $cmd + '"'

$command = $plink + " " + $plinkoptions + " " + $User + "@" + $computer + " " + $remoteCommand

Invoke-Expression -command $command

Here is a sample execution listing devices:

There are a few issues with this script:

Assumes SSH connectivity is open to the host, which on classic ESX w/Service Console is probably a valid reason but what about ESXi? We know that we should not be using the unsupported Busybox console on ESXi and we can not make the assumption that it will be opened

Here is the same example when executing against an ESXi host:

The script can only execute against one host at a time.

In a perfect world, the esxcli API would be properly exposed and integrated into the vSphere API. PowerCLI and other vSphere SDKs would be able to automate these configurations without this hack, but unfortunately this is not the case. However, we can leverage the esxcli utility that is bundled up with the vCLI which can be installed on either a Linux or Windows platform.

With this information, I decided to write my own PowerShell script (yes you heard right, it was slightly difficult without knowing the syntax and format) matching the capabilities similar to that of the vSphere SDK for Perl implementation running on vMA. So here is the second revision of the script called esxcli-automation2.ps1:

esxcli-automation2.ps1

$Username = "root"
$esxcli = "C:\Program Files\VMware\VMware vSphere CLI\bin\esxcli.exe"
#$Params = "swiscsi nic list -d vmhba33"

#list of ESX and ESXi host to perform operation against
$vihosts = @("esxi4-1.primp-industries.com","esxi4-3.primp-industries.com")

#prompt user for password
$passwordInput = Read-Host -AsSecureString:$true "Please enter your common password for hosts"

#convert secure password to normal string
$Ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($passwordInput)
$Password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($Ptr)
[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($Ptr)

#loop through array of hosts
for($i = 0; $i -le $vihosts.length -1; $i++) {
 write-Host Executing operation on $vihosts[$i]
 #could not find a way to store the params in a variable you must
 #insert esxcli parameters at the very end of the statement below
 &amp; $esxcli --server $vihosts[$i] --username $Username --password $Password swiscsi nic list -d vmhba33
}

$Username = "root"

$esxcli = "C:\Program Files\VMware\VMware vSphere CLI\bin\esxcli.exe"

#$Params = "swiscsi nic list -d vmhba33"

#list of ESX and ESXi host to perform operation against

$vihosts = @("esxi4-1.primp-industries.com","esxi4-3.primp-industries.com")

#prompt user for password

$passwordInput = Read-Host -AsSecureString:$true "Please enter your common password for hosts"

#convert secure password to normal string

$Ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($passwordInput)

$Password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($Ptr)

[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($Ptr)

#loop through array of hosts

for($i = 0; $i -le $vihosts.length -1; $i++) {

write-Host Executing operation on $vihosts[$i]

#could not find a way to store the params in a variable you must

#insert esxcli parameters at the very end of the statement below

& $esxcli --server $vihosts[$i] --username $Username --password $Password swiscsi nic list -d vmhba33

}

Note: PowerCLI is not required, just PowerShell.

This new version of the script allows you specify a list of ESX or ESXi host to perform a common esxcli operation. The script also prompts for the password as the previous script, so you do not have to hard code the password to the host. You will need to update the script with the path to the plink.exe and the list of hosts you want to perform the operation against.

Here is an example of verifying iSCSI multipathing configuration against two hosts (1 ESX and 1 ESXi) without the use of SSH:

As you can see, you can easily integrate this with other existing PowerCLI scripts that perform configuration changes across a given set of host(s). Until VMware provides access to the esxcli APIs, these are some of the methods/hacks that need to be implemented when automating configuration changes using esxcli.

Here are some PowerShell references that were used in creating the two scripts:

http://technet.microsoft.com/en-us/library/ee176935.aspx

http://www.rickcable.com/blog/default.asp?Display=33

http://bsonposh.com/archives/247

esxcli Part2 – Automating esxcli using vMA

06/17/2010 by William Lam 3 Comments

In the previous article, esxcli Part1, we discussed what esxcli is and what it can be used for. In this article, we will show you how you can automate and perform a common esxcli operation against a set of ESX and/or ESXi hosts using vMA. As mentioned in the previous article, esxcli needs to be executed against a specific host, it is not vCenter aware and cannot connect to vCenter to perform an operation against a host.

One of the pre-requisites before starting is that all ESX and ESXi hosts need to be managed by vMA and accessible using vi-fastpass authentication. Please refer to the vMA documentation on configuring this on vMA.

Download:esxcli-automation.pl

The following script is based off of useVIFastpassOnvMAToRunPerlScriptWithoutClearTextPassword.pl. The modified script utilizes the vi-fastpass library to properly access a set of target(s) and performs the specified esxcli operation without having to provide credentials to each and every host.

The following example will demonstrate iSCSI multipath configuration on two hosts using the method described by Duncan Epping in this article.

1) Create a file containing the list host to perform the operations against:

[[email protected] ]$ cat hosts
esxi4-1.primp-industries.com
esxi4-3.primp-industries.com

2. Run the script with the specific esxcli parameters as you would normally by passing it into the --cmd_option argument (double quote the arguments):

Binding iSCSI interface to vmk0

[[email protected] ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic add -n vmk0 -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
Executing script on "esxi4-3.primp-industries.com" ...
Binding iSCSI interface to vmk1

[[email protected] ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic add -n vmk1 -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
Executing script on "esxi4-3.primp-industries.com" ...

3. Verify iSCSI multipathing has been properly configured:

[[email protected] ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic list -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
vmk0
pNic name: vmnic0
ipv4 address: 172.30.0.183
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:69:7a
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 167051261
packets sent: 232132
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
vmk1
pNic name: vmnic1
ipv4 address: 172.30.0.184
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:68:6c
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 145924
packets sent: 54502
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
Executing script on "esxi4-3.primp-industries.com" ...
vmk0
pNic name: vmnic0
ipv4 address: 172.30.0.233
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:33:95
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 138535
packets sent: 8026
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
vmk1
pNic name: vmnic1
ipv4 address: 172.30.0.234
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:27:65
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 145924
packets sent: 112
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A

Now you will be able to automate any of the supported esxcli operations against multiple hosts!

In Part3, we will take a look at automating esxcli operations in a Windows environment using Powershell.

esxcli Part1 – What is esxcli?

06/16/2010 by William Lam 8 Comments

esxcli is a new CLI (commandline interface) framework in vSphere that provides a modular architecture for various components called namespaces running in the VMkernel. Some of these namespaces are nmp (Native Multipathing Plugin) for the new VMware Pluggable Storage Architecture, corestorage for claim rules used for masking certain devices to a host, and swiscsi for managing iSCSI interface.

esxcli can be executed within the classic ESX Service Console, the unsupported Busybox console in ESXi or using the vCLI's remote version of esxcli. There are currently 3 namespaces (nmp,corestorage and swiscsi) with the current release of vSphere and we may see others introduced in future releases of vSphere. One important thing to note is that because these modules run within the host, using the vCLI's version of esxcli, you will need to authenticate to the host first to see what modules will be available for access. Currently, esxcli is not vCenter aware, meaning you must connect to a specific ESX or ESXi host when performing an operation.

Here is an example of esxcli executed without connecting to the host first:

Here is an example of esxcli being executed after connecting to the host:

When invoking the esxcli command, you may also notice an esxcli.log is generated. If the command is successfully executed, the log will generally be empty, but if there was an error you may want to take a look at esxcli.log if the command does not provide any output to the screen.

Here is an example of using an auth configuration file and because of the case sensitivity of esxcli, the VI_PROTOCOL entry is failing with HTTPS vs https:

[[email protected] ~]$ cat esxcli.log

[root CRITICAL] Exception:Unsupported protocol
[root CRITICAL] Traceback (most recent call last):
File "esxcli.py", line 387, in _GetStub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 239, in stub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 299, in Login
Exception: Unsupported protocol

There is not a whole lot of information available to the public about esxcli. From what I understand after talking to a few VMware engineers, esxcli has an API, but it is currently not exposed to the public for consumption. Not only is there an API, but 3rd party providers or users can potentially create their own modules and install it using the VIB format also known as vSphere Installation Bundle.

Some well known packages that are currently being distributed in the VIB format today are: Cisco Nexus 1000V VEM, HP Insight Manager Agent, EMC PowerPathV/E, Xsigo ESX IB drivers and VMware ESX/ESXi/vMA updates, to name a few. Hopefully in the future, VMware will expose the esxcli API functionality to the developer ecosystem.

Here are a few blog posts with detail examples on using esxcli with the various namespaces:

Stay tuned for Part2 and Part3 where we will look at automating esxcli operations using both the vSphere SDK for Perl and Windows PowerShell!

Is the release vSphere 4.0 Update 2 imminent?

06/10/2010 by William Lam Leave a Comment

There's already been some chatter in the community regarding vSphere 4.0 Update 2, including a VMware KB article:

http://kb.vmware.com/kb/1021023
http://www.yellow-bricks.com/2010/03/30/whats-the-point-of-setting-iops1/
http://www.vladan.fr/vsphere-4-update-2-is-next/
http://virtualization.info/en/news/2010/03/vsphere-40-update-2-to-add-new-ha.html
http://www.yellow-bricks.com/2010/03/29/cool-new-ha-feature-coming-up-to-prevent-a-split-brain-situation/

While recently downloading a copy of ESX 4.0u1 for testing, I noticed the download URL was the following:

http://www.vmware.com/downloads/download.do?downloadGroup=ESX40U1

The link will generally take you to landing page to login if you have not ready logged in:

I thought I try to plug-in "2" and see what happens, to my surprise it actually displayed some interesting text:

http://www.vmware.com/downloads/download.do?downloadGroup=ESX40U2

When you login, it will throw an error that the download group does not exists.

However, if you tried another invalid entry say "ESX40U3", you'll get an error all together such as the following even before logging in:

http://www.vmware.com/downloads/download.do?downloadGroup=ESX40U3

So does this mean vSphere 4.0 Update 2 about to be released? Your guess is as good as mine, we will just have to wait and see.

How to remove stale targets from vMA

06/10/2010 by William Lam Leave a Comment

If you have used vMA's vi-fastpass authentication, you will know how easy it is to setup using vifp utility which supports both ESX/ESXi and vCenter targets.

Here's an example of adding ESXi target:

[[email protected] ~]$ sudo vifp addserver esxi3-1.primp-industries.com
*protected email*'s password:

Here's an example of the listing of the available fastpass targets:

[[email protected] ~]$ sudo vifp listservers
esxi3-1.primp-industries.com ESXi

During this process, two accounts (vi-userXX & vi-adminXX) are created on the target host with a password that vMA management creates and caches it locally in an obfuscated but not encrypted form. This will allow you to initialize a fastpass target using vifpinit utility and execute commands against the target host without having to manually type in the credentials.

The fastpass targets are stored in 2 configuration files on vMA:

1) The obfuscated cached credentials is stored in /home/vi-admin/.vmware/credstore/vicredentials.xml

If you cat out the contents, it will look something like this:

   esxi3-1.primp-industries.com
   vi-admin00
   XXXXXXXXXXXXXXXXXXXXXXX

esxi3-1.primp-industries.com

vi-admin00

XXXXXXXXXXXXXXXXXXXXXXX

2) A More detailed configuration for each of the targets along is stored in /etc/vmware/viconfig/viconfig.xml

If you cat out the contents, it will look something like this:

   esxi3-1.primp-industries.com
   443
   524d18f6-8bbb-2c5f-a366-6d191813fbe3
   https
   /sdk
   vi-admin00
   vi-user00
   true
   ESX
   1276121961

esxi3-1.primp-industries.com

443

524d18f6-8bbb-2c5f-a366-6d191813fbe3

https

/sdk

vi-admin00

vi-user00

true

ESX

1276121961

What happens when you rebuild your host, or the system is no longer available because it has been decommissioned or being used for another purpose? vMA will still think it's managing the host and the fastpass credentials will no longer function as the account is no longer valid the host. If you try to remove the old target, you will see the following error:

[[email protected] ~]$ sudo vifp removeserver esxi3-1.primp-industries.com
*protected email*'s password:

Error: Failed to connect. Please make sure the server is up and is of supported version.

The reason this occurs is that vMA is unable to login to the host and remove the two accounts that were initially created and fails to remove the target. What you will need to do is actually pass in an additional parameter to vifp command "--force" which will forcefully remove the target from vMA management. This command actually does not require the user to enter the correct password to the host even if it is still reachable by vMA. By specifying this flag and providing some input when prompted for the password, vMA will purge the target from it is system.

[[email protected] ~]$ sudo vifp removeserver esxi3-1.primp-industries.com --force
*protected email*'s password:

After a target is removed from vMA, it is also removed from the two above files. You do not manually tweak either of these configuration files or it may lead to issues on your vMA host.

Best practice for decommissioning a host that has been added to vMA's management is the following:

Disable vilogger if you've enabled it for the host
Remove target from vMA management
Verify the host is no longer being managed by vMA
Decomission host