Monitoring vCenter SSO User Account Expiration

Did you know that user accounts created in the vCenter SSO Server automatically expire by default after 365 days? If you do not update your password prior to the expiration date, in about a years time you could potentially be locked out of your vCenter SSO Server which also applies to the admin@system-domain account.

You can change the default password expiration policy by logging into the vSphere Web Client with an SSO Administrator account. Under the configuration section of "Sign-On and Discovery", there is a Password Policies tab that allows you can modify password policies. By default, this is set to 365 days. I would also recommend that after you have installed and setup your vCenter SSO Server, you add at least one user or group from your directory service such as Active Directory and assign it to the SSO Administrator group. This will ensure that you can still log in to the SSO configuration in the event the local SSO user accounts are locked out.

Even though you can change the password expiration policy, there is still no automated notification or alerting built-in for user accounts that are going to expire. The best you can do is to create a calendar event to remind you update your passwords prior to the expiration date. I am sure that many of you are anxious to add another color event to your already busy schedule :)

While investigating alternative options a few weeks back, the only method that I have found to retrieve the status for each SSO user is to directly connect to the vCenter SSO Database. There are two specific tables of interest, one which provides the current password policy and the other providing the last password changed date for each SSO user:

• ims_authn_password_policy
• ims_principal_data

Disclaimer: This "may" not be officially supported by VMware.

Instead of having you manually dig around in the SSO Database, I have created a Perl script called getSSOUserExpiration.pl which can connect to either a MSSQL or vPostgress backend SSO database. The script which will automatically list out the current password policy as well as user accounts that will be expiring in N days, where N is input provided by the user. You also have the ability to configure the script to automatically email you the results which is nice for a daily or weekly report and can be setup using a cronjob or a scheduled task. There are several configuration variables that will need to be adjusted based on your environment and these are all located within the script itself. For more details on how to setup and use the script, please take a look at the Setup and Configuration section below.

Note: To reduce any negative impact to the vCenter SSO Database, you should add or ask your DBA's to create a limited read-only account and limit access to the following tables above. You may even be able to have your DBA's create a scheduled routine for the specific queries and have that emailed to you internally.

Here is a screenshot of connecting to a vPostgres backend Database:

Here is a screenshot of connecting to a MSSQL backend database:

Here is a screenshot of what the email report looks like:

Note: The email body should contain the specific vCenter SSO Database, but I am not sure why it is not showing up in Gmail, but it does work for other email clients.

Setup and Configuration

vPostgres

To connect to a vPostgres DB, you will need to install the following two perl packages: perl-DBI and perl-DBD-Pg. In this example, I am using the vMA appliance and the zypper package installer. For more details on how to add a SLES repo, please take a look at the following article. I also assume if you are connecting to a vPostgres DB, then you are using the VCSA (vCenter Server Appliance) and by default it does not accept remote connections. We will need to also make two configuration changes to the VCSA for our script to be able to connect to the database. 

Step 1 - Run the following two commands to install both perl packages:

sudo zypper in perl-DBI
sudo zypper in perl-DBD-Pg

Step 2 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/pg_hba.conf you will need to add the network in which you will be connecting from:

host    all             all             172.30.0.0/24           trust

Step 3 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/postgresql.conf you will need to add the IP Address(s) that you want vPostgres to listen for remote connection. If you use "*", it will allow all addressees:

listen_addresses = '*'

Step 4 - For the changes to go into effect, you will need to restart the vPostgres DB by running the following command:

service vmware-vpostgres restart

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB. If you need help in identifying the vCenter SSO DB credentials, please refer to this article for the details.

MSSQL

To connect to an MSSQL DB, there are a few additional steps and packages that will be required. We will be using FreeTDS which provides libraries to connect to an MSSQL DB for UNIX/Linux platforms. There was a bit of trial and error in getting the MSSQL solution working and I would like to thank Reuben Stump for his assistance. The following article was used as a reference for the setup below.

Step 1 - Run the following two commands to install the required packages:

sudo zypper in perl-DBI
sudo zypper in gcc

Step 2 - Download and extract the contents of the FreeTDS package:

wget ftp://ftp.astron.com/pub/freetds/stable/freetds-stable.tgz
tar -zxvf freetds-stable.tgz
cd freetds-0.91

Step 3 - Compile and install FreeTDS under /usr/local/freetds:

export SYBASE=/usr/local/freetds/
./configure --prefix=/usr/local/freetds
make
sudo make install

Step 4 - Add your vCenter SSO Server details into the FreeTDS configuration file located in /usr/local/freetds/etc/freetds.conf

[sso]
        host = 172.30.0.239
        port = 1433
        tds version = 7.0

In the example above, I named my database entry "sso" but you can use any name and this will be referenced when editing the script in step 5.

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB.

Detecting A Duplicate IP Address For Your ESXi Hosts Using a vCenter Alarm

The motivation for this article was a tweet I noticed from Duncan Epping this morning. Per Duncan's tweet, it looks like he may have accidentally assigned an IP Address to one of his virtual machines which was already being used by an existing ESXi host causing a duplicate IP Address error. We probably have all experienced this once in our lives and it can be quite difficult and frustrating to troubleshoot. Similar to a Windows OS, ESXi can also detect a duplicate IP Addresses but instead of a notification window, it is just logged in the VMkernel logs which looks like the following:

2013-01-21T15:52:35.989Z cpu1:2049)Tcpip_Vmk: 112: arp: 00:50:56:bd:3b:2b is using my IP address 172.30.0.213 on vmk1!

The biggest challenge of course is to identify which ESXi host actually has a conflict and then taking a look at the logs to find the offending MAC Address and shutting them down yourself or with the help of a Network Administrator. Wouldn't it be great if we had an alarm to automatically notify us when a duplicate IP Address is detected? Well I am glad you asked and the answer is YES! :)

In addition to logging to the VMkernel logs, ESXi also logs this "observation" in /var/log/vobd.log which stands for the VMkernel Observation. These "observations" can provide critical identifying information in case of an error and is usually used during troubleshooting. In our case, we are seeing an intermittent network connectivity to our ESXi host which is in result of a duplicate IP Address. The really neat thing about these VOBs is that you can create vCenter Alarms when a specific VOB has been detected. I have shown an example of this before in my Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm article.

We can do exactly the same for detecting a duplicate IP Address for an ESXi host. The first thing we need to do is identify the VOB ID by looking in /var/log/vobd.log:

2013-01-21T15:02:07.513Z: [netCorrelator] 917174784727us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for 172.30.0.83 on interface vmk0, current owner being 00:50:56:bd:3b:2b

We can see the VOB ID for this is esx.problem.net.vmknic.ip.duplicate and this will be used in our vCenter Alarm trigger.

Step 1 - Create a new Alarm and specify a name, the Monitor type will be Hosts and Monitor For will be for a specific event:

Step 2 - Copy the VOB ID that we have identified from above and specify that as our alarm Trigger:

Step 3 - If you wish to receive an email notification or send an SNMP trap go ahead and configure additional actions, else just click next which will just display a vCenter Server alert in the UI.

Now that our alarm has been created, we will want to give this a test drive .... who can we ask? Well it just happens that I have a new user in my environment and I provisioned him a new VM which is already connected to the network. Let's hope he does not try to change the IP Address (because this never happens, right?)

After the user statically assigns the IP Address of an existing ESXi host in the VM, we should see our new alarm trigger in vCenter.

As you can see, we have quickly identified the ESXi host that is impacted and we can then login to DCUI via the console to take a look at the logs to find the offending MAC Address. Hopefully duplicate IP Addresses is not a common problem in your environment but it does happen from time to time and having an alarm to help you quickly narrow down the culprit can be quite useful.

Retrieving vscsiStats Using the vSphere 5.1 API

In my previous article, I talked about the new Service Manager API that was introduced in vSphere 5.1 and how you can retrieve ESXTOP performance data using this new vSphere API. In this article I will show you how to collect vscsiStats data using this same interface. If you are not familiar or have not used vscsiStats before, I would highly recommend you take a look at the Using vscsiStats for Storage Performance Analysis as it goes over some of the basics of vscsiStats and how it works.

Disclaimer: You should try to limit the use of these interfaces for statistics collection or debugging/troubleshooting purposes as there is a certain amount of overhead when running these commands. It is also important to note that since the output is based on the implementer of the service, there is no guarantee that output would not change from one release to the other.

The first step is to get a reference to the vscsiStats service via the Service Manager (must connect directly to an ESXi 5. host, this is not supported when connecting to vCenter Server) and to invoke an operation for vscsiStats, you will need to use the ExecuteSimpleCommand. For vscsiStats, there are four valid operations:

• StartVscsiStats
• FetchAllHistograms
• ResetVscsiStats
• StopVscsiStats

To demonstrate the vscsiStats interface, I have written a sample vSphere SDK for Perl script called getVscsiStats.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

StartVscsiStats

This operation starts the vscsiStats collection for ALL virtual machines residing on your ESXi hosts. This is exactly the same operation if you were to only specify the -s option to the vscsiStats command-line. Here is a screenshot of the "start" operation implemented in the script:

You should see a response of OK from the output and this would indicate the vscsiStats collection has started.

FetchAllHistograms

This operation fetches ALL the vscsiStats histogram data similar to specifying the -p All option in the vscsiStats command-line. The output contains the following:

The <VM> tag denotes the details about each Virtual Machine:

• VM Display Name
• VM VMX Configuration Path
• VM BIOS UUID
• vCenter Server UUID

This is then followed by the <VirtualDisk> tag which provides the VMDK name in the format of scsi:X:Y and within each virtual disk section it will contain 13 <histo> tags which represents each of the statistics type and their associated values:

1. VSCSIVsi_DistanceHistogram: Histogram: distance (in LBNs) between successive commands
2. VSCSIVsi_DistanceLast16Histogram: Histogram: distance (in LBNs) between each command from the closest of previous 16
3. VSCSIVsi_DistanceReadsHistogram: Histogram: distance (in LBNs) between successive Read commands
4. VSCSIVsi_DistanceWritesHistogram: Histogram: distance (in LBNs) between successive Write commands
5. VSCSIVsi_IoLatencyHistogram: Histogram: latency of IOs in Microseconds (us)
6. VSCSIVsi_IoLatencyReadsHistogram: Histogram: latency of Read IOs in Microseconds (us)
7. VSCSIVsi_IoLatencyWritesHistogram: Histogram: latency of Write IOs in Microseconds (us)
8. VSCSIVsi_IoLengthHistogram: Histogram: IO lengths of commands
9. VSCSIVsi_IoLengthReadsHistogram: Histogram: IO lengths of Read commands
10. VSCSIVsi_IoLengthWritesHistogram: Histogram: IO lengths of Write commands
11. VSCSIVsi_OutstandingIOsHistogram: Histogram: number of outstanding IOs when a new IO is issued
12. VSCSIVsi_OutstandingIOsReadsHistogram: Histogram: number of outstanding Read IOs when a new Read IO is issued
13. VSCSIVsi_OutstandingIOsWritesHistogram: Histogram: number of outstanding Write IOs when a new Write IO is issued

Here is a screenshot of the "getstats" operation implemented in the script:

Note: In comparing the output between the vscsiStats command-line and this interface, I found the following three statistics are not available:

• Histogram: latency of IO interarrival time in Microseconds (us)
• Histogram: latency of IO interarrival time for Reads in Microseconds (us)
• Histogram: latency of IO interarrival time for Writes in Microseconds (us) 

 

ResetVscsiStats

This operation will reset the vscsiStats collection similar to the -r option in the vscsiStats command-line. Here is a screenshot of the "reset" operation implemented in the script:

StopVscsiStats 

This operation will stop the vscsiStats collection similar to the -x option in the vscsiStats command-line. Make sure you perform this operation once you are done retrieving your vscsiStats data. Here is a screenshot of the "stop" operation implemented in the script:

In addition to the four operations, you can also save the output to a file by specifying the --output option along with the name of the file. vscsiStats is an extremely useful tool to help vSphere administrators profile their virtual machine's IO workload and now you can easily collect this information using the vSphere API. Some really cool things you can do with this data is to create some nifty graphs such as the ones here and here.

Retrieving ESXTOP Performance Data Using the vSphere 5.1 API

In vSphere 5.1, VMware introduced a new managed object called the Service Manager. The Service Manager is a generic object that wraps the execution of a single command and it requires a specific set of inputs to invoke a particular service command. This is particularly interesting as it allows users to access both the ESXTOP and vscsiStats interface using the vSphere API. Prior to vSphere 5.1, to use ESXTOP you would need to either login to the ESXi Shell to run the local ESXTOP command or connect remotely using the RESXTOP utility which is only available on a Linux system. For vScsiStats, you would need to login to the ESXi Shell as a remote version of this tool does not exist. The Service Manager used to be a private interface, an interesting tidbit is that some of you may have already interacted with this interface without even realizing it if you have used PowerCLI's Get-Esxtop cmdlet. In this article I will show you how to programmatically access ESXTOP using the vSphere API.


Disclaimer: You should try to limit the use of these interfaces for statistics collection or debugging/troubleshooting purposes as there is a certain amount of overhead when running these commands. It is also important to note that since the output is based on the implementer of the service, there is no guarantee that output would not change from one release to the other.

Both the ESXTOP and vscsiStats services are only available when connecting directly to an ESXi 5.1 host, it is not available when connecting to a vCenter Server. If we browse over to the vSphere MOB, we can clearly see the two services:

The first step is to get a reference to the ESXTOP service via the Service Manager and to invoke an operation for ESXTOP, you will need to use the ExecuteSimpleCommand. For ESXTOP, there are three valid operations:

• CounterInfo
• FetchStats
• FreeStats

To demonstrate the ESXTOP interface, I have written a sample vSphere SDK for Perl script called getEsxtop.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

CounterInfo

This operation only needs to be invoked once and it will provide you with the list of available counters and their associated properties and data types for a given ESXi host. Here is an example of this using the "getcounters" operation implemented in the script:

Each line represents a specific counter type followed by each property name and their data type. For example, the first line is for the Server counter and has the following properties and types:

Property Name	Type
MinFetchIntervalInUsec	U64
IsVMVisor	B
TimeStampInUsec	U64
Time	S64

Here is a quick diagram to help you visualize the hierarchy of all the ESXTOP counters and their relationships with one another:

Note: This diagram was created using yuml.me and here is the raw text in JSON format if you are interested.

FetchStats

This operation fetches a single snapshot of ALL the ESXTOP statistics which contains two pieces of information:

• The topology of the counter instances
• The actual counter instances values

The first section is denoted by ==NUM-OF-OBJECTS== which contains either inventory data that does not change or counter instance structure which describes the relationship between the different counter instances. Here is an example of the first section using the "getstats" operation implemented in the script:

If we take a look at the second line as an example |PCPU|LCPU,24|Core,12|Package,2| we can see that PCPU counter contains 24 LCPU that you would need to then enumerate as well as inventory information describing the CPU's logical cores and physical socket.

To view the enumerated counter instances and their instance values, we need to look in the second portion of the data which is denoted by ==COUNTER-VALUE== within the output. Here is a screenshot of this section and we can see the enumerated LCPU's (24 in total as denoted earlier) and their associated instance values:

Remember you will need to correlate with the counter definitions that was extracted earlier from the "getcounters" and this will help you build up the data. I do have to say it can be a bit confusing when you first look at the raw data, but as you start to play with it a bit more, it will start to make sense. Two useful references that can help with parsing the data is the ESXTOP bible and an article that Luc Dekens wrote awhile back exploring the Get-Esxtop cmdlet which I mentioned earlier leverages this exact interface.

FreeStats

Lastly, once you are done collecting the ESXTOP data, you will need to run the "freestats" operation and this will release any server side resources used during the collection. When this command is invoked, it will free up all resources even for past collections where you might have forgotten to perform this last step. There is no output from this operation as you can see from the example screenshot below:

Even though it is nice to see the ESXTOP interface be accessible via the vSphere API, it is not the easiest interface to use and is definitely geared more towards a developer. For extracting general performance data, I would still recommend using the Performance Manager managed object or one of the above mentioned command-line interfaces. In the next article, I go into more detail about the vscsiStats interface and how to consume it using the vSphere API. 

New Updates for ghettoVCB & ghettoVCB-restore

I know it has taken me awhile, but I finally found some time over the last week and half to go through my entire backlog of bugs, issues, feature requests, etc. and have updated both my ghettoVCB and ghettoVCB-restore script. Here is a quick summary of the new enhancements in this release:

• ghettoVCB & ghettoVCB-restore supports ESXi 5.1
• This was a fairly simple change by modifying the version number in the script which I noticed several users sharing the details in the VMTN forums. However, there were a few changes with the release of ESXi 5.1 that caused some initial issues which now have all been resolved in this release. For some of those details, take a look at the "Fixes" section of the change log.
• Support for individual VM backup via command-line and added new -m flag
• This has been requested a few times and the idea is if you have a single VM to backup, it was extra work to create a file containing the name of the VM. So now you can specify a single VM backup via the command-line
• Support VM(s) with existing snapshots and added new configuration variable called ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP 
•  This is probably the most requested feature I have seen and as many of you know my personal stance on snapshots, though they can be extremely useful they can also be dangerous when miss-used. I have seen too many VMTN posts about admins finding out they have a 300-500GB snapshot they need to commit and that they are also out of space/etc. However, I decided to support this use case as it was recently brought to my attention that some of the commercial backup solutions that support VMs with existing snapshots just consolidate all snapshots prior to backup. If this feature is enabled, it will consolidate ALL existing snapshots on the VM prior to running a backup.
• Support multiple running instances of ghettoVCB running and added a new -w flag
• Today, you can only run a single instance of ghettoVCB, there have been several requests to support running multiple instance and this was a feature submitted which allows you to specify a separate working directory for ghettoVCB. Users should try to minimize the number of instances running as there is a limited amount of resources in the ESXi Shell which can potentially impact your host. Note, this is an experimental feature.
• Configure VM shutdown/startup order and added two new configuration variables called VM_SHUTDOWN_ORDER and VM_STARTUP_ORDER 
• This was another submitted feature which allows you to specify the order if which VMs should be shutdown and started back up for which have a dependency between each other. 
• Support changing custom VM name during restore
• ghettoVCB-restore has been updated to allow for an optional 4th parameter in the restore file for users to specify an alternate VM name for the restored VM. This can be useful if want to restore the VM alongside the original VM for backup validation purposes (which everyone should do) and by disconnecting the network, you would not be impacting your existing VM while you perform your verification.
•  Documentation updates

I highly recommend you take a look at the change log for more details (or Github diff more exact changes in the code) as well as the ghettoVCB documentation which has been updated for all the latest changes including feedback from the community. I would also like to thank the ghettoVCB/ghettoVCB-restore community for the feedback and comments you have provided as well as the following users: daviderickson, sethsp, vlooy, gvalkov, jonmchan, fryrpc and aspineux who have all submitted patches for bug fixes and feature requests via the ghettoVCB github repository

I hope you enjoy these two releases and if you run into any troubles, please post in the ghettoVCB VMTN group forum.

vCloud Suite Virtual Appliances: Passwords, Databases, URLs, etc

I recently re-organized my home lab and I got rid of a bunch of VMs for random projects that I have been working on last year. Part of this re-organization was to re-deploy a few of the virtual appliances found within the vCloud Suite. As part of the deployment, I often find myself scouring various documents looking for default credentials to the OS, VAMI interface or the application. It is not always easy to find and I often end up going to Google or the VMTN forums for the answer.

As a fun little exercise, I thought why not deploy all of the latest virtual appliance that are available in the vCloud Suite and just document the latest usernames/passwords for the application, OS, VAMI interface, database configurations, URLs, etc.? This would primarily be a reference for myself, but thought it might also benefit others as well. Duncan Epping had done this awhile back for vCloud Director and few other virtual appliance and funny enough, his site was one of the first ones I found for the default vCloud Director password.

Not only have I deployed all the virtual appliances from the vCloud Suite, which can be seen from the screenshot below,  but I also went through each appliance and validated the credentials for the application, OS or VAMI interface if applicable as well as identify all database credentials and configurations which are not all publicly documented (this took a bit of digging in the appliances, but was not too difficult if you know where to look).

Note: All credentials and configurations were identified by going through public documentation and exploring the virtual appliances, internal VMware documentation and Wikis were not used (that would have been too easy)

Below is a quick summary of the credentials for the each of the application, OS, VAMI interface as well as applicable database configurations for the 15 virtual appliances within the vCloud Suite. To view the full report on all the virtual appliances in the vCloud Suite, you can refer to either the Spreadsheet or HTML report.

vCenter Infrastructure Navigator 2.0.0

Default App Username: N/A
Default App Password: N/A
Default OS Username: root
Default OS Password: SET DURING DEPLOYMENT
Default VAMI Username: root
Default VAMI Password: SAME AS OS PASSWORD
Database Type: N/A
Database Name: N/A
Default Database Port: N/A
Default DB Username: N/A
Default DB Password: N/A

vCenter Operations Manager (UI) 5.6.0

Default App Username: admin
Default App Password: admin
Default OS Username: root
Default OS Password: vmware
Default VAMI Username: N/A
Default VAMI Password: N/A
Database Type: vPostgres
Database Name: cmapp
Default Database Port: 5432
Default DB Username: cm
Default DB Password: RANDOMLY-GENERATED (refer to spreadsheet/HTML report below for more details)

vCenter Operations Manager (Analytics) 5.6.0

Default App Username: N/A
Default App Password: N/A
Default OS Username: root
Default OS Password: vmware
Default VAMI Username: N/A
Default VAMI Password: N/A
Database Type: vPostgres
Database Name: alivevm
Default Database Port: 5432
Default DB Username: alive
Default DB Password: RANDOMLY-GENERATED (refer to spreadsheet/HTML report below for more details)

vCenter Orchestrator 5.1.0

Default App Username: vmware
Default App Password: vmware
Default OS Username: root
Default OS Password: vmware
Default VAMI Username: root
Default VAMI Password: vmware
Database Type: postgres
Database Name: vmware
Default Database Port: 5432
Default DB Username: vmware
Default DB Password: vmware

vCenter Server Appliance 5.1.0b

Default App Username: N/A
Default App Password: N/A
Default OS Username: root
Default OS Password: vmware
Default VAMI Username: root
Default VAMI Password: vmware
Database Type: vPostgres
Database Name: VCDB
Default Database Port: 5432
Default DB Username: vc
Default DB Password: RANDOMLY-GENERATED (refer to spreadsheet/HTML report below for more details)

vCloud Connector (Server) 2.0.0

Default App Username: N/A
Default App Password: N/A
Default OS Username: root
Default OS Password: vmware
Default VAMI Username: admin
Default VAMI Password: vmware
Database Type: vPostgres
Database Name: hcs1
Default Database Port: 5432
Default DB Username: postgres
Default DB Password: postgres

vCloud Connector (Node) 2.0.0

Default App Username: N/A
Default App Password: N/A
Default OS Username: root
Default OS Password: vmware
Default VAMI Username: admin
Default VAMI Password: vmware
Database Type: vPostgres
Database Name: hcs
Default Database Port: 5432
Default DB Username: postgres
Default DB Password: N/A

vCloud Director 5.1.1

Default App Username: N/A
Default App Password: N/A
Default OS Username: root
Default OS Password: Default0
Default VAMI Username: root
Default VAMI Password: vmware
Database Type: Oracle
Database Name: XE
Default Database Port: 1521
Default DB Username: vcloud
Default DB Password: VCloud

vCloud Networking and Security 5.1.2

Default App Username: admin
Default App Password: default
Default OS Username: admin
Default OS Password: default
Default VAMI Username: N/A
Default VAMI Password: N/A
Database Type: N/A
Database Name: N/A
Default Database Port: N/A
Default DB Username: N/A
Default DB Password: N/A

vFabric Application Director 5.0.0

Default App Username: admin
Default App Password: SET DURING DEPLOYMENT
Default OS Username: root or darwin_user
Default OS Password: SET DURING DEPLOYMENT
Default VAMI Username: root
Default VAMI Password: SAME AS OS PASSWORD
Database Type: postgres
Database Name: darwin
Default Database Port: 5432
Default DB Username: darwin
Default DB Password: N/A

vFabric Hyperic Server (Server) 5.0.0

Default App Username: hqadmin
Default App Password: SET DURING DEPLOYMENT
Default OS Username: root
Default OS Password: SET DURING DEPLOYMENT
Default VAMI Username: root
Default VAMI Password: SET DURING DEPLOYMENT
Database Type: N/A
Database Name: N/A
Default Database Port: N/A
Default DB Username: N/A
Default DB Password: N/A

vFabric Hyperic Server (DB) 5.0.0

Default App Username: N/A
Default App Password: SET DURING DEPLOYMENT
Default OS Username: root
Default OS Password: SET DURING DEPLOYMENT
Default VAMI Username: root
Default VAMI Password: SAME AS OS PASSWORD
Database Type: vPostgres
Database Name: HQ
Default Database Port: 5432
Default DB Username: hqadmin
Default DB Password: SET DURING DEPLOYMENT

vMA 5.1.0

Default App Username: N/A
Default App Password: N/A
Default OS Username: vi-admin
Default OS Password: SET DURING DEPLOYMENT
Default VAMI Username: vi-admin
Default VAMI Password: SAME AS OS PASSWORD
Database Type: N/A
Database Name: N/A
Default Database Port: N/A
Default DB Username: N/A
Default DB Password: N/A

vSphere Data Protection 5.1.1

Default App Username: root
Default App Password: changeme
Default OS Username: root
Default OS Password: SET DURING DEPLOYMENT
Default VAMI Username: N/A
Default VAMI Password: N/A
Database Type: postgres
Database Name: vdrdb
Default Database Port: 5555
Default DB Username: admin
Default DB Password: N/A

vSphere Replication 5.1.0.1

Default App Username: N/A
Default App Password: N/A
Default OS Username: root
Default OS Password: SET DURING DEPLOYMENT
Default VAMI Username: root
Default VAMI Password: SAME AS OS PASSWORD
Database Type: vPostgres
Database Name: vrmsdb
Default Database Port: 5432
Default DB Username: vrmsdb
Default DB Password: N/A

What Are the Shadow of vmnic in ESXTOP?

In ESXi 5.1, you might have noticed something new called Shadow of vmnic under the USED-BY column in the Network view of ESXTOP.

I initially heard about this from a few followers on twitter and I was curious myself since I could not find any documentation regarding this topic. It took a bit of digging but it turns out these shadow vmnics is actually related to the new VDS Health Check feature released in vSphere 5.1. A shadow port is created automatically for every uplink in your ESXi host and is used for transmitting and receiving health check related packets for each uplink. In ESXTOP, you can monitor the statistics for these shadow ports which can be used to debug/troubleshoot the network health check feature and this is why they are present.

One thing to note, these shadow ports are created regardless of whether or not your ESXi host is connected to a VDS or if you have the network health check features enabled. This is by design and there are four VMkernel modules that are responsible for the network health check feature:

• vlanmtucheck
• teamcheck
• heartbeat
• healthchk

Disclaimer: Do not modify or disable any of the above VMkernel modules else you can potentially disable network connectivity to your ESXi host (trust me, I know).

After some investigation and testing in my lab, I found that I could unload the above VMkernel modules and the shadow vmnics entries would no longer appear in ESXTOP. To do this, you will need an ESXi 5.1 host which is not running any virtual machines and you will need to run the following commands in this exact order (as there are module dependencies):

vmkload_mod -u vlanmtucheck
vmkload_mod -u teamcheck
vmkload_mod -u heartbeat
vmkload_mod -u healthchk

Once you have successfully unloaded all four VMkernel modules, if you open up ESXTOP, you should no longer see the shadow vmnics. To restore the shadow vmnics, you can either reboot (since the unload is not persistent) OR you can run the following commands in this exact order:

vmkload_mod heartbeat
vmkload_mod teamcheck
vmkload_mod vlanmtucheck

Note: By loading the heartbeat VMkernel module, it also loads the healthchk module.

vCloud Director Simulator

I have received several questions from folks asking if there are other VMware solutions that could work with the vCenter Server Simulator (vcsim) which I recently wrote an article about back in 2012. Well, the answer is I do not know, as other solutions have not really been tested out with vcsim. The goal of vcsim was primarily focused at the vSphere layer and I suspect that other VMware solutions can probably leverage what vcsim has done but probably will not work (today) as some solutions require a fair amount of communication between itself and the vCenter Server. Now having said all that, you might be wondering about the title of this article?

The one solution that I am aware of today that works with vcsim is vCloud Director and this is merely a proof of concept than anything else, so I would like to set the proper (low) expectations. The available functionality from vCloud Director is quite limited when using it with vcsim and I have also ran into several issues which I will go into more detail later. For vCloud Director to be able to consume the simulated inventory from vcsim, a few tweaks are required in both vcsim and the vCloud Director database.

Disclaimer: This is not officially supported by VMware, use at your own risk.

Requirements:

• VCSA 5.1 (vCenter Server Appliance)
• VCNS 5.1 (vCloud Networking and Security)
• VCD 5.1 Appliance
• configureVCSimulator.sh script

I highly recommend you take a look at my vCenter Server Simulator article before getting started as it gives you some background that is required later on but you will not be required to configure the VCSA in advanced as there are some additional configurations that are needed. I will also be using the vCloud Director appliance which I recommend for quick ease of deployment and setup.

Configurations: 

Step 1 - Deploy VCSA and configure it as you would normally (do not touch the vcsim configurations, as that is handled in the next step using a script that is provided)

Step 2 - Download the configureVCSimulator.sh script and upload it to the VCSA. By default, vcsim's XML configuration files is set to ESXi 4.0 and we will need to adjust the version numbers to reflect either 5.0 or 5.1 as well as modify the hypervisor type from embeddedEsx to esx which is required by the simulator code. Go ahead and run the script on the VCSA which will setup the vcsim as well as modify the necessary files. Once the script has completed, if you wish to change the default inventory, go ahead and modify /etc/vmware-vpx/vcsim/model/initInventory.cfg and then restart the vCenter Server service by running service vmware-vpxd restart

Here is a screenshot of the script running:

Note: If you are interested in the files that are being modified, you can take a look at the shell script for more details. 

Step 3 - Deploy and configure VCNS appliance as you would normally. Make sure you also register the vCenter Server in the VCNS UI that you just deployed as this is needed before moving forward to the next step.

Step 4 - Deploy the VCD appliance using the embedded database and power on the system. Once the system is ready, go ahead and SSH to the host, we will need to execute a few SQL queries which will allow vCloud Director to support simulated hosts as well as the particular ESXi versions. First we need to switch over to the Oracle user, run the following command:

su - oracle

Next we will login to the VCD database using sqlplus and the default credentials which should be vcloud/VCloud, run the following command:

sqlplus vcloud/VCloud

The first SQL statement will allow VCD to support simulated ESXi hosts, type in the following SQL statement:

update config set value='1' where name='vzSim50Supported';

The next two SQL statements will define the version of ESXi that VCD will support, type in the following SQL statements that are applicable to you:

insert into os values (seq_config.NextVal, 'vmnix-x86', 'VMware ESX', '5.1.0', 0, 1);
insert into os values (seq_config.NextVal, 'vmnix-x86', 'VMware ESX', '5.0.0', 0, 1);

Finally, we just need to type "quit" to exit from sqplus and then type "exit" to logoff as the oracle user and this will bring us back to root context. The last thing we need to do is restart the vCloud Director service for the changes to go into effect by running the following command

service vmware-vcd restart

Here is a screenshot of running through the VCD database changes:

Step 5 - Once the vCloud Director service has been successfully restarted, we are now ready to setup our vCloud Director Simulator environment. Start off by adding both your VCSA and VCNS to the vCloud Director under Resources tab as you would normally.

Next, click on Provider VDCs to create your Pvdc and when you arrive at the ESXi host preparation step, you can type in anything for the username/password (since these are simulated ESXi host).

After you click next or finish, you then click into the Hosts tab to watch the ESXi host preparation. This is where you will notice the first issue, the preparation should finish almost immediately as thgey are simulated hosts, so no VCD agent is installed or services needing to be enabled, but from what I have seen this process can take up to 30minutes if not longer. You will need to be patient when you get to this point and do not try to cancel the process and just let it sit and finish. To ensure your environment is setup correctly, make sure the following three columns show green check marks: Enable, Ready and Available during the preparation.

This is probably a good time to get some coffee or tea and check out other cool articles on virtuallyghetto.com ;)

Once the ESXi hosts have completed preparation, you should see a green check box under the Status column which means you are now ready to create an Organization which is a pre-requisite before creating an Organization VDC. I will assume you know how to proceed from here and depending if you setup an ESXi 5.0 or ESXi 5.1 environment in vcsim, you will need to choose the appropriate virtual machine compatibility (virtual hardware) to support in your OrgVDC.

Note: If you setup an ESXi 5.0 environment, you may see an error with too many connected datastores, to resolve this, you need to either decrease the number of hosts in your simulated environment or use ESXi 5.1

You can probably guess the last step is to start provisioning VMs/vApps in our vCloud Director instance, but sadly this is where you will face another issue. What I have found is that when you deploy a new vApp, an exception is thrown regarding a NULL pointer and VCD will fail on the deployment.

However, if we take a look at our vcsim environment, you will see that the VM actually does get created but this still poses a problem for us in VCD, as the task is still not considered successful.

I have not been able to figure out why and I know from running through several tests, there was a case where I was able to successfully provision a VM in the VCD simulator. This might be related to the version of virtual hardware selected, I had more luck using either 7 or 8 but not so much with 9. Unfortunately, without provisioning capabilities, this may not be as useful as one would like but I encourage you to still give this a try if you are interested.

The vCloud API can also be access in the simulated environment, but remember that the operations will be limited to what has been shown and will probably work for most of the inventory based API calls, but again, YMMV.

Though vCloud Director Simulator is somewhat limited today and with a few known issues, it is a proof of concept that other VMware solutions can leverage what vcsim has provided as a base foundation. If you think this is something that is useful to have or have other use cases for, please leave a comment. You never know, this could be a VMware Fling one day if there is enough interest from the community. 

Configure Apple Mac Mini to Default Boot ESXi

If you are running ESXi on an Apple Mac Mini and it is installed on a USB key, you probably have noticed that the Mac Mini tries to boot from disk by default and instead of using the USB device. This means when you reboot your ESXi host each time, you will need to hold down the "ALT/OPTION" key which will present you with a boot menu to select the device you wish to boot from.

This can be quite annoying if you have a headless setup for your Mac Mini and you just want it to automatically boot off of the right device containing your ESXi installation. To fix this, you can configure the default boot device which can be done by first selecting the device you wish to boot off of as shown in the screenshot above. Next, hold down on the "CONTROL" key which will turn the straight arrow into circular arrow icon as shown in the screenshot below.

Now you just need to either hit enter or if you have a mouse, click on the circular arrow icon and this will configure the default boot device the Apple Mac Mini will use going forward. It is that simple! If you want to boot off of another device after configuring the default boot device, you can still do so by holding down "ALT/OPTION" key while the Mac Mini is still booting up.

Credit goes to this site for solution.