virtuallyGhetto

vSphere 6.0

How to bootstrap vCenter Server onto a single VSAN node Part 2?

by 47 Comments

In this article, I will provide a step by step walk through on how to setup and configure single VSAN node that will allow you to deploy a vCenter Server onto a VSAN datastore. This initial "bootstrapping" can help when initially building out your VSAN cluster and can come in handy for greenfield deployments and potentially for brownfield deployments as well. Before getting started, make sure you have taken a look at How to bootstrap vCenter Server onto a single VSAN node Part 1.

Environment:

  • 3 physical host
  • Each host as a small iSCSI boot LUN for ESXi installation (this can be another local disk or USB/SD card)
  • Each host has single SSD and SATA disk (minimum)

Step 1 -  Install ESXi 5.5 onto your physical hosts, we technically only need one host to begin the process but you will probably want to have two additional hosts ready unless you do not care about your vCenter Server being able to recover if there is any hardware issues.

Step 2 - You will need to modify the default VSAN storage policy on the ESXi host in which you plan to provision your vCenter Server. It looks like this behavior changed during the VSAN beta and when VSAN was GA'ed yesterday with vSphere 5.5 Update 1. You will need to run the following two ESXCLI commands to enable "force provisioning":

esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"
esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"

You can confirm you have the correct VSAN default policy by running the following ESXCLI command:

~ # esxcli vsan policy getdefault
Policy Class  Policy Value
------------  --------------------------------------------------------
cluster       (("hostFailuresToTolerate" i1))
vdisk         (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmnamespace   (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmswap        (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))

We start off with our first ESXi host and as you can see from the screenshot below, we do not have additional datastores that we can use to provision our vCenter Server.

Step 3 - You will need to identify the disks that you will be using on the first ESXi host to contribute to the VSAN datastore. You can do so by running the following ESXCLI command:

esxcli storage core device list

To get specific details on a particular device such as identifying whether it is an SSD or regular HDD, you can specify the -d option and the device name.

Once you have identified the disks you will be using, make a note of the the disks names as they will be needed in the upcoming steps. As mentioned in my environment, I only have a single SSD and single HDD and their respective device names are naa.50026b72270126ff & naa.5000c500331bca77

Step 4 - Before we can create our VSAN datastore, we need to first create a VSAN cluster. One of the parameters that is needed when going through this "bootstrapping" method without a vCenter Server is a unique UUID to identify the VSAN cluster. The UUID is in the format of "nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn" where n is a hexidecimal value. You can easily generate one within the ESXi Shell by leveraging the following Python snippet

python -c 'import uuid; print(str(uuid.uuid4()));'

Step 5 - To create a VSAN cluster, we will use the following ESXCLI command and specify the UUID from the previous step for the -u option:

esxcli vsan cluster join -u UUID

UPDATE (02/11/15) - In vSphere 6, you no longer have to perform step 4 to generate a UUID. There is now a new ESXCLI command which will automatically create a VSAN Cluster and generate a UUID automatically by running the following command:

esxcli vsan cluster new

Once the VSAN cluster has been created, you can retrieve information about the VSAN cluster by running the following ESXCLI command:

esxcli vsan cluster get

Step 6 - Next we need to add the disks from our ESXi host to create our single node VSAN datsatore. To do so, we will need the disk device names from our earlier step for both SSD and HDDs and run the following ESXCLI command:

esxcli vsan storage add -s SSD-DISK-ID -d HDD-DISK-ID

The -d option specifies regular HDD disks and the -s option specifies an SSD disk. If you have more than one HDD disk, you will need to specify multiple -d entries. You can also take a look at the disks being contributed to the VSAN datatore by running the following ESXCLI command:

esxcli vsan storage list

Step 7 - To save us one additional step, you can also enable the VSAN traffic type on the first ESXi host using ESXCLI and you can also do this for the other two hosts in advance. This step does not necessary have to be done now as it can be done later when the vCenter Server is available and using the vSphere Web Client. You will need to either create or select an existing VMkernel interface to enable the VSAN traffic type and you can do so by running the following ESXCLI command:

esxcli vsan network ipv4 add -i VMK-INT

At this point, you now have a valid VSAN datastore for your single ESXi host! You can verify this by logging into the vSphere C# Client and you should see the VSAN datastore mounted to your ESXi host.

At this point, you are now ready to deploy your vCenter Server 5.5 onto the VSAN datastore. The next series of steps outline the deployment of the VCSA for completeness of the article.

Step 8 - Deploy the VCSA 5.5 OVA/OVF onto the VSAN datastore and power on the VM.

UPDATE: You skip Steps 9-11 by leveraging ovftool 4.0 to inject the required OVF properties when deploying the VCSA, take a look at this article for more details.

Step 9 - Since you can not configure the OVF properties for the VCSA, you will notice that networking is not configured (unless you happen to have DHCP on the network). If you are like most Enterprise customers, you will not have DHCP running in your environment and you will need to configure a static IP.

Step 10 - Login to the VCSA console and we will use the following VAMI CLI /opt/vmware/share/vami/vami_set_network to configure the IP Address for the VCSA. Here is an example of what that command would look like:

/opt/vmware/share/vami/vami_set_network eth0 STATICV4 172.24.68.14 255.255.255.0 172.24.68.1

For more details on the syntax, you can refer to this blog article here. At this point, you should be able to ping your VCSA and verify connectivity.

Step 11 (Optional) - In addition to IP connectivity, you may also want to configure your DNS Server and DNS search domain before configure the VCSA application. You can also do this by using the following VAMI CLI /opt/vmware/share/vami/vami_set_dns and for search domain, you would need to add the entry to /etc/resolve.conf

Step 12 - You now are ready to configure the VCSA. Open a browser and connect to https://[VCSA-IP]:5480 and proceed through the VCSA setup wizard.

Step 13 - Once the VCSA has been configured, you can now login to the vSphere Web Client and create a Datacenter object and then a vSphere Cluster and enable VSAN. Make sure you also enter your VSAN beta license key under the "Manage" section of the vSphere Cluster before you can use VSAN.

Step 14 - Add all three of your ESXi hosts to the vSphere Cluster. If you recall earlier we had enabled the VSAN traffic type on our first ESXi host and if you did not run the command on the remainder ESXi hosts, you will need to do so using the vSphere Web Client under the "Networking" section of each ESXi host

Step 15 - Once all three ESXi hosts have been added to the vSphere Cluster, we should now see their local storage contributed to the VSAN datastore under the "General" tab

Step 16 (Optional) - If for whatever reason the disks do not get automatically claimed, you can click on "Disk Management" and manually claim them. If you selected "Automatic" mode when enabling VSAN, the disks on each ESXi host should automatically be handled by VSAN. However, they may not be claimed if the disks are being seen as "remote" versus "local" devices.

Step 17 - The final thing I would recommend is to configure the VCSA to automatically startup and shutdown when the ESXi host reboots. To do so, login to the ESXi host using the vSphere C# Client and click on "Virtual Machine Startup/Shutdown" under the Configuration tab.

So there you have it! You are now running the vCenter Server on top of the VSAN datastore without having to initially setup a local VMFS or rely on an external NFS volume to deploy your vCenter Server and build up to the full VSAN cluster. By leveraging this bootstrap method, you can easily standup a fully self contained storage and compute cluster which is ideal for an SMB or ROBO environment. The best part of about this setup is that the VCSA will use the default VSAN storage policy which is to tolerate at least one failure and as you add your 2nd and 3rd ESXi host, you will automatically have resiliency for the VCSA.

Monitoring vCenter SSO User Account Expiration

by 2 Comments

Did you know that user accounts created in the vCenter SSO Server automatically expire by default after 365 days? If you do not update your password prior to the expiration date, in about a years time you could potentially be locked out of your vCenter SSO Server which also applies to the [email protected] account.

You can change the default password expiration policy by logging into the vSphere Web Client with an SSO Administrator account. Under the configuration section of "Sign-On and Discovery", there is a Password Policies tab that allows you can modify password policies. By default, this is set to 365 days. I would also recommend that after you have installed and setup your vCenter SSO Server, you add at least one user or group from your directory service such as Active Directory and assign it to the SSO Administrator group. This will ensure that you can still log in to the SSO configuration in the event the local SSO user accounts are locked out.

Even though you can change the password expiration policy, there is still no automated notification or alerting built-in for user accounts that are going to expire. The best you can do is to create a calendar event to remind you update your passwords prior to the expiration date. I am sure that many of you are anxious to add another color event to your already busy schedule 🙂

While investigating alternative options a few weeks back, the only method that I have found to retrieve the status for each SSO user is to directly connect to the vCenter SSO Database. There are two specific tables of interest, one which provides the current password policy and the other providing the last password changed date for each SSO user:

  • ims_authn_password_policy
  • ims_principal_data

Disclaimer: This "may" not be officially supported by VMware.

Instead of having you manually dig around in the SSO Database, I have created a Perl script called getSSOUserExpiration.pl which can connect to either a MSSQL or vPostgress backend SSO database. The script which will automatically list out the current password policy as well as user accounts that will be expiring in N days, where N is input provided by the user. You also have the ability to configure the script to automatically email you the results which is nice for a daily or weekly report and can be setup using a cronjob or a scheduled task. There are several configuration variables that will need to be adjusted based on your environment and these are all located within the script itself. For more details on how to setup and use the script, please take a look at the Setup and Configuration section below.

Note: To reduce any negative impact to the vCenter SSO Database, you should add or ask your DBA's to create a limited read-only account and limit access to the following tables above. You may even be able to have your DBA's create a scheduled routine for the specific queries and have that emailed to you internally.

Here is a screenshot of connecting to a vPostgres backend Database:

Here is a screenshot of connecting to a MSSQL backend database:

Here is a screenshot of what the email report looks like:

Note: The email body should contain the specific vCenter SSO Database, but I am not sure why it is not showing up in Gmail, but it does work for other email clients.

Setup and Configuration

vPostgres

To connect to a vPostgres DB, you will need to install the following two perl packages: perl-DBI and perl-DBD-Pg. In this example, I am using the vMA appliance and the zypper package installer. For more details on how to add a SLES repo, please take a look at the following article. I also assume if you are connecting to a vPostgres DB, then you are using the VCSA (vCenter Server Appliance) and by default it does not accept remote connections. We will need to also make two configuration changes to the VCSA for our script to be able to connect to the database.

Step 1 - Run the following two commands to install both perl packages:

sudo zypper in perl-DBI
sudo zypper in perl-DBD-Pg

Step 2 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/pg_hba.conf you will need to add the network in which you will be connecting from:

host    all             all             172.30.0.0/24           trust

Step 3 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/postgresql.conf you will need to add the IP Address(s) that you want vPostgres to listen for remote connection. If you use "*", it will allow all addressees:

listen_addresses = '*'

Step 4 - For the changes to go into effect, you will need to restart the vPostgres DB by running the following command:

service vmware-vpostgres restart

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB. If you need help in identifying the vCenter SSO DB credentials, please refer to this article for the details.

MSSQL

To connect to an MSSQL DB, there are a few additional steps and packages that will be required. We will be using FreeTDS which provides libraries to connect to an MSSQL DB for UNIX/Linux platforms. There was a bit of trial and error in getting the MSSQL solution working and I would like to thank Reuben Stump for his assistance. The following article was used as a reference for the setup below.

Step 1 - Run the following two commands to install the required packages:

sudo zypper in perl-DBI
sudo zypper in gcc

Step 2 - Download and extract the contents of the FreeTDS package:

wget ftp://ftp.astron.com/pub/freetds/stable/freetds-stable.tgz
tar -zxvf freetds-stable.tgz
cd freetds-0.91

Step 3 - Compile and install FreeTDS under /usr/local/freetds:

export SYBASE=/usr/local/freetds/
./configure --prefix=/usr/local/freetds
make
sudo make install

Step 4 - Add your vCenter SSO Server details into the FreeTDS configuration file located in /usr/local/freetds/etc/freetds.conf

[sso]
host = 172.30.0.239
port = 1433
tds version = 7.0

In the example above, I named my database entry "sso" but you can use any name and this will be referenced when editing the script in step 5.

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB.

A Pretty Cool Method of Upgrading to ESXi 5.1

by 40 Comments

I recently came across an interesting article by Andreas Peetz which shows you how to patch an ESXi host using an image profile that is directly available on VMware's online depot within the ESXi shell. I knew that VMware had online depots for use with VUM and Auto Deploy but I was not aware of this particular method, especially directly from the host.

Disclaimer: This method assumes you can install the default ESXi Image Profile with no additional drivers or packages, else you may have connectivity issue after the upgrade. If you still need to customize the ESXi Image Profile before installation, you will still need to use something like Image Builder and then upload that to your online depot.

Note: There are many ways that you can patch/upgrade your ESXi hosts, here is another article that provides more details for command-line only methods.

Before you get started, you will need to make sure that your ESXi host has the httpClient firewall rule enabled, else you will not be able to connect to VMware's online depot. To enable this, run the following ESXCLI command:

esxcli network firewall ruleset set -e true -r httpClient

Also make sure that your ESXi host can reach the following URL (you can specify a proxy if needed):

https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

To view the available ESXi Image Profiles, run the following ESXCLI command (use the --proxy if you need to specify a proxy to reach VMware's online depot):

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

If you are able to successfully connect to the online depot, you see a list of all the ESXi Image Profiles that are available to you. You will see two ESXi 5.1 Image Profiles (these were recently published), one with VMware Tools and one without VMware Tools.

Note: Before you begin, make sure you do not have any running VMs and put your host into maintenance mode.

Let's go ahead and upgrade our ESXi 5.0 Update 1 host to latest ESXi 5.1. To install the new Image Profile, run the following command:

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.1.0-799733-standard

This can take a few minutes to complete depending on how fast you can pull down the Image Profile. Once it is done, you will see all the new VIBs that have been updated and you will be asked to reboot for the changes to go into effect and then you will be running ESXi 5.1! Pretty cool IMO!

Disabling IPv6 via Command-Line For ESXi 5.1 (Without Automatic Host Reboot)

by 15 Comments

IPv6 for the VMkernel interface is now automatically enabled by default for the latest release of ESXi 5.1 and you may have also noticed the additional IP Address in DCUI after the host boots up.

IPv6 support has been around for awhile now and you can enable IPv6 by using the old vSphere C# Client or the new vSphere Web Client. If you enable or disable IPv6, you will need to perform a system reboot for the changes to go into effect. You also have the ability to enable/disable it via the DCUI, which also has been around for awhile as well.

UPDATE: 07/20/15 - For ESXi 6.0, the VMkernel module is name is now tcpip4 instead of tcpip3.

There is one very important thing to note if you do enable/disable IPv6 on the DCUI, after you made your changes and you wish to apply, there is a very important confirmation box that is displayed.

Carefully read the last sentence which is underline in red "In case IPv6 has been enabled or disabled this will restart your host". If you are not careful in reading the confirmation screen, you may hit yes and your host will issue a reboot. If you are going to use the DCUI to enable or disable IPv6, make sure you do not have any running VMs on your host and you should already have your host maintenance mode when making configuration changes to your host.

In addition to the two methods listed above (vSphere Web Client/C# CLient and DCUI) you can easily enable/disable IPv6 using ESXCLI (my preferred method) and restart the ESXi host when you get a chance.

To view whether IPv6 is currently enabled, run the following ESXCLI command (ESXi 5.5 Update 1 the VMkernel module is now called tcpip4):

esxcli system module parameters list -m tcpip3

As you can see from the screenshot above, ipv6 property is set to 1 which means it is enabled.

To disable IPv6, you just need to set the property to 0, run the following ESXCLI command:

esxcli system module parameters set -m tcpip3 -p ipv6=0

We can now reconfirm by re-running our list operation to ensure the changes were made successfully. All that is left is to perform a system reboot, you can either type in "reboot" or use the new ESXCLI 5.1 command:  

esxcli system shutdown reboot -d 60 -r "making IPv6 config changes"

Note: You can run the ESXCLI command locally on the ESXi Shell or you can run the same command remotely by specifying additional connection options & proxy through vCenter Server if you wish. Take a look here for additional connection options for ESXCLI.

Configuring New vSphere Web Client Session Timeout

by 5 Comments

Just like you could in the old vSphere C# Client, users can also configure a session timeout for the new vSphere Web Client in the latest release of vSphere 5.1. This not only ensures that idle sessions automatically disconnect after a certain period of time but also helps reduce the resources consumed on the vCenter Server as each session allocates a certain amount of resources.

To configure the session timeout, you will need to login to the server running the vSphere Web Client service (which is usually your vCenter Server) and find the webclient.properties file and change the default timeout and then restart the vSphere Web Client service. For the VCSA, the default timeout value is 120 minutes and I assume it is the same default for the Windows vCenter Server.

Step 1 - Locate the webclient.properties file:

    VCSA 5.x

/var/lib/vmware/vsphere-client/webclient.properties

    VCSA 6.x

/etc/vmware/vsphere-client/webclient.properties

    Windows vCenter Server 5.x

%ALLUSERPROFILE%\VMware\vCenterServer\cfg\vsphere-client\webclient.properties

    Windows vCenter Server 6.x

%ALLUSERPROFILE%\VMware\vSphere Web Client\webclient.properties

Step 2 - Un-comment and change session.timeout value to desired value:

session.timeout = 120

Step 3 - Restart the vSphere Web Client Service:

/etc/init.d/vsphere-client restart

You will need to restart the vSphere Web Client service for the changes to go into effect. For the Windows vCenter Server, just restart the vSphere Web Client service and for the VCSA, run the above command.

In my lab, I configured the time out to be 1 minute, once the session has gone idle for the configured period, you will automatically be logged out and brought back to login page with the following message: