virtuallyGhetto

vcenter

Quick Tip – Steps to shutdown/startup VSAN Cluster w/vCenter running on VSAN Datastore

by 11 Comments

I know Cormac Hogan already wrote about this topic awhile ago, but there was a question that was recently brought up that included a slight twist which I thought it would be useful to share some additional details. The question that was raised: How do you properly shutdown an entire VSAN Cluster when vCenter Server itself is also running on the VSAN Datastore? One great use case for VSAN in my opinion is a vSphere Management Cluster that would contain all your basic infrastructure VMs including vCenter Server which can be bootstrapped onto a VSAN Datastore. In the event that you need to shutdown the entire VSAN Cluster which may also include your vCenter Server, what is the exact procedure?

To help answer this question, I decided to perform this operation in my own lab which contains a 3-Node (physical) VSAN Cluster that had several VMs running on the VSAN Datastore including the vCenter Server VM that was managing the VSAN Cluster.

stutdown-vsan-cluster-with-vcenter-on-vsan-datastore-0
Below are the steps that I took to properly shutdown down a VSAN Cluster as well as powering everything back on.

UPDATE (4/27) - Added instructions for shutting down a VSAN 6.0 Cluster when vCenter Server is running on top of VSAN.

Shutdown VSAN Cluster (VSAN 6.0)

Step 1 - Shutdown all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM, that will be the last VM you shutdown.

stutdown-vsan-cluster-with-vcenter-on-vsan-datastore-1
Step 2 - To help simplify the startup process, I recommend migrating the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.

Step 3 - Ensure that there are no vSAN Components being resync'ed before proceeding to the next step. You can find this information by going to the vSAN Cluster and under Monitor->vSAN->Resyncing Components as shown in the screenshot below.

Step 4 - Shutdown the vCenter Server VM which will now make the vSphere Web Client unavailable.

stutdown-vsan-cluster-with-vcenter-on-vsan-datastore-4
Step 5 - Next, you will need to place ALL ESXi hosts into Maintenance Mode. However, you must perform this operation through one of the CLI methods that supports setting the VSAN mode when entering Maintenance Mode. You can either do this by logging directly into the ESXi Shell and running ESXCLI locally or you can invoke this operation on a remote system using ESXCLI.

Here is the ESXCLI command that you will need to run and ensure that "No Action" option is selected when entering Maintenance Mode:

esxcli system maintenanceMode set -e true -m noAction

Step 5 - Finally, you can now shutdown all ESXi hosts. You can login to each ESXi hosts using either the vSphere C# Client / ESXi Shell or you can also perform this operation remotely using the vSphere API such as leveraging PowerCLI as an example.

Shutdown VSAN Cluster (VSAN 1.0)

Step 1 - Shutdown all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM.

stutdown-vsan-cluster-with-vcenter-on-vsan-datastore-1
Step 2 - To help simplify the startup process, I recommend migrating the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.

stutdown-vsan-cluster-with-vcenter-on-vsan-datastore-2
Step 3 - Place all ESXi hosts into Maintenance Mode except for the ESXi host that is currently running the vCenter Server. Ensure you de-select "Move powered-off and suspend virtual machines to other hosts in the Cluster" as well as selecting the "No Data Migration" option since we do not want any data to be migrated as we are shutting down the entire VSAN Cluster.

Note: Make sure you do not shutdown any of the ESXi hosts during this step because the vCenter Server VSAN Components are distributed across multiple hosts. If you do this, you will be unable to properly shutdown the vCenter Server VM because its VSAN components will not available.

stutdown-vsan-cluster-with-vcenter-on-vsan-datastore-3
Step 4 - Shutdown the vCenter Server VM which will now make the vSphere Web Client unavailable.

stutdown-vsan-cluster-with-vcenter-on-vsan-datastore-4
Step 6 - Finally, you can now shutdown all ESXi hosts. You can login to each ESXi hosts using either the vSphere C# Client / ESXi Shell or you can also perform this operation remotely using the vSphere API such as leveraging PowerCLI as an example.

Startup VSAN Cluster

Step 1 - Power on all the ESXi hosts that is part of the VSAN Cluster.

Step 2 - Once all the ESXi hosts have been powered on, you can then login to the ESXi host that contains your vCenter Server. If you took my advice earlier from the shutdown procedure, then you can login to the first ESXi host and power on your vCenter Server VM.

Note: You can perform steps 2-4 using the vSphere C# Client but you can also do this using either the API or simply calling vim-cmd from the ESXi Shell. To use vim-cmd, you need to first search for the vCenter Server VM by running the following command:

vim-cmd vmsvc/getallvms

startup-vsan-cluster-with-vcenter-on-vsan-datastore-0
You will need to make a note of the Vmid and in this example, our vCenter Server has Vmid of 6

Step 3 - To power on the VM, you can run the following command and specify the Vmid:

vim-cmd vmsvc/power.on [VMID]

startup-vsan-cluster-with-vcenter-on-vsan-datastore-1
Step 4 - If you would like to know when the vCenter Server is ready, you can check the status of VMware Tools as that should give you an indication that system is up and running. To do so, you can run the following command and look for the VMware Tools status:

vim-cmd vmsvc/get.guest [VMID]

startup-vsan-cluster-with-vcenter-on-vsan-datastore-2
Step 5 - At this point, you can now login to the vSphere Web Client and take all of your ESXi hosts out of Maintenance Mode and then power on the rest of your VMs.

startup-vsan-cluster-with-vcenter-on-vsan-datastore-3
As you can see the process to shutdown an entire VSAN Cluster even with vCenter Server running on the VSAN Datastore is fairly straight forward. Once you are comfortable with the procedure, you can even automate this entire process using the vSphere API/CLI, so you do not even need a GUI to perform these steps. This might even be a good idea if you are monitoring a UPS and have an automated way of sending remote commands to shutdown your infrastructure.

How to recover VCSA 5.5 from an expired administrator account?

by 9 Comments

Last week I wrote about a new security feature in the new VCSA 5.5 where the administrator account (root) password will now expire automatically after 90 days of powering on the VCSA if the password is not changed before then. This new enhancement is to ensures that administrative passwords are rotated routinely for good security practices. However, in the event that you forget to change the password before the expiration, you can still recover the VCSA and this article will walk you through that process.

As a lab exercise, I have configured my root password to expire in one day and purposely let it expire. If you try to login to the VAMI UI, you will get an "Unable to authenticate user" error and you will see something similar if you login to the SSH console. Ideally, this message should be a bit more descriptive to say something like the password has expired (which I have filed an internal bug for).

Requirements:

  • You will need console access to your VCSA
  • You will also need a Linux LiveCD, I personally like using KNOPPIX

Step 1 - Mount the Linux LiveCD to your VCSA and boot into the image. You will need to bring up a terminal shell. The version I am using has a menu and I just select the "shell" option.

Step 2 - Once you are in the terminal, you will need to switch to the root user by running the following command:

su -

Step 3 - Next, we need to mount the VCSA root partition which will be /dev/sda3 to /mnt directory by running the following command:

mount /dev/sda3 /mnt

Step 4 - We now need to edit /etc/shadow file on our VCSA which is located in /mnt/etc/shadow to disable the account lock. You will need to use an editor such as vi to open up the file.

You need to delete "x" in the 2nd field and the numeric value on the 5th field (if it exists, this should be the number of days for expiration, default is 90) for the root user account. The screenshot above shows what values needs to be deleted. Once you have made the changes, go ahead and save the file.

Step 5 - Reboot the VCSA and now you can login to both the VAMI UI interface as well as the SSH console.

Note: If you had the password expiration feature enabled, it has now been disabled for you to login. If you wish to re-enable it, you will need to configure it in the VAMI UI or through the CLI. Please refer to this article here for more details.

Administrator password expiration in new VCSA 5.5

by 4 Comments

A new security enhancement that you should be aware of when deploying the new vCenter Server Appliance (VCSA) 5.5 is that there is now a password expiration that is enabled for the administrator account (root) after powering on the VCSA. By default, the password will expire 90 days after and if the password is not changed before the expiration, the account will be locked out of the VAMI interface and the SSH console. From a security point of view, this is a nice feature to have to ensure administrative passwords are automatically rotated, however this can also be an administrative challenge if you are not aware of this new change and you suddenly notice you can no longer login after 90 days.

You can find the password expiration settings under the Admin tab of the VAMI interface. You have the ability to enable or disable the feature as well as change the number of days the password is valid for. If you decide to change the default number of days, you will be required to enter an email address which will be used to email you 7 days prior to expiration which is the default.

In addition to using the VAMI interface to configure these settings, I was also interested to see if these settings can be automated through the command-line and with a bit of digging, these options can be completely controlled through the CLI!

We will be using the chage utility which manages user account expiry. To view the default settings for the root account or any other account, run the following command:

chage -l root

We can see from the screenshot above, the maximum days before expiration is 90 and the number of days to warn before expiration is 7 which matches the VAMI UI.

Lets say we want to change the maximum days before expiration to 120 and instead of warning 7 days before expiration, we want to change it to 12, you can do so by running the following command:

chage -M 120 -W 12 root

If you wish to completely disable account password expiry, you can do so by running the following command:

chage -M -1 -E -1 root

You can also configure the email address through the command-line which is used to warn X days before password expiry. To add or update the email address, you will need to create a file called /etc/vmware-vpx/root.email that contains the email address.

From an operational perspective, you will want to ensure you configure an SMTP server in your vCenter Server after deploying the VCSA and ensure you add an email address so you can be notified before the root account password expires. You should also configure the maximum number of days before the password expire and the number of days to warn to match your internal security policies.

In the event that you lock yourself out, how do you go about recovering from this since you will not be able to login to the VAMI interface nor the SSH console? I have purposely configured one of my VCSA to expire the password in 1 day, so stay tune for a future article on how to recover from this.

Here is How to recover VCSA 5.5 from an expired administrator account article.

New vCenter Server Simulator 2.0 enhancements in VCSA 5.5

by 46 Comments

Last year I wrote about a very interesting tool called vCenter Server Simulator (VCSIM) which allows a user to quickly simulate a VMware environment that can be comprised of thousands of ESXi hosts and virtual machines. VCSIM can benefit a variety of use cases such as learning about the vSphere API, creating reports for vSphere or vCloud Director to building vSphere Web Client plugins to help visualize large inventories. There was an overwhelming interest in VCSIM from last year and I received some great feedback and feature requests which I fed back to the VMware engineers who developed this internal tool.

With the upcoming version of vSphere 5.5 to be released very soon, I was wondering if there were going to be any new features for VCSIM in VCSA 5.5? I reached out to one of the engineers, Haiping Yang, who works in the Performance Engineering team who is currently taking over some of the development of VCSIM. Some of you might be familiar with some of her work such as the recent visualEsxtop, esxtop and resxtop to just name a few. In talking to Haiping, I found that she has been quite busy adding cool new features to VCSIM and this is on top of her regular day job!

Disclaimer: This is not officially supported by VMware, please use at your own risk.

Here is a quick summary of the new features of VCSIM 2.0:

Distributed Virtual Switch (VDS) Support:

  • Add / Remove ESXi hosts from VDS
  • Create / Delete Distributed Virtual Portgroup
  • Reconfigure Distributed Virtual Portgroup
    • Add / Remove VM from Distributed Portgroup

vCloud Networking & Security (vCNS) Support:

  • Create / Delete vCNS Gateway
  • Create / Delete Isolated/Routed Org Networks
  • Create / Delete vApp Networks
  • Deploy / Undeploy vApp with DHCP service enabled

Persistent Inventory Configuration upon restart:

  • Folder, Cluster, Resource Pool, Host, Datastore, Virtual Machine, Network and VDS

Custom Configuration Support:

  • ESXi version template
  • ESXi configuration template
  • Datastore configuration
  • Virtual Machine datastore

Easy startup commands:

  • vmware-vcsim-start
  • vmware-vcsim-stop [true|false] - Determines whether the inventory is cleared after stopping VCSIM

Note: Before you can use VCSIM, you will need to configure the VCSA as you normally would by going through the VAMI interface or running through the SSH commands noted in this article.

I will not go over every single feature mentioned above, but I did want to take a look at a few noteworthy features such as the new VCSIM start/stop command, datastore configuration and ESXi host configuration templates.

VCSIM Start/Stop Commands:

With the previous version of VCSIM, you had to manually edit the vCenter Server configuration file (vpxd.conf) and append the necessary VCSIM configurations. In this release, we now have an easy to use command-line utility to start and stop VCSIM. The vmware-vcsim-start command supports several startup options.

To view the list of supported options, just run the following command:

vmware-vcsim-start help

Option 1 - You can specify a VCSIM configuration file and you can find several examples located in /etc/vmware-vpx/vcsim/model

Option 2 - You can specify either the keyword "empty" for a blank vSphere inventory or "default" which will automatically use /etc/vmware-vpx/vcsim/model/vcsim-default.cfg inventory configuration

Option 3 - You can just specify the inventory layout on the command-line. An example would be "custom:dc=1,cluster=1,rp=1,host=1,vm=1,vm_on=1,latency=true"

To get a list of all the available VCSIM configurations, take a look at /etc/vmware-vpx/vcsim/model/vcsim.cfg.template

Here is an example of starting VCSIM using the "default" mode:

vmware-vcsim-start default

 

Datastore Configuration:

Custom datastore configuration was something that was much sought after with VCSIM 1.0 and unfortunately, there was only a single global datastore that was automatically "connected" to all simulated ESXi host. The new version of VCSIM now supports custom datastore configurations that can be defined globally, at the cluster level, local storage as well as string prefix which can help you separate out different VCSIM instances.

Here is an example of the configuration that would need to be added to the VCSIM configuration file:

Here is what one of the simulated ESXi hosts would show for its datastores:

 

ESXi Configuration Template:

Another useful feature that I personally have asked for is the ability to customize an individual simulated ESXi host. Though this is still currently a work in progress, what you can do with VCSIM 2.0 is to customize the ESXi host version as well as the datastores on a per host basis. If you take a look vcsim.cfg.template, you will find a configuration line that looks like:

vcsim/model/hostConfig

This specifies a directory that would contain custom simulated ESXi host templates and their configurations. A sample host template is provided at /etc/vmware-vpx/vcsim/model/hostConfig.xml.template and currently, you need to specify the default simulated hostname (e.g. DC0_C0_H0.xml).

Here is an example of what that host template can look like:

Now if we go back to our DC0_C0_H0 ESXi host, you will see that the host template will override the global configuration:

For the two examples above, here is what I used in my custom VCSIM configuration file that I called vcsim-virtuallyghetto.cfg if you are interested in what I used:

I have already asked for the ability to fully customize the simulated ESXi host display name and have already been told that this is something they would consider for a future release. VCSIM 2.0 has also been improved to better operate with vCloud Networking & Security and vCloud Director. I was able to quickly test VCSIM 2.0 with the latest version of vCloud Director 5.5 and everything seems to be working fine. You can follow the existing instructions here for vCloud Director setup with VCSIM.

As you can see VCSIM 2.0 contains many new features and I highly encourage you to give it a spin when vSphere 5.5 is made generally available. There are definitely some additional fit and finish features that Haiping just could not get into this release. Hopefully we will get those updates in a future release of VCSIM and include additional ESXi template versions. If you have any feedback, comments or feature requests feel free to leave a comment and I will make sure it reaches Haiping and the development team. I do not want to spoil the surprise, but I just want to say one of the features coming in VCSIM 3.0 will be quite AWESOME! 😀 (sorry for the tease)

How To Configure vCenter Server 5.0 To Work With VIN 2.0?

by 8 Comments

Many of you know that I am a huge fan of VIN (vSphere Infrastructure Navigator) and the value it can bring to vSphere administrators and their organizations. With the latest release of VIN 2.0, there are even more exciting features and integration with both the vSphere and vCenter Operations Manager platforms. However, one of the prerequisites for using the latest version of VIN 2.0 is that you will need to be running a vSphere Web Client 5.1 Server which can be a challenge for customers still on vSphere 5.0.

There was a question raised internally awhile back ago on whether it would be possible to have VIN 2.0 function with a vCenter Server 5.0? In the VIN 2.0 release notes, there is a statement that seems to indicate this is possible:

The user interface of the vCenter Infrastructure Navigator 2.0 virtual appliance that is deployed on vCenter Server 5.0 can only be viewed from the vSphere Web Client 5.1

A feature that may not be very well known with the release of vSphere 5.1 is that the vSphere Web Client Server also supports vCenter Server 5.0 which must be manually added through the vSphere Web Client admin application. This means that vSphere administrators not only benefit from all the new feature enhancements of the new vSphere Web Client but will would also be able to get a single view of their entire vSphere 5.x infrastructure.

Given all this information, I suspect this should work and I had an idea on how I could implement this. Since VIN 2.0 can only be used from a 5.1 version of the vSphere Web Client, we can simply deploy a VCSA 5.1 (vCenter Server Appliance) and configure it to point to our vCenter Server 5.0 environment. This will then allow us to use VIN 2.0 with our vCenter Server 5.0 environment while still maintaining our vCenter Server 5.0 environment.

Note: Though I have opted to use the VCSA as that is the simplest method IMHO, you are not required to. The only requirement is access to a vSphere 5.1 Web Client Server which you can also install on a separate Windows server.

Here is a quick diagram of what this would look like:

For some background here is what the environment looks like:

  • VCSA 5.0 managing ESXi 5.0 hosts with running VMs
  • VCSA 5.1 (configured, but no inventory)
  • VIN 2.0 deployed onto the ESXi 5.0 hosts being managed by the vCenter Server 5.0

Here are the steps to get this working:

Step 1 - Deploy the VCSA 5.1 and configure the system as you would normally. We will only be using the vSphere Web Client from this VCSA.

Step 2 - Register your vCenter Server 5.0 environment using the admin app in the vSphere Web Client. If you are using the VCSA 5.1, you will need to follow the instructions here.

Step 3 - Deploy VIN 2.0 into the vCenter Server 5.0 environment if you have not already.

Step 4 - Open a browser and connect to the VCSA's 5.1 vSphere Web Client. The URL should be https://[VC-IP]:9443/vsphere-client and provide the vCloud Suite License key which is required to license VIN 2.0

Step 5 - Enable discovery for the vCenter Server 5.0 under the "Infrastructure Navigator" tab on the left hand side of the vSphere Web Client.

Step 6 - Once the initial discovery has completed, you should now be able to see VIN information displayed for your virtual machines.

So there you have it! VIN 2.0 functioning with a vCenter Server 5.0 environment with a bit of help from the 5.1 version of the VCSA. You will still be able to connect to the vCenter Server 5.0 environment using either the vSphere C# Client and even the 5.0 vSphere Web Client. Though with so many new features in the new vSphere Web Client, this a great way to start getting comfortable with the new interface and enjoy all the benefits from VIN.

Automating SSL Certificate Regeneration in VCSA 5.1 & 5.5 (vCenter Server Appliance)

by 4 Comments

The VCSA (vCenter Server Appliance) provides a very simple way of regenerating the self-signed SSL Certificate by using the VAMI web management interface. This is extremely useful if you change the IP Address or hostname of your VCSA and want a proper SSL certificate with the correct common name, especially important if you are plan on using something like vCenter Orchestrator which validates this. To regenerate the SSL Certificate, you just need to login to the VAMI web interface by pointing your browser to the following address: https://[VC-IP]:5480 and under the Admin tab there is a option to "Toggle certificate setting".

After enabling this option, you will need to reboot your VCSA for the new SSL certificate to be generated. Once the VCSA is booted up, you will need to go back into the VAMI interface and disable this setting, else another SSL certificate will be generated upon the next reboot.

I was recently asked if it was possible to automate the SSL regeneration via the command-line without using the GUI which would be very useful for automated VCSA deployments. In looking into this, it turns out the process is quite simple and the present of a file within the VCSA will determine whether a certificate regeneration is required.

To enable certificate regeneration, run the following command which will "touch" (create) allow_regeneration file under /etc/vmware-vpx/ssl directory:

touch /etc/vmware-vpx/ssl/allow_regeneration

To disable certificate regeneration, you just need to remove the file after the VCSA has rebooted. Behind the scenes, this is what is happening when you are toggling the option in the VAMI interface and now you can automate this from the CLI without using the GUI!

UPDATE (09/04/13)

For the new VCSA 5.5, there is a new option that you can specify which will re-generate the SSL certificate and then delete the file without requiring manual intervention after reboot. You would still need to create the /etc/vmware-vpx/ssl/allow_regeneration file but if the contents of the file contains "only-once", it will delete the file automatically which is nice from an Automation perspective.

To re-generate the SSL certificate and automatically have it clean itself up, run the following command:

echo only-once > /etc/vmware-vpx/ssl/allow_regeneration