Project Nanosphere

The #NotSupported event at the VMworld Community Lounge ended with a very special presentation by our very own Randy Keener about a project that a few VMware engineers have been working on called Nanosphere. For those of you who could not make the session or attend VMworld this year, here is some additional information about what Project Nanosphere is all about.  

What is Nanosphere?

First off, Nanosphere is not a product, it is a proof-of-concept. The idea is to make ESXi easier to deploy and manage for non-technical users in small environments (SOHO, remote/branch office, family) to get the same benefits of virtualization that enterprises have. Nanosphere provides an ultra-lightweight management layer on top of an ESXi host that will offer a basic set of features including self-configuration, VM provisioning, VM lifecycle management, and console access.

Today, connecting to a VM console typically requires both server-side dependencies (a VDI broker, a Windows stack, or specialized guest customization) and client side dependencies (installing a special ActiveX browser plugin that works only on Windows, and only in IE or Firefox browsers). By deploying WSX on ESXi, it makes it possible to connect to any VM (any guest OS) with any modern browser (e.g. including iPad) without any special software. 

What can Nanosphere do?

• Network auto-configuration
• Automatic network configuration without ever typing an IP address
• Web Management Interface
• Provision, Delete, Power On/Off Virtual Machines with pure HTML5 interface
• Console access without special apps or plugins
• WSX remote console running on ESXi
• Dead-simple installation
• Just install a tiny VIB onto any ESXi host and you’re good to go. The VIB can also be integrated into a vanilla ESXi ISO image

During Randy’s session, a demo of the network autoconfiguration of Nanosphere and its web interface was given and here is how it works.

Assuming you have a simple cable-model-like setup:

1. The physical host has ESXi and Nanosphere installed.
2. You "unbox" it (take it home from Staples) and plug it in on your home LAN, headless.
3. It gets DHCP but you have no idea what the address is because it's headless.
4. Nanosphere "phones home" to a broker running at nanosphere.cloudfoundry.com (custom application written on Cloudfoundry) to report its local LAN address (e.g. '192.168.0.4') and its UUID. The broker also records the WAN address.
5. You use a plain browser on any device on the same LAN - we used an iPad - to connect to the same broker. It matches the WAN addresses and redirects the browser to the Nanosphere’s LAN address.

Here are a few screenshots of the Nanosphere web interface:

 

What's next for Nanosphere? 

 

As mentioned earlier, nanosphere is still a proof-of-concept but the VMware engineers have some interesting ideas on where it could go and would love to get your feedback if the following use cases interests you.

• Early adopters and hobbyists playing with ESXi for fun
• VARs delivering Nanosphere-based servers in selected vertical markets
• Nanosphere-based appliances delivering NAS and media streaming
• Nanosphere-based servers for developing markets and nonprofit organizations
• Hybrid public/Nanosphere clouds with bidirectional app portability
• OEMs delivering Nanosphere-based servers through a retail channel
• Value-added services like cloud backup and remote admin (including VMware GO)

Other work includes tracking ongoing WSX improvements. If any of these use cases interests you, please leave a comment below or if you have other ideas/feedback for Nanosphere, feel free to leave a comment as well.

I think the Nanosphere project is a really cool initiative and hopefully we will get to see more in the future. I wanted to also give a big thanks to folks who worked on the Nanosphere project and made it possible to show off at the #NotSupported event: Steve Strassmann (VMware Staff Engineer), Shivam Tiwari (VMware Intern) and of course Randy Keener (VMware TechOps) for presenting on Project Nanosphere!

How to Enable Nested ESXi & Other Hypervisors in vCloud Director 5.1

The process to enable  "Nested Virtualization" in the latest release of vCloud Director 5.1 and create your own virtual lab similar to VMware's vSEL (Virtual Sales Enablement Cloud) is very similar to the previous steps outlined for vCloud Director 1.5 release. The only change is how VHV (Virtual Hardware-Assisted Virtualization) aka "Nested Virtualization" is enabled in vCloud Director 5.1 and ESXi 5.1.

In the vCloud Director 1.5, to enable VHV, you needed to add a special SQL statement that would enable VHV for the underlying ESXi 5.0 hosts. With the latest release of vCloud Director 5.1, that is no longer necessary and you now enable it on a Per VM basis within the vCloud Director 5.1 UI.

Here are the steps for enabling VHV for vCloud Director 5.1

• Insert SQL statements into VCD Database that perform the following:
• Enable new "VMware" guestOS Family
• Enable new guestOS Type ESXi 4.x and 5.x 
• Enable host preparation to enable VHV (vSphere 5.0 & vCloud 1.5 only)
• Enable promiscuous mode
• Insert SQL statement into VCD Database for Network Pool that is being used for your ESXi VMs
• Enable both Promiscuous Mode and Forged Transmit for vSphere Backed Portgroup within vCenter Server or ESXi host

The SQL statements can be found in this article and have not changed for vCloud Director 5.1

Here is a screenshot of what you should see in the vCloud Director 5.1 UI for creating a new VM and you should now have the ability to select a new guestOS Type called "VMware" and select either an ESXi 4.x or ESXi 5.x guestOS Version.

To enable VHV for the VM, you will need to also check the box "Exposed hardware-assisted CPU virtualization to guestOS" and this will allow you to run a nested ESXi VM as well as 64-bit nested VMs, assuming your physical CPUs support it. To learn more about running VHV on ESXi 5.1, take a look at this article here for more details. 

How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1

There are a ton of new features with the latest release of vSphere 5.1, but the one "unsupported" feature I always test first is "Nested Virtualization" (aka Nested ESXi) and with the latest release, it seems to have gotten even better. You will still need to have the same physical CPU prerequisites as you did in the past to run "Nested Virtualization" as well as nesting 64-bit VMs.

• Intel VT-x or AMD-V is required for running "Nested Virtualization" which supports nested 32-bit VMs
• Intel EPT or AMD RVI is required for running nested 64-bit VMs.

A quick way to verify whether your CPU truly supports both Intel-VT+EPT or AMD-V+RVI, you can paste the following into a browser:  

https://[your-esxi-host-ip-address]/mob/?moid=ha-host&doPath=capability 

You will need to login with your root credentials and then look for the "nestedHVSupported" property and if it states false, it means you maybe able to install nested ESXi or other hypervisor, but you will not be able to run nested 64-bit VMs, only 32-bit VMs, assuming you have either Intel-VT or AMD-V support on your CPUs.

For more details, take a look at this article for troubleshooting: Having Difficulties Enabling Nested ESXi in vSphere 5.1?

Disclaimer: This is not officially supported by VMware, please use at your own risk.

There are some changes with Nested Virtualization in vSphere 5.1 also officially known as VHV (Virtual Hardware-Assisted Virtualization). If you are using vSphere 5.0 to run Nested ESXi or other nested Hypervisors, then please take a look at the instructions in this article. With vSphere 5.1, there have been a few minor changes to enable VHV.

1. The new Virtual Hardware 9 compatibility will be required when creating your nested ESXi VM, Virtual Hardware 8 will not work if you are running ESXi 5.1 on your physical host. You will still need to enable promiscuous mode on the portgroup that will be used for your nested ESXi VM for network connectivity.
2. vhv.allow = "true" is no longer valid for ESXi 5.1 to enable VHV. A new parameter has been introduced called vhv.enable = "true" that is now defined on a per VM basis to provide finer granularity of VHV support. This also allows for better portability between VMware's hosted products such as VMware Fusion and Workstation as they also support the vhv.enable parameter. 
3. You can now enable VHV on a per VM basis and using the new vSphere Web Client which basically adds the vhv.enable = "true" parameter to the VM's .VMX configuration file.

Note: You can run a nested ESXi 5.1 VM on top of a physical ESXi 5.0 host, just follow the instructions here.

Enabling VHV (Virtual Hardware-Assisted Virtualization)

 

Step 1 - Create a new Virtual Hardware 9 Virtual Machine using the new vSphere Web Client that's available with vCenter Server 5.1.

Step 2 - Select "Linux" as the guestOS Family and "Other Linux (64-bit)" as the guestOS Version.

Step 3 - During the customize hardware wizard, expand the "CPU" section and select "Hardware Virtualization" box to enable VHV.

Note: If this box is grayed out, it means that your physical CPU does not supported Intel VT-x + EPT or AMD-V + RVI which is required to run VHV OR that you are not using Virtual Hardware 9. If your CPU only supports Intel-VT or AMD-V, then you can still install nested ESXi, but you will only be able to run nested 32-bit VMs and not nested 64-bit VMs.

Step 4 - It is still recommended that you change the guestOS Version to VMware ESXi 5.x after you have created the VM shell, as there are some special settings that are applied automatically. Unfortunately with the new vSphere Web Client, you will not be able to modify the guestOS after creation, so you will need to use the C# Client OR manually go into the .VMX and update guestOS = "vmkernel5"

Now you are ready to install nested ESXi VMs as well as run nested 64-bit VMs within.

If you have followed my previous article about How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5 you may recall a diagram about the levels of "Inception" that can be performed with nested ESXi. That is, the number of times you could nest ESXi and still have it be in a "functional" state. With vSphere 5.0, the limit that I was able to push was 2 levels of nested ESXi. With latest release of vSphere 5.1, I have been able to push that limit now to an extraordinary 3 levels of inception!

You might ask why would someone want to do this ... well I don't have a good answer other than ... because I can? ;) VHV is one of the coolest "unsupported" feature in my books and I'm glad it is working beyond what it was designed for.

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.  

Nesting "Other" Hypervisors

For those of you who feel inclined to run other hypervisors such as Hyper-V, you can do so with latest release of ESXi 5.1. The process if very straight forward just like running nested ESXi host.

Step 1 - Create a Virtual Hardware 9 VM and select the appropriate guestOS. In this example, I selected Windows Server 2012 (64-bit) as the guestOS version.

Step 2 - Enable VHV under the CPU section if you wish to create and run nested 64-bit VMs under Hyper-V

Step 3 - You will need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE

There's a CloudPhysics Card for that

“There’s a CloudPhysics card for that” … is what you could be saying pretty soon with all the great ideas that have been suggested so far in the CloudPhysics Lab for the CloudPhysics VMworld 2012 Challenge. One of the coolest aspect of the challenge, is it is 100% community driven and the idea is really simple. Suggest an idea (using a card) that describes an issue or challenge you have faced or are facing in your vSphere environment.

Note: To learn more about CloudPhysics, check out Frank Denneman’s CloudPhysics in a nutshell article.

An example card suggestion could be automatically validate all Virtual Machines against the vSphere Security Hardening Guide or provide a DRS vCPU analyzer and recommend or automate Virtual Machine migration based on vCPU count.  
As you can see, the possibilities are endless and there are currently over 500 cards that have been suggested so far. You still have time to join in on the fun (contest ends on Aug 26th, 2012), just sign up at at CloudPhysics.com and start suggesting your ideas and you can win some really cool prizes if your card is rated as one of the best by the community.

Note: Due to U.S legislation, non-U.S. residents are excluded from winning prizes

The voting system on CloudPhysics just went live yesterday and there is also a new layout for viewing all the cards in the system. The cards are now organized in the following manner:

• My Suggested Cards (these are cards you created)
• Most Popular
• Newest Cards
• Random Cards
• All Community Cards

So after you are done submitting your card ideas, be sure to vote on some of your favorite community cards or leave comments on the cards. The more voting and commenting you do, the more points you earn!

Here are a few cards that I personally thought were interesting and voted for:
 
So hopefully in the near future, you can say “There is a CloudPhysics Card for that” and it just might be your card!

Configuring ESXi Power Management Policy Using the CLI

An interesting question on the VMTN forum caught my eye today, which was around configuring ESXi's Power Management Policy using the command-line via a kickstart script. I found this question to be interesting as I never had to tweak this configuration and was curious myself to see how you might be able to perform this via the command-line as I never recall seeing a command relating to the power management settings.

After a few minutes of digging, I found that the standard set of CLI's such as ESXCLI, vim-cmd, etc. do not provide a way to configure ESXi's power management settings but did find it was possible using my other favorite "not officially supported" CLI called vsish. Now, you can of course create a remote script using the vSphere API to configure this setting, but if you are looking to modify this within a kickstart script, this is route you will want to take.

To check whether your ESXi host supports power management, run the following command:

~ # vsish -e get /power/hardwareSupport 

Hardware power management support {
   CPU power management:Enhanced Intel SpeedStep(R)
   Memory power management:Not available
}

To view the current power management setting, run the following command:

 ~ # vsish -e get /power/currentPolicy

Host power management policy {
   ID: 2
   Short name:dynamic
   Long name:Balanced
   Description:Reduce energy consumption with minimal performance compromise
}

Just like the vSphere Client, you have 4 options which maps to the "ID" property as seen above. You can get more details by querying each of the policy (1-4), here is an example:

~ # vsish -e get /power/policy/1

Host power management policy {
   ID: 1
   Short name:static
   Long name:High Performance
   Description:Do not use any power management features
}

Here's a quick table that maps the policy ID to power management policy which is the same order as shown in the vSphere Client:

Policy ID	Power Management Policy
1	High Performance
2	Balanced
3	Low Power
3	Custom

To change the power management policy, run the following command:

~ # vsish -e set /power/currentPolicy 1

So now you can integrate power management settings in your ESXi kickstart script for automated deployment and configurations!

#NotSupported Sessions at VMworld 2012, A Dream Come True

There is going to so much awesomeness at this year's VMworld, but one new event that I am most excited for this year is the #NotSupported Sessions which will be held at the VMware Community Lounge. The concept of this event was thought of by no other than Randy Keener, who works in TechOps at VMware. Randy's idea was to put together an event where people from all around could share tips/tricks, best practices, etc. on some of the neat and cool things  people have done with VMware technologies that may not be "officially supported". This can include installing ESXi on not supported white box configurations, to running nested ESXi for home lab environments. 

I am really glad Randy pushed for this event this year, as this is like a dream come for myself as I tend to walk on the "Not Supported" road quite frequently for those who know me well. I enjoy pushing the boundaries of VMware's products and it will be great to meet other people who also enjoy doing the same. I will also be presenting at #NotSupported which will take place on Wednesday 8/29, so be sure to come check out my session as well as Duncan Epping's session from 1-3pm or even coming by and just saying hello. 

For more details about #NotSupported sessions, including the vBrownBags which will also be taking place in the VMware Community Lounge, take a look at the schedule here.

In addition to the awesome session line-up, there will also be some NEW stuff that I have been working on with Randy and some of our engineers which will be unveiled by Randy during the #NotSupported event, so you won't want to miss out! #NotSupported will be the PLACE to be at VMworld 2012! Hope to see you there!

Forwarding vCenter Server Logs to a Syslog Server

I was recently asked if it was possible to forward vCenter Server logs to a regular syslog server and if so, how difficult would it be to setup? I had researched this topic several years back, but did not find an ideal solution as vCenter Server was only available on the Windows platform and vCenter Server itself did not provide any syslogging capabilities. With the release of vSphere 5.0, VMware introduced the VCSA (vCenter Server Appliance) and realized I never revisited this question for the VCSA.

After a bit of digging, I found that the VCSA comes installed with syslog-ng by default which is used to provide the vSphere Syslog Collector functionality as well as the local syslog client for the VCSA itself. Given this information, it was pretty trivial to source the local /var/log/vmware/vpx/vpxd.log (symlink to latest vCenter Server log as well as other important vCenter logs) and automatically forward that to a remote syslog server.

VCSA Syslog Configuration

You will need to edit the following configuration file on the VCSA - /etc/syslog-ng/syslog-ng.conf and add the following lines at the bottom of the file (remember to replace the syslog host with your own):

Note: If you are interested in more details about "sourcing" a local log, take a look at this article here which I used as a reference.

Once you have saved the configuration file, you just need to restart the syslog client by running the following command:

service syslog restart

If you login to your remote syslog server, you should now see that your VCSA is forwarding it's vpxd logs over. Pretty simple, right? :) You can of course forward over other vCenter Server logs by adding additional source files. The main key is that there is a symlink that automatically points to the latest log file which you map as the source file.

I am sure many of you are probably asking what about vCenter Server for Windows? Well, I did also looked into a similar solution but it's a bit more complex than just adding a few configuration entries.

Windows vCenter Server Syslog Configuration

Disclaimer: This is not supported by VMware, please use at your own risk.

There are a few challenges with the Windows version, by default there are no syslog clients installed and there is no automatic symlink to the latest vCenter Server log. Having said that, you can still get the above solution working using the free syslog-ng, but it takes a few more steps. The solution will be leveraging Cygwin, so we can run the free version of syslog-ng on a Windows system.

Step 1 - Install Cygwin and configure syslog-ng service on your vCenter Server as described in this article. You will need to add an additional package which is "Admin/Cron" that will be used in the subsequent steps. In the example, I ran syslog-ng under default system account, but if you need to run it under a different user, you may find these two articles to be helpful

• http://linux.subogero.com/894/cron-on-cygwin/
• http://www.davidjnice.com/articles/cygwin_cron-service.html

Step 2 - Just as before, we will need to edit /etc/syslog-ng/syslog-ng.conf and add the following lines at the bottom of the file (remember to replace the syslog host with your own):

 

You will notice this time, we are accessing the Windows C drive by using the /cygdrive path

Step 3 - As mentioned earlier, there is no symlink that points to the latest vCenter Server log, which makes it difficult to map to static log file. What we can do is basically identify the latest vpxd-#.log and automatically create a symlink and that is what is being monitored by syslog-ng to forward the log. We will be using a cronjob and a very simple shell script.

You can place the script in the current home directory /home/Administrator (or whatever default user you happen to have installed Cygwin on)

Here is the shell script which I have called latest.sh:

Make sure to set the script to be executable: chmod +x latest.sh

Step 4 - Create a cronjob which will run every minute (you might be able to set a longer delay depending on your environment and it's rotation frequency) by editing the following file /var/cron/tabs/Administrator or using crontab -e
Step 5 - Start or restart syslog-ng by running one of the following commands:

Start - cygrunsrv -S syslog-ng
Restart - cygrunsrv -E syslog-ng;cygrunsrv -S syslog-ng

If everything was successful, you should start seeing your vCenter Server logs from your Windows system forward to your remote syslog server. When the latest vpxd-#.log changes, the cronjob will automatically take care of re-linking to the latest vpxd-#.log to ensure you continue forwarding your vCenter Server logs.

As you can see, it is not trivial to set this up for the Windows vCenter Server as it is for the VCSA, but you now have a way to centrally store all your important vCenter Server logs for archival or analysis purposes without having to manually copy them off to a remote volume.

Few additional notes:

• I believe the paid version of syslog-ng supports file globbing, so you do not need to setup a cronjob and just watch for all vpxd-*.log, but in this example, I went with a completely free solution
• You might also be able to leverage Splunk to monitor vCenter Server logs as noted in this Splunkbase entry, but I have not verified and I am not sure if you have to pay for this feature in Splunk
• Here is an easier way of forwarding vCenter Server logs on Windows using Snare by Raphael Schitz.