govmomi

Record and Replay vSphere Inventory using govc and vcsim

01/04/2021 by William Lam 2 Comments

Happy New Year! 🥳

I wanted kick off 2021 with something I had just learned about right at the end of 2020 which I think this will be useful going into the new year for a number of different use cases. Back in 2017, I wrote about a new and lighter weight version of the vCenter Simulator (vcsim) which had been developed as part of the govmomi (vSphere SDK for Go) project. Since then, the govmomi project has grown exponentially and is now integral to a number of popular open source projects such as Packer builder for vSphere, Terraform provider for vSphere and Kubernetes Cluster API for vSphere (CAPV) to just name a few.

Govmomi is also heavily used internally by VMware for both development and testing purposes. In fact, it has been used to build a number of new VMware features such as the vSphere Integrated Containers (VIC) solution and most recently the vSphere with Tanzu capability that was introduced in vSphere 7.0.

Getting back to vcsim, this has been an invaluable tool for both our VMware developers but also the general VMware community. The ability to "simulate" a mocked vSphere environment with a basic inventory can be extremely useful for learning about the vSphere API and interacting with this endpoint using any vSphere SDK including PowerCLI. For automation folks, this can be useful for designing and creating your scripts in an offline mode before testing it against a real environment. For folks building 3rd party solutions that includes a visual interface, this is an easy way to test out your UI and ensure that there are no issues for large vSphere inventories which can be difficult to validate in a development environment.

Simulating a fake vSphere inventory is great, but it also has its limitations. There are so many unrealized use cases if you could capture a real vSphere inventory and then replay that back using vcsim. Just think about a bug reproduction use case and being able to share a real vSphere inventory with a development or QA team without needing to provide them direct access to the production environment?

In my opinion, this was the missing key feature from the original vcsim. To my surprise, this functionality was actually added to govc/vcsim earlier last year and I was quite happy with its implementation! Let's now take a closer look at how the record and replay functionality of govc/vcsim works.

govcsim – Neat incubation project (vCenter Server & ESXi API based simulator)

04/21/2017 by William Lam 10 Comments

I know many of my readers have inquired about VCSIM (vCenter Server Simulator) which was a really useful tool that served a variety of use cases, but unfortunately it had stopped working with the VCSA 6.0 release. VCSIM is another topic that is near and dear to me and it is something I continue to push and advocate for internally at VMware. Earlier this week, I came to learn about a cool new incubation project that Doug MacEachern had been working on for some time now. Doug is an awesome VMware developer working on the vSphere Integrated Containers (vIC) project and he is also well known for his active contributions to both govmomi (vSphere SDK for Go) and govc CLI.

As you can probably guess from the title, the name of the project is called govcsim and it is a vCenter Server and ESXi API based simulator written using govmomi. It creates a vCenter Server model with a datacenter, hosts, cluster, resource pools, networks and a datastore. The naming of the objects is similar to that of the original VCSIM mode that was included with the VCSA. The number of resources can be increased or decreased using the various resource type flags. Resources can also be created and removed using the API. Doug had developed the tool to provide an easy way for their team to test some of the work they are doing with vIC. The tool is still under incubation but continues to received enhancements. In fact, the other day when I had used it for the first time, I had found a couple of issues which Doug resolved immediately.

I got a chance to give govcsim a spin the other day and currently you can connect to it using govmomi, govc, pyvomi (vSphere SDK for Python) or rbvmomi (vSphere SDK for Ruby). It currently does not work with PowerCLI (connects but no inventory), I know this is something Doug is currently looking into. You might also be able to connect using other vSphere SDKs but these are the ones that Doug and I have tried so far.

Slick way of deploying OVF/OVA directly to ESXi & vCenter Server using govc CLI

04/08/2016 by William Lam 9 Comments

I have been meaning to write about this neat little feature that was added to the govc CLI late last year that allows you to easily deploy any OVF/OVA without the need of ovftool. You might ask, why not use ovftool? Well, if you just need to perform a very basic OVF/OVA deploy and prefer not to install anything on your desktop, this can be a nice alternative. govc is provided as a simple binary that is platform agnostic and supports Windows, Linux & Mac OS X. govc is built using govmomi which is also known as the vSphere SDK for Go and this also means you can consume this capablitity beyond just the CLI but also programmatically if you wish. Obviously, the CLI is the easiest method which I will demonstrate below.

Just to be clear, there is still a huge amount of value in using ovftool as it contains a large mount of functionality that is not found any where else. It is still the recommended tool for deploying OVF/OVA across all VMware based Hypervisors and is extensively used by other VMware's products for general OVF/OVA deployment.

Here are the steps on using govc to deploy an OVF/OVA:

Step 1 - Download the latest govc binary for your platform which can be found here. In this example, I will be running govc on my Mac OS X system and you can easily download it by running the following commands:

curl -L https://github.com/vmware/govmomi/releases/download/v0.5.0/govc_darwin_amd64.gz | gunzip -d > /usr/local/bin/govc
chmod +x /usr/local/bin/govc

Step 2 - To verify that govc is working, go ahead and run the following command to retrieve the version which should display v0.5.0:

govc version

Note: If you are new to govc and want a quick primer on how it works, please head over to this article here for more details.

Step 3 - Next, we need to setup a couple of GOVC environmental variables which will define the vSphere environment (ESXi or VC) you plan to deploy to. You also have the option to specify this in the command-line using the --help option for more details. In this example, I will just set them as environmental variables. Below is an example of my environment which is going to be deploying directly to an ESXi host, you will want to update these variables based on your environment.

export GOVC_INSECURE=1
export GOVC_URL=nuc.primp-industries.com
export GOVC_USERNAME=root
export GOVC_PASSWORD=vmware123
export GOVC_DATASTORE=vsanDatastore
export GOVC_NETWORK="VM Network"
export GOVC_RESOURCE_POOL='*/Resources'

Note: If you are new to govc and want a quick primer on how it works, please head over to this article here for more details.

Step 4 - To ensure that govc can properly communicate with your vSphere enviornment, you can run the following command which should return details about the system you are connected to:

govc about

Step 5 - To deploy an OVF/OVA, we first need to create a spec file by inspecting the actual OVF/OVA using the import.spec command which will output in a JSON format. To make it easier to read, we will format the output using python. To do so, run the following command (replace with the OVF/OVA you wish to deploy):

govc import.spec /Volumes/Storage/Images/Current/VMware-VCSA-all-6.0.0-3634788/vcsa/vmware-vcsa | python -m json.tool

As you can see from the screenshot above, we are inspecting the OVA for the latest vCenter Server Appliance (VCSA) 6.0 and it has automatically generated all the OVF properties that can be specified for this OVA. We then just need to re-direct the output to a file in which we can then edit the properties and pass that configuration when we deploy. We can do so by running the following command:

govc import.spec /Volumes/Storage/Images/Current/VMware-VCSA-all-6.0.0-3634788/vcsa/vmware-vcsa | python -m json.tool > vcsa.json

Step 6 - Now, we just need to open up the JSON file we saved and edit the values before we can use it in our deployment. You may have also noticed that there is an InjectOvfEnv param which means you can use govc to also inject OVF properties when deploying directly to ESXi host 🙂

Here is an example of the vcsa.json for deploying an Embedded VCSA:

{

 "Deployment": "tiny",

 "DiskProvisioning": "thin",

 "IPAllocationPolicy": "dhcpPolicy",

 "IPProtocol": "IPv4",

 "InjectOvfEnv": true,

 "Name": "VCSA-60u2-Embedded",

 "PowerOn": true,

 "PropertyMapping": [

 {

 "Key": "guestinfo.cis.appliance.net.addr.family",

 "Value": "ipv4"

 },

 {

 "Key": "guestinfo.cis.appliance.net.mode",

 "Value": "static"

 },

 {

 "Key": "guestinfo.cis.appliance.net.addr",

 "Value": "192.168.1.40"

 },

 {

 "Key": "guestinfo.cis.appliance.net.prefix",

 "Value": "24"

 },

 {

 "Key": "guestinfo.cis.appliance.net.gateway",

 "Value": "192.168.1.1"

 },

 {

 "Key": "guestinfo.cis.appliance.net.dns.servers",

 "Value": "192.168.1.1"

 },

 {

 "Key": "guestinfo.cis.appliance.net.pnid",

 "Value": "192.168.1.40"

 },

 {

 "Key": "guestinfo.cis.vmdir.password",

 "Value": "VMware1!"

 },

 {

 "Key": "guestinfo.cis.vmdir.domain-name",

 "Value": "vghetto.local"

 },

 {

 "Key": "guestinfo.cis.vmdir.site-name",

 "Value": "virtuallyGhetto"

 },

 {

 "Key": "guestinfo.cis.vmdir.first-instance",

 "Value": "True"

 },

 {

 "Key": "guestinfo.cis.appliance.root.passwd",

 "Value": "VMware1!"

 },

 {

 "Key": "guestinfo.cis.appliance.root.shell",

 "Value": "False"

 },

 {

 "Key": "guestinfo.cis.appliance.ssh.enabled",

 "Value": "False"

 },

 {

 "Key": "guestinfo.cis.appliance.ntp.servers",

 "Value": "pool.ntp.org"

 },

 {

 "Key": "guestinfo.cis.deployment.node.type",

 "Value": "embedded"

 },

 {

 "Key": "guestinfo.cis.ceip_enabled",

 "Value": "False"

 }

 ],

 "WaitForIP": false

}

view raw
vcsa-embed.json
hosted with ❤ by GitHub

Note: There is a known bug currently in which the "Deployment" pararm is always generated. You will need to delete this if you have an OVA that does not use this parameter. I have already been reported here.

Step 7 - Finally, we just need to use the import.ova command and specify the path to our JSON configuration file as well as the path to our OVA. To do so, run the following command:

govc import.ova -options=vcsa.json /Volumes/Storage/Images/Current/VMware-VCSA-all-6.0.0-3634788/vcsa/vmware-vcsa

Note
Note: There is known bug in which you will see a warning about Invalid value for the network which you can ignore. I have already reported the issue here.

Once the VM powers on, the OVF properties will automatically be injected into the VM and in a few minutes, you will have a fully configured VM. Pretty cool right!? What I really love about this solution beyond its simplicity is that the tool allows you to easily generate the OVF property spec file. This is much easier than having to manually inspect an OVF/OVA and then extract the params by hand which can be very error prone. Lastly, I want to give a big shoutout to Pieter Noordhuis who was the Engineer who had implemented this awesome feature and is also one of the Engineers who created govmomi. Thanks Peter!

How to deploy a Kubernetes Cluster on vSphere?

09/05/2014 by William Lam 18 Comments

In the previous article, we walked through the installation of govmomi which is the vSphere SDK for Go and govc which is a command-line interface that uses the SDK to expose vSphere functionality which is used in the Kubernetes vSphere Provider. Now that we have all the prerequisites installed, we are now ready to deploy a Kubernetes Cluster onto a vSphere based infrastructure.

UPDATE (10/26/15) - It looks like the instructions on setting up a Kubernetes Cluster has since changed and I have updated the instructions below. One of the main changes is instead of building from source we are just downloading the Kubernetes binaries.

tep 1 - You will need to download the latest Kubernetes binary (kubernetes.tar.gz) which can be found here. At the time of updating this article, the latest is v1.2.0-alpha2.

Step 2 - Go ahead and extract the contents of kubernetes.tar.gz file by running the following command:

tar -zxvf kubernetes.tar.gz

Step 2 - Download the Kubernetes VMDK using either "wget" or "curl" depending on what is available on your system. Since I am on a Mac, it only has curl by default. Here are the two commands depending on the which download utility you have access to:

wget https://storage.googleapis.com/govmomi/vmdk/kube.vmdk.gz{,.md5}
curl -O https://storage.googleapis.com/govmomi/vmdk/kube.vmdk.gz{,.md5}

Once the download has completed, you should now see two files in your working directory: kube.vmdk.gz and kube.vmdk.gz.md5

Step 3 - Next we need to un-compress the VMDK by running the following command:

gzip -d kube.vmdk.gz

Step 4 - Once the VMDK has been extracted, we will need to upload it to a vSphere datastore. Before doing so, we need to set a couple of environmental variables that provide connection details to your vSphere environment. Below are the commands to set the environmental variables, you will need to replace the information from your own environment.

export GOVC_URL='https://[USERNAME]:[PASSWORD]@[ESXI-HOSTNAME-IP]/sdk'
export GOVC_DATASTORE='[DATASTORE-NAME]'
export GOVC_DATACENTER='[DATACENTER-NAME]'
export GOVC_RESOURCE_POOL='*/Resources'
export GOVC_GUEST_LOGIN='kube:kube'
export GOVC_INSECURE=true

You can leave the last three variables as-is. The GOVC_RESOURCE_POOL defines the full path to root Resource Pool on an ESXi host which will always exists and for vCenter Server, it is the name of the vSphere Cluster or Resource Pool the GOVC_GUEST_LOGIN is the credentials to the Kubernetes Master/Node VMs which are defaulted in the VMDK that was downloaded. The last variable GOVC_INSECURE is if you have an ESXi or vCenter Server using self-signed SSL Certificate, you will need to ensure this variable is set.

To upload the kube.vmdk to the vSphere Datastore and under the kube directory which will be created for you, you will run the following command:

govc datastore.import kube.vmdk kube

Step 5 - We now have our base kube.vmdk uploaded to our ESXi host and before we are ready to deploy our Kubernetes Cluster, we need to set the provider by running the following command:

export KUBERNETES_PROVIDER=vsphere

Step 6 - We are now ready to deploy the Kubernetes Cluster which consists of Kubernetes Master and 4 Kubernetes Minions and they will be derived from the kube.vmdk that we just uploaded. To do so, you will run the following command:

kubernetes/cluster/kube-up.sh

Note: If you see a message about "Docker failed to install on kubernetes-minion-N" it is possible that this related to a timing issue in which the Minion may not be up when the Master is checking. You can verify this by running the next command, else you can follow the instructions to bring down the Kubernetes Cluster and re-creating it.

Step 7 - In the previous step, we deployed the Kubernetes Cluster and you will see the assigned IP Addresses for the Master/Minions along with the credentials (auto-generated) for the Docker Containers. We can confirm that the everything was created successfully by checking the number of running Minions by running the following command:

cluster/kubecfg.sh list minions

Step 8 - Once we have confirmed we have 4 running Minions, we can now deploy a Docker Container onto our Kubernetes Cluster. Here is an example of deploying 2 nginx mapping from port 8080 to 80 instances by running the following command:

cluster/kubecfg.sh -p 8080:80 run dockerfile/nginx 2 myNginx

Step 9 - We should expect to see two "Pods" for the nginx instances we have instantiated and we can do so by running the following command:

cluster/kubecfg.sh list pods

Here are some additional commands to stop and remove the Pods:

cluster/kubecfg.sh stop myNginx
cluster/kubecfg.sh rm myNginx

You can also bring down the Kubernetes Cluster (which will destroy the Master/Minion VMs) by running the following command:

cluster/kube-down.sh

Hopefully this gave you a good introduction to the new Kuberenetes vSphere Provider and I would like to re-iterate that this is still being actively developed on and the current build is an Alpha release. If you have any feedback/requests or would like to contribute, be sure to check out the Kubernetes and govmomi Github repository and post your issues/pull requests.

govmomi (vSphere SDK for Go), govc CLI & Kubernetes on vSphere

09/04/2014 by William Lam 15 Comments

One of the exciting announcements that was made last week at VMworld was the joint partnership between Docker, Google, Pivotal and VMware. Paul Strong (Office of the CTO) wrote a great blog post Better Together – Containers are a Natural Part of the Software-Defined Data Center where he goes into more details about the partnership. The really neat part of the announcement which I think some people may have missed is that this was more than just an announcement. There are actually active projects currently being worked on, most notably a working prototype for a Kubernetes vSphere Provider.

For those of you who are not familiar with Kubernetes, it is an open-source project that was started by Google which provides Container Cluster Management. You can think of Kubernetes as a placement engine/scheduler for Containers, similar to how vSphere DRS is responsible for scheduling Virtual Machines. The Kubernetes vSphere Provider allows you to run a Kubernetes Cluster on top of a vSphere based infrastructure and provides a platform for scheduling Docker Containers running on top of vSphere.

Kubernetes is completely written in Go (short for Golang), a programming language developed by Google. To be able to easily integrate with Kubernetes, a Go library needed to be written for the vSphere API and hence govmomi was born! Similar to pyvmomi and rbvmomi which are vSphere SDKs for Python and Ruby respectively, govomimi is the vSphere SDK equivalent for Go. The govmomi project is an open source project lead by VMware and you can find the Github repository at https://github.com/vmware/govmomi.

In addition to govmomi, I also learned about a neat little CLI that was built on top of the SDK called govc (currently an Alpha release) which provides a simplified command-line interface to a vSphere environment leveraging govmomi. You can find the source code under the govmomi Github repository https://github.com/vmware/govmomi/tree/master/govc. The Kubernetes vSphere Providers leverages govc to be able to orchestrate the deployment of a Kubernetes Cluster on top of vSphere leveraging the vSphere API.

To use govc, you will need to ensure you have Go 1.2+ installed on your system. Here are the steps on installing Go and govc

Step 1 - Download the latest Go package installer for your OS here and once you have Go installed, you can verify that everything is working by running the following command:

go version

Step 2 - Setup your build environment by running the following commands:

export GOPATH=$HOME/src/go
mkdir -p $GOPATH
export PATH=$PATH:$GOPATH/bin

Step 3 - Check out govc source code by running the following command:

go get github.com/vmware/govmomi/govc

At this point, govc has been installed. You can now connect to either a vCenter Server or ESXi host. The easiest way is to specify the vSphere API endpoint and credentials is by setting a couple of environmental variables, so you do not have to specify it on the command-line.

Step 4 - Run the following command and specify the username, password and either the hostname or IP Address of your vCenter Server or ESXi host:

export GOVC_URL='https://[USERNAME]:[PASSWORD]@[ESXI-OR-VCENTER-HOSTNAME-OR-IP]/sdk'

Step 5 - To verify that everything is working, you can run the following command to query the endpoint you have connected to:

govc about

If everything was successful, you should see some basic information about the vSphere API endpoint you have connected to. In the example above, I am connected to a VCSA (vCenter Server Appliance). The govc CLI is quite similar to RVC with commands broken up into various namespaces. However, one feature that is not there today is the ability to tab complete the commands which is something I just love about RVC!

You can also just run "govc" and it will provide a list of available commands:

You can get more details about each command by specifying --help command, here is an example of the host.info

To get information about one of my ESXi hosts, I need to specify --host.ip option along with the IP:

As you can see from the screenshot above, some basic information is displayed about my ESXi host which is running on a Mac Mini. If you would like to learn more about govc, I highly recommend you check out the govc repository on Github which has additional documentation. You can also file any bugs or feature requests you would like to see on the project page.

At this point you are now ready to proceed to the next steps which is to setup Kuberentes and deploy a Kubernetes Cluster onto your vSphere environment. Unfortunately I ran into a problem while going through the Kubernetes deployment and I did not know where to go next and decided to file a Github issue here. To my surprise, I immediately got a response back from the VMware Engineers who are working on the project. I had a couple of email exchanges with the team to debug the problem. It looks like we found the culprit and I was able to get Kubernetes up and running. There are a couple of minor caveats which I will explain in more detail in Part 2 of this post and walk you through the steps of deploying a Kubernetes Cluster running on top of vSphere.