vcsim

Record and Replay vSphere Inventory using govc and vcsim

01/04/2021 by William Lam 2 Comments

Happy New Year! 🥳

I wanted kick off 2021 with something I had just learned about right at the end of 2020 which I think this will be useful going into the new year for a number of different use cases. Back in 2017, I wrote about a new and lighter weight version of the vCenter Simulator (vcsim) which had been developed as part of the govmomi (vSphere SDK for Go) project. Since then, the govmomi project has grown exponentially and is now integral to a number of popular open source projects such as Packer builder for vSphere, Terraform provider for vSphere and Kubernetes Cluster API for vSphere (CAPV) to just name a few.

Govmomi is also heavily used internally by VMware for both development and testing purposes. In fact, it has been used to build a number of new VMware features such as the vSphere Integrated Containers (VIC) solution and most recently the vSphere with Tanzu capability that was introduced in vSphere 7.0.

Getting back to vcsim, this has been an invaluable tool for both our VMware developers but also the general VMware community. The ability to "simulate" a mocked vSphere environment with a basic inventory can be extremely useful for learning about the vSphere API and interacting with this endpoint using any vSphere SDK including PowerCLI. For automation folks, this can be useful for designing and creating your scripts in an offline mode before testing it against a real environment. For folks building 3rd party solutions that includes a visual interface, this is an easy way to test out your UI and ensure that there are no issues for large vSphere inventories which can be difficult to validate in a development environment.

Simulating a fake vSphere inventory is great, but it also has its limitations. There are so many unrealized use cases if you could capture a real vSphere inventory and then replay that back using vcsim. Just think about a bug reproduction use case and being able to share a real vSphere inventory with a development or QA team without needing to provide them direct access to the production environment?

In my opinion, this was the missing key feature from the original vcsim. To my surprise, this functionality was actually added to govc/vcsim earlier last year and I was quite happy with its implementation! Let's now take a closer look at how the record and replay functionality of govc/vcsim works.

govcsim – Neat incubation project (vCenter Server & ESXi API based simulator)

04/21/2017 by William Lam 10 Comments

I know many of my readers have inquired about VCSIM (vCenter Server Simulator) which was a really useful tool that served a variety of use cases, but unfortunately it had stopped working with the VCSA 6.0 release. VCSIM is another topic that is near and dear to me and it is something I continue to push and advocate for internally at VMware. Earlier this week, I came to learn about a cool new incubation project that Doug MacEachern had been working on for some time now. Doug is an awesome VMware developer working on the vSphere Integrated Containers (vIC) project and he is also well known for his active contributions to both govmomi (vSphere SDK for Go) and govc CLI.

As you can probably guess from the title, the name of the project is called govcsim and it is a vCenter Server and ESXi API based simulator written using govmomi. It creates a vCenter Server model with a datacenter, hosts, cluster, resource pools, networks and a datastore. The naming of the objects is similar to that of the original VCSIM mode that was included with the VCSA. The number of resources can be increased or decreased using the various resource type flags. Resources can also be created and removed using the API. Doug had developed the tool to provide an easy way for their team to test some of the work they are doing with vIC. The tool is still under incubation but continues to received enhancements. In fact, the other day when I had used it for the first time, I had found a couple of issues which Doug resolved immediately.

I got a chance to give govcsim a spin the other day and currently you can connect to it using govmomi, govc, pyvomi (vSphere SDK for Python) or rbvmomi (vSphere SDK for Ruby). It currently does not work with PowerCLI (connects but no inventory), I know this is something Doug is currently looking into. You might also be able to connect using other vSphere SDKs but these are the ones that Doug and I have tried so far.

Quick Tip – VCSIM no longer works in VCSA 6.0

07/01/2015 by William Lam 2 Comments

I figured I do a quick blog post on the topic of VCSIM and VCSA 6.0 since I have received several questions regarding its functioanlity in VCSA 6.0. The simple and unfortunate answer is that VCSIM no longer functions in the latest VCSA 6.0. There have been enough changes to the vSphere 5.x and vSphere 6.0 code base that this internal tool will no longer function. As far as I know, there are no plans of changing this as VCSIM was primarily used internally by our Developers for certain types of functional testing. I know I will miss using the tool as it did come in handy!

UPDATE (04/23/17) - For those interested in VCSIM, there is a new project called govcsim. For more details check it out here.

VCSIM should continue to work in both VCSA 5.1 and 5.5 and you can find more details in the articles below.

Feedback needed for the future of VCSIM

01/29/2015 by William Lam 2 Comments

Last week, I had a chance to catchup with a couple of folks over in our Performance Engineering team to talk about VCSIM. For those of you who have not heard of VCSIM before or would like to know how to get started, I highly recommend you check out this article here and here for more details. We had a discussion on a variety of topics including how I and some of our customers are using VCSIM today.

As some of you already know, VCSIM is an internal tool originally developed by VMware Engineering for a very specific set of use cases, it was never intended to be used by our customers. Having said that, I think after talking to Engineering, they understood the value in having such a tool which could be useful to both our customers and partners. In fact, I have even shared a couple of use cases that I believe VCSIM can greatly benefit everyone:

Exploring and learning about the vSphere API and the basic inventory hierarchy of vSphere objects
Environment to develop and create various inventory reporting scripts (vCLI, PowerCLI, etc)
Developing performance metric gathering tools
Developing vSphere Web Client plugins and being able visualize large inventory of objects

There were some thoughts on how VCSIM could evolve over time from its current implementation as it was designed for a very specific purpose, but it was too early to tell and it would be based on some assumptions. Instead, I thought it might be useful to get feedback into Engineering so they could better understand how VCSIM was currently being used today. To help with this, I have created a very short survey below or you can also directly access it using the link here. Please take a few minutes to provide some feedback on how you currently use VCSIM today. Thanks in advance!

Additional VCSIM Resources:

Simulating vSphere Performance Metrics using VCSIM

04/01/2014 by William Lam 7 Comments

A really useful tool that I leverage from time to time is VCSIM (vCenter Simulator) which can be found within the VCSA (vCenter Server Appliance). VCSIM allows you to easily "simulate" a custom vSphere Inventory that can be used for a variety of use cases including custom reports using the vSphere API/CLIs. There are additional capabilities that VCSIM provides and one that I have not explored much is the simulation of vSphere Performance Metrics. Having received a couple of inquires regarding VCSIM and performance metrics, I figure this would be a good opportunity to explore this feature in a bit more detail.

Disclaimer: This is not officially supported by VMware, use at your own risk.

Before getting starting, you should familiarize yourself with VCSIM by reading these two articles here and here.

By default when running VCSIM, there are no Performance Metrics. If you wish to include it, you will need to ensure the following two lines are added to your VCSIM configuration file:

<perfCounterInfo>vcsim/model/PerfCounterInfo.xml</perfCounterInfo>
<metricMetadata>vcsim/model/metricMetadata.cfg</metricMetadata>

The first file contains the performance metric definitions that are supported and the second file will specify which metrics will be simulated. To demonstrate the performance metric capabilities of VCSIM, I will be using the following configuration files which you can just copy/paste:

vghetto-perf-vcsim.cfg - This will be our VCSIM configuration file

<simulator>
  <enabled>true</enabled>
  <initInventory>vcsim/model/vghetto-perf-inventory.cfg</initInventory>
  <hostConfigLocation>vcsim/model/hostConfig</hostConfigLocation>
  <perfCounterInfo>vcsim/model/PerfCounterInfo.xml</perfCounterInfo>
  <metricMetadata>vcsim/model/metricMetadata.cfg</metricMetadata>
  <datastore>
     <global>1</global>
     <cluster>2</cluster>
     <local>1</local>
     <prefix>vghetto</prefix>
  </datastore>
</simulator>

<initInventory>vcsim/model/vghetto-perf-inventory.cfg</initInventory>

<hostConfigLocation>vcsim/model/hostConfig</hostConfigLocation>

<perfCounterInfo>vcsim/model/PerfCounterInfo.xml</perfCounterInfo>

<metricMetadata>vcsim/model/metricMetadata.cfg</metricMetadata>

<prefix>vghetto</prefix>

</datastore>

</simulator>

vghetto-perf-inventory.cfg - This will be our VCSIM inventory configuration file

<config>
  <inventory>
    <dc>1</dc>
    <host-per-dc>0</host-per-dc>
    <vm-per-host>0</vm-per-host>
    <poweron-vm-per-host>0</poweron-vm-per-host>
    <cluster-per-dc>1</cluster-per-dc>
    <host-per-cluster>2</host-per-cluster>
    <rp-per-cluster>1</rp-per-cluster>
    <vm-per-rp>3</vm-per-rp>
    <poweron-vm-per-rp>3</poweron-vm-per-rp>
    <dv-portgroups>0</dv-portgroups>
  </inventory>
  <prefix>vGhetto-</prefix>
  <worker-threads>1</worker-threads>
  <synchronous>true</synchronous>
</config>

<host-per-dc>0</host-per-dc>

<vm-per-host>0</vm-per-host>

<poweron-vm-per-host>0</poweron-vm-per-host>

<cluster-per-dc>1</cluster-per-dc>

<host-per-cluster>2</host-per-cluster>

<rp-per-cluster>1</rp-per-cluster>

<vm-per-rp>3</vm-per-rp>

<poweron-vm-per-rp>3</poweron-vm-per-rp>

<dv-portgroups>0</dv-portgroups>

</inventory>

<prefix>vGhetto-</prefix>

<worker-threads>1</worker-threads>

</config>

To ensure everything is working, we can start VCIM by issuing the following command on the VCSA:

vmware-vcsim-start /etc/vmware-vpx/vcsim/model/vghetto-perf-vcsim.cfg

If everything is working, you should be able to login using the vSphere Web/C# Client to view the small inventory we just created. If you are able to see the inventory, then go ahead and stop VCSIM by issusing the following command:

vmware-vcim-stop false

Note: To ensure our inventory is not destroyed each time, you should pass in the 'false' flag else it will automatically be deleted each time. This is useful if you want to preserve your inventory on subsequent reboots.

For all Performance Metric configurations, you will only need to edit the metricMetadata.cfg file. VCSIM supports four different types of Stats model:

Constant
Linear
Square
Triangle

We will take a look at Stats model for 2-4 since the Constant is not all that interesting 🙂 For our examples, we will take a look at each Stats model for the datastore.datastoreIops.average metrics.

Linear Stats Model:

<Metric id="datastore.datastoreIops.average">
  <Instance id="Default"/>
    <StatsModel>
      <Type>Triangle</Type>
      <Values>0,20,10,30,0</Values>
      <Periods>600,300,600,900</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

<Type>Triangle</Type>

</StatsModel>

</Instance>

</Metric>

Square Stats Model:

<Metric id="datastore.datastoreIops.average">
  <Instance id="Default"/>
    <StatsModel>
      <Type>Square</Type>
      <Values>0,10,0,20,0</Values>
      <Periods>300,300,600,600,300</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

<Type>Square</Type>

</StatsModel>

</Instance>

</Metric>

Triangle Stats Model:

<Metric id="datastore.datastoreIops.average">
  <Instance id="Default"/>
    <StatsModel>
      <Type>Triangle</Type>
      <Values>0,20,10,30,0</Values>
      <Periods>600,300,600,900</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

<Type>Triangle</Type>

</StatsModel>

</Instance>

</Metric>

I thought this was pretty neat that the VCSIM developers included a couple of Stats Models that could be leveraged right out of the box! As you can see, it is pretty easy to enable various performance metrics simply by identifying the metric(s) you are interested in and specifying the Stats Model and then starting up VCSIM. The other neat thing that I have been asked about before is can VCSIM simulate performance metrics for specific vSphere entities? I originally thought the answer was no until I started to play with the performance metric simulator a bit more and realize there is a List property that you can use to specify the specific objects in which you want data to be displayed.

Here is an example of of the same performance metric we have been looking at but only enabling it for two Datastores:

<Metric id="datastore.datastoreIops.average">
  <Instance id="List">
    <List>vghettoDS_vGhetto-DC0_C0_0,vghettoDS_vGhetto-DC0_C0_1</List>
    <StatsModel>
      <Type>Triangle</Type>
      <Values>0,20,10,30,0</Values>
      <Periods>600,300,600,900</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

<List>vghettoDS_vGhetto-DC0_C0_0,vghettoDS_vGhetto-DC0_C0_1</List>

<Type>Triangle</Type>

</StatsModel>

</Instance>

</Metric>

Note: One thing I noticed while playing with the performance metric simulator is that some times the object in the UI is blank when using the vSphere Web Client. If I use the vSphere C# Client, it is perfectly fine and that is also true if you are using the vSphere API to query for these metrics.

Hopefully this was a good overview of how the VCSIM performance metrics feature works. I know there are a couple of internal folks who have used VCSIM injunction with vCenter Operations and I am also curious to see what other neat uses cases exists for the performance metrics feature. Also, if you have created a really cool metricMetadata.cfg configuration file, feel free to share with the rest of the community!

Quick Tip – Adding a custom prefix to your VCSIM environment

03/16/2014 by William Lam 3 Comments

I just discovered this neat little feature of VCSIM (vCenter Server Simulator) which allows you to append a global string prefix to all vSphere objects in your inventory. I am not sure if this capability has always been there but definitely can be useful if you plan on running multiple VCSIM environments, this can be an easy way to uniquely identify a particular configuration.

To add the custom prefix, you will need to add the <prefix></prefix> tags to your VCSIM inventory configuration file. You can take a look at the default template /etc/vmware-vpx/vcsim/model/initInventory.cfg to get an idea of the available configurations.

Here is a the sample VCSIM configuration file for the inventory in the screenshot above:

<config>
  <inventory>
    <dc>1</dc>
    <host-per-dc>0</host-per-dc>
    <vm-per-host>0</vm-per-host>
    <poweron-vm-per-host>0</poweron-vm-per-host>
    <cluster-per-dc>1</cluster-per-dc>
    <host-per-cluster>2</host-per-cluster>
    <rp-per-cluster>1</rp-per-cluster>
    <vm-per-rp>3</vm-per-rp>
    <poweron-vm-per-rp>3</poweron-vm-per-rp>
    <dv-portgroups>0</dv-portgroups>
  </inventory>
  <prefix>vGhetto-</prefix>
  <worker-threads>1</worker-threads>
  <synchronous>true</synchronous>
</config>

<host-per-dc>0</host-per-dc>

<vm-per-host>0</vm-per-host>

<poweron-vm-per-host>0</poweron-vm-per-host>

<cluster-per-dc>1</cluster-per-dc>

<host-per-cluster>2</host-per-cluster>

<rp-per-cluster>1</rp-per-cluster>

<vm-per-rp>3</vm-per-rp>

<poweron-vm-per-rp>3</poweron-vm-per-rp>

<dv-portgroups>0</dv-portgroups>

</inventory>

<prefix>vGhetto-</prefix>

<worker-threads>1</worker-threads>

</config>