virtuallyGhetto

powercli

vGhetto Automated Pivotal Container Service (PKS) Lab Deployment 

by 3 Comments

While working on my Getting started with VMware Pivotal Container Service (PKS) blog series awhile back, one of the things I was also working on was some automation to help build out the required infrastructure NSX-T (Manager, Controller & Edge), Nested ESXi hosts configured with VSAN for the Compute vSphere Cluster and Pivotal Ops Manager. This was not only useful for my own learning purposes, but that I could easily rebuild my lab if I had messed something up and allowed me to focus more on the PKS solution rather than standing up the infrastructure itself.

To be honest, I had about 95% of the script done but I was not able to figure out one of the NSX-T APIs and I got busy and had left the script on the back burner. This past weekend while cleaning out some of my PKS research documents, I came across the script and funny enough, in about 30minutes I was able to solve the problem which I was stuck for weeks prior. I just finished putting the final touches on the script along with adding some documentation. Simliar to my other vGhetto Lab Automation scripts, I have created a Github repo vGhetto Automated PKS Lab Deployment

UPDATE (06/19/18) - I have just updated the script to also include the deployment and configuration of the PKS components (Ops Manager, BOSH Director, Harbor & Stemcell). The script by default will now configure everything end-2-end and you will have a fully functional PKS environment that you can start playing around with. For complete details, please see the Github repo which has the updated requirements and documentation. Below is a screenshot of the PKS deployment and configuration which requires the use of the Ops Manager CLI (OM).


The script will deploy the following components which will be placed inside of a vApp as shown in the screenshot below:

  • NSX-T Manager
  • NSX-T Controller x 3 (though you technically only need one for lab/poc purposes)
  • NSX-T Edge
  • Nested ESXi VMs x 3 (VSAN will be configured)
  • Ops Manager


The script follows my PKS blog series and automates Part 3 (NSX-T) and the start of Part 4 (Ops Manager deploy), please refer to these individual blog posts for more information. The goal of the script is to enable folks to jump right into the PKS configuration workflows and not have to worry about setting up the actual infrastructure that is needed for PKS. Once the script has finished, you can jump right into Ops Manager and start your PKS journey.

Here is a sample execution of the script which took ~29 minutes to complete.


The full requirements for using the script be found on the Github repo and below are the software versions that I had used to deploy and configure PKS:

How do you “log a reason” using PowerCLI when rebooting or shutting down ESXi host?

by 1 Comment

I am sure many of you have seen this UI prompt asking you to specify a reason before issuing a reboot or shutdown of an ESXi host and I assume most of you spend a few seconds to type in a useful message and not just random characters, right? 😉


Have you ever tried performing the same reboot or shutdown operation using the vSphere API or PowerCLI (which leverages the API)? Have noticed, there is not a way to specify a message like you can in the UI?

Here is a table of the PowerCLI cmdlets and the respective vSphere API that is used to perform these two operations:

Operation Cmdlet vSphere API
Reboot  Restart-VMHost  RebootHost_Task
Shutdown  Stop-VMHost  ShutdownHost_Task

When looking at either the PowerCLI and/or vSphere API documentation, we can confirm that there are no fields to specify a message which can lead to an assumption that this is simply not possible or that the functionality might be provided by a private API. Fortunately, this is not the case and the functionality is in fact in the public vSphere API and has been for quite some time.

When you specify a message prior to rebooting or shutting down, this message is actually persisted and implemented as an Event within vCenter Server as shown in the screenshot below.

Instead of being able to specify a message that is only applicable to an ESXi host, I believe the original vSphere API designers thought that this functionality could also be useful and applied more broadly across any number of the vSphere Inventory objects, not just ESXi hosts. As such, this functionality which the vSphere UI uses is provided by the LogUserEvent() method which is part of the EventManager API. Customers or solutions can leverage this mechanism to log custom user defined events which is then persisted with the lifecycle fo the vSphere Inventory Object or as far back as your retention period for vCenter Server Events.

Going back to our original question, if you want to specify a message prior to rebooting or shutting down an ESXi host, the following snippet below demonstrates the use of the vSphere API via PowerCLI:

Converting a vSphere Content Library from Subscriber to a Publisher?

by Leave a Comment

This is a blog post on some old Content Library research which I had looked into several months back for a customer inquiry and realized I never got a chance to write about. When you create a Subscriber Content Library (CL) which synchronize content from a Published CL, the Subscriber CL is basically in read-only mode. Not only can you not edit or modify the content, but the CL itself is also locked as a Subscriber forever. Why might this be a concern or an issue?

Consider the following scenario, where you have a single master CL that contains all the images you wish to distribute amongst your remote sites as efficiently as possible and ideally leveraging CL's built-in replication mechanism. Lets say the master CL is in Palo Alto and you have several vCenter Servers that can be configured as Subscriber CLs in Cork, Ireland. Instead of having configuring each Subscriber to pull remotely from PA, why not replicate that to a single Subscriber located in Ireland and re-publish that content locally to the remainder CL? The ideal workflow would look something like the diagram below.

However, today you can not simply convert a Subscriber CL to also become a Publisher CL. The only option that I could think of without relying on something like externally replicating a CL is to create an additional Published CL from the Subscriber CL. The idea is you would clone from your Subscriber CL in the same vCenter Server to a new Publisher CL and then that CL can then be used by the other local Subscriber CL as depicted in the diagram below.

This functionality currently does not exist in the vSphere Web/Flex Client, however it can be automated using the Content Library REST APIs. I had created a Content Library PowerCLI Module a few months back and one of the functions that would be useful for this workflow is the Copy-ContentLibrary.

The function is pretty straight forward to use, it accepts the name of a source CL that has already been configured as a Subscriber CL and the name of a new Published CL that you have created but is empty.


After that, the contents of the source CL will be copied into the destination CL as shown in the screenshot below.


Depending on the size of your CL, this can take some time but once it has completed, you should that your Published CL is now populated with content from your source CL.

The PowerCLI function also does a few additional things such as checking if a file already exists and will skip the file and move on to help with the copy.


If your intention is to only replicate from your master CL and then create a local publisher CL, there is an additional flag that you can set to $true called -DeleteSourceFile which will delete the files from the Subscriber CL. This is useful if you only have a single Datastore in which both the Subscriber and Publisher CL is storing, which means after the CL copy, you would be consuming 2x the storage as everything has been duplicated. For this particular customer, they only required an initial sync to populate content in their EMEA office and after that, they would be managing it local and not require additional replications from their master CL.

How to troubleshoot NSX-T API errors when using PowerCLI?

by Leave a Comment

In the last few days, I have been ramping back up on NSX-T usage, especially as I take a closer look at VMware Pivotal Container Service (PKS) which hopefully I will be able to share more with an upcoming blog series (at least, thats the plan when I find some spare time). While working on some Automation using the NSX-T REST APIs and the new NSX-T PowerCLI cmdlets, I found it to be pretty challenging and frustrating to troubleshoot NSX-T error messages thrown from the  PowerCLI cmdlets.

For example, below is a PowerCLI snippet which I wrote to create a new T1 Logical Router in NSX-T.

However, when I ran the code, PowerCLI threw back a pretty generic and useless error which I am sure some of you have probably seen before "A server error occurred: 'com.vmware.vapi.std.errors.invalid.frequest': . Check $Error[0].Exception.ServerError". Even checking the error variable as stated in the error message did not yield any further details or specific to the issue I was running into.

[Read more...] about How to troubleshoot NSX-T API errors when using PowerCLI?

Automating Intel Sighting remediation using PowerCLI (SSH not required)

by 22 Comments

In case you may not be aware, Intel recently notified VMware that certain Intel Broadwell and Haswell CPUs are affected by Intel Sighting after applying the latest microcode update to remediate against the Spectre vulnerability. VMware has published the following KB 52345 which provides more details on the affected Intel CPUs along with the recommended workaround in case you have already applied the latest ESXi patches containing the faulty microcode. I highly recommend you carefully read over the KB before, even if you have not applied the ESXi patches proceeding further.

With this updated news, I have also updated my existing Spectre verification script (found here) to include the additional Intel Sighting information which can help customers easily identify whether they have ESXi hosts that are impacted. In this article, I will provide a solution to help automate the deployment of the Intel Sighting remediation as outlined in the KB article, but unlike the manual steps outlined in the KB, SSH access to the underlying ESXi host will NOT required.

Step 1 - Download the VerifyESXiMicrocode.ps1 PowerCLI script as well as the new PowerCLI script IntelSightingWorkaround.ps1

Step 2 - Run the "Verify-ESXiMicrocodePatch" function against a specific vSphere Cluster or ESXi host to determine if you are impacted by the Intel Sighting issue. Below is a screenshot for a system which is affected by Intel Sighting and we can determine this by seeing a value of "True" under the IntelSighting column.


Step 3 (Optional) - This step is optional, but I wanted to demonstrate how you can tell whether the Intel Sighting workaround has been applied correctly. You can use the "Verify-ESXiMicrocodePatchAndVM" function which provides information from a Virtual Machine's perspective and whether the new CPU instructions are exposed to the VM. In the screenshot below, I have a Test VM called TestVM-03 that has been powered on and as you can see, the three new CPU instructions (IBR,IBPB and STIB) are present as I have not applied the Intel Sighting workaround.


Step 4 (Optional) - This step is also optional, but I wanted to demonstrate how you can quickly check the contents of /etc/vmware/config without requiring SSH access or direct ESXi Shell access. You can use the "Get-Esxconfig" function and specify an ESXi host to query. The screenshot below confirms that we have not deployed the Intel Sighting workaround.


Step 5 - Once we have identified the ESXi hosts that are impacted by Intel Sightin (See step 2), we need to create a text file that contains the Hostname/IP of ESXi hosts (one on each line) that we wish to remediate. This will then be used as an input to the remediation function. In the example here, I have a single host to remediate and have added its name to a file that I have called "affected_hosts".txt (it can be named anything).

Note: The remediation does NOT require the ESXi host to reboot for the changes to go into effect, but you may still want to consider following standard procedures of putting the host into Maintenance Mode if you wish to control when VMs will see the masked instruction which will require a VM power cycle (OS restart is not sufficient).

To apply the remediation, you will use the "Set-IntelSightingsWorkaround" function which has a single mandatory parameter called AffectedHostList. Below is a screenshot of running the function and the remediation is fairly quick as it is merely appending a single string to the /etc/vmware/config configuration file. After the remediation has completed, we can then use the "Get-Esxconfig" function to confirm that we have added the masked CPU instruction to the ESXi host as shown in screenshot below.


Step 6 - As the KB mentions, for the changes to go into affect, you will need to perform a complete power cycle of your VMs before they can see the new masked CPU instructions. If you recall earlier, I had a VM called TestVM-03 which saw the three new CPU instructions. If we power cycle the VM and re-run the "Verify-ESXiMicrocodePatchAndVM" function against our VM, we should now see that the three CPU instructions have been properly masked away as shown in the screenshot below.


When it is time to remove the workaround, the process is also very straight forward as well. You can use the "Remove-IntelSightingsWorkaround" function which also accepts a list of ESXi hosts to remediate. Once the remediation has completed and you wish to expose the new CPU instructions to the VMs, you will also need to power cycle the VMs for CPU instructions to be unmasked.

Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI

by 80 Comments

VMware recently published a new knowledge base (KB) article 52085 that outlines instructions for enabling the Hypervisor-Assisted Guest Mitigation (CVE-2017-5715), also known as the Spectre vulnerability. This KB also provides steps to verify the updated microcode (included in the ESXi patch) has been applied along with Virtual Machine verification (those applicable) to ensure that they are seeing the new CPU features. While following the KB and patching one of my vSphere environments, I had noticed the verification steps was not only manual but it also to difficult to scale beyond a few VMs as it required customers to look for a specific set of strings from within the vmware.log file which is generated for each powered on VM, which could easily be several hundreds or thousands of VMs.

I figured there had to be a better way to help customers automate this at scale and remove the human factor. The other reason I was not fond of the current method is that it would require customers to either enable ESXi Shell/SSH access or to manually or through automation to download every single vmware.log file to inspect for specific log entries which can take quite a bit of time for any sizable environment. I had an idea on how this could be done without having to look at the vmware.log file while leveraging our vSphere APIs and did some investigation. Before proceeding further, please familiarize yourself with KB 52085. For complete background on both Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) as it relates to all VMware products, please carefully read through this top level KB 52245 which is being updated as new information is available, so you may want to subscribe to the KB for all the latest updates.

UPDATE 4 (01/23/18) - VMware has just updated KB 52345 with the current list of Intel CPUs affected by Intel Sightings. I have also updated my script to reflect all these changes. Make sure to download the latest version to ensure you have the latest changes.

UPDATE 3 (01/16/18) - I have just enhanced the script further to also collect the current microcode version running on a given ESXi host. Unfortunately, this information can only be collected when SSH is enabled and is something a user must explicitly allow. The benefit here is that Intel Sighting impact reporting is more robust as the code now checks for both impacted CPU signature as well as the microcode affected by Intel Sighting as outline in KB 52345. See below for more details.

UPDATE 2 (01/14/18) - As mentioned in the last update, I have been working on a Intel Sighting remediation script which can help customers automate the temporary workaround as recommended in KB 52345. Please see this blog post for complete details.

UPDATE 1 (01/13/18) - VMware just published a new KB 52345 outlining certain Intel Broadwell and Haswell CPUs being affected by Intel Sightings after applying latest microcode updates. Please refer to the KB for the complete details. I am currently working on a script to help with the remediation as the remediation method outlined in the KB would require customers to enable SSH access and manually update /etc/vmware/config. In the meantime, I wanted to provide a way for customers to easily identify whether their system are affected by Intel Sightings and thanks to community member Adam Robinson who just submitted a patch this morning to update my existing script to include these details. I have also added the CPU model to the output as additional information.

Note: This script only provides information about your existing vSphere environment, it does not make any changes or provides any remediation steps, please follow the KB for those instructions.

The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and it contains the following two functions:

  • Verify-ESXiMicrocodePatchAndVM
  • Verify-ESXiMicrocodePatch

[Read more...] about Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI