Like most folks, I was excited about the release of vSphere 7.0 Update 2 and I was ready to upgrade my personal homelab, which was running on vSphere 7.0 Update 1c. However, after starting my VCSA upgrade in the VAMI UI, it quickly failed with the following error message: Exception occurred in install precheck phase
Joy … I just attempted to upgrade my VCSA (7.0u1c) in my personal homelab to #vSphere70Update2 and ran into “Exception occurred in install precheck phase” … pic.twitter.com/4mkvxHxdRl
— William Lam (@lamw) March 9, 2021
Given the release had just GA’ed less than an hour ago and everyone was probably hammering the site, I figured I would wait and then try again.
Issue
I normally use the “Stage and Install” option and after accepting the EULA and providing the required credentials, the upgrade began.
Almost immediately, it failed with the same error and the only option was to click on the Resume button which I did.
The upgrade seemed to have resumed and pre-check results passed as you can see from the screenshot below. However, after hitting continue, I would be taken right back to the pre-check error as noted earlier. I repeated this a few times and realize it was stuck in this loop.
I even attempted to reboot the VCSA, maybe my environment had some issue but the results were the same. There were also a couple of suggestions on removing the following state file /etc/applmgmt/appliance/software_update_state.conf on the VCSA filesystem which apparently was a solution that had worked. However, in this case it did not do anything and I still ran into the issue.
At this point, I was not interested poking around or randomly removing files as I heard few other folks were running into this problem. I generated a support bundle and file an internal bug to have Engineering take a look.
Disclaimer: If you are currently facing this problem in a Production environment, please stop and file a VMware Support Request so that this can be officially tracked and you can have GSS reference PR 2732490. The workaround below has only been confirmed by the community and should only be attempted on non-production or homelabs environment where you are comfortable with the workaround.
Workaround
Several folks have shared with me a workaround that they have verified which requires the upgrading the VCSA using the VCSA Shell. I have personally not applied this workaround as I wanted to leave my environment in its existing state, in case VMware Engineering need to troubleshoot live. For those interested, below are the instructions to apply the workaround.
Step 1 – Login to the VAMI UI and under Updates, only select the “Stage” option to download the 7.0 Update 2 updates.
Step 2 – SSH to the VCSA and remove /etc/applmgmt/appliance/software_update_state.conf file
Step 3 – Run the following command to install the staged 7.0 Update 2 software:
software-packages install –url –acceptEulas
Once this operation completes, you should now have a successfully upgraded VCSA to 7.0 Update 2. I will update this blog post as I learn more from Engineering and if this workaround is official, I will ask the team to publish a KB and also mention this in the 7.0 Update 2 release notes.
I ran into this today. The fix for me was staging and then upgrading in two steps, rather than one. The install then proceeded until 80% when it threw an error that I was able to continue past. No issues with my second vCenter.
Two vCenters have been upgraded today. It should be easier than this. Don’t they test?!
William – It would be helpful to know what version your VCSA was at prior to the upgrade attempt to 7.0.2? I recently ran into this same issue when upgrading VCSA 7.0.1a to 7.0.1d. I’ll refer readers to your Disclaimer above – STOP and call GSS if you’re experiencing this on a production system! In my case, GSS had me follow the procedure in https://kb.vmware.com/s/article/67179 to remove the software_update_state.conf file, then mount the full patch -fp.iso to my VCSA vm appliance, then staging the packages and upgrading in two separate steps via CLI as described here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-5FCA78EC-8637-43A4-8B28-24624E4D5EBA.html . One thing not mentioned is that you will get prompted to enter SSO credentials during the upgrade process, but the CLI upgrade went smoothly. GSS indicated to me that this was caused a bug in the source VCSA bits of my previous 7.0.1a install, and that this bug was resolved in the newer versions – so future upgrades from VAMI GUI should work as normal. I have NOT tried this 7.0.2 upgrade yet in that environment.
Thanks for sharing Patrick. I actually did mention my version, it’s 7.0u1c 🙂
My apologies – my eyes skimmed right over it. Reading Is Fundamental! Now I need to test from 7.0.1d and see if I run into the same issue 😉
Fixed for me as well, thanks!
I’ve tried the workaround to update from 7.0 Update 1d. The update ends with the following error: Mismatch:
summary: PostgreSQL failed to run upgrade command. So even it’s my lab, I decide to wait for an update of vCSA 7.0 Update2. 🙂
Hit this exact issue twice on the last two times I patched the VCSA in our lab.
The same workaround (patch ISO) worked each time, and appears to be working this time.
I wanted to update my lab yesterday to 7.0 U2. from 7.0 U1a
First I tried via VAMI and got the same error like above.
Removing the file etc/applmgmt/appliance/software_update_state.conf did not resolve the issue.
But also via CLI I could not update, if I use the –ulr switch I got:
“Answers for following questions have to be provided to install phase:
Question:
ID: vmdir.password
Text: Single Sign-On administrator password
Description: For the first instance of the identity domain, this is the password given to the Administrator account. Otherwise, this is the password of the Administrator account of the replication partner.
Allowed values:
Default value:
”
I do not get a prompt to enter something.
(Command> software-packages stage –url https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.1.00100.latest/ –acceptEulas
)
Then I downloaded the 7.0 U2 iso and mounted it to VCSA and run the update via CLI with the –iso switch, I got:
“ISO unmounted successfully
CD drives do not have valid patch iso.”
It sounds like you may have downloaded the install .iso and not the Full Patch iso? Try the VMware-vCenter-Server-Appliance-7.0.2.00000-17694817-patch-FP.iso available on the VMware patch site: https://my.vmware.com/group/vmware/patch
@ Patrick Long: thx, with the patch iso the upadte to 7.0 U2 worked.
There has been a running bug in 7.0 since GA just in a slightly different form. If you install any updates using the CLI then VAMI will no longer install updates at all failing with errors or not showing any updates available.
I had cases open and was told this would be fixed in U2 sounds like fixes for it may have broken something else.
I was able to complete the update today. Here is a short document, I created for future reference.
http://www.infotechram.com/index.php/2021/03/16/how-to-update-vcenter-manually-using-cli/
Ram
Thank you William, the workaround worked as a charm. In my case I stumbled on some other issues like:
“Error in method invocation {‘id’: ‘com.vmware.appliance.update.checksum_verification_failed’, ‘default_message’: ‘Checksum verification failed’, ‘args’: []}”; so changing the URL at update Section overcomes this one: https://10.15.9.4:5480/#/ui/update/availableUpdates
https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.1.00100.latest/
Afterwards, once staged and prechecked/validated the Install process starts to never end. After an snapshot rollback the only way to resolve it so far is to follow on the posted solution at the article.
Got the exact same issue in a prod vcenter; going to raise a ticket right now, will reference your GSS ticket. thanks for that!
Get the same issue but still no luck with your instruction, always failed in the “copying software packages” stage when it comes to 66/67 package out of 150.
Had an odd experience with this issue just now. I got the error as everyone else here. Then started to google the issue, and of course stumbed upon this post.
Then after a while, without doing anything,i noticed i got tossed out into the login screen of the VCSA, and when trying to log in again, it failed.
I then tried opening vcenter normally, and got a message about upstream issue.
This reminded me a bit about what happens while an update is going on, so i let it rest for a while, and god different errors when trying to enter vcenter.
Left it for about an hour, and when i tried again, it was updated! So even though i got the error, and didn’t try any of the workarounds, it ended up updating to Update 2 after all!