Upgrading an Amazon EVS environment from VCF version 5.2.x to 5.2.4

Background

At the time of publication, Amazon EVS environments are deployed with VMware Cloud Foundation (VCF) running version 5.2.2.0. This article highlights the considerations discovered while validating the upgrade path from 5.2.x to VCF 5.2.4.

Customers should always follow the Broadcom documentation for detailed upgrade procedures.

Components impacted by the upgrade

Component	Current Version	Current Build	New Version	New Build
SDDC Manager	5.2.3.0	25219033	5.2.4.0	25437063
VMware vCenter Server Appliance	8.0 Update 3i	25197330	8.0 Update 3j	25413364
VMware ESXi	8.0 Update 3i	25205845	8.0 Update 3j	25429389
VMware NSX	4.2.3.0	25171318	4.2.4.0	25410638

Tasks

1. Review Release Notes
2. Reconfigure Backups for SDDC Manager, NSX Manager and vCenter
3. Upgrade SDDC Manager
4. Run upgrade pre-checks and upgrade NSX, vCenter and ESXi

1: Review Release Notes

• Check the official VMware Cloud Foundation 5.2.4 Release Notes for prerequisites, known issues, and specific upgrade steps.

2: Reconfigure Backups for SDDC Manager, NSX Manager and vCenter

Note: By default, the backup repository for the above components is set to the SDDC Manager, which will cause errors during VCF environment upgrades.

• Before attempting to upgrade the environment, ensure that backups for SDDC Manager, NSX Manager, and vCenter are configured to use an external SFTP server repository.
• Refer to the Broadcom documentation for how to configure backups for SDDC Manager, NSX Manager and vCenter.
• Perform a manual backup of SDDC Manager, NSX Manager and vCenter components
• For customers who require an easy SFTP server deployment on EC2, refer to the following AWS re:Post article for external SFTP server setup.

3: Upgrade SDDC Manager

• Begin by upgrading SDDC Manager to version 5.2.4.0 using Lifecycle Manager within the SDDC Manager web interface to download and install the 5.2.4.0 update file.
• Follow the Broadcom documentation for the upgrade process.

4: Run upgrade pre-checks and upgrade NSX, vCenter and ESXi

After upgrading SDDC Manager, run the precheck for the workload domain you plan to upgrade from the Updates tab. Continue to upgrade VCF components or resolve any issues as directed by the Lifecycle Manager.

Figure 1: Update pre-check example

During the pre-check phase, the following issues or warnings may appear:

1. Default upgrade policy mismatch

• Checks if any upgrade policies configured for the cluster match the default SDDC Manager ESXi upgrade policy.
• This is a known warning and can be safely ignored. See Broadcom KB for details.

2. Third-party VIBs detected

• Amazon EVS includes a custom VIB within the EVS-deployed ESX image. Any warnings related to this within Vendor Addon can be safely disregarded, as any new ESX installation will already contain the latest required custom VIBs.
• For customers using NetApp FSx for ONTAP to provide supplemental storage for their EVS environment, you may see this warning if you have not added the NetApp VIB to the vCenter cluster image as a Component. Other third-party VIBs must also be added to vCenter cluster image. See this Broadcom KB for more details about creating new cluster images.

Note: Example command to view list of installed VIBs on an ESXi Host: "esxcli software vib list | grep -i vibname"

3. No backups configured for SDDC Manager, NSX Manager or vCenter

• Ensure that backups have been reconfigured as per Task 2: Re-configure Backups for SDDC Manager, NSX Manager and vCenter.

4. ERROR vSphere SHA-1 validation failed

• This error may be present in the pre-check output if the EVS environment is using the default self-signed certificates.
• To fix this issue customers can either use SDDC Manager to configure the environment to use customer provided signed certificates, or follow this Broadcom KB to resync the vCenter VECS store and continue with the existing self-signed certificates.

5. No image available for ESXi host upgrade

• During the ESXi host upgrade section, customers could encounter an issue whereby there is no image available to select to apply to the hosts.

• This Broadcom KB provides steps to fix this issue by creating a new vCenter cluster image. Ensure you include all required VIBs as part of the new cluster image.

• Ensure when downloading the ESXi image from Broadcom support portal, you import the correct image for Amazon EVS running VCF 5.2.4.0 (version 8.0.3 Update 3j Build 25429389)

Note: Example command to view the currently active image profile from an ESXi Host: "esxcli software profile get"

6. NSX Edge Cluster credentials check failed internally

• NSX edges in Amazon EVS are not deployed using SDDC Manager and are instead deployed directly from NSX manager. The edges are fully manageable from with NSX manager.
• Additionally the SDDC Manager upgrade should complete successfully since the SDDC Manager runs the upgrade workflow through the NSX built-in upgrade workflow.
• Broadcom provides a KB article indicating the warning can be ignored.

7. ESXi Host hardware compatibility warnings

• You may see one or all of the following three warnings after performing the ESXI host upgrade pre-check:

• The image has hardware compatibility issues. Check hardware compatibility for more details
• Firmware information is unavailable without a Hardware Support Manager configured as part of the cluster's image. Firmware compatibility must be verified manually

• These warnings are caused by AWS not having a "Hardware Support Manager" which is used to determine firmware versions. Without it, the pre-check cannot validate if the driver in the image is compatible with the firmware.
• Broadcom provides a KB article indicating the warning can be ignored.

8. Legacy boot detected. ESXi servers running legacy BIOS are encouraged to move to UEFI. Please refer to KB 84233 for more details

• AWS currently has exceptions with Broadcom for the i7i.metal and i4i.metal hosts to continue running in legacy BIOS mode. This warning can be safely ignored.

9. Non-Volatile memory controller: Device is incompatible with the driver/firmware in cluster's image

• Since the NVMe firmware is not exposed to vCenter, there is no way for vLCM to perform validation checks against it.
• This warning can safely be ingored.

Troubleshooting Summary

Issue	Cause	Resolution
Upgrade policy mismatch	Cluster policy differs from default	Safe to ignore; see Broadcom KB
Third-party VIB warning	Custom or vendor VIBs detected	Ensure VIBs are added to vCenter cluster image as component or vendor addon
Backup repository configured to SDDC Manager	SDDC Manager, NSX Manager and vCenter backups pointing to SDDC Manager	Re-configure backups to point to external SFTP server before upgrade following Broadcom KB
SHA-1 certificate error	Default self-signed certificates	Replace with signed certs or follow Broadcom KB to resync VECS store
No ESXi image available	Cluster image not created	Create new vCenter cluster image and include required vendor addon and component VIBs following Broadcom KB
NSX Edge Cluster credentials	NSX edges in Amazon EVS are not deployed using SDDC Manager	Safe to ignore; see Broadcom KB
ESXi Host hardware compatibility warnings	No HSM available in the environment	Safe to ignore; see Broadcom KB
Legacy boot detected. ESXi servers running legacy BIOS are encouraged to move to UEFI. Please refer to KB 84233 for more details	i4i. metal & i7i.metal hosts still run legacy BIOS mode	AWS have an exception in place with Broadcom, safe to ignore
Non-Volatile memory controller: Device is incompatible with the driver/firmware in cluster's image	NVMe firmware is not exposed to vCenter	Safe to ignore

Related information

• Broadcom VMware Cloud Foundation 5.2 Documentation
• Broadcom Knowledge Base
• AWS re:Post article: Deploying an SFTP server
• Amazon EVS Service Overview