Recently I was performing an upgrade to a UCS environment to support some new B200 M4 blades. The current firmware version only supported the B200 M3 blades.
With UCS firmware there is a specific order to follow during the upgrade process. The order to follow is:
- Upgrade UCS Manager
- Upgrade Fabric Interconnect B (subordinate) & Fabric B I/O modules
- Upgrade Fabric Interconnect A (primary) & Fabric A I/O modules
- Upgrade Blade firmware by placing each ESXi Host
During the upgrade process, and particularly during the Fabric Interconnect and I/O module upgrades you will see a swarm of alerts coming from UCSM. This is expected as some of the links will be unavailable as part of the upgrade process so wait until all the steps have been completed before raising an issue .
You will need to check the release requirements from Cisco regarding the upgrade path from your current firmware version to the desired version and also the capabilities that the desired firmware version contains. The upgrade process takes some time so it’s best to review everything in advance and not have to do this on the day of the upgrade itself.
The steps to carry out in the planning phase are:
Step 1: Verify the Upgrade requirements
Check the release version notes on Cisco’s website for the version you want to upgrade to. For this example we are upgrading to version 2.2
Check the required minimum software version as this should be the one that you should update to. This depends on what new infrastructure you are adding such as new blade type or VIC module.
Also check the Capability Catalog support to ensure that the CPU and RAM.
Check the VIC type support also, we are using the latest VIC 1340 which is support in release 2.2 (3a). You may notice that that version also supports the UCS B200 M4 with the CPU listed above so we could go to 2.2 (3a) but it’s best to move to the minimum suggested software release from Cisco.
Review the resolved caveats to see if any issues from a previous release
Step 2: Verify Interoperability
Check the interoperability matrix site also to check if there are any specifics relating to your version of ESXi
Read the VMware guide to see if there are any extra items required to get the system working:
Step 3: Review Faults
Launch a browser and log into UCSM with your Administrative account. Once you login you will be presented with the UCS Manager screen. By default it will open on the Equipment tab from where you can expand each component to get a more in-depth view of the infrastructure
Review the faults on each UCS Chassis or just click on one of the links in fault summary in the top left. This will open up the list of faults for the fabric.
To view the full alerts go to Admin -> Faults, Events and Alerts. It is important to remove critical and major alerts before completing the UCS upgrade.
For the critical issues it is best to open a support call with TAC to advise that a firmware upgrade is going to be performed to ensure that there are no recommended steps missing.
The major alert you sometime get is due to Keyrings certificate is invalid. This is showed how to resolve this issue in a separate blog post
STEP 4: DOWNLOAD FIRMWARE
Download the Cisco UCS Firmware from cisco.com. You will need a valid Cisco Support login to download this firmware.
- Open a Web browser to software.cisco.com. Select Upgrade and Update
- 2. In the Products window select Servers –Unified Computing, then select Cisco UCS B-Series Blade Server Software
- In the Download Software screen select Unified Computing System (UCS) Server Software Bundle.
- A new window will appear displaying the versions of software for the Cisco UCS B-Series Blade Server Software. Select the highest version of suggested firmware, which in this example is 2.2(3d).
- There are three binaries to download, an Infastructure Software Bundle (ucs-k9-bundle-infra.2.2.3d.A.bin), a Software for the UCS B-Series blade server products (ucs-k9-bundle-b-series.2.2.3d.B.bin) and a Software for the UCS C-Series server products (ucs-k9-bundle-b-series.2.2.3d.C.bin). Click Download for each binary file.
STEP 5: UPLOAD FIRMWARE BINARIES to UCS MANAGER
1. Select the Equipment tab -> select the Equipment tree ->then select Firmware Management tab.
2. Select the Download Tasks tab and press the Download Firmware button.
- Select location of the file as Local File System.
- Press the Browse button and locate the three binary files downloaded earlier.
- Select the first binary (A.bin) and press the OK button.
Click ok to view the Transfer State
Click Ok on the Download Task dialog to continue
Under Download Tasks you can now see the new file has been uploaded. It is confusingly labelled as having a Transfer State as Downloaded
- Do the above steps to import the B and C bin files.
- Next go to Firmware Management -> select the Packages tab. The three binaries should now be available
Select the Installed Firmware tab. Expand all the items to view the Running, Startup and Backup Firmware versions
STEP 6: TAKE BACKUP
Take a backup of the UCS configuration before continuing. Select the Admin tab -> select the General tab -> then select Backup Configuration.
- Select Download to Local System, select type as Full State and ensure the file extension on the local file system is set to .xml.
- Select Browse and go to c:/Temp on your local drive and enter the file name. Click ok
- Click ok to complete the backup
STEP 7: UPGRADE UCS MANAGER
Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> then select the Installed Firmware tab.
- Select UCS Manager at the top and click the Activate Firmware button.
- In the Activate Firmware dialog select the new version of firmware from the Version to Be Activated drop-down menu and click ok (Note: The screenshot below shows 2.2(1d) but the version selected to activate was 2.2(3d))
UCS Manager will now upgrade. You will lose access to UCSM during the upgrade. Sit tight for a few minutes and it will be available again.
Log in again to UCSM and accept any Java warnings.
6. Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> then select the Installed Firmware tab and verify that the Running Version has changed (2.2(3d))
STEP 8: UPGRADE THE FABRIC INTERCONNECTS AND I/O MODULES
Select the Equipment tab -> select the Equipment tree -> select Fabric Interconnects
Select Fabric Interconnect A (primary) -> select High Availability Details on General Tab to expand
Verify that HA settings are as follows: Ready: Yes, State: Up, Leadership: Primary, Cluster Link State: Full.
Select Fabric Interconnect B (subordinate) -> click Activate Firmware in Actions.
5. From the drop down menus select the desired Kernel and System Versions and then click OK.
6. Fabric Interconnect B will now upgrade including the I/O modules. This process will take approximately 20 to 25 minutes. You can monitor the progress of the firmware upgrade from the FSM tab.
- Use the FSM tab to monitor the progress of the firmware upgrade. Once the progress reaches 100% you can continue with Fabric Interconnect A.
- Connect via SSH directly to Fabric Interconnect IP address. Once connected you can run the below commands to make the newly updated fabric interconnect the lead FI within the environment. This stops your systems from going offline during the upgrade of the A fabric interconnect. The change over will be pretty much instantaneous.
UCS-A# connect local-mgmt UCS-A(local-mgmt)# cluster lead b UCS-A(local-mgmt)#
- Now that Fabric Interconnect B has taken over the primary role, back in the UI select Fabric Interconnect A (primary) -> click Activate Firmware in Actions.
10. From the drop down menus select the desired Kernel and System Versions and then click OK.
11. Fabric Interconnect A will now upgrade including the I/O modules. This process will take approximately 20 to 25 minutes. You can monitor the progress of the firmware upgrade from the FSM tab. You may lose connectivity to UCSM during this upgrade process as it’s the primary Fabric Interconnect that you are connected to and it will need to reboot. If this occurs just log into UCSM again and it will connect to the remaining Fabric Interconnect.
12. Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> then select the Installed Firmware tab.
- Check that the Fabric Interconnects and I/O Modules have a Running Version of 2.2(3d).
- Now that both Fabric Interconnects have been upgraded you can go ahead and return to primary function back to Fabric Interconnect A. Please note that this is an optional step. Connect via SSH directly to Fabric Interconnect B IP address. Once connected you can run the below commands to make the newly updated fabric interconnect the lead FI within the environment. This stops your systems from going offline during the upgrade of the A fabric interconnect. The change over will be pretty much instantaneous.
UCS-B# connect local-mgmt UCS-B(local-mgmt)# cluster lead a UCS-B(local-mgmt)#
STEP 9: UPGRADE BLADE SERVERS
- Select the Servers tab -> select the Policies tree -> expand Host Firmware Packages.
2. Enter the Name as per your companies naming convention e.g FWP2.2-3d. Enter a description and select Simple for How would you like to configure the Host Firmware Package?
3. From the drop down menus select the desired firmware version for Blade and Rack packages and then click ok.
- Select Maintenance Policies and verify that the Reboot Policy is set to User Ack. Otherwise the ESXi hosts will reboot automatically once the firmware upgrade process begins
5. Now you are ready to start upgrading the Blade firmware of your ESXi hosts. Go into vCenter and select the ESXi host that you want to upgrade. Place the ESXi host into Maintenance Mode
6. For Service Profiles that are bound to a service profile template the change can be made across the board quite easily. Select the “Servers” tab, select Service Profile Template, select root, select Sub-organizations and then select the Service Profile Template Name. Select Service Profile Template -> select root -> select Sub-organizations -> select the Service Profile Template Name of the associated ESXi hosts that you want to update.
- Select the Policies tab and then expand Firmware Policies. From drop down menu select the newly created Host Firmware policy and click Save Changes.
- Select the General tab and press Reboot now from the Pending Activities section. You can select multiple hosts to reboot
- As part of the blade upgrade the following will be upgraded – BIOS, CIMC Controller and Board Controller. This can take between 35 and 60 minutes per blade
9. Use the FSM tab to monitor the progress of the firmware upgrade.
- Once the FSM progress shows 100% you can exit Maintenance Mode of the upgraded ESXi host.
- Verify that all firmware has been upgraded. Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> select the Installed Firmware tab. Ensure that the same version appears for all components