Potential issues with concurrent firmware upgrade with Dell Compellent Storage
Preventive Service Planning
Issues have been seen with Dell Compellent Storage Virtualized behind SAN Volume Controller, Storwize V Series and FlashSystem V Series Controllers which may lead to loss of access during a controller firmware upgrade
When upgrading the firmware on a Dell Compellent Storage Controller which is Virtualized behind one of the products listed below an issue may occur whereby the mdisks can potentially go offline when the second Dell Compellent controller goes offline at the start of its upgrade.
This issue may occur on the following products when virtualizing Dell Compellent Storage and when performing a concurrent firmware upgrade.
SAN Volume Controller
In order to avoid the potential of this issue occurring or if previous issues have been seen the following upgrade process can be followed in order to allow the SAN Volume Controller code to re-balance the first controller prior to the second controller starting its upgrade.
This will slow down the upgrade process allowing mdisk smoothing to complete and avoid offline mdisks
1.) Verify ALL Storage Arrays are optimal with no degraded paths.
2.) Choose ONLY 1 Dell Compellent Storage array .
3.) For the Compellent Storage Array chosen in step 2 for the upgrade, install the Maintenance ISO on the first controller, which will fail the paths over to the other controller so the storage is online overall with only one controller offline for upgrade.
4.) Verify SVC error log has Message : 981022 : Managed Disk offline imminent, offline prevention started
5.) Wait for 15 minutes from the time of the previous Message
6.) Upgrade the second controller only after waiting 15 minutes. Mdisk smoothing should now have completed and the update of the second controller will now fail the paths to the first controller while this second controller is offline for upgrade.
7.) Verify SVC error log has message : 981022 : Managed Disk offline imminent, offline prevention started
8.) Wait for 15 minutes from the time of the previous Message.
9.) Rebalance the Dell Compellent Storage Array chosen in step 2.
10.) Verify SVC error log has message : 981022 : Managed Disk offline imminent, offline prevention started
11.) Wait for 15 minutes form previous message.
12.) Repeat at Step 1, for any additional Arrays.
Note : If any Mdisk or Controller path is seen "degraded" in the above steps, the following two steps can be used to address this before moving on.
E1.) Run include mdisk for any degraded mdisk
E2.) Run detectmdisk (not more then 1 time per 5 mins)
If the above 2 steps do not remove the degraded states, please stop and report the status
to IBM support.
DO NOT proceed with command if any controller
or mdisk is marked degraded until that is resolved.
This will prevent an overlap of time when disk smoothing is going on and should prevent further outages based on this condition in the future.