PowerScale: Isilon: Isilon On-Cluster Analysis: Network Pools - Parallel Upgrade Support

Network Pools - Parallel Upgrade Support          FAIL

CRITICAL: A parallel upgrade runs the risk of making one or more external networks temporarily unavailable. Affected pools: groupnet0.subnet0.pool1, groupnet0.subnet1.pool1, groupnet0.subnet1.pool2

• This is by design as the cluster has disjoint external networks where there is a possible risk that all nodes in the impacted network pools reboot simultaneously.
• In which case, clients cannot access the cluster through the SmartConnect Zone Name (SCZN) associated with that network pool. 

Important definitions:
 

• Diskpool:  Collection of disks distributed among a subset of cluster nodes. 
• Protection Levels/Policies: All files on the /ifs file system are assigned a protection level. OneFS stripes or mirrors the content of files across nodes and drives of a single diskpool. A file with a protection level of 2d+1n guarantees that the file will still be available if any two drives, or one node in the enclosing diskpool goes down.
• Neighborhoods:  Sets of nodes that wholly contain one or more diskpools. A subset of nodes assigned to a diskpool may overlap with a subset of nodes that are assigned to another neighborhood. There is no overlapping of diskpools between neighborhoods. 
• Diskpools DB API and reservations The Diskpools database API has a reservation feature that allows a client application such as the Upgrade Framework to remove a set of nodes (ie reboot) or drives from the cluster without breaching protection on any diskpool.
• Disjoint external networks: A cluster has disjoint external networks if there a set of nodes with external interfaces that do not overlap across all diskpools

• A parallel upgrade installs the new operating system on all nodes and then reboots a subset of nodes simultaneously, up to one node in each diskpool.
• Each node attempts to make a reservation for their turn to reboot until all nodes are upgraded.  Node subsets and reservations are based on diskpool and node availability.
• During a parallel upgrade, node subsets that are not being rebooted remain online and can continue serving clients.

• The cluster has disjoint external networks (KB example: groupnet0.subnet0.pool1, groupnet0.subnet1.pool1, groupnet0.subnet1.pool2)
• During the parallel upgrade, there is a possible DU risk on those network pools as all nodes in the affected network pools could reboot at a same time (i.e, they could all take a diskpools reservation to reboot simultaneously). 
• If that occurs, access to the affected network pool's SCZN will not be possible. 

• As every cluster has a different network configuration, the solution is not the same for all clusters.
• To avoid having a possible DU on the disjoint network pool during the parallel upgrade, a network configuration change on the cluster and the external environment will be needed which usually requires several approvals on the customer's end.
• This KB article will not include a direct resolution to the IOCA check failure due to the above points.
• However, the general guideline is that at least two nodes’ external member interfaces in the affected network pools exist on the same neighborhood/failure domain to avoid a disjoint network/DU risk to that SmartConnect network pool (i.e, add nodes from the same neighborhood/failure domain to the affected network pool). 
• The KB article provides commands on how to determine the cluster's neighborhood/failure domains and the member nodes in affected network pools.

• Use the below command determine the member nodes in affected network pools (Note: Replace the "<network-pool 1 ID> | <network-pool 2 ID> | <network-pool 3 ID>" with the affected network pool IDs from the IOCA check details and separate between them using the '|' character). 

# isi network pools list -v | egrep -i 'ID: |ifaces' | egrep -A1 '<network-pool 1 ID> | <network-pool 2 ID> | <network-pool 3 ID>'

• Run the below command to determine the cluster's neighborhoods (Note: Replace the <IOCA script location> and <New OneFS version> with their correct values) 

# perl <IOCA script location> -o <New OneFS version>,parallel  -e -r "checkNetworkParallelUpgrade"  

• Cross reference the outputs of the above commands to determine which changes needs to be done to the cluster's network pools.
• A quick fix would be to add all clusters nodes to the impacted network pools but again, that is usually not easy as the external network environment might not allow this change.
• After the needed changes are applied, rerun the IOCA check again.
• If unable to determine the needed changes, open a case with Support for further assistance.
• Example below to determine the needed changes in a test scenario:

1) Determine member nodes in affected network pools:
# isi network pools list -v | grep -i 'ID: |ifaces' | egrep -A1 'groupnet0.subnet0.pool1| groupnet0.subnet1.pool1| groupnet0.subnet1.pool2'
                     ID: groupnet0.subnet0.pool1
                 Ifaces: 1:10gige-agg-1, 2:10gige-agg-1, 4:10gige-agg-1, 3:10gige-agg-1
                     ID: groupnet0.subnet1.pool1
                 Ifaces: 37:10gige-agg-1, 38:10gige-agg-1, 39:10gige-agg-1, 40:10gige-agg-1
                     ID: groupnet0.subnet1.pool2
                 Ifaces: 37:10gige-agg-1, 38:10gige-agg-1, 39:10gige-agg-1, 40:10gige-agg-1

2) Run the IOCA check to determine cluster neighborhoods:
# perl IOCA -o 9.2.1.5,parallel -e -r "checkNetworkParallelUpgrade"
Isilon On-Cluster Analysis                        0.1395
Cluster Name                                      TestCluster
Cluster GUID                                      0050569b6db2ad086861001a2f1dd1d02473
Node Count                                        52
Current OneFS Version                             8.2.2.0
Destination OneFS Version                         9.2.1.5
Destination OneFS Version                         WARN
  WARN: There is a newer patch release available for OneFS 9.2.1: 9.2.1.9
Network Pools - Parallel Upgrade Support          FAIL
  CRITICAL: A parallel upgrade runs the risk of making one or more external networks temporarily unavailable. Affected pools: groupnet0.subnet0.pool1, groupnet0.subnet1.pool1, groupnet0.subnet1.pool2
  
  ==============================
  Node Neighborhoods
  ==============================
  1: [ 1, 7, 10, 16, 19, 24, 27, 31, 35, 40, 43, 45, 47 ]
  2: [ 2, 6, 9, 15, 18, 23, 26, 29, 33, 38, 41, 46, 48 ]
  3: [ 3, 5, 12, 14, 17, 22, 25, 30, 34, 37, 42, 50, 51 ]
  4: [ 4, 8, 11, 13, 20, 21, 28, 32, 36, 39, 44, 49, 52 ]

3) The possible resolution in this case would be to :

a) A quick fix would be to add all clusters nodes to the impacted network pools groupnet0.subnet0.pool1 & groupnet0.subnet1.pool1, groupnet0.subnet1.pool2
b) Add more nodes to affected network pools ( suggested nodes are based on neighborhood command output ) :

- Possible resolution to groupnet0.subnet0.pool1 : at least add node 7  to the network pool as node 7 exists in the same neighborhood as nodes 1
- Possible resolution to groupnet0.subnet1.pool1 : at least add node 34 to the network pool as node 34 exists in the same neighborhood as nodes 37
- Possible resolution to groupnet0.subnet1.pool2 : at least add node 33 to the network pool as node 33 exists in the same neighborhood as nodes 38

4) After applying the network changes, re-run the IOCA check to confirm that there are no issues:
# perl IOCA -o 9.2.1.5,parallel -e -r "checkNetworkParallelUpgrade"