PowerStore: Cluster Creation fails with error Failed to create ICM network

When the mezzanine card is missing, the initial configuration wizard (ICW) can be started without errors.
However, it fails later with a generic error message on the PowerStore Manager UI.

The service script svc_diag list --icw_hardware shows no failed components.

Run the command svc_cluster_diag to identify the issue:

svc_cluster_diag
...
----scanning for current hardware failures--------

Running svc_diag list --icw_hardware from svc_cluster_diag
hw_type Warnado-EX
Running on Node A

Local Node - no Expansion Enclosures detected
Local Node - no Expansion Enclosures detected
Lab Mode Flags: unsupported_drives (False), unsupported_exp_enclosures (False), unsupported_node_hw_configs (False)
Node A FRU Status
        OK | Peer Node        | 0x0f80
        OK | Local Node       | 0x1480
        OK | Embedded Module  | 0x8b81
     Empty | 4-Port Card      | 0x0080     <------
        OK | I/O Module 0     | 0x8b81
     Empty | I/O Module 1     | 0x0080
        OK | Internal Backup Battery Module | 0x3380

Node B FRU Status
        OK | Peer Node        | 0x0f80
        OK | Local Node       | 0x1580
        OK | Embedded Module  | 0x8b81
     Empty | 4-Port Card      | 0x0080     <------
        OK | I/O Module 0     | 0x8b81
     Empty | I/O Module 1     | 0x0080
        OK | Internal Backup Battery Module | 0x3380

IO Module Consistency Check = Success
Node Consistency Check = Success
Battery Check = OK


Node A Fault Status Register Status = Success
        OK | Node            

        OK | Embedded Module 
        OK | Internal Backup Battery Module
        OK | DIMM00          
        ...
        OK | DIMM23          
        OK | I/O Module 0    
        OK | I/O Module 1    
        OK | 4-Port Card     

Node B Fault Status Register Status = Success
        OK | Node            
        OK | Embedded Module 
        OK | Internal Backup Battery Module
        OK | DIMM00          
        ...
        OK | DIMM23          
        OK | I/O Module 0    
        OK | I/O Module 1    
        OK | 4-Port Card     

The node Fault Status Register Status shows Success and no faults are reported. The 4-port card slot is reported as empty for both node A and B.

Messages that indicate the problem:

======Scanning for clustering error messages======
Mar 14 11:46:11 xxxx-A control-path[11375]: 2022-03-14 11:46:11.896 [CC] [ERROR] [com.emc.cyclone.contexts.clustering.services.ClusterConfigurationAS|vert.x-eventloop-thread-3] [xxxx|configureTemporaryNetwork_execute] Failed to configure temporary network
                                       General error in networking validation service.

...

Mar 14 11:55:05 xxxx-A control-path[11375]: 2022-03-14 11:55:05.337 [CC] [ERROR] [com.emc.cyclone.contexts.clustering.services.ClusterConfigurationAS|vert.x-eventloop-thread-3]  Clustering internal error, Reason: Platform failure: Node A: Failed to create ICM network
Mar 14 11:55:05 xxxx-A control-path[11375]: 2022-03-14 11:55:05.577 [CC] [ERROR] [com.emc.cyclone.contexts.clustering.services.JoinClusterAS|vert.x-eventloop-thread-3]  Error from Join Cluster: 
                                      Clustering internal error, Reason: Platform failure: Node A: Failed to create ICM network

And:

Mar 14 11:54:57.342515 xxxx-A cyc_config[17710]: A: # Creating ICM network
Mar 14 11:54:58.494138 xxxx-A cyc_config[17710]: A: # ICM VLAN:         '0'
Mar 14 11:54:58.494207 xxxx-A cyc_config[17710]: A: # ICM prefix:       '64'
Mar 14 11:54:58.494247 xxxx-A cyc_config[17710]: A: # ICM appliance IP: 'xxxx:xxxx:xxxx::xxx:xxxx:xxxx:xxxxx'
Mar 14 11:54:58.494277 xxxx-A cyc_config[17710]: A: # ICM node A IP:    'xxxx:xxxx:xxxx::xxx:xxxx:xxxx:xxxxx'
Mar 14 11:54:58.494302 xxxx-A cyc_config[17710]: A: # ICM node B IP:    'xxxx:xxxx:xxxx::xxx:xxxx:xxxx:xxxxx'
Mar 14 11:54:58.495112 xxxx-A cyc_config[17710]: A: Parameters: network_type=NWType.ICM, vlan=0, appliance_ip=xxxx:xxxx:xxxx::xxx:xxxx:xxxx:xxxxx, ip_a=xxxx:xxxx:xxxx::xxx:xxxx:xxxx:xxxxx, ip_b=xxxx:xxxx:xxxx::xxx:xxxx:xxxx:xxxxx, prefix=64, gateway=, debug=False
Mar 14 11:54:58.501819 xxxx-A cyc_config[17710]: A: Base net device was not found to create network (capability = ICM): list index out of range
Mar 14 11:54:58.501994 xxxx-A cyc_config[17710]: A: Could not get base net device: list index out of range
Mar 14 11:54:58.569967 xxxx-A cyc_config[17710]: A: Error: # failed to create ICM network:
Mar 14 11:54:58.586940 xxxx-A cyc_config[17710]: A: Debug: # Task Time: 365 seconds to execute 16_create_base_stack_cluster
Mar 14 11:54:58.586994 xxxx-A cyc_config[17710]: A: Error: # [16/23]: Create Base Stack Cluster -- FAILED
Mar 14 11:54:58.870177 xxxx-A cyc_config[17710]: A: Error: reinit_install: ICM network creation failed:
Mar 14 11:54:58.870243 xxxx-A cyc_config[17710]: A: Error: EC:293
Mar 14 11:54:58.908290 xxxx-A cyc_config[16976]: Error: # Configuration sync: reinit_install failed															 

---

For the PowerStore 1000, 3000, 5000, 7000 and 9000 T and X the mezzanine card is always required, even if the system is to be deployed block-optimized and the Ethernet ports are not going to be connected.

The PowerStore is unable to create the internal ICM network, if the mezzanine card is not present.
To verify the mezzanine card's presence:

[SVC:service@xxxx-A user]$ svc_diag list --hardware --sub_options inventory | grep -i mezzanine0 -A 2
FRU Device Description : SLIC1 (ID 14)
 Device not present (Timeout)

With the mezzanine card present, this is the expected output:

[SVC:service@xxxx-A user]$ svc_diag list --hardware --sub_options inventory | grep -i mezzanine0 -A 2
FRU Device Description : Mezzanine0 (ID 27)
 Chassis Type          : Rack Mount Chassis
 Chassis Part Number   :                     

---

These PowerStore systems cannot be ordered and deployed without mezzanine cards. 

Power off the PowerStore Appliance to reinsert the mezzanine card, and then factory reset the PowerStore for another initialization: PowerStore: How to reinitialize a PowerStore system to factory state using the svc_factory_reset command

Verify with the above command on both nodes that the mezzanine card is properly detected before the cluster creation.

---