PowerFlex 4.X: Add a standby MDM Error: MDM failed command Communication error

This procedure only applies to a PowerFlex 4.x system already managed and connected to a 4.x PowerFlex Management Platform.

This also assumes the MDM RPM has already been installed on a new node and is waiting to be pulled into the MDM cluster.

If adding the standby MDM to the current MDM cluster results in the following error, use this KB:

Error: MDM failed command.  Status: Communication error

Steps

1. Log in to the PowerFlex 4.x system to obtain the "System ID."

Example:

# scli --login --management_system_ip <INGRESS_IP> --username admin --password <PASSWORD>
Successfully generated login certificate at /root/.scli/cli_cert.p12
Logged in. User role is SuperUser. System ID is xxxxxxxxxxxxxxxx

2. Connect to one of the PowerFlex Manager Platform (PFMP) servers through SSH.
3. Create the following YAML file. Replace the "commonName" value with your own MDM <System ID> value obtained in step 1.

vi mdm_cert_req.yaml

Example:

apiVersion: cert-manager.io/v1 
kind: Certificate 
metadata: 
  name: mdm-cert-req 
spec: 
  secretName: mdm-cert-secret 
  duration: 43830h 
  renewBefore: 360h 
  subject: 
    organizations: 
      - dellemc 
    organizationalUnits: 
      - superuser 
  commonName: MDM-xxxxxxxxxxxxxx 
  isCA: false 
  usages: 
    - server auth 
    - client auth 
    - digital signature 
    - key encipherment 
  dnsNames: 
  - mdm 
  issuerRef: 
    kind: ClusterIssuer 
    name: crtmgr-vault-cluster-issuer

4. Apply the YAML file.

kubectl apply -f mdm_cert_req.yaml -n powerflex

5. Pull the PFMP CA private and public certificates. 

kubectl get secret mdm-cert-secret -n powerflex -o jsonpath='{.data.tls\.key}' | base64 --decode > mdm_certificate.pem 
kubectl get secret mdm-cert-secret -n powerflex -o jsonpath='{.data.tls\.crt}' | base64 --decode >> mdm_certificate.pem 
kubectl get secret mdm-cert-secret -n powerflex -o jsonpath='{.data.ca\.crt}' | base64 --decode > mgmt_ca.pem

6. Copy the two new certificates from step 5 to the new MDM node. Put the certificates in the /opt/emc/scaleio/mdm/cfg/ directory 

7. The mdm_certificate.pem should be renamed to mdm_mno_certificate.pem after step 7 is complete. Use SCLI on the primary MDM to add this new MDM node as a standby. If a different syntax is needed, see the documentation.

scli --add_standby_mdm --new_mdm_ip <MDM_IPs> --mdm_role manager --new_mdm_management_ip <MGMT_IP> --new_mdm_name <NAME>