ScaleIO Cluster went down and after the MDM's came back both are in Secondary state How do I fix this

Have you tried running the command on the MDM to swap between primary and secondary?  I was wondering if your just having a display issue and by issuing the command it might correct that.  Page 324 of the user guide.

scli --switch_mdm_ownership (--new_master_mdm_id |

--new_master_mdm_ip [--new_master_mdm_port

]| --new_master_mdm_name )

You may try to restart the MDM service on both the MDMs one by one.

Stop service:

/opt/emc/scaleio/mdm/bin/delete_service.sh

Start service:

/opt/emc/scaleio/mdm/bin/create_service.sh

Thereafter again check the output of "scli --query_cluster"

Can you please post the output of "scli --query_cluster" here. You may mask the IP addresses in the output before posting them here.

Can't use the command since both are secondary

This is the error I get after service restart :

scli --query_cluster --use_nonsecure_communication

Error: MDM failed command.  Status: The command used for Tie-Breaker is not allowed

I guess now all the three are showing as Tie Breaker

Is the the same error when you run "scli --query_cluster" on all the 3 MDMs?

Since both MDM's are in secondary state you can't issue scli commands

I will suggest to collect get-info from all the MDMs and open a Service Request with Tech Support and we can investigate.

Sanjeev thanks for your help .Keep up the good work.

Seems like it would be more helpful to the community if the answer was posted vs an SR number.