How to forcefully remove a sds or sds device?

1. Try clearing the device error 1st using the following command:
scli --clear_sds_device_error --sds_ip [SDS_IP] --clear_all

2. Run the remove_sds_device command again (with/without force flag)

What is the exact command you are running and what is the exact error message?

Yes, the error-ed drive is in a storage pool. There are currently mapped volumes in this storage pool, so it has many other sds devices in the pool.

No luck, I tried clearing the error in the GUI as well. It just goes back to an error state right away and I cannot remove the drive still. Maybe the only solution is to wipe everything and start fresh

If this was added to a storage pool, was any volume created from this Storage pool that was not deleted or any snapshot created that was not deleted and you are trying to delete the last disk from the Storage pool?

Yes it is.

Is this the last device in the Storage pool that you want to remove?

I am trying to remove a error-ed sds device. There's no error after issuing the removal command, it's just stuck at "Remove Pending" state. The drive has already been physically removed from the server and unmapped in VMware.

Note: If the capacity of this SDS is still used by volumes, and cannot be replaced, the command will fail.

Allen,

Before trying the command Tomer has advised, did you run abort_remove_sds_device?

Yes, I did. After aborting sds, and clearing the error, it still becomes an error-ed device. I tried adding another device in order to removed the error-ed one but it's saying it needs at least one sds device?

Few more Qs:

1. How many SDSs did you have in your setup? What was the capacity of each SDS?

2. Is that the sequence of operation that you did that got U into this state?

a. Perform remove SDS device (from which interface? CLI/GUI/vSphere plugin) - BTW, was that the last SDS device at that time? if that it the case and there were volumes on that SDS it should not have allowed to remove the last SDS device.

b. U did not wait for the rebalance to complete and removed the hdd? got a device error alert

Can you please send us the ShowEvents printout from both MDMs?

go to /opt/emc/scaleio/mdm/bin and run the ./showevents.py script (for both MDMs)

Can you send a picture of the GUI dashboard - how much spare capacity do you have? was the spare configured properly?

Thanks,

Tomer

1. 6 SDS's. There's 4 servers with 10TB and 2 servers with 16TB. 2. Yes, I issued the remove command in the scli to remove 6 drives, 2 from 3 SDS’s. The drives were probably removed in an hr or 2 after issuing the command. Now I wish to remove the 3rd SDS entirely. So in vcenter, I removed all the Mapped Raw LUN’s to this SVM. 2a. From the CLI. I realize now that it would've been safer to remove in vSphere plugin. It was not the last SDS device at the time. 2b. That's correct; I issued the command and didn't watch the rebalance to see that everything was completed. I thought the removal command would be quick and hoped the data would be able to recover from the parity bits from the other SDS's. It looks like there was around 5.1GB of data that hadn't been rebalanced yet before the drive had been physically removed. 3. Sure, please see attachment.

How many PD (protection domains) + SP (Storage pools) do you have?

Can you please supply also the following:

1. Screenshots of the GUI BackEnd view + Alerts view?

2. MDM (both) + relevant SDS logs -> you can run the getInfo under /opt/emc/scaleio/mdm/diag (same for SDS), it will collect all logs on the host on which you run it.

If you have a SIO-GW installed, you can also login to the IM-Web and from the Maintain view run the GetInfo which will collect all logs from all your SIO system nodes.

Did you try to perform SDS / MDM restart since U got into this state and check if it clears our the problem?
pkill sds / mdm

or

under /opt/emc/scaleio/sds/bin/ -> delete_service -> create_service

same for MDM (note: this will trigger a switch-over)

There's 1 pd and 3 storage pools.

Please check your email for the screenshots and logs.

I did try restarting the SVM for the 3rd SDS, no luck.

Thanks!