ScaleIO Node failure scenario question

HI Gary,

Whether in a 3-node setup or a larger setup, the answer is going to (most likely) be the same. In your 3-node setup, what you are describing is a double failure. If the rebuild from the first node failure has not completed by the time the HDD in node 2 fails, then you would likely be looking at a data unavailable scenario. Meaning, there were blocks of data that only existed on that HDD from node 2 that had yet been rebuilt fully before it failed. I say likely as the disks in this case would all be participating in the rebuild from the original failure (node 1) and they will all be sending and receiving blocks of data equally to keep things balanced. If that rebuild has not finished, then you will have some primary data blocks unavailable to be sent from that HDD.

If the rebuild finished from node 1 and then the HDD from node 2 fails, if you don't have enough spare capacity then you will have an issue with not being able to complete the rebuild. But there will still be at least 1 copy of those data blocks from the failed disk residing elsewhere and clients will still be able to access their data.

It all depends on timing and when the failures occurred relative to the rebuild.

Hope that helps,

Rick