ECS: Erasure Coding rebuild with node outage on a four node ECS cluster

In ECS versions prior to 3.4, if only three healthy nodes remained, the ECS initiated restoring customer data to three mirrored copies, one per node. This design decision was made to maximize data protection for customer data. The process is also known as EC (Erasure Coding) retiring.

EC is an algorithm that reduces storage space while protecting data against disk or node failures.

When data is erasure coded, the physical space required on an ECS is approximately 1.33x for storing customer data (12 data segments + 4 parity segments).  However, if the ECS begins EC retiring, the physical space required for customer data increases from 1.33x to 3x.

This may lead to the ECS reaching the 90% capacity utilization threshold, causing the ECS to go into read-only mode and data unavailability. 

To enhance data protection and avoid exceeding the 90% capacity threshold, the default behavior for ECS clusters with only three healthy nodes was changed.

In 3.4, the design change was made such that ECS will no longer implement EC Retiring automatically when only three nodes are healthy/online. The system runs in a degraded state and may encounter performance issues but is likely to avoid a DU. New writes continue to be written as three mirrored copies and will be erasure coded once there are 4+ nodes online and available to write to.

Any additional drive failures may cause isolated DUs, it may also slightly increase exposure to a potential Data Loss (DL), however it is still unlikely.

Also, consider expanding the ECS to five or more nodes. It decreases ECS exposure to performance degradation, DU, and DL situations during node failure. For more details on ECS architecture, see the ECS Admin Guide.