Simulation of possible failure scenarios

Hi all,

sometimes we need to show that all EMC Isilon properties which EMC offers and we of course 'sell' are true.

For example: if we sell an Isilon cluster with the property that a protection scheme of +2n withstands the failure of either two hard drive disks or two complete nodes without data loss - than we have to prove that statement. Most of the customers believe that 'as it is' and don't want us to show them. But there are always customers who want to see such features according to compliance tests.

Most failure scenarios can easily be shown or reproduced as for example

• link aggregation >> disconnecting network cables
• dual power supply >> disconnecting power cords
• Infiniband redundancy >> disconnecting Infiniband cables or power off one IB switch
• SmartConnect features >> doing some mounts from one client to the same SmartConnect zone; client distribution over Nodes
• etc.

But however - some scenarios are hard to simulate.

How could we simulate a HDD failure? EMC advised against simple pulling HDDs out of the chassis and putting it later back. I think at least we have to wait the FlexProtect job finished and format the disk before putting it back to cluster.

The same for a Node failure. Is just disconnecting the IB cables enough? Or maybe shutting the Node down and shown the customer that all data is still there and accessible with one Node off.

Any ideas or experiences with this kind of compliance tests?

We don't want real data loss caused by a strange compliance test. In that case the customer has to believe what EMC is claiming.

By

Phil