One thing to add is that quorum is also a component within node pools.  If you have multiple pools, you need at least 3 nodes to make the pool, and then need to maintain the quorum within the pool to keep it operational. 

So if you have 10 nodes - a 7 node pool and 3 node pool, and you shutdown 2 of the nodes in the 3-node pool, that pool will no longer have quorum, so the remaining node will go into read-only mode, while the 7-node pool remains fully operational.

Quorum is defined as more than 50% of the node are online and available to read and write data. If you have a 6 node cluster, quorum requires at least 4 nodes to be online. If you have a 7 node cluster, quorum also requires at least 4 nodes to be online. The other nodes may be in read only mode and are considered to be split from the cluster. This means that one can only write data to the cluster quorum, instead of writing different data to the different parts of the split cluster, resulting in cluster inconsistency.

When you encounter cluster splits, below are some solutions to resolve a node split:

1. The majority of the time, you can reboot the cluster resolves a node split.

2. Reconnect the cables of the node that has stopped communicating with the cluster.

3. If new nodes have recently been added or removed, confirming they are connected correctly.

4. If there is a hardware error, you may be able to replace the components which failed.

Thank you all ☺

At Jeffrey,

If there are less than 4 nodes online in a cluster that is known to have 6 nodes, will the cluster be in ‘read only’ mode or ‘write only’ mode, because in the previous response, Adam reckons the nodes will be in ‘read-only’ mode as opposed to your response of ‘write only’. Please clarify.

Br,

Gaurav

Perhaps the wording was a bit confusing.  In the event of loss of quorum - a cluster or node pool will go immediately to Read-Only mode, no writing activity will resume until quorum is restored in an effort to prevent conflicts in the stored data. 

Think again of this 7 node cluster.  Split it in half with a bad IB cable, so you have 3 nodes working together on one side, and 4 nodes working together on the other.  Ostensibly these are the same cluster, same data sets, and the nodes can still take incoming connections from the external network interfaces.  In order to protect data, only the 4 nodes working together will still be able to write data, while the spit off 3 nodes will go to a read only state. 

After fixing the IB connection to restore the 7 node communication - the 3 nodes that had split will be updated on the state of data and will resume read/write activity. 

Thank you Chris. My concepts are now clear ☺

Br,

Gaurav Parekh

Hi gaurav,

Back to your question. If you have a 6 node cluster, then the cluster encounter the network issue. you have 3 nodes working together on one side, other side also have 3 nodes. This means two sides will be in "read-only" mode.

I always wondered how to break an IB switch into two pieces, both still functioning...

Ok, with two redundant switches, and removing some cables

one might get into such situation.

But actually, it could get worse, e.g. for a 6 node cluster:

switch 1: nodes 1,2,3,4 online, quorum ok

switch 2: nodes 3,4,5,6 online, quorum ok

How does the cluster behave now?

-- Peter

Hi Peter,

We give these examples for explaining what's Quorum in Isilon, There are several things that can cause cluster split:

• cabling issues
• software bug
• configuration changes
• Hardware errors

In general, when more than one node are splitting from cluster, it will work as individual "read-only" node. So these examples are just explaining the conception of Quorum.

Back to your question, we can configure an internal switch as a failover network to provide redundancy, but it doesn't increase performance.  Backing to your example, if switch 1 is primary, node 1,2,3 and 4 work well in internal-a network, then node 5 and 6 suddenly lost their connection with internal-a network, then node 1,2,3 and 4 will use internal-b network to communicate with node 5 and 6. Meanwhile, node 1, 2,3 and 4 use internal 1 to communication. Node 1,2,3,4,5 and 6 have quorum.

Will the IB failover only occur if the primary is completely down, or when the secondary

has "better" connectivity than the primary?

Like nodes 1,2,3,4 (quorum) connected on primary,

but 2,3,4,5,6 (quorum, but at higher count) on secondary?