Highlighted
PKPhu
1 Nickel

Question for Isilon data protect level

Jump to solution

Hi all!

I have some question for Isilon, please help me answer, thanks!

- I have 3 node, in one node have 35 drive (2TB), but when I show in system, I onnly see total 181TB, why?

- I check protect level +2d:1n => 2 drives or 1 node failure. In case, system have 3  drives failure, data destroyed? and how to recover?

- SSD failed, how is system?

0 Kudos
1 Solution

Accepted Solutions
carlilek
2 Iron

Re: Question for Isilon data protect level

Jump to solution

1. That seems well in line given the vagaries of HDD capacities.

2. If you have 3 drive failures, yes, data is destroyed. In that case, you would contact Isilon and they might be able to help you recover it. Note that it wouldn't be all data destroyed, just data that was striped (at least partially) onto those 3 drives. It's important to replace drives quickly at +2d:1n.

3. If SSD fails, the system should be just fine, particularly if you're using L3. If you're using metadata on SSD schemes, the data is either protected on HDD (metadata read) or on other SSDs. I don't recall exactly, but I suspect it's probably at 4x mirroring for metadata at +2d:1n.

0 Kudos
5 Replies
carlilek
2 Iron

Re: Question for Isilon data protect level

Jump to solution

1. That seems well in line given the vagaries of HDD capacities.

2. If you have 3 drive failures, yes, data is destroyed. In that case, you would contact Isilon and they might be able to help you recover it. Note that it wouldn't be all data destroyed, just data that was striped (at least partially) onto those 3 drives. It's important to replace drives quickly at +2d:1n.

3. If SSD fails, the system should be just fine, particularly if you're using L3. If you're using metadata on SSD schemes, the data is either protected on HDD (metadata read) or on other SSDs. I don't recall exactly, but I suspect it's probably at 4x mirroring for metadata at +2d:1n.

0 Kudos
kipcranford
2 Iron

Re: Question for Isilon data protect level

Jump to solution

Just want to add a few more details to carlilek's response.

> - I have 3 node, in one node have 35 drive (2TB), but when I show in system, I onnly see total 181TB, why?

This is mostly due to two things:  base-2 conversion and "virtual hot spare".  The base-2 conversion is simply converting from the base-10 units the drive manufactures like to use, i.e. a 2 TB drive is really 2 * (1000/1024)^4 =~ 1.82 TiB.

The OneFS "virtual hot spare" (VHS) concept (the ability for OneFS to use all available free space in the cluster during a device repair) means we set aside a (configurable) amount of space to ensure that we can always do a repair even on very full filesystems (in our UI, it's under StoragePools -> SmartPools Settings in most recent releases).  By default, we set aside enough free space to equal about twice a drive size.

In any event, an easy way to reconcile all this is to do this from the command line:

isilon#  isi status -q

and look at the first 7 or so lines.  On one of my clusters, it looks like this:

x4101-1# isi stat -q

Cluster Name: x4101

Cluster Health:     [  OK ]

Cluster Storage:  HDD                 SSD Storage

Size:             360.3T (368.0T Raw) 0 (0 Raw)

VHS Size:         7.6T

Used:             1.2T (< 1%)         0 (n/a)

Avail:            359.1T (> 99%)      0 (n/a)

This is a 3-node X410 with 34 4 TB drives.  That's really 3 * 34 * 4 * (1000/1024)^4 =~ 371 TiB raw.  There's a bit of other overhead (space on drives to store the filesystem, etc) that eats a little bit more, so that's where the "368.0T Raw" number in the above output comes from.  Subtracting out my VHS space gives about 360T, which is what most of our UI's will report.

0 Kudos
kipcranford
2 Iron

Re: Question for Isilon data protect level

Jump to solution

> I don't recall exactly, but I suspect it's probably at 4x mirroring for metadata at +2d:1n.

At +2d:1n, file metadata (inodes, extension blocks, etc) is 3x mirrored.  Directories are protected one level higher than the in-place protection, so in this example would be 4x mirrored.

0 Kudos
PKPhu
1 Nickel

Re: Question for Isilon data protect level

Jump to solution

Hi Calilek,

Thank for your support! In case, number of failed disks is over protection level, can client read/write data in Isilon system or not and Isilon system is blocked all?

Best Regards,

0 Kudos
AdamFox
2 Iron

Re: Question for Isilon data protect level

Jump to solution

The cluster is read/write as long as there is quorum.  Quorum is defined at a majority of the nodes in the cluster are available.  So if you have a 7 node cluster you can still write data to the cluster as long as at least 4 nodes are available.  If there is not quorum, the cluster will be read-only.  This is to prevent any kind of "split-brain" scenario.

However, if you have more failed elements than your protection allows, there may be individual files that are not available because they rely on the missing elements (disks and/or nodes).  Files that are not reliant on the failed elements will still be available r/w, assuming there is free space in the cluster for writes, of course.

As the FlexProtect job runs, it will log LIN #s that could not be re-protected due to an excess of failed elements.  You can find the path to those LINs by running # isi get -L <LIN#>.  You could then use that path to restore from backup or DR copies if needed.

This is not a common scenario so if you ever find yourself in this rare situation, always work with support before proceeding as they may be able to figure out why so many elements have failed and work around it.

0 Kudos