This post is more than 5 years old
41 Posts
0
3110
Question for Isilon data protect level
Hi all!
I have some question for Isilon, please help me answer, thanks!
- I have 3 node, in one node have 35 drive (2TB), but when I show in system, I onnly see total 181TB, why?
- I check protect level +2d:1n => 2 drives or 1 node failure. In case, system have 3 drives failure, data destroyed? and how to recover?
- SSD failed, how is system?
carlilek
205 Posts
0
July 28th, 2016 07:00
1. That seems well in line given the vagaries of HDD capacities.
2. If you have 3 drive failures, yes, data is destroyed. In that case, you would contact Isilon and they might be able to help you recover it. Note that it wouldn't be all data destroyed, just data that was striped (at least partially) onto those 3 drives. It's important to replace drives quickly at +2d:1n.
3. If SSD fails, the system should be just fine, particularly if you're using L3. If you're using metadata on SSD schemes, the data is either protected on HDD (metadata read) or on other SSDs. I don't recall exactly, but I suspect it's probably at 4x mirroring for metadata at +2d:1n.
kipcranford
125 Posts
0
July 28th, 2016 08:00
> I don't recall exactly, but I suspect it's probably at 4x mirroring for metadata at +2d:1n.
At +2d:1n, file metadata (inodes, extension blocks, etc) is 3x mirrored. Directories are protected one level higher than the in-place protection, so in this example would be 4x mirrored.
kipcranford
125 Posts
0
July 28th, 2016 08:00
Just want to add a few more details to carlilek's response.
> - I have 3 node, in one node have 35 drive (2TB), but when I show in system, I onnly see total 181TB, why?
This is mostly due to two things: base-2 conversion and "virtual hot spare". The base-2 conversion is simply converting from the base-10 units the drive manufactures like to use, i.e. a 2 TB drive is really 2 * (1000/1024)^4 =~ 1.82 TiB.
The OneFS "virtual hot spare" (VHS) concept (the ability for OneFS to use all available free space in the cluster during a device repair) means we set aside a (configurable) amount of space to ensure that we can always do a repair even on very full filesystems (in our UI, it's under StoragePools -> SmartPools Settings in most recent releases). By default, we set aside enough free space to equal about twice a drive size.
In any event, an easy way to reconcile all this is to do this from the command line:
isilon# isi status -q
and look at the first 7 or so lines. On one of my clusters, it looks like this:
x4101-1# isi stat -q
Cluster Name: x4101
Cluster Health: [ OK ]
Cluster Storage: HDD SSD Storage
Size: 360.3T (368.0T Raw) 0 (0 Raw)
VHS Size: 7.6T
Used: 1.2T (< 1%) 0 (n/a)
Avail: 359.1T (> 99%) 0 (n/a)
This is a 3-node X410 with 34 4 TB drives. That's really 3 * 34 * 4 * (1000/1024)^4 =~ 371 TiB raw. There's a bit of other overhead (space on drives to store the filesystem, etc) that eats a little bit more, so that's where the "368.0T Raw" number in the above output comes from. Subtracting out my VHS space gives about 360T, which is what most of our UI's will report.
PKPhu
41 Posts
0
August 2nd, 2016 00:00
Hi Calilek,
Thank for your support! In case, number of failed disks is over protection level, can client read/write data in Isilon system or not and Isilon system is blocked all?
Best Regards,
AdamFox
254 Posts
0
August 2nd, 2016 07:00
The cluster is read/write as long as there is quorum. Quorum is defined at a majority of the nodes in the cluster are available. So if you have a 7 node cluster you can still write data to the cluster as long as at least 4 nodes are available. If there is not quorum, the cluster will be read-only. This is to prevent any kind of "split-brain" scenario.
However, if you have more failed elements than your protection allows, there may be individual files that are not available because they rely on the missing elements (disks and/or nodes). Files that are not reliant on the failed elements will still be available r/w, assuming there is free space in the cluster for writes, of course.
As the FlexProtect job runs, it will log LIN #s that could not be re-protected due to an excess of failed elements. You can find the path to those LINs by running # isi get -L . You could then use that path to restore from backup or DR copies if needed.
This is not a common scenario so if you ever find yourself in this rare situation, always work with support before proceeding as they may be able to figure out why so many elements have failed and work around it.