Moderator
Moderator

n+2:1 Protection Parity Calc

Jump to solution

HI there ...

I have read below threads but still not sure about n+2:1 ...

so please advise .

With 3 nodes system .... this means Data ( or File ) will be spread over 3X2=6  disks with 3 stripes ?

if so ...
what is the physical space used for below files ....

160KB/513KB/1025KB

I have done below calc .. but not sure ....

Labels (1)
0 Kudos
1 Solution

Accepted Solutions
Highlighted
8 Krypton

Re: n+2:1 Protection Parity Calc

Jump to solution

> With 3 nodes system .... this means Data ( or File ) will be spread over 3X2=6  disks with 3 stripes ?

That depends on the size of the file.  As of the OneFS 7.0 release, OneFS uses the concept of diskpools to create "failure domains", and files are confined to these diskpools when written.  A diskpool is at most 6 drives per node.  So, for example, an X400 node with 36 HDD will have those drives carved up into 6 diskpools.  Each diskpool then spans up to 40 nodes before additional diskpools are created.

So in your 3 node cluster (let's imagine it's the 36 drive X400 in my example), you have 6 diskpools, each with 18 drives (6 drives/node * 3 nodes).  When you write file #1, it goes to diskpool 1.  File #2 goes to diskpool 2, etc.  So each file will be striped within that diskpool's set of 18 drives, and so each file could use up to 18 drives.

If you write a really big file (> 32 MiB), you will see that this file is striped across many of the drives in its diskpool, and possibly all of them if it's big enough.  That's because OneFS uses "drive rotation" to avoid using the same set of drives for all blocks of large file.  Exactly when drive rotation happens depends on various things (including certain filesystem tunables like 'streaming'), but "at around 32 MiB" is close enough for this discussion.

If you write smaller files (in this example that would be files >384 KiB but less than about 32 MiB), OneFS will use 2 drives per node, or 6 drives total, to store the file.  The number of stripes the file has depends on the file's size: in the 3 node example, a 512 KiB file will have 1 stripe; a 1024 KiB file will have 2 stripes, etc.

If you write files that are smaller than 384 KiB (in this example), you will use less than 6 drives to store that file.  In the 3 node, +2:1 example, the smallest number of drives you will ever use is 3, one drive from each node needed to store at minimum a single block each, from, say, a 1 byte file.

> if so ...
> what is the physical space used for below files ....

>

> 160KB/513KB/1025KB

The actual logical space a file consumes depends on several things, e.g. raw + protection + inodes + other metadata like extension blocks, etc.  The exact logical space consumed is tricky to predict from first principles, since some of the metadata, like extension blocks, are only created at certain file sizes *and* that size can depend on the OneFS release and what features you're using in that release (the time when extension blocks are created depends on the "fullness" of the inode, which depends on the version of the inode we're using, etc).

You can, however, simply take a theoretical approach to calculating utilization and get pretty close.  All you need is the cluster size, the file's raw size, and the OneFS protection.  Let's go through your example file sizes:

- 160 KiB, 3 nodes, +2:1

160 KiB is 20 8 KiB blocks.  16 of those 20 blocks fill a one stripe unit, a full 128 KiB.  That goes on 1 drive within 1 diskpool on node 1 (for example).  That leaves 4 8 KiB blocks.  These partially fill a second stripe unit of 32 KiB, and are placed on 1 drive on node 2.  OneFS then calculates FEC using those two stripe units.  The FEC calculation creates two additional stripe units, each 128 KiB in size, and places them "elsewhere" on the cluster (one of them will be on node 3, then other will be on either node 1 or node 2 on a different drive than the stripe units containing the file data).  From a utilization standpoint, you have this:

1 * 128 KiB data stripe unit (DSU) + 1 * 32 KiB DSU + 2 * 128 KiB FEC stripe units (FSU) = 416 KiB

With +2:1 protection on a file, you also have 3 inodes, which is 3 * 512 B = 1.5 KiB.  So, roughly, you have 416 KiB + 1.5 KiB = 417.5 KiB for your 160 KiB file.

- 513 KiB (this file uses 2 stripes given the example configuration)

That's 65 8 KiB blocks (64 fully allocated blocks and 1 block with 1 KiB in it, though it's still fully 8 KiB allocated in OneFS):

stripe 1: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

stripe 2: 1 * 8 KiB DSU + 2 * 8 KiB FSU = 24 KiB

which is

768 KiB + 24 KiB + 3 * 512 B = 793.5 KiB

(if you create this file on a real cluster, then run du, you might get 818 KiB for utilization.  If so, OneFS created extension blocks, which would be, in this example, 3 * 8 KiB.  So, 793.5 + 24 = 817.5 KiB =~ 818 KiB).

- 1025 KiB (this file uses 3 stripes)

That's 129 8 KiB blocks:

- stripe 1: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

- stripe 2: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

- stripe 3: 1 * 8 KiB DSU + 2 * 8 KiB FSU = 24 KiB

which is

768 KiB  + 768 KiB + 24 KiB + 3 * 512 B = 1561.5 KiB


and if you assume the extension blocks will be created here too, that's 1561.5 + 24 = 1585.5 KiB.

4 Replies
Moderator
Moderator

Re: n+2:1 Protection Parity Calc

Jump to solution

sorry ... this is my calc ..

 

160KB =>128+32+64=244  513KB->384+126+256=766KB &3X3=9KB    1025KB =768+256+512=1536 &1X3 = 3 KB

0 Kudos
8 Krypton

Re: n+2:1 Protection Parity Calc

Jump to solution

I gave up tracking disk usage down to single kilobytes after plotting some result from a real cluster...

It was a very "interesting" chart and I would recommend to the experiment by yourself (I haven't kept it).

Practical rule of thumb (for a 3 node cluster):

files up to 128 KB use 3x,

files of 512 KB or multiples of 512 KB use 2+1 which means factor 1.5x -- maximum efficiency for 3 nodes

files between 128 and 512 KB gradually gain efficiency from 3x to 1.5x

files slightly larger than 512 KB loose a bit of efficiency again

files much larger than 512 KB essentially use 2+1 or 1.5x

Again, do some tests and plot the data!

hth

-- Peter

Highlighted
8 Krypton

Re: n+2:1 Protection Parity Calc

Jump to solution

> With 3 nodes system .... this means Data ( or File ) will be spread over 3X2=6  disks with 3 stripes ?

That depends on the size of the file.  As of the OneFS 7.0 release, OneFS uses the concept of diskpools to create "failure domains", and files are confined to these diskpools when written.  A diskpool is at most 6 drives per node.  So, for example, an X400 node with 36 HDD will have those drives carved up into 6 diskpools.  Each diskpool then spans up to 40 nodes before additional diskpools are created.

So in your 3 node cluster (let's imagine it's the 36 drive X400 in my example), you have 6 diskpools, each with 18 drives (6 drives/node * 3 nodes).  When you write file #1, it goes to diskpool 1.  File #2 goes to diskpool 2, etc.  So each file will be striped within that diskpool's set of 18 drives, and so each file could use up to 18 drives.

If you write a really big file (> 32 MiB), you will see that this file is striped across many of the drives in its diskpool, and possibly all of them if it's big enough.  That's because OneFS uses "drive rotation" to avoid using the same set of drives for all blocks of large file.  Exactly when drive rotation happens depends on various things (including certain filesystem tunables like 'streaming'), but "at around 32 MiB" is close enough for this discussion.

If you write smaller files (in this example that would be files >384 KiB but less than about 32 MiB), OneFS will use 2 drives per node, or 6 drives total, to store the file.  The number of stripes the file has depends on the file's size: in the 3 node example, a 512 KiB file will have 1 stripe; a 1024 KiB file will have 2 stripes, etc.

If you write files that are smaller than 384 KiB (in this example), you will use less than 6 drives to store that file.  In the 3 node, +2:1 example, the smallest number of drives you will ever use is 3, one drive from each node needed to store at minimum a single block each, from, say, a 1 byte file.

> if so ...
> what is the physical space used for below files ....

>

> 160KB/513KB/1025KB

The actual logical space a file consumes depends on several things, e.g. raw + protection + inodes + other metadata like extension blocks, etc.  The exact logical space consumed is tricky to predict from first principles, since some of the metadata, like extension blocks, are only created at certain file sizes *and* that size can depend on the OneFS release and what features you're using in that release (the time when extension blocks are created depends on the "fullness" of the inode, which depends on the version of the inode we're using, etc).

You can, however, simply take a theoretical approach to calculating utilization and get pretty close.  All you need is the cluster size, the file's raw size, and the OneFS protection.  Let's go through your example file sizes:

- 160 KiB, 3 nodes, +2:1

160 KiB is 20 8 KiB blocks.  16 of those 20 blocks fill a one stripe unit, a full 128 KiB.  That goes on 1 drive within 1 diskpool on node 1 (for example).  That leaves 4 8 KiB blocks.  These partially fill a second stripe unit of 32 KiB, and are placed on 1 drive on node 2.  OneFS then calculates FEC using those two stripe units.  The FEC calculation creates two additional stripe units, each 128 KiB in size, and places them "elsewhere" on the cluster (one of them will be on node 3, then other will be on either node 1 or node 2 on a different drive than the stripe units containing the file data).  From a utilization standpoint, you have this:

1 * 128 KiB data stripe unit (DSU) + 1 * 32 KiB DSU + 2 * 128 KiB FEC stripe units (FSU) = 416 KiB

With +2:1 protection on a file, you also have 3 inodes, which is 3 * 512 B = 1.5 KiB.  So, roughly, you have 416 KiB + 1.5 KiB = 417.5 KiB for your 160 KiB file.

- 513 KiB (this file uses 2 stripes given the example configuration)

That's 65 8 KiB blocks (64 fully allocated blocks and 1 block with 1 KiB in it, though it's still fully 8 KiB allocated in OneFS):

stripe 1: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

stripe 2: 1 * 8 KiB DSU + 2 * 8 KiB FSU = 24 KiB

which is

768 KiB + 24 KiB + 3 * 512 B = 793.5 KiB

(if you create this file on a real cluster, then run du, you might get 818 KiB for utilization.  If so, OneFS created extension blocks, which would be, in this example, 3 * 8 KiB.  So, 793.5 + 24 = 817.5 KiB =~ 818 KiB).

- 1025 KiB (this file uses 3 stripes)

That's 129 8 KiB blocks:

- stripe 1: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

- stripe 2: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

- stripe 3: 1 * 8 KiB DSU + 2 * 8 KiB FSU = 24 KiB

which is

768 KiB  + 768 KiB + 24 KiB + 3 * 512 B = 1561.5 KiB


and if you assume the extension blocks will be created here too, that's 1561.5 + 24 = 1585.5 KiB.

Moderator
Moderator

Re: n+2:1 Protection Parity Calc

Jump to solution

hi guys ...

thanks you so much for the great update ...

In short ,,, the parity is defined how bit data was , and Parity is actually include whole data its self and how many Parity is needed if 2disks are failed or 1node is failed ...

thanks you !

aya

0 Kudos