n+2:1 Protection Parity Calc

> With 3 nodes system .... this means Data ( or File ) will be spread over 3X2=6  disks with 3 stripes ?

That depends on the size of the file.  As of the OneFS 7.0 release, OneFS uses the concept of diskpools to create "failure domains", and files are confined to these diskpools when written.  A diskpool is at most 6 drives per node.  So, for example, an X400 node with 36 HDD will have those drives carved up into 6 diskpools.  Each diskpool then spans up to 40 nodes before additional diskpools are created.

So in your 3 node cluster (let's imagine it's the 36 drive X400 in my example), you have 6 diskpools, each with 18 drives (6 drives/node * 3 nodes).  When you write file #1, it goes to diskpool 1.  File #2 goes to diskpool 2, etc.  So each file will be striped within that diskpool's set of 18 drives, and so each file could use up to 18 drives.

If you write a really big file (> 32 MiB), you will see that this file is striped across many of the drives in its diskpool, and possibly all of them if it's big enough.  That's because OneFS uses "drive rotation" to avoid using the same set of drives for all blocks of large file.  Exactly when drive rotation happens depends on various things (including certain filesystem tunables like 'streaming'), but "at around 32 MiB" is close enough for this discussion.

If you write smaller files (in this example that would be files >384 KiB but less than about 32 MiB), OneFS will use 2 drives per node, or 6 drives total, to store the file.  The number of stripes the file has depends on the file's size: in the 3 node example, a 512 KiB file will have 1 stripe; a 1024 KiB file will have 2 stripes, etc.

If you write files that are smaller than 384 KiB (in this example), you will use less than 6 drives to store that file.  In the 3 node, +2:1 example, the smallest number of drives you will ever use is 3, one drive from each node needed to store at minimum a single block each, from, say, a 1 byte file.

> if so ...
> what is the physical space used for below files ....

>

> 160KB/513KB/1025KB

The actual logical space a file consumes depends on several things, e.g. raw + protection + inodes + other metadata like extension blocks, etc.  The exact logical space consumed is tricky to predict from first principles, since some of the metadata, like extension blocks, are only created at certain file sizes *and* that size can depend on the OneFS release and what features you're using in that release (the time when extension blocks are created depends on the "fullness" of the inode, which depends on the version of the inode we're using, etc).

You can, however, simply take a theoretical approach to calculating utilization and get pretty close.  All you need is the cluster size, the file's raw size, and the OneFS protection.  Let's go through your example file sizes:

- 160 KiB, 3 nodes, +2:1

160 KiB is 20 8 KiB blocks.  16 of those 20 blocks fill a one stripe unit, a full 128 KiB.  That goes on 1 drive within 1 diskpool on node 1 (for example).  That leaves 4 8 KiB blocks.  These partially fill a second stripe unit of 32 KiB, and are placed on 1 drive on node 2.  OneFS then calculates FEC using those two stripe units.  The FEC calculation creates two additional stripe units, each 128 KiB in size, and places them "elsewhere" on the cluster (one of them will be on node 3, then other will be on either node 1 or node 2 on a different drive than the stripe units containing the file data).  From a utilization standpoint, you have this:

1 * 128 KiB data stripe unit (DSU) + 1 * 32 KiB DSU + 2 * 128 KiB FEC stripe units (FSU) = 416 KiB

With +2:1 protection on a file, you also have 3 inodes, which is 3 * 512 B = 1.5 KiB.  So, roughly, you have 416 KiB + 1.5 KiB = 417.5 KiB for your 160 KiB file.

- 513 KiB (this file uses 2 stripes given the example configuration)

That's 65 8 KiB blocks (64 fully allocated blocks and 1 block with 1 KiB in it, though it's still fully 8 KiB allocated in OneFS):

stripe 1: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

stripe 2: 1 * 8 KiB DSU + 2 * 8 KiB FSU = 24 KiB

which is

768 KiB + 24 KiB + 3 * 512 B = 793.5 KiB

(if you create this file on a real cluster, then run du, you might get 818 KiB for utilization.  If so, OneFS created extension blocks, which would be, in this example, 3 * 8 KiB.  So, 793.5 + 24 = 817.5 KiB =~ 818 KiB).

- 1025 KiB (this file uses 3 stripes)

That's 129 8 KiB blocks:

- stripe 1: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

- stripe 2: 4 * 128 KiB DSU + 2 * 128 KiB FSU  = 768 KiB

- stripe 3: 1 * 8 KiB DSU + 2 * 8 KiB FSU = 24 KiB

which is

768 KiB  + 768 KiB + 24 KiB + 3 * 512 B = 1561.5 KiB

and if you assume the extension blocks will be created here too, that's 1561.5 + 24 = 1585.5 KiB.