Number of nodes for 200 gigabytes per second

They are what I like to know since the data sheet from EMC just claim that X410 supports 200 gigabytes per second what criteria this throughput is on?

http://www.emc.com/collateral/software/specification-sheet/h10639-isilon-x-series-ss.pdf

Can you elaborate a bit on the type of I/O that makes up the 200 GB/s?  For example:

- What protocol(s), and if multiple protocols, about how much of each would be in use?

- Reads or writes, or some mix?

- Is this random I/O, sequential I/O, or some mix of all or some of these?

- What about request sizes?  Large block, small block, etc.

- Are the files in the filesystem large (> ~2 MB) or small (< ~ 500 KB)?

Using the same methodologies described above, it breaks down like this:

  X410:  1467 MB/s/node

  S210:  1313 MB/s/node

  X400:  904 MB/s/node

  S200:  874 MB/s/node

NL400:  675 MB/s/node

HD400:  786 MB/s/node

To reiterate, these per-node results are from large block sequential read benchmarks, done in a lab.  Performance comes in all shapes and sizes, and this is but one aspect of a complex concept...

> Sorry but how is this going to happen where average xfer sizes are way below 100KB in practice? Or was that mostly data from cache (L1+L2 RAM, L3 SSD)?

There actually were only 34 HDD per node (and 2 SSD per node).  The cluster in question was using L3 cache, but that doesn't come into play in this type of test (i.e. no blocks read in this test made it to L3).

The big contribution from cache in a test like this is from L2, and the aggressive prefetch being done on the test files (which are 50 GiB each).  The aggressive prefetch was set by applying the 'streaming' access template to the test directory and test files.  These files also had a 'streaming' layout, meaning they used the most drives per-stripe that OneFS allows...

The amount of data that leaves the NIC (per unit of time) must have been read from the drives, wether prefetched or not.

So what did the sustained disk IO traffic look like in this test, average disk read ops/s and disk read sizes?

Prefetching helps to overcome latency issues and to deliver a continuous stream of data to the client, but in the end the (aggregated) client throughput and the (aggregated) disk throughput must match...

That's 211241 MB/s from 144*36 drives, or 40 MB/s from each single drive (even more with protection overhead),

or 150+ disk read ops at 256 KB xfer size. Sorry but how is this going to happen where average xfer sizes are way below 100KB in practice? Or was that mostly data from cache (L1+L2 RAM, L3 SSD)?

-- Peter

> So what did the sustained disk IO traffic look like in this test, average disk read ops/s and disk read sizes?

I looked back at the statistics taken during the x410 run.  Average MB/s out per-disk was ~42 MB/s.  Average per-disk ops out was ~225.  These are statistics as reported by "isi statistics drive".

In these benchmark tests, I cross-check the throughput as reported by OneFS with bytes transferred over the NICs under test, and also cross-check those with throughput reported at the client, to ensure that different aspects of the larger test system all agree on the performance seen...

Thank you, very interesting. A sustained ops rate of ~225 means there is some data locality in the overall layout; i.e. movements of drive head are not completely random. The average xfer size 180KB (42MB/225) is quite high, which indicates minimal fragmentation within files.

Looking behind the scenes helps better explaining the differences between marketing propaganda and real world observations, and it should /reduce/ annoyance not to say frustration... As you said "Performance comes in all shapes and sizes"   and my take is that the actual condition of the file system layout is one more determining factor, unsurprisingly.

Thanks again!

-- Peter

Thank you!