1 Nickel

Isilon storage performance issue

Hi all,

We have setup of four X 200 storage nodes, where each node has 6 Gig RAM,12 disks of 7200 RPM SATA (except storage node #4 , which has 11 disks), total 47 disk.

We are accessing storage cluster using 4 compute nodes via NFS protocol.Front end networking from compute node to storage cluster is through 1Gbps switch (each compute node can read and write data to storage cluster using 1Gbps bandwidth) ,backend networking is done using QDR infiband.

We did performance evaluation to get Maximum random read IOPS for 128K chunk read(we choose 128K , because our elasticsearch(application) read data in 128 K chunk).

We have use FIO tool to check performance.

Our observations for the same is :

1.With 20 FIO jobs(20 parallel reading thread) , running on single compute node, gives us max 850 IOPS of 128 K chunk (max out network utilization of 1 Gbps) , when we run same FIO job for 2 compute nodes in parallel(total 40 reading thread, 20 on each)  , performance drops to 673 & 691 respectively , total 1364 IOPS. When we run same FIO job for 4 compute nodes in parallel(total 80 reading thread, 20 on each) , performance further drops to 252,255,567,550,total 1624 IOPS. Whereas with 8K random read chunk from 4 compute nodes(80 reading threads,20 on each compute node) it was giving 4602 IOPS.

My question :

Why we are stuck at max 1624 IOPS in case of 128 K chunk random read whereas total 47 disk can give much higher IOPS(as with 8k random read chunk ,it is giving 4602 IOPS)

I understand , when we read 128K chunk , we effectively reading more, but in this case : neither network from compute to storage nor CPU on storage and compute node was bottleneck. In all above cases, we used 8K disk block size on isilon ,random access pattern to optimize IOPS , protection was +2:1.

We also tried to increase disk block size in isilon but with increase in disk block size , the performance was decreasing.

It will be nice if you people will recommend tuning parameters to optimize IOPS for 128 K chunk.




Tags (3)