Poor iSCSI performance in an PC8024 Stack

Hi all,

We've recently noticed poor 8K random performance in our iSCSI infrastructure. I've isolated the degradation to traffic passing through our stack of four 8024s (two 8024Fs and two 8024 vanilla). These are stacked via 10Gb DAC cables, four between the two 8024Fs, two to each 8024, and then a single link connecting the 8024s. Diagram attached. I realize this isn't best practice, but I don't believe it to be a throughput contention issue as those links never come close to being saturated. 

Stack ports don't show any errors:

VCISCSICORE-Stack#show switch stack-ports diag

Legend:

RBYT : Received Bytes RPKT : Received Packets
TBYT : Transmitted Bytes TPKT : Transmitted Packets
RFCS : Received Frame Check Sequence Errors RFRG : Received Fragment Errors
RJBR : Received Jabber Errors
RUND : Received Underrun Errors ROVR : Received Overrun Errors
TFCS : Transmit Frame Check Sequence Errors TERR : Transmit Errors

1 - 0/13:
RBYT:-309567802 RPKT:1196161818 TBYT:517154851 TPKT:-1663192043
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

1 - 0/14:
RBYT:224407688 RPKT:863198887 TBYT:-272195895 TPKT:548004213
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

1 - 0/15:
RBYT:1870769700 RPKT:683716200 TBYT:1126388597 TPKT:889180346
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

1 - 0/16:
RBYT:-1113773387 RPKT:1541462782 TBYT:-798205983 TPKT:607832173
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

1 - 0/19:
RBYT:1904948766 RPKT:2054328448 TBYT:624773940 TPKT:494053724
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

1 - 0/20:
RBYT:281286632 RPKT:-1066021893 TBYT:-524631390 TPKT:-1004797821
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

2 - 0/13:
RBYT:513843049 RPKT:-1663193100 TBYT:-317426810 TPKT:1196159889
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

2 - 0/14:
RBYT:-287092973 RPKT:548001622 TBYT:218939556 TPKT:863196367
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

2 - 0/15:
RBYT:1124732020 RPKT:889179262 TBYT:1865995352 TPKT:683715077
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

2 - 0/16:
RBYT:-799228653 RPKT:607831478 TBYT:-1114219369 TPKT:1541462466
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

2 - 0/19:
RBYT:-1955792372 RPKT:-814222735 TBYT:-227489482 TPKT:-1912236459
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

2 - 0/20:
RBYT:-1161510467 RPKT:435374518 TBYT:1289974733 TPKT:495061944
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0
--More-- or (q)uit

3 - 0/21:
RBYT:1501554135 RPKT:-2063211625 TBYT:-1995644571 TPKT:2047533354
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

3 - 0/22:
RBYT:-970854253 RPKT:-2116055448 TBYT:1276263151 TPKT:1315189644
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

3 - 0/23:
RBYT:0 RPKT:0 TBYT:0 TPKT:0
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

3 - 0/24:
RBYT:1524986447 RPKT:32900470 TBYT:-371435059 TPKT:1924303997
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

4 - 0/21:
RBYT:-227839654 RPKT:-1912236587 TBYT:-1957346236 TPKT:-814223075
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

4 - 0/22:
RBYT:1267346377 RPKT:495058927 TBYT:-1162056151 TPKT:435373243
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

4 - 0/23:
RBYT:0 RPKT:0 TBYT:0 TPKT:0
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

4 - 0/24:
RBYT:429511139 RPKT:542561299 TBYT:-588280182 TPKT:643446802
RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
TFCS:0 TERR:0

VCISCSICORE-Stack#show switch

 Management Standby Preconfig Plugged-in Switch Code
SW Status Status Model ID Model ID Status Version
--- ---------- --------- ------------- ------------- ------------- -----------
1 Stack Mbr Cfg Stby PC8024F PC8024F OK 4.2.2.3
2 Stack Mbr PC8024F PC8024F OK 4.2.2.3
3 Mgmt Sw PC8024 PC8024 OK 4.2.2.3
4 Stack Mbr PC8024 PC8024 OK 4.2.2.3

Performance metrics seem OK:

VCISCSICORE-Stack#show process cpu

Memory Utilization Report

status bytes
------ ----------
 free 673590704
 alloc 324752912

CPU Utilization:


 PID Name 5 Secs 60 Secs 300 Secs
-----------------------------------------------------------------
 4e7e8d0 tNet0 0.42% 0.22% 0.22%
 50aa510 ipnetd 0.00% 0.00% 0.01%
 5108c80 envMonTask 0.00% 2.96% 1.81%
 5112450 osapiTimer 0.00% 0.08% 0.12%
 5394470 bcmL2X.0 3.17% 3.06% 3.04%
 53bec70 bcmCNTR.0 0.42% 0.77% 0.75%
 5413830 bcmTX 0.00% 0.03% 0.04%
 5ad1340 bcmRX 2.75% 3.02% 3.16%
 5af1530 bcmNHOP 0.21% 0.10% 0.07%
 5b03880 bcmATP-TX 0.00% 0.10% 0.10%
 5b0cd80 bcmATP-RX 0.21% 0.21% 0.28%
 60c2d00 MAC Send Task 0.00% 0.02% 0.06%
 60cc200 MAC Age Task 0.00% 0.00% 0.01%
 6ac6910 bcmLINK.0 0.63% 0.29% 0.31%
 920b910 tL7Timer0 0.00% 0.02% 0.01%
 9231200 osapiMonTask 0.00% 0.02% 0.05%
 a390730 servPortMonTask 0.00% 0.00% 0.01%
 a537f60 simPts_task 0.00% 0.10% 0.11%
 ac3a190 emWeb 0.21% 0.03% 0.01%
 ac43460 dtlTask 0.21% 0.10% 0.12%
 ac95c20 hapiRxTask 0.00% 0.12% 0.10%
 b850520 SNMPTask 0.00% 0.14% 0.63%
 c6ade00 dot1s_timer_task 0.42% 0.66% 0.63%
 d3871f0 snoopTask 0.21% 0.03% 0.02%
 d40a770 dot3ad_timer_task 0.00% 0.04% 0.03%
 e505230 sFlowTask 0.00% 0.13% 0.15%
 e5af720 spmTask 0.00% 0.04% 0.04%
 e6272c0 ipMapForwardingTask 0.21% 0.11% 0.09%
152b3ae0 lldpTask 0.42% 0.36% 0.36%
154fb4a0 tCptvPrtl 0.00% 0.04% 0.04%
15a17de0 isdpTask 0.00% 0.07% 0.08%
161f29e0 RMONTask 0.21% 0.15% 0.14%
16202450 boxs Req 0.00% 0.01% 0.02%
17a6ba30 sshd 0.00% 0.00% 0.02%
17d93af0 sshd[0] 0.00% 0.00% 0.03%
-----------------------------------------------------------------
 Total CPU Utilization 9.74% 13.20% 12.85%

Jumbos/flowcontrol are configured as per best practice. Only thing I can find in our config that deviates from best practice is our STP method which is currently set to mst (I believe I've read RTSP is best practice) and our wiring. 

Anyone run into something like this before? It seems kind of odd to me that switch 3 is the management switch, although perhaps that's because of how we have this wired. 

Any suggestions would be helpful, thanks.