Issues with jumbo frames on FX2s into Dell SC3200 SAN
Hello everyone, hopefully this is the correct place for this as this is the first time dealing with FX2s support and our first foray into 10GB iScsi linked SANs.
Our new setup is 2 PowerEdge FC640 servers running server 2016 in an FX20CI with Intel 10Gbe 4p x710-k bNDC cards connecting them to 2 PowerEdge FN410S IOM switches in the back. There are then 2 fibre cables per switch, so 4 in total, connecting these to the 2 controllers on the back of an SCv3000series SAN, so 2 fibre connections into 2 iSCSI fault domains. The top switch is part of the 10.10.10.0 fault domain, bottom switch is the 10.10.20.0 fault domain. Separately we also have a Quest DL4300 running server 2012 R2 with a QLogic BCM57810 10Gigabit Ethernet card that is connected to port 3 on the top FN410S IOM switch.
Now for the issue. The FN410S IOM switch ports are all configured for jumbo frames with an MTU of 9216, these are both the 8 internal and 4 external switch ports. Both fault domain controllers on the SAN are configured for jumbo frames with an MTU of 9000. The single fibre connection from the standalone Quest DL4300 is configured for jumbo frames with an MTU of 9014. This all works absolutely fine and I can ping with a packet size of 8972 from the Quest box to the san without any dropped packets. As soon as we enable jumbo frames on either of the PowerEdge FC640 servers, so this is going into the Network and Sharing Centre in Windows on each server, selecting each of the 2 iSCSI links that are connected and enabling jumbo packets with a MTU of 9014 bytes, the majority of the packets when doing a ping test with a size of 8972 are dropped, a few work but most are dropped. Dropping the ping size down to 8000 and most of the packets are still lost, to get solid ping results we need to go down to a ping packet size of around 2500. This happens when pinging the 10.10.10.1 and 10.10.20.1 ip addresses, which are the addresses of the controllers on the SAN. Pinging higher than 8972 and we get the 'Packet needs to be fragmented but DF set.' Does anyone have any ideas at all?
The entire platform was setup by a Dell engineer but he couldn't figure it out so he logged a job with Dell Storage support and requested them to remote on to ensure the settings were correct but they delayed so long before turning round and saying if the MTU size is configured properly in the SAN, they won't do anything at all, that we've had to put production servers on the 2 FC640s so troubleshooting has to be done out of hours. The 2 servers are currently set up with jumbo frames off and everything is working fine.