We have SBR enabled and some static routes statements.  We are on Isilon version 8.0.0.4.   It only happens on the 1st initiated packet.    What did you do to fix the issue?

look at your routes / default gateway.  Our firewall block asymmetrical routing as well.  Also look into "Source Based Routing" to help with static route nightmare.

we don't have SBR enabled so had to add additional static routes.  Run tcpdump on all interfaces and see if initial request is coming in on a different interface than expected.

Hi PJurisprudencia,

I have found this to be helpful when talking about routing, either destination or source based. Routing and Isilon, how to get from A to B and back again

Are you referring to "sysctl net.inet.ip.choose_ifa_by_ipsrc" ?    Ours is set to 0 or is not enabled.    How does this work or help?

BTW, how does NIC affinity played on this picture?   Does it also cause packets come in from one interface and leave to another?

"NIC affinity" mentioned in the quoted blog post could matter here,

but how to check it on OneFS 8.0? Can't find a sysctl with an obvious name... ?

Cheers

-- Peter

We ran the tcpdump in Isilon on all nodes and from the packet capture, we don't see a different mac-address the packet goes in and out.   But in the firewall, the 1st packet is being blocked.   I was wondering if the Isilon capture we had is already the 2nd communication after Smartconnect pass the traffic communication to this particular node.  I have a feeling that the blocked "isn't SYN" packet is the 1st communication from SmartConnect to the client.   

We are using LACP for multiple NICs.   I was reading about NIC affinity.  Do you guys enable NIC affinity?   Any danger enabling NIC affinity?

Hi PJurisprudencia,

This is from the Best Practices Guide for Isilon External Network Connectivity regarding NIC affinity:

https://support.emc.com/docu58740

NIC affinity is a sysctl that can be configured in OneFS. The NIC affinity setting applies only when there are multiple NICs on the same node connected to the same subnet. The NIC affinity setting is enabled automatically when there are multiple NICs on the same subnet to enable response packets to go out using the same NIC that they arrived on, based on the source IP address of the response packet. The interface that is currently configured with that IP address is the interface that the packet will be sent on.

So if you are using LACP chances are that it doesn't apply unless your management network is not a separate subnet or you have a third NIC (For example, if your node has two 1GB interfaces and four 10GB interfaces).

I would also make a test without LACP, just in case...