Isilon First packet isn't SYN - Smartconnect issue?

We have SBR enabled and some static routes statements.  We are on Isilon version 8.0.0.4.   It only happens on the 1st initiated packet.    What did you do to fix the issue?

look at your routes / default gateway.  Our firewall block asymmetrical routing as well.  Also look into "Source Based Routing" to help with static route nightmare.

we don't have SBR enabled so had to add additional static routes.  Run tcpdump on all interfaces and see if initial request is coming in on a different interface than expected.

Hi PJurisprudencia,

I have found this to be helpful when talking about routing, either destination or source based. Routing and Isilon, how to get from A to B and back again

Are you referring to "sysctl net.inet.ip.choose_ifa_by_ipsrc" ?    Ours is set to 0 or is not enabled.    How does this work or help?

BTW, how does NIC affinity played on this picture?   Does it also cause packets come in from one interface and leave to another?

"NIC affinity" mentioned in the quoted blog post could matter here,

but how to check it on OneFS 8.0? Can't find a sysctl with an obvious name... ?

Cheers

-- Peter

We ran the tcpdump in Isilon on all nodes and from the packet capture, we don't see a different mac-address the packet goes in and out.   But in the firewall, the 1st packet is being blocked.   I was wondering if the Isilon capture we had is already the 2nd communication after Smartconnect pass the traffic communication to this particular node.  I have a feeling that the blocked "isn't SYN" packet is the 1st communication from SmartConnect to the client.   

We are using LACP for multiple NICs.   I was reading about NIC affinity.  Do you guys enable NIC affinity?   Any danger enabling NIC affinity?

Hi PJurisprudencia,

This is from the Best Practices Guide for Isilon External Network Connectivity regarding NIC affinity:

https://support.emc.com/docu58740

NIC affinity is a sysctl that can be configured in OneFS. The NIC affinity setting applies only when there are multiple NICs on the same node connected to the same subnet. The NIC affinity setting is enabled automatically when there are multiple NICs on the same subnet to enable response packets to go out using the same NIC that they arrived on, based on the source IP address of the response packet. The interface that is currently configured with that IP address is the interface that the packet will be sent on.

So if you are using LACP chances are that it doesn't apply unless your management network is not a separate subnet or you have a third NIC (For example, if your node has two 1GB interfaces and four 10GB interfaces).

I would also make a test without LACP, just in case...

Hi,

I'm a little bit confused in your statement "If you are using LACP changes are that it doesn't apply unless your management network is not a separate subnet or you have a third NIC (for example, if your node has two 1GB and 4 10GB interface"

So we have two 1GB NIC for management (only 1 NIC is used) and four 10GB NIC (only 2 are used).  Our management network is on separate subnet from our data subnet.  Our 10GB NIC for the data are LACP'd.    Should we consider enabling the NIC Affinity then?

i don't think so,  you are using LACP.  If you had 10G interfaces where each interface was assigned a dedicated IP address and those addresses happen to be on the same subnet, then you would use NIC affinity.

Can you explain how you are testing? Maybe the teardowns are only occuring on certain nodes (left out of the ACL on the firewall or something) also have you tried restarting the dns service (smart connect) on the Isilon, we had some funky issues after a previous OneFS patch and firmware upgrade to our 10g nics?

Test procedure:

1. Disconnect existing network connection to the mapped network shares from my PC using "net use /d *".

2. Start packet capture in all Isilon nodes

3. Start wireshark on my PC

4. Start packet capture in the firewall.

5. Access the network share

Unfortunately, we were not able to replicate the issue.   It was a hit or miss. 

It's important to remember that while SmartConnect is part of how you establish an SMB connection, it operates on a different port and protocol.

It is also important to remember that since the Isilon is a filer, it does not send the first packet of any conversation, that would be on your client.

Could you provide details of the error message indicating the problem? Is the Source interface or host for the message the Client or the Isilon?

Normally these errors will indicate the port/service the issue is observed on.

If the problem was a reply from SmartConnect coming from the wrong interface, you would see port 53 or service DNS.

If the source is the client, (and the client doesn't have actual issues mounting) it would seem that your firewall is having trouble correctly tracking traffic passing through it.