Yesterday i added 5 nodes into one of our Isilon External Subnets so they would be available for client connections. After adding them we noticed that some applications were having issues accessing filesystems. The application involved would not be able to access the filesystem initially but after several re-tries it would be okay. We also had some alerts concerning a share being accessed from another client that said it did not exist, but it did and i could see from the modification times it was being accessed.
This would suggest that there was not actually and issue with the node accepting client connections but if it is not that then not sure where to look. I say that because after suspending the nodes from the IP pools to stop them accepting new client connections everything went back to as it was with no issues being reported by the applications.
I was wondering if anyone in the past had seen such behaviour and figured out what the issue was. Or if anyone has any thoughts on what this could be as im a bit stuck. I want to blame it on the applications but unless i have some evidence to support that i can't.
A good thing to do before you add all nodes to all pools is to add each node to a single test or management pool to confirm that each node is accessible on the network.
I would take a more systematic approach and add one node to one pool at a time, confirm from a client that the addresses assigned are available, then move onto the next one.
Some things to look at if you add a node's interface into a pool and the IP(s) for its interface is not accessible: Did you add the correct interface: 1gige vs 10gige vs 40gige vs aggregates? Are you using VLAN tagging and are all of the necessary VLANs trunked into the interface on the switch for the nodes? Are you using LACP, is this set up properly on the switch, did you add the aggregate interface and not the individual interfaces, did you choose the correct aggregation type? Physical connectivity: switch port enabled, cables plugged in, etc.