What SmartConnect connection policy should I use?

The responses are generalizations, but should be good reference points.

1) Connection Count : Allocates connections by counting TCP sessions. This may also include management and other types of TCP based connections.

Connection Count is useful when the mounts are fairly persistent and the workload is similar between connections. If a client needs to be unmounted for some reason, it will be remounted to a node with the least amount of connections and continue to get the same level of service as before.

2)  CPU Utilization : Allocates connections by assessing the CPU load of each node.

CPU Utilization is useful when there are lots of connections that are short lived and transfer a relatively small amounts of data, after which the connection is torn down. Establishing and tearing down the TCP socket requires CPU resources.  As many hosts attempt to connect to the Isilon, they may experience latency or inconsistent responses when some cluster CPU's are at high utilization, and others are not. By directing clients to the node with the lowest CPU utilization at that time, the clients will experience consistent response times and CPU utilization over the entire cluster should be more even.

3) Network Throughput: Allocates connections based on the network throughput of each node.

Network Throughput is useful when the mount pattern is dynamic, the IO is sequential, and the application is throughput centric and perhaps bursty. When the application or user makes a request for data and requires high bandwidth, you want to direct them to the node that can provide the most bandwidth, but also prevent that new connection from interfering with an existing connection with similar requirements. The reason it is helpful that the mount pattern is dynamic, is to avoid having a connection made to a node that has low network throughput at the time, only to find that an idle connection has woken up, and started making aggressive use of bandwidth.  

4) Round Robin: Allocates connections by order of IP, regardless of other activities on the cluster.

Round Robin is useful in many cases for general purpose access. It is particularly useful in some HPC like applications when many clients mount at almost the same time.

Why wouldn't one of the above, more granular metrics be more appropriate?  It is because the above policies require some amount of time to assess the cluster condition in order to choose the appropriate IP. During this time, the same IP will be provided, even while the conditions are changing. In a batch mount environment this will cause many clients to be directed to some nodes but not others.

Round Robin does not need to assess the cluster condition and can issue unique IPs in rapid succession ensuring an even distribution of connections.