SMB Multichannel with Isilon

Question

Hi all, Isilon supports SMB Multichannel starting with SMB 3.0 since OneFS 7.1.1+. If all conditions for SMB Multichannel are met, how can I figure out if it is used for SMB connections? We have an Isilon cluster connected with all 10gige ports to the network. There are some Windows Server 2012 machines that connects to the 10gige SmartConnect Zone. Every Node has two IP addresses associated with that access zone. At the Isilon I've observed at the WebUI that these Windows 2012 machines established exact eight SMB connections to single Nodes, four to each IP address. At the WebUI it looks like SmartConnect isn't working fine because the big amount of SMB connections are not distributed over all Nodes. You can see for example 16 SMB connections going to only two Nodes while other Nodes are bored. An isi smb session list shows only two SMB sessions for this two Nodes 5 and 6. One at each Node. Looking with netstat shows the following output for Node 5. *.14 and *.21 are the currently assigned IP addresses of Node 5. tcp4       0      0 xx.xx.xx.21.microsoft-d   xx.xx.xx.10.61288       ESTABLISHED tcp4       0      0 xx.xx.xx.21.microsoft-d   xx.xx.xx.10.61289       ESTABLISHED tcp4       0      0 xx.xx.xx.21.microsoft-d   xx.xx.xx.10.61287       ESTABLISHED tcp4       0      0 xx.xx.xx.21.microsoft-d   xx.xx.xx.10.61286       ESTABLISHED tcp4       0      0 xx.xx.xx.14.microsoft-d   xx.xx.xx.10.60471       ESTABLISHED tcp4       0      0 xx.xx.xx.14.microsoft-d   xx.xx.xx.10.60470       ESTABLISHED tcp4       0      0 xx.xx.xx.14.microsoft-d   xx.xx.xx.10.60469       ESTABLISHED tcp4       0      0 xx.xx.xx.14.microsoft-d   xx.xx.xx.10.60465       ESTABLISHED The same connection behaviour do the other two servers have: Exact four tcp connections to each node IP address. Are these many connections from one server to one Node a indication for multichannel SMB? Otherwise there is the question why the connections are not distributed equal over all nodes. Phil

kipcranford · Answer

Hi Phil,

SMB-MC only works between a client and a particular node. In other words, for the purposes of load balancing I/O, the client will only "know" about the IP addresses on the node it is connected to, and will not use other nodes' IP addresses in its balancing logic. Additionally, when a share is mapped from a client to a node, only a single TCP connection will be seen regardless of the presence of Multi-channel. To see SMB-MC in action, you'll have to generate some I/O and then look at either TCP connection via netstat, and/or client statistics via 'isi statistics client' ('isi smb sessions' will always show just a single connection).

Here's an example from my lab. 192.168.200.112 and 192.168.200.136 are the two 10G IPs on my Win2012 client, and 192.168.200.31 and 192.168.200.34 are the two 10G IPs on the Isilon node I connected to. All four NICs are connected to the same flat 10G network.

After initially mapping a share from the client, I see this:

x200-1# netstat -an | grep EST | grep '.445'

tcp4 0 0 192.168.200.31.445 192.168.200.136.59615 ESTABLISHED

and

x200-1# isi smb session list

Computer User

---------------------

192.168.200.136 root

---------------------

Total: 1

However, initiating a large-file copy on the client from that node, I see this:

x200-1# netstat -an | grep EST | grep '.445'

tcp4 0 0 192.168.200.34.445 192.168.200.112.59622 ESTABLISHED

tcp4 0 0 192.168.200.34.445 192.168.200.112.59621 ESTABLISHED

tcp4 0 0 192.168.200.34.445 192.168.200.112.59620 ESTABLISHED

tcp4 0 0 192.168.200.34.445 192.168.200.112.59619 ESTABLISHED

tcp4 0 0 192.168.200.31.445 192.168.200.136.59618 ESTABLISHED

tcp4 0 0 192.168.200.31.445 192.168.200.136.59617 ESTABLISHED

tcp4 0 0 192.168.200.31.445 192.168.200.136.59616 ESTABLISHED

tcp4 0 0 192.168.200.31.445 192.168.200.136.59615 ESTABLISHED

and

x200-1# isi statistics client

Ops In Out TimeAvg Node Proto Class UserName LocalName RemoteName

----------------------------------------------------------------------------------------

231.0 27.0k 161.3M 5674.4 1 smb2 read root 192.168.200.31 192.168.200.136

229.3 26.8k 160.8M 5659.8 1 smb2 read root 192.168.200.34 192.168.200.112

----------------------------------------------------------------------------------------

Total: 2

x200-1# isi smb session list

Computer User

---------------------

192.168.200.136 root

---------------------

Total: 1

You can see the I/O being load balanced in the 'netstat' and 'isi statistics' output between Isilon IPs .31 and .34 and client IPs .112 and .136, even though there's still technically a single SMB session between that client and that node. That's SMB-MC at work.

So you might want to try looking at your statistics again when I/O is occurring (and you might want to also look at 'isi statistics client' from our CLI).

As for the balancing of SmartConnect amongst nodes, that might be a separate issue.

philippspohr · Answer

kipcranford schrieb:

You can see the I/O being load balanced in the 'netstat' and 'isi statistics' output between Isilon IPs .31 and .34 and client IPs .112 and .136, even though there's still technically a single SMB session between that client and that node. That's SMB-MC at work.

Ok, thank you kipcranford.

That's it. Nearly the same behaviour as at our cluster. One SMB session for OneFS but 1x4 TCP (or 2x 4?) connections to one single node from/to one single client.

Isilon

Was this post helpful?