Unsolved
This post is more than 5 years old
1 Rookie
•
107 Posts
0
3133
February 19th, 2016 07:00
SMB Multichannel with Isilon
Hi all,
Isilon supports SMB Multichannel starting with SMB 3.0 since OneFS 7.1.1+.
If all conditions for SMB Multichannel are met, how can I figure out if it is used for SMB connections?
We have an Isilon cluster connected with all 10gige ports to the network. There are some Windows Server 2012 machines that connects to the 10gige SmartConnect Zone. Every Node has two IP addresses associated with that access zone.
At the Isilon I've observed at the WebUI that these Windows 2012 machines established exact eight SMB connections to single Nodes, four to each IP address. At the WebUI it looks like SmartConnect isn't working fine because the big amount of SMB connections are not distributed over all Nodes. You can see for example 16 SMB connections going to only two Nodes while other Nodes are bored.
An isi smb session list shows only two SMB sessions for this two Nodes 5 and 6. One at each Node.
Looking with netstat shows the following output for Node 5. *.14 and *.21 are the currently assigned IP addresses of Node 5.
tcp4 0 0 xx.xx.xx.21.microsoft-d xx.xx.xx.10.61288 ESTABLISHED
tcp4 0 0 xx.xx.xx.21.microsoft-d xx.xx.xx.10.61289 ESTABLISHED
tcp4 0 0 xx.xx.xx.21.microsoft-d xx.xx.xx.10.61287 ESTABLISHED
tcp4 0 0 xx.xx.xx.21.microsoft-d xx.xx.xx.10.61286 ESTABLISHED
tcp4 0 0 xx.xx.xx.14.microsoft-d xx.xx.xx.10.60471 ESTABLISHED
tcp4 0 0 xx.xx.xx.14.microsoft-d xx.xx.xx.10.60470 ESTABLISHED
tcp4 0 0 xx.xx.xx.14.microsoft-d xx.xx.xx.10.60469 ESTABLISHED
tcp4 0 0 xx.xx.xx.14.microsoft-d xx.xx.xx.10.60465 ESTABLISHED
The same connection behaviour do the other two servers have: Exact four tcp connections to each node IP address.
Are these many connections from one server to one Node a indication for multichannel SMB? Otherwise there is the question why the connections are not distributed equal over all nodes.
Phil
0 events found
kipcranford
125 Posts
0
February 24th, 2016 09:00
Hi Phil,
SMB-MC only works between a client and a particular node. In other words, for the purposes of load balancing I/O, the client will only "know" about the IP addresses on the node it is connected to, and will not use other nodes' IP addresses in its balancing logic. Additionally, when a share is mapped from a client to a node, only a single TCP connection will be seen regardless of the presence of Multi-channel. To see SMB-MC in action, you'll have to generate some I/O and then look at either TCP connection via netstat, and/or client statistics via 'isi statistics client' ('isi smb sessions' will always show just a single connection).
Here's an example from my lab. 192.168.200.112 and 192.168.200.136 are the two 10G IPs on my Win2012 client, and 192.168.200.31 and 192.168.200.34 are the two 10G IPs on the Isilon node I connected to. All four NICs are connected to the same flat 10G network.
After initially mapping a share from the client, I see this:
x200-1# netstat -an | grep EST | grep '.445'
tcp4 0 0 192.168.200.31.445 192.168.200.136.59615 ESTABLISHED
and
x200-1# isi smb session list
Computer User
---------------------
192.168.200.136 root
---------------------
Total: 1
However, initiating a large-file copy on the client from that node, I see this:
x200-1# netstat -an | grep EST | grep '.445'
tcp4 0 0 192.168.200.34.445 192.168.200.112.59622 ESTABLISHED
tcp4 0 0 192.168.200.34.445 192.168.200.112.59621 ESTABLISHED
tcp4 0 0 192.168.200.34.445 192.168.200.112.59620 ESTABLISHED
tcp4 0 0 192.168.200.34.445 192.168.200.112.59619 ESTABLISHED
tcp4 0 0 192.168.200.31.445 192.168.200.136.59618 ESTABLISHED
tcp4 0 0 192.168.200.31.445 192.168.200.136.59617 ESTABLISHED
tcp4 0 0 192.168.200.31.445 192.168.200.136.59616 ESTABLISHED
tcp4 0 0 192.168.200.31.445 192.168.200.136.59615 ESTABLISHED
and
x200-1# isi statistics client
Ops In Out TimeAvg Node Proto Class UserName LocalName RemoteName
----------------------------------------------------------------------------------------
231.0 27.0k 161.3M 5674.4 1 smb2 read root 192.168.200.31 192.168.200.136
229.3 26.8k 160.8M 5659.8 1 smb2 read root 192.168.200.34 192.168.200.112
----------------------------------------------------------------------------------------
Total: 2
x200-1# isi smb session list
Computer User
---------------------
192.168.200.136 root
---------------------
Total: 1
You can see the I/O being load balanced in the 'netstat' and 'isi statistics' output between Isilon IPs .31 and .34 and client IPs .112 and .136, even though there's still technically a single SMB session between that client and that node. That's SMB-MC at work.
So you might want to try looking at your statistics again when I/O is occurring (and you might want to also look at 'isi statistics client' from our CLI).
As for the balancing of SmartConnect amongst nodes, that might be a separate issue.
philippspohr
1 Rookie
•
107 Posts
0
February 24th, 2016 23:00
Ok, thank you kipcranford.
That's it. Nearly the same behaviour as at our cluster. One SMB session for OneFS but 1x4 TCP (or 2x 4?) connections to one single node from/to one single client.