PowerStore: embedded ESXi node cannot present any protocol endpoints on PowerStore X
Summary: PowerStore: embedded ESXi node cannot present any protocol endpoints on PowerStore X
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
When the Maximum Transmission Unit (MTU) configured on Ethernet switches is not equal to or larger than the MTU configured on the PowerStore management network, it can cause intermittent management issues.
In this specific example PowerStore X VASA connectivity fails for one of the embedded ESXi hosts only (ESXi Host that is installed on Node B hardware)
When jumbo frames (typically MTU size of 9000 bytes) are enabled, they must be set consistently end to end. Misconfigured jumbo frames may result in connection failures or reduced IO performance.
Ensure the error in vvold.log is err=Connection timed out and not err=SSL Exception. If the error is SSL Exception, follow VMware KB 67744 instead.
When testing connectivity with jumbo frames, subtract the ICMP header of 8 bytes and the minimum IP header size of 20 bytes. 9000 - 28 = 8972. These 2 headers will be added automatically increasing the frame size.
These ping tests were run from an ssh session to the ESXi hosts, see VMware KB 1003728 for more info on vmkping.
This looks like we cannot ping across the VLTi port-channel. Whether pings succeed or not in the examples above depends the source interface that is chosen, as each source interface is connected to a different switch.
On a Dell Networking OS10 or OS9 switch, the MTU of all interfaces that are connected to PowerStore should set to 9216. Misconfiguration will result in this issue.
There is an issue in OS10 versions prior to 10.5.0 (August 2019) where the VLTi interface Port-channel 1000 does not pass frames with an MTU larger than 1500 without fragmenting. It is expected that by default the VLTi should pass frames up to 9216 MTU.
After correcting the MTU mismatch and rescanning the storage on the ESXi host, we can see all protocol endpoints as expected.
In this specific example PowerStore X VASA connectivity fails for one of the embedded ESXi hosts only (ESXi Host that is installed on Node B hardware)
When jumbo frames (typically MTU size of 9000 bytes) are enabled, they must be set consistently end to end. Misconfigured jumbo frames may result in connection failures or reduced IO performance.
Table of Contents
1. Issue
These connectivity errors are observed in /var/log/vvold.log on ESXi host affected:2020-06-24T09:33:08.114Z info vvold[2104948] [Originator@6876 sub=Default] VasaSession::Initialize url is empty 2020-06-24T09:33:08.114Z warning vvold[2104948] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (PowerStore)! 2020-06-24T09:33:08.114Z info vvold[2104948] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://xx.xx.xx.xx:8443/version.xml 2020-06-24T09:33:08.114Z error vvold[2104948] [Originator@6876 sub=Default] Initialize: Unable to init session to VP PowerStore state: 0 2020-06-24T09:33:08.117Z info vvold[2104947] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://xx.xx.xx.xx:8443/version.xml 2020-06-24T09:34:28.895Z warning vvold[2104947] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=Connection timed out, using default 2020-06-24T09:34:28.896Z info vvold[2104947] [Originator@6876 sub=Default] VasaSession::Initialize url is empty
This is a different example from the log below from a different system. The following log is a certificate failure which is a different issue completely. However, in the above example it is a connectivity error even though large parts of the log are identical.
These certificate errors observed in /var/log/vvold.log on a different ESXi host for a different issue with different symptoms:
2019-12-26T16:57:03.396Z info vvold[2139844] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://xxxxxxxx.com:8443/version.xml 2019-12-26T16:57:03.401Z warning vvold[2139844] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters: --> PeerThumbprint: 0B:01:C4:F2:16:E0:10:C9:63:B5:F2:92:D3:36:B5:65:5C:59:DB:17 --> ExpectedThumbprint: --> ExpectedPeerName: xxxxxxxx.com --> The remote host certificate has these problems: --> --> * Host name does not match the subject name(s) in certificate., using default 2019-12-26T16:57:03.401Z info vvold[2139844] [Originator@6876 sub=Default] VasaSession::Initialize url is empty 2019-12-26T16:57:03.401Z warning vvold[2139844] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (xxxxxxxxx)! 2019-12-26T16:57:03.401Z info vvold[2139844] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://xxxxxxxx.com:8443/version.xml 2019-12-26T16:57:03.401Z error vvold[2139844] [Originator@6876 sub=Default] Initialize: Unable to init session to VP xxxxxxxxx state: 0
Ensure the error in vvold.log is err=Connection timed out and not err=SSL Exception. If the error is SSL Exception, follow VMware KB 67744 instead.
When testing connectivity with jumbo frames, subtract the ICMP header of 8 bytes and the minimum IP header size of 20 bytes. 9000 - 28 = 8972. These 2 headers will be added automatically increasing the frame size.
[root@Powerstore1000X-host-2:~] vmkping -I vmk1 1.2.3.4 -s 8972 -c 2 PING 1.2.3.4 (1.2.3.4): 8972 data bytes 8980 bytes from 1.2.3.4: icmp_seq=0 ttl=64 time=0.327 ms 8980 bytes from 1.2.3.4: icmp_seq=1 ttl=64 time=0.376 ms --- 1.2.3.4 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.327/0.352/0.376 ms [root@Powerstore1000X-host-2:~] vmkping -I vmk1 1.2.3.5 -s 8972 -c 2 PING 1.2.3.5 (1.2.3.5): 8972 data bytes --- 1.2.3.5 ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss [root@Powerstore1000X-host-2:~] vmkping -I vmk2 1.2.3.5 -s 8972 -c 2 PING 1.2.3.5 (1.2.3.5): 8972 data bytes 8980 bytes from 1.2.3.5: icmp_seq=0 ttl=64 time=0.303 ms 8980 bytes from 1.2.3.5: icmp_seq=1 ttl=64 time=0.411 ms --- 1.2.3.5 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.303/0.357/0.411 ms [root@Powerstore1000X-host-2:~] vmkping -I vmk2 1.2.3.4 -s 8972 -c 2 PING 1.2.3.4 (1.2.3.4): 8972 data bytes --- 1.2.3.4 ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss
These ping tests were run from an ssh session to the ESXi hosts, see VMware KB 1003728 for more info on vmkping.
[root@Powerstore1000X-host-2:~] vmkping -I vmk2 1.2.3.4 -s 1472 -c 2 PING 1.2.3.4 (1.2.3.4): 1472 data bytes 1480 bytes from 1.2.3.4: icmp_seq=0 ttl=64 time=0.278 ms 1480 bytes from 1.2.3.4: icmp_seq=1 ttl=64 time=0.296 ms --- 1.2.3.4 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.278/0.287/0.296 ms [root@Powerstore1000X-host-2:~] vmkping -I vmk2 1.2.3.5 -s 1472 -c 2 PING 1.2.3.5 (1.2.3.5): 1472 data bytes 1480 bytes from 1.2.3.5: icmp_seq=0 ttl=64 time=0.189 ms 1480 bytes from 1.2.3.5: icmp_seq=1 ttl=64 time=0.272 ms --- 1.2.3.5 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.189/0.231/0.272 ms [root@Powerstore1000X-host-2:~] vmkping -I vmk1 1.2.3.4 -s 1472 -c 2 PING 1.2.3.4 (1.2.3.4): 1472 data bytes 1480 bytes from 1.2.3.4: icmp_seq=0 ttl=64 time=0.218 ms 1480 bytes from 1.2.3.4: icmp_seq=1 ttl=64 time=0.220 ms --- 1.2.3.4 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.218/0.219/0.220 ms [root@Powerstore1000X-host-2:~] vmkping -I vmk1 1.2.3.5 -s 1472 -c 2 PING 1.2.3.5 (1.2.3.5): 1472 data bytes 1480 bytes from 1.2.3.5: icmp_seq=0 ttl=64 time=0.188 ms 1480 bytes from 1.2.3.5: icmp_seq=1 ttl=64 time=0.173 ms --- 1.2.3.5 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.173/0.180/0.188 ms
This looks like we cannot ping across the VLTi port-channel. Whether pings succeed or not in the examples above depends the source interface that is chosen, as each source interface is connected to a different switch.
2. Solution
On a Dell Networking OS10 or OS9 switch, the MTU of all interfaces that are connected to PowerStore should set to 9216. Misconfiguration will result in this issue.
There is an issue in OS10 versions prior to 10.5.0 (August 2019) where the VLTi interface Port-channel 1000 does not pass frames with an MTU larger than 1500 without fragmenting. It is expected that by default the VLTi should pass frames up to 9216 MTU.
SWITCH# ping -M do -s 8972 1.2.3.6 -c 3 PING 1.2.3.6 (1.2.3.6) 8972(9000) bytes of data. ping: local error: Message too long, mtu=1500 ping: local error: Message too long, mtu=1500 ping: local error: Message too long, mtu=1500 --- 1.2.3.6 ping statistics --- 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2042ms SWITCH# ping -M do -s 2472 1.2.3.6 -c 3 PING 1.2.3.6 (1.2.3.6) 2472(2500) bytes of data. ping: local error: Message too long, mtu=1500 ping: local error: Message too long, mtu=1500 ping: local error: Message too long, mtu=1500 --- 1.2.3.6 ping statistics --- 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2039ms SWITCH# ping -M do -s 1472 1.2.3.6 -c 3 PING 1.2.3.6 (1.2.3.6) 1472(1500) bytes of data. 1480 bytes from 1.2.3.6: icmp_seq=1 ttl=64 time=1.05 ms 1480 bytes from 1.2.3.6: icmp_seq=2 ttl=64 time=0.966 ms 1480 bytes from 1.2.3.6: icmp_seq=3 ttl=64 time=1.00 ms --- 1.2.3.6 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.966/1.008/1.059/0.046 ms SWITCH#
- The -s switch is used to define the payload size for the frame
- In the above output sending a frame with a payload of 8972 which would correspond to an MTU of 9000 Fails.
- After this a payload of 2472 corresponding to a 2500 MTU is sent which also fails
- Finally a successful payload of 1472 corresponding to 1500 MTU is successful
- In this case, it was confirmed that the network path could not accept a Frame larger than 1500 MTU
- In this specific example, the problem was the VLTi port-channel 1000 between 2 x S4148U-ON due to the OS10 defect described earlier.
After correcting the MTU mismatch and rescanning the storage on the ESXi host, we can see all protocol endpoints as expected.
Cause
NA
Resolution
NA
Affected Products
PowerStoreArticle Properties
Article Number: 000125860
Article Type: Solution
Last Modified: 19 Apr 2021
Version: 5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.