Skip to main content
Sign Out

Welcome to Dell

My Account
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
Sign In Create an Account Premier Sign In Dell Financial Services Partner Program Sign In
Contact Us

Your Dell.com Carts

Dell VxRail Architecture Overview

PDF

Network and latency

Both L2 and L3 support VMware vSAN connectivity between the data nodes. L2 does not require a static route, but L3 requires a static route for versions earlier than VxRail 7.0.480 or VxRail 8.0.200.

The following table provides a description of the intersite communication for L2 and L3:

Table 1. Intersite communicationThe following table provides a description of the intersite communication for L2 and L3:
Inter-site Deployment scenario Layer Routing
Site-to-site Default L2 Not required.
L3 Static routes are required.
Site-to-witness Default L3 Static routes are required.
L3 Static routes are required when using an interface other than the management (vmk0) interface.
Witness traffic separation L2 for 2-node Static routes are not required.
Figure 1. Network layer connectivity between VMware vSAN nodes
Network layer connectivity between VMware vSAN nodes

Static routes are required because VMware vSAN uses the default TCP stack. A VMkernel specific gateway is not supported. The VMkernel management interface uses the default gateway that is also used by any other VMkernel interface using the same TCP stack. Isolate the VMware vSAN network from normal infrastructure traffic on a dedicated network. The VMware vSAN network cannot use the default gateway and static routes must be used to connect with L3 addresses. While used for site-witness communication over L3, it can also include site-site communication when using L3 for data node addresses.

Use L3 for connectivity between the data sites and the witness. Witness Traffic Separation (WTS) can be used with stretched cluster configurations under the following conditions:

  • If a VMkernel interface other than the management VMkernel interface is tagged with witness traffic, static routes are required to communicate with the VMware vSAN witness host VMkernel interface tagged for VMware vSAN traffic.
  • If the management VMkernel interface is tagged with witness traffic, static routes are not required if the host can already communicate with the VMware vSAN witness host VMkernel interface using the default gateway.

The MTU can be different. For data nodes, the MTU can be 9000 and between data nodes and the witness, the MTU can be 1500.

Witness traffic separation

The witness traffic separation offers the following features:

  • An alternate VMkernel interface can be designated to carry traffic that is destined for the witness rather than the VMware vSAN tagged VMkernel interface.
  • More flexible network configurations are separated by allowing separate networks for node-to-node and node-to-witness traffic.
  • Two independent subnets and routes can be advertised from each data node site to the witness site for routing purposes.
  • You can set up a VxRail 2-node cluster.

Supported geographical distances

For VMware vSAN stretched clusters, support is based on network latency and bandwidth requirements instead of distance. The key requirement are the latency numbers between sites.

Bandwidth

The bandwidth requirement between the main sites depends on the workload on VMware vSAN, the amount of data, and the handling of failure scenarios.

Figure 2. Inter-site bandwidth and latency
Inter-site bandwidth and latency

The following table describes the basic bandwidth and latency requirements under normal operating conditions:

Table 2. Basic bandwidth and latency requirementsBasic bandwidth and latency requirements
Site Bandwidth Latency
Site-to-site VMware vSAN OSA: Minimum of 10 Gbps

VMware vSAN ESA: Minimum of 10 GbE.

<5 millisecond latency RTT
Site-to-witness 2 Mbps per 1000 VMware vSAN components < 500 millisecond latency RTT for 1 host per site

<200 millisecond latency RTT (up to 10 hosts per site)
  

<500 millisecond latency RTT (1 host per site)
  

Site-to-witness network latency

In most VMware vSAN stretched-cluster configurations, latency or RTT between sites hosting VM objects and the witness nodes should not be greater than 200 msec (100 millisecond one way).

The latency to the witness depends on the number of objects in the cluster. Broadcom recommends that on VMware vSAN stretched cluster configurations up to 10+10+1, a latency of less than or equal to 200 milliseconds is acceptable. If possible, a latency of less than or equal to 100 milliseconds is preferred. For configurations that are greater than 10+10+1, Broadcom requires a latency of less than or equal to 100 milliseconds.

For site-to-witness calculation examples, see Appendix B: Bandwidth calculation examples.

Intersite MTU

Unless witness traffic separation is used, maintain a consistent MTU between data nodes and the witness in a stretched cluster configuration.

Ensure that each VMkernel interface designated for VMware vSAN traffic is set to the same MTU to prevent traffic fragmentation. The VMware vSAN health check looks for a uniform MTU across the VMware vSAN data network, and reports on any inconsistencies.

Figure 3. Intersite MTU
Intersite MTU

Connectivity

The following connectivity options are available:

  • The management network provides connectivity to all three sites.
  • VM network connectivity between the data sites (the witness does not run VMs that are deployed on the VMware vSAN cluster).
  • VMware vSphere vMotion network connectivity between the data sites (VMs are never migrated from a data host to the witness host).
  • VMware vSAN network connectivity to all three sites.
  • VMware vSAN data traffic on the ISL.
  • VMware vSAN metadata traffic on the uplinks to the witness.

Rate this content

Accurate
Useful
Easy to understand
Was this article helpful?
0/3000 characters
  Please provide ratings (1-5 stars).
  Please provide ratings (1-5 stars).
  Please provide ratings (1-5 stars).
  Please select whether the article was helpful or not.
  Comments cannot contain these special characters: <>()\