Alex10
1 Copper

Not able to mount volume using multiple SDC networks

Hello all. I have 4 ESX hosts running SDC, 4 VMs running SDS (one per ESX host), and 3 VMs outside of the ESX cluster running all the MDM components. All are connected via two separate VLANs on separate interfaces. I have completed all the steps to get this setup, however, I am not able to mount the volume using BOTH networks (one works).

ESX1     10.209.46.192     10.209.47.192     /24

ESX2     10.209.46.193     10.209.47.193     /24

ESX3     10.209.46.194     10.209.47.194     /24

ESX4     10.209.46.195     10.209.47.195     /24


SDS1     10.209.46.1          10.209.47.1     /24

SDS2     10.209.46.2          10.209.47.2     /24

SDS3     10.209.46.3          10.209.47.3     /24

SDS4     10.209.46.4          10.209.47.4     /24


MDM1     10.209.46.150     10.209.47.150     /24     also has public IP for MDM cluster use

MDM1     10.209.46.151     10.209.47.151     /24     also has public IP for MDM cluster use

MDM1     10.209.46.152     10.209.47.152     /24     also has public IP for MDM cluster use


I setup the SDC using "scli --add_sdc --sdc_ip IP" and enabled --set_restricted_sdc_mode. I then mapped to the volume using the "allow_multi_map" option :


scli --map_volume_to_sdc --volume_name VOLUME01 --sdc_ip 10.209.46.192  --allow_multi_map

scli --map_volume_to_sdc --volume_name VOLUME01 --sdc_ip 10.209.47.192  --allow_multi_map

scli --map_volume_to_sdc --volume_name VOLUME01 --sdc_ip 10.209.46.193  --allow_multi_map

scli --map_volume_to_sdc --volume_name VOLUME01 --sdc_ip 10.209.47.193  --allow_multi_map

for all 8 IPs


# scli --query_all_sdc

MDM restricted SDC mode: Enabled

Query all SDC returned 8 SDC nodes.

SDC ID: 2620886900000000 Name: N/A IP: 10.209.47.194 State: Connected GUID: 5693D1EF-0F57-6D00-25F4-44D3CADA2622 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  1 IOPS 1.0 KB (1024 Bytes) per-second

SDC ID: 2620886a00000001 Name: N/A IP: 10.209.46.195 State: Connected GUID: 5693D856-74A0-E69F-C16F-44D3CADA2682 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  1 IOPS 1.0 KB (1024 Bytes) per-second

SDC ID: 2620886b00000002 Name: N/A IP: 10.209.46.193 State: Connected GUID: 5693D31A-A660-C180-5FA6-44D3CADA4E34 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  1 IOPS 1.0 KB (1024 Bytes) per-second

SDC ID: 2620886c00000003 Name: N/A IP: 10.209.46.192 State: Connected GUID: 56938649-B16B-2DBC-2BD7-44D3CADA4D60 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  1 IOPS 1.0 KB (1024 Bytes) per-second

SDC ID: 2620886d00000004 Name: N/A IP: 10.209.47.192 State: Disconnected GUID: 00000000-0000-0000-0000-000000000000 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 2620886e00000005 Name: N/A IP: 10.209.46.194 State: Disconnected GUID: 00000000-0000-0000-0000-000000000000 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 2620886f00000006 Name: N/A IP: 10.209.47.193 State: Disconnected GUID: 00000000-0000-0000-0000-000000000000 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 2620887000000007 Name: N/A IP: 10.209.47.195 State: Disconnected GUID: 00000000-0000-0000-0000-000000000000 Approved: yes

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second


Notice the bottom four are disconnected and have no GUID. Also notice it's not always ".47" that is not working! I went ahead and unmapped everything and started over:


# scli --query_all_sdc

MDM restricted SDC mode: Disabled

Query all SDC returned 4 SDC nodes.

SDC ID: 2620886900000000 Name: N/A IP: 10.209.47.194 State: Connected GUID: 5693D1EF-0F57-6D00-25F4-44D3CADA2622

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 2620886a00000001 Name: N/A IP: 10.209.46.195 State: Connected GUID: 5693D856-74A0-E69F-C16F-44D3CADA2682

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 2620886b00000002 Name: N/A IP: 10.209.46.193 State: Connected GUID: 5693D31A-A660-C180-5FA6-44D3CADA4E34

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 2620887100000003 Name: N/A IP: 10.209.46.192 State: Connected GUID: 56938649-B16B-2DBC-2BD7-44D3CADA4D60

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second


(you cannot remove an SDC once it is connected, only when it goes away)


# scli --query_sdc --sdc_ip 10.209.46.194

Error: MDM failed command.  Status: Could not find the SDC

# scli --query_sdc --sdc_ip 10.209.47.195

Error: MDM failed command.  Status: Could not find the SDC

# scli --query_sdc --sdc_ip 10.209.47.193

Error: MDM failed command.  Status: Could not find the SDC

# scli --query_sdc --sdc_ip 10.209.47.192

Error: MDM failed command.  Status: Could not find the SDC

(pinged all these IPs from MDM and SDS with success)

# scli --query_all_sds

Query-all-SDS returned 4 SDS nodes.

Protection Domain ca25d2cf00000000 Name: pd001

SDS ID: 0f16a72e00000005 Name: SDS4 State: Connected, Joined IP: 10.209.46.4,10.209.47.4 Port: 7072

SDS ID: 0f16a72d00000004 Name: SDS3 State: Connected, Joined IP: 10.209.46.3,10.209.47.3 Port: 7072

SDS ID: 0f16a72b00000002 Name: SDS2 State: Connected, Joined IP: 10.209.46.2,10.209.47.2 Port: 7072

SDS ID: 0f16a72900000000 Name: SDS1 State: Connected, Joined IP: 10.209.46.1,10.209.47.1 Port: 7072

(notice all IPs are joined)

From an ESX host :

# esxcli system module parameters list -m scini     (cleaned up output)

Name                           Type    Value

IoctlIniGuidStr                string  56938649-b16b-2dbc-2bd7-44d3cada4d60

IoctlMdmIPStr                  string  10.209.46.150,10.209.46.151,10.209.47.150,10.209.47.151

I checked the ESX host for local firewall rules - the SDC VIB created a rule to allow 7072, 9011, 6611 outbound. I would assume a firewall issue would affect both IPs though. Any ideas?

-Alex

Tags (3)
0 Kudos
3 Replies
daverush
1 Nickel

Re: Not able to mount volume using multiple SDC networks

Alex,

This is a common misunderstanding regarding SDC configuration, made more unclear due to restricted SDC mode requiring the manual addition of an SDC via SCLI. 


Without that mode enabled, SDCs just need a GUID and the MDM IPs to register themselves.

Restricted-mode SDCs do not need all IPs added to the MDM's configuration, only one. In fact, you could use the management network IP of the SDC. The SDC then pulls the MDM and SDS clusters' layout and connects to the SDS IPs that it can reach.

In your case, if you can vmkping -I vmkX 10.209.46.1 (and 2,3,4, etc) as well as vmkping -I vmkY 10.209.47.1 (and 2,3,4, etc), then you have connectivity to the SDSes on both subnets.


The SDC will use all available subnets to reach the SDSes. If you have 3 subnets for data on the SDSes and only 2 configured on the SDCs, then you will connect via the two available subnets. If you add the 3rd subnet to the SDCs, they will then also connect to the SDSes on the third subnet.  (The exception to this being SDS IP role configuration, which dedicates an IP/subnet to either SDC-only traffic, or SDS-only traffic.)

Best practice is to maintain a 1:1 ratio of NICs/vSwitches to subnets.

0 Kudos
daverush
1 Nickel

Re: Not able to mount volume using multiple SDC networks

Also,

https://support.emc.com/kb/466097 details ESX SDC config and troubleshooting, minus the restricted SDC mode.

0 Kudos
Alex10
1 Copper

Re: Not able to mount volume using multiple SDC networks

This makes a lot more sense. I have no real way to know if it is using those networks though. I'll get this all setup and see how it performs. I really appreciate the detailed answer, Rush!