Article Number: 000173831
IDPA: IDPA Deployment fails at 36% or 45% with error "FAILED: Configuring Protection Storage. Error: Unable to ping gateway server x.x.x.x using the IP address x.x.x.x configured on the ethVx network interface of the Protection Storage server. "
Summary: This KB Article walks you through the IDPA Deployment failure at 36% or 45% with the error "FAILED: Configuring Protection Storage." This issue happens when the Data Interfaces on Data Domain (DD) are unable to ping the default gateway. ...
Article Content
Symptoms
IDPA Deployment may fail at 36% or 45% with error "FAILED: Configuring Protection Storage. Error: Unable to ping gateway server x.x.x.x using the IP address x.x.x.x configured on the ethVx network interface of the Protection Storage server."
The deployment fails at 36% if the DD Interface ethV1 is unable to ping gateway. Deployment fails at 45% if the DD interface ethV2 is unable to ping the gateway.
This issue may be present due to the switch configurations. Verify that Physical VMNIC ports 6 and 1 are connected to the switch and have single VLAN traffic that is allowed and configured in access mode.
The following error may be seen on the IDPA ACM UI:
If the deployment fails on ethV2, the following error may be seen:
Analyze the server.log on ACM to confirm that the following errors are present:
2020-02-26 14:17:39,934 INFO [pool-23-thread-1]-ddadapter.ConfigDataDomainTask: checkConnectionWithGateway-->Non-exit code(41) while running ping.Output:[Ljava.lang.String;@56ce2982
2020-02-26 14:17:39,934 ERROR [pool-23-thread-1]-ddadapter.ConfigDataDomainTask: ApplianceException occurred while executing Datadomain config task.
com.emc.vcedpa.common.exception.ApplianceException: Unable to ping the gateway server <GATEWAY_IP_ADDRESS> using the IP address <DD Data Interface IP Address> configured on the ethV2 network interface of the Protection Storage server. Check the vSwitch1 connectivity on the IDPA Virtualization Server (ESXi server) and verify the switch settings in the customer environment.
at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.checkConnectionWithGateway(ConfigDataDomainTask.java:242)
at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.configureDD(ConfigDataDomainTask.java:330)
at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.run(ConfigDataDomainTask.java:1680)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Review the "/var/log/vobd.log" file on ESXi for the following error messages:
[root@xxxxxxx:/var/log] grep -i 128 vobd.log
2019-10-12T04:24:30.888Z: [netCorrelator] 133533756us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2019-10-12T04:24:30.889Z: [netCorrelator] 133534817us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2019-10-12T04:24:32.827Z: [netCorrelator] 135473005us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2019-10-12T04:28:03.941Z: [netCorrelator] 346586293us: [vob.net.pg.uplink.transition.down] Uplink: vusb0 is down. Affected portgroup: iDRAC Network. 0 uplinks up. Failed criteria: 128
2019-10-12T04:34:10.179Z: [netCorrelator] 712824422us: [vob.net.firewall.config.changed] Firewall configuration has changed. Operation 'addIP4' for rule set snmpTrapsOut succeeded.
2019-10-12T04:34:10.179Z: [netCorrelator] 712824515us: [esx.audit.net.firewall.config.changed] Firewall configuration has changed. Operation 'addIP4' for rule set snmpTrapsOut succeeded.
2020-06-16T14:47:11.896Z: [netCorrelator] 25270540046us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: DP-appliance-mgmt. 1 uplinks up. Failed criteria: 128
2020-06-16T14:47:11.896Z: [netCorrelator] 25270540052us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2020-06-16T14:47:12.725Z: [netCorrelator] 25271369162us: [vob.net.pg.uplink.transition.down] Uplink: vmnic6 is down. Affected portgroup: DP-appliance-backup. 1 uplinks up. Failed criteria: 128
Make a note of the vmnic ID from the vobd.log showing above and run the following command to check firmware and driver version on the NIC:
esxcli network nic get -n vmnic<id>
For Example:
[root@xxxxx4400:/var/log] esxcli network nic get -n vmnic7
Advertised Auto Negotiation: false
Advertised Link Modes: 10000BaseT/Full
Auto Negotiation: false
Cable Type: DA
Current Message Level: 0
Driver Info:
Bus Info: 0000:3b:00:3
Driver: i40en
Firmware Version: 6.80 0x80003d73 18.8.9
Version: 1.7.11
Link Detected: true
Link Status: Up
Name: vmnic7
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: DA
Supports Auto Negotiation: false
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:53:d4:cf
Wakeon: None
The deployment fails at 36% if the DD Interface ethV1 is unable to ping gateway. Deployment fails at 45% if the DD interface ethV2 is unable to ping the gateway.
This issue may be present due to the switch configurations. Verify that Physical VMNIC ports 6 and 1 are connected to the switch and have single VLAN traffic that is allowed and configured in access mode.
The following error may be seen on the IDPA ACM UI:
In the following snippet, the deployment failed for ethV1.
If the deployment fails on ethV2, the following error may be seen:
Analyze the server.log on ACM to confirm that the following errors are present:
2020-02-26 14:17:39,934 INFO [pool-23-thread-1]-ddadapter.ConfigDataDomainTask: checkConnectionWithGateway-->Non-exit code(41) while running ping.Output:[Ljava.lang.String;@56ce2982
2020-02-26 14:17:39,934 ERROR [pool-23-thread-1]-ddadapter.ConfigDataDomainTask: ApplianceException occurred while executing Datadomain config task.
com.emc.vcedpa.common.exception.ApplianceException: Unable to ping the gateway server <GATEWAY_IP_ADDRESS> using the IP address <DD Data Interface IP Address> configured on the ethV2 network interface of the Protection Storage server. Check the vSwitch1 connectivity on the IDPA Virtualization Server (ESXi server) and verify the switch settings in the customer environment.
at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.checkConnectionWithGateway(ConfigDataDomainTask.java:242)
at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.configureDD(ConfigDataDomainTask.java:330)
at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.run(ConfigDataDomainTask.java:1680)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Review the "/var/log/vobd.log" file on ESXi for the following error messages:
[root@xxxxxxx:/var/log] grep -i 128 vobd.log
2019-10-12T04:24:30.888Z: [netCorrelator] 133533756us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2019-10-12T04:24:30.889Z: [netCorrelator] 133534817us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2019-10-12T04:24:32.827Z: [netCorrelator] 135473005us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2019-10-12T04:28:03.941Z: [netCorrelator] 346586293us: [vob.net.pg.uplink.transition.down] Uplink: vusb0 is down. Affected portgroup: iDRAC Network. 0 uplinks up. Failed criteria: 128
2019-10-12T04:34:10.179Z: [netCorrelator] 712824422us: [vob.net.firewall.config.changed] Firewall configuration has changed. Operation 'addIP4' for rule set snmpTrapsOut succeeded.
2019-10-12T04:34:10.179Z: [netCorrelator] 712824515us: [esx.audit.net.firewall.config.changed] Firewall configuration has changed. Operation 'addIP4' for rule set snmpTrapsOut succeeded.
2020-06-16T14:47:11.896Z: [netCorrelator] 25270540046us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: DP-appliance-mgmt. 1 uplinks up. Failed criteria: 128
2020-06-16T14:47:11.896Z: [netCorrelator] 25270540052us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2020-06-16T14:47:12.725Z: [netCorrelator] 25271369162us: [vob.net.pg.uplink.transition.down] Uplink: vmnic6 is down. Affected portgroup: DP-appliance-backup. 1 uplinks up. Failed criteria: 128
Make a note of the vmnic ID from the vobd.log showing above and run the following command to check firmware and driver version on the NIC:
esxcli network nic get -n vmnic<id>
For Example:
[root@xxxxx4400:/var/log] esxcli network nic get -n vmnic7
Advertised Auto Negotiation: false
Advertised Link Modes: 10000BaseT/Full
Auto Negotiation: false
Cable Type: DA
Current Message Level: 0
Driver Info:
Bus Info: 0000:3b:00:3
Driver: i40en
Firmware Version: 6.80 0x80003d73 18.8.9
Version: 1.7.11
Link Detected: true
Link Status: Up
Name: vmnic7
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: DA
Supports Auto Negotiation: false
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:53:d4:cf
Wakeon: None
Cause
This issue may be present due to the switch configurations. Verify that Physical VMNIC ports 6 and 1 are connected to the switch and have single VLAN traffic that is allowed and configured in access mode.
The network team should verify the switch configuration to verify that all four IDPA ports are set in access mode and have single VLAN allowed.
Review the cabling and configuration guidelines below:
For single network use case, review the information below:
For separate management network use case, review the information below:
The network team should verify the switch configuration to verify that all four IDPA ports are set in access mode and have single VLAN allowed.
Review the cabling and configuration guidelines below:
For single network use case, review the information below:
- Ports 2,9,3,8 are connected to Management network VLAN ONLY (Single VLAN allowed).
- All ports on switch should be in "Access mode or untagged."
- All switch ports should be active and not configured in LACP.
- MTU size of 1518 or higher is configured on the external customer switch.
- MTU size of 1500 is configured along the network path. (Jumbo frames are not supported)
- If Cisco switches, port security is disabled over these ports.
For separate management network use case, review the information below:
- Ports 2 and 9 are connected to Management network VLAN ONLY (Single VLAN allowed).
- Ports 3 and 8 are connected to Backup network VLAN ONLY (Single VLAN allowed).
- All ports on switch should be in "Access mode or untagged."
- All switch ports should be active and not configured in LACP.
- MTU size of 1518 or higher is configured on the external customer switch.
- MTU size of 1500 is configured along the network path. (Jumbo frames are not supported)
- If Cisco switches, port security is disabled over these ports.
Resolution
Note: This issue may be due to incorrect cabling and switch configuration issues at the switch end. Review cabling and configuration guidelines in 'Cause' section before proceeding.
Resolution:
1. This issue may be due to incorrect cabling and switch configuration. Review KB article 192911: PowerProtect DP Series appliance and IDPA: DP4400 Networking and Initial Connectivity Resolution Path. to troubleshoot and fix DP4400 networking issues.
2. This issue may also be encountered if we use a duplicate IP address for backup interfaces ethV1 or ethV2 on Protection Storage (Data Domain). Perform ping tests to confirm if the IP is duplicate. Also verify there no stale records of this IP on Firewall, etc if this IP was in use earlier.
3. If the NIC i40en driver version is lower than 1.9.5, contact Dell EMC Support to update firmware. Also, if the i40en firmware version is lower than 19.5.12, contact Dell EMC Support to update firmware.
Resolution:
1. This issue may be due to incorrect cabling and switch configuration. Review KB article 192911: PowerProtect DP Series appliance and IDPA: DP4400 Networking and Initial Connectivity Resolution Path. to troubleshoot and fix DP4400 networking issues.
2. This issue may also be encountered if we use a duplicate IP address for backup interfaces ethV1 or ethV2 on Protection Storage (Data Domain). Perform ping tests to confirm if the IP is duplicate. Also verify there no stale records of this IP on Firewall, etc if this IP was in use earlier.
3. If the NIC i40en driver version is lower than 1.9.5, contact Dell EMC Support to update firmware. Also, if the i40en firmware version is lower than 19.5.12, contact Dell EMC Support to update firmware.
Article Properties
Affected Product
PowerProtect DP4400, Integrated Data Protection Appliance Family
Product
PowerProtect Data Protection Software, Integrated Data Protection Appliance Family, Integrated Data Protection Appliance Software
Last Published Date
13 Oct 2022
Version
5
Article Type
Solution