Data Domain: Direct-Connect Networking Between Data Protection Systems

• Directly connected systems cannot ping each other.
• Replication reports "No route to host", connection failed, or destination unreachable errors.
• Interface state shows DOWN, Disconnected, or Not Connected.
• Interface Connectivity Down alerts are generated.
• Link speed negotiates lower than expected.
• Throughput is lower than expected during testing.
• Interfaces flap or periodically lose connectivity.
• MTree replication, collection replication, backup, restore, migration, or vault operations fail over the direct-connect network.
• Direct-connect ports come online only during specific windows in Cyber Recovery environments.
• Newly installed direct-connect links remain offline after configuration.

• Network Configuration Issues
  • Directly connected interfaces are configured on different subnets.
  • Incorrect subnet masks create routing problems.
  • Static routes are missing when multiple network paths exist.
  • Traffic is routed through another interface instead of the direct-connect path.
  • Connection-host or destination configuration references the wrong IP address.
• Physical Connectivity Issues
  • Incorrect physical ports are connected.
  • Faulty cables, optics, DAC cables, or SFP/QSFP modules.
  • Unsupported or mismatched transceivers.
  • Link speed auto-negotiation failures.
  • NIC firmware or hardware defects.
• Interface Configuration Issues
  • Interface administratively disabled.
  • Bonding configuration mismatch.
  • MTU mismatch between endpoints.
  • Incorrect veth configuration.
  • Improper LACP settings.
• Cyber Recovery Behavior
  • Vault interfaces may be intentionally disabled between synchronization windows.
  • Replication contexts may be disabled by Cyber Recovery automation.
  • Interface-down alerts may be expected behavior when the vault is locked.
• Platform-Specific Issues
  • Certain 25Gb NIC and SFP combinations have exhibited instability or interface flapping in direct-connect deployments.
  • FC direct-connect configurations may experience topology or initiator discovery issues.
  • Incorrect hostname or replication configuration may prevent multiple direct-connect paths from being utilized as intended.

1. Layer 1 Validation

Always begin troubleshooting at the physical layer.

Verify:

• • Link LEDs are illuminated on both systems.
  • Correct interfaces are cabled together.
  • The cable is certified for the negotiated speed.
  • SFP/QSFP modules match on both ends.
  • The issue follows the cable or remains with the port during testing.

2. Validate Physical Port Mapping

In direct-connect deployments, it is not always obvious which physical NIC port on the rear of the appliance corresponds to the interface being configured in DDOS. This is especially common on systems with multiple NICs, expansion cards, bonded interfaces, or after hardware upgrades.

A frequent issue occurs when the administrator believes the cable is connected to one interface while it is actually connected to a different physical port.

Troubleshooting steps:

• • Configure temporary (dummy) IP addresses on multiple candidate interfaces on both systems.
  • Administratively enable the interfaces.
  • Observe which interfaces establish carrier and transition to an UP state.
  • Verify physical link LEDs on both appliances.
  • Use interface and hardware status commands to identify the correct port mapping.
  • Once the correct ports have been identified, remove the temporary configuration and apply the intended production settings.

This approach can quickly eliminate uncertainty around physical port selection and prevent unnecessary investigation into routing, replication, performance, or software-related issues when the actual problem is simply an incorrect cable-to-interface mapping.

Why This Helps

Many direct-connect issues are eventually traced back to:

• • Cable connected to the wrong NIC.
  • Incorrect assumptions about interface-to-port mapping.
  • Multiple NIC cards with similar port labeling.
  • Interface configured in DDOS does not match the port used by the installer.
  • Bonded members connected differently than expected.

Confirming physical port mapping before proceeding with higher-layer troubleshooting can significantly reduce time spent investigating routing, subnetting, replication, or Cyber Recovery behavior when these are not the actual root cause.

3. Verify Network Configuration

The most common direct-connect issue is improper subnetting.

For direct-connect communication:

• • Both interfaces must reside on the same subnet.
  • Dedicated direct-connect interfaces should use a subnet separate from all other interfaces.
  • A /30 network (255.255.255.252) is recommended because it provides only the two required host addresses and simplifies routing.

Example:

System A: 192.168.100.1/30
System B: 192.168.100.2/30

Benefits of a /30:

• • • Simplified troubleshooting.
    • Minimal ARP traffic.
    • No default gateway requirement.
    • Clear point-to-point routing.

If a dedicated subnet cannot be used:

• • • Configure appropriate static routes.
    • Verify traffic is using the intended direct-connect interface.

4. Validate Interface State

Check that interfaces are:

• • Enabled on both systems.
  • Assigned the correct IP address.
  • Operating at the expected speed.
  • Configured with matching MTU values.

A direct-connect interface cannot communicate if:

• • The remote interface is disabled.
  • The link has not established carrier.
  • One endpoint is configured incorrectly.

5. Verify Bonding Configuration

When multiple direct-connect cables are used:

• • Ensure bonding settings match on both systems.
  • Verify member interfaces belong to the correct bond.
  • Validate load-balancing and aggregation configuration.

Field experience has shown that some environments negotiating lower-than-expected link speeds when using Round Robin bonding were resolved after migrating to LACP.

LACP is generally the preferred bonding method when supported on both endpoints.

6. Understand Cyber Recovery Behavior

Cyber Recovery deployments frequently generate what appears to be a networking issue but is actually expected behavior.

The Cyber Recovery application may:

• • Disable the vault interface.
  • Enable the interface only during synchronization windows.
  • Disable replication contexts after synchronization completes.

As a result:

• • Production systems may report InterfaceConnectivityDown alerts.
  • Ping failures may occur outside synchronization windows.
  • Replication may report "No route to host."

Before escalating:

• • Verify the vault is unlocked.
  • Verify the synchronization window is active.
  • Confirm the interface was not intentionally disabled by Cyber Recovery automation.

7. Troubleshoot Lower Than Expected Link Speeds

If a 100Gb link negotiates at 25Gb or another reduced speed:

Verify:

• • Matching optic types.
  • Supported cable lengths.
  • Compatible transceivers.
  • Bonding configuration.
  • NIC firmware levels.

A bonding or negotiation mismatch can prevent interfaces from operating at their intended speed.

8. 8. Troubleshoot Low Throughput

Lower throughput does not always indicate a network problem.

Consider:

• • CPU limitations on test tools such as iPerf.
  • Single-core bottlenecks.
  • Session distribution across processor cores.
  • Source and destination system utilization.

When evaluating performance:

• • Review application or replication statistics.
  • Check for actual backlog or lag.
  • Compare workload performance against synthetic test results.

Do not rely solely on iPerf results when determining overall data transfer performance.

9. Hardware-Specific Issues

Field cases have identified issues involving:

• • 25Gb NIC instability.
  • SFP compatibility problems.
  • Interface flapping requiring port resets.
  • Direct-connect links requiring interface bounce operations before recovery.

If software configuration appears correct:

• • Review hardware compatibility.
  • Check known defects and release notes.
  • Replace suspect optics or cables.
  • Validate firmware and DDOS versions.

Additional Information

Direct-connect networking can be used for:

• MTree replication
• Collection replication
• Cyber Recovery vault connectivity
• IDPA-to-IDPA communication
• IDPA-to-Data Domain communication
• Data migration
• Backup and restore isolation
• Performance testing
• Network troubleshooting
• Temporary deployment connectivity

MRepl and CRepl can operate over a direct-connect network. From the Data Domain perspective, the connected device does not need to be a switch, router, or other network appliance. As long as carrier is established and the interfaces are configured correctly, communication can occur directly between the connected endpoints.

For directly connected interfaces:

• Both endpoints must be configured within the same subnet unless routing is intentionally introduced.
• A dedicated subnet should be used whenever possible.
• If the direct-connect network overlaps with other interfaces, static routes may be required to ensure traffic uses the intended path.
• Direct-connect troubleshooting is commonly used to isolate external networking devices as a potential source of connectivity or performance issues.

Data Domain: Troubleshooting Interfaces Down or Intermittent for Users
Data Domain - Configuring physical interfaces with Graphical User Interface (UI)
Data Domain - Configuring physical interfaces through command-line interface (CLI)

Data Domain: Intel E810 10Gb/25Gb Dual Rate NIC may not link up at 10Gb with M14MK Module
Use this KB when an Intel E810 dual-rate adapter does not establish a link at 10Gb speeds while using an M14MK module. It covers compatibility considerations, symptoms, and corrective actions to restore proper link negotiation.
Data Domain: Intel X710 NIC may fail to VLAN tag properly if it enters recovery mode.
Reference this KB if VLAN traffic is not being passed correctly through an Intel X710 interface. The article explains how NIC recovery mode can affect VLAN tagging and provides steps to identify and resolve the issue.
Data Domain: Error "SMS command failed: No interfaces found" from the DD CLI and GUI
Use this KB when the DD CLI or GUI reports that no interfaces are found. It helps identify interface discovery issues and outlines troubleshooting steps to restore normal network interface visibility.
Data Domain: New kernel version 5.4.x can cause DDFS HMON/CRON/RPC timeout panics
This KB should be referenced when system instability, panics, or timeout-related errors occur after a kernel update. It details the impact of specific 5.4.x kernel versions and the recommended remediation.
Data Domain: Troubleshooting a "Net Set Up Flag Failure" Error
Consult this KB when network configuration changes fail with a "Net Set Up Flag Failure" message. It provides guidance on diagnosing configuration inconsistencies and restoring interface functionality.
PowerProtect DD and IDPA: After Upgrading, ethVX Fails to Communicate
Use this KB if ethVX interfaces stop communicating following a software upgrade. It outlines common causes, validation steps, and corrective actions to restore connectivity.
Data Domain: Intel Interfaces link down with tx_timeout
Reference this KB when Intel-based network interfaces unexpectedly go down and tx_timeout errors are observed. The article helps determine whether the issue is driver, firmware, or hardware related and provides recovery procedures.
Data Domain: Cyber Recovery FEC setting off is not allowed
This KB applies when Cyber Recovery network configurations fail because Forward Error Correction (FEC) is disabled. It explains the FEC requirements and how to configure supported settings.
Data Domain: Intel X710 card not detected when booting with connected network cables
Use this KB when an Intel X710 adapter is missing or not detected during system boot when network cables are attached. The article discusses the conditions that trigger the issue and the recommended resolution.
Data Domain: Changing Bond Type with LACP Switch configuration Disables Interfaces
Reference this KB when modifying bonding modes on interfaces connected to LACP-configured switches. It describes how bond type changes can cause interfaces to become unavailable and how to safely perform the transition.
Data Domain: Invalid network configuration setting error caused by qlogic cards
This KB is useful when network configuration changes fail due to invalid settings associated with QLogic adapters. It provides troubleshooting steps and guidance for correcting the configuration.
Data Domain: Invalid MTU Causes VLAN Interface Failure After Reboot
Consult this KB if VLAN interfaces fail to come online after a reboot due to MTU-related configuration issues. The article explains the symptom, root cause, and proper MTU validation requirements.
Data Domain: Bonded Interface Alert Due to Member Interface Speed Mismatch
Use this KB when bonded interfaces generate alerts because member ports are operating at different speeds. It outlines how speed mismatches affect bonding health and the steps needed to resolve the condition.
Data Domain: Network Interfaces May Go Down Due to Intel Workaround Configuration
Reference this KB when network interfaces unexpectedly become unavailable after applying Intel-related workaround settings. It explains the behavior, affected configurations, and recommendations to maintain stable network connectivity.