Data Domain: Issues while adding a route in DDMC 2.0

Due to a defect in the DataDomain Management Center (DDMC) code in version 2.0.x, a customer will be unable to add static routes through either the CLI or the GUI.

All attempts and variations of the commands to add a route will fail, for example the following through either admin or SE mode:

# net route add -net 10.154.x.x netmask 255.255.255.0 gw 10.157.x.x dev ethV1
# net route add ipversion ipv4 -net 10.154.x.x netmask 255.255.255.0 gw 10.157.x.x dev ethV1
# net route add ipversion ipv4 -net 10.154.x.x netmask 255.255.255.0 gw 10.157.x.x ethV1

All commands returning the same result:

Error: Route command system error.

If a customer or support checks the messages.engineering file (through "log view debug/messages.engineering", if using the CLI) one can find the following error messages:

Dec 8 13:18:17 GDS-BEGIA-0418 sms: INFO: sms_net_add_route_job: Route command: /ddr/bin/ddroute.py add -net 10.154.x.x netmask 255.255.255.0 gw 10.157.x.x ethV1 Dec 8 13:18:17 GDS-BEGIA-0418 sms: NOTICE: Run: /ddr/bin/ddroute.py add - net 10.154.33.0 netmask 255.255.255.0 gw 10.157.x.x ethV1 Dec 8 13:18:17 GDS-BEGIA-0418 sms: NOTICE: exec route error:127 Dec 8 13:18:17 GDS-BEGIA-0418 sms: WARNING: sms_net_add_route_job: Route command failed: **** An error occured while running an internal command.: Dec 8 13:18:17 GDS-BEGIA-0418 ddsh: NOTICE: MSG-DDSH-00017: (tty=pts/0, session=21095) root: command "net route add -net 10.154.x.x netmask 255.255.255.0 gw 10.157.64.94 ethV1" exited with code: 41

Some customers may even notice the DDMC , despite booting and being accessible through the VMware console, can't be reached through SSH, and even ping doesn't work. As the DDMC default gateway, if configured, is a static route, it will fail adding when starting DDMC 2.0.0.x, hence if a customer tries to access the DDMC from a network the VM is not directly attached to, the DDMC will not be reachable for ping, SSH or the GUI.
 

 

The cause for this problem is some extensive changes to the underlying routing engine was performed as of DDMC 2.x and DDOS 6.x, to implement "policy based routing". When carrying over routing configuration from earlier releases to the new ones, a wrapper script called "/ddr/bin/ddroute.py" would be run by DDMC when adding persistent routes as configured in the registry, but the script wouldn't properly parse input data, and may fail to add some routes.

This defect has been partially addressed and is now fixed for some cases in the DDMC 2.0.1.0 and 2.1.0.0 releases.

If even after upgrading to the mentioned releases the default gateway is not automatically added to the DD when booting, then a customer can use the following CLI command from the DDMC host to add the route manually, and it will both be added then and make it persist across reboots:

# net route set gateway DEFAULT_GW_IP_ADDRESS

For other static routes different to the default gateway, a customer can use the following CLI as well (example to route all traffic to the private 192.168.1.0/24 network through interface ethV0, using 192.168.1.1 as the IP address for the next hop):

# net route add net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.1 dev ethV0