ECS: Service Console: "Static routes validation' failed" after IP address changed

IP Addresses had been changed.

After the change, the Service Console health check fails with error 'Static routes validation' failed.

20200205 17:23:52.703: |   Static routes validation
INFO: On node 1XX.2XX.1.1 NAN managed static routes configured for networks: ['mgmt']
20200205 17:23:54.563: |   | FAIL (1 sec)
[ERROR] 'Static routes validation' failed: [1XX.2XX.1.1]: Command failed: 'sshpass ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=30  -T  root@1XX.2XX.1.1 $'(1>&2 echo connect7); wicked ifstatus public.mgmt 2>&1' < /dev/null'
Output:
public.mgmt     device-not-running
      link:     #10, state up, mtu 1500
      type:     vlan public[1474], hwaddr a4:bf:01:1f:c3:ea
      config:   compat:suse:/etc/sysconfig/network/ifcfg-public.mgmt
      leases:   ipv4 static failed
      leases:   ipv6 auto granted
      addr:     ipv4 10.1.2.3/24 [static]
      route:    ipv4 10.4.5.6/27 via 10.1.2.1 [static]
      route:    ipv4 10.4.5.7/27 via 10.1.2.1 [static]
      route:    ipv6 default via fe80::218:74ff:fe1c:8640 metric 1024 proto ra

In the above example, the IP Addresses for network separation enabled public.mgmt has been changed.

The wicked ifstatus for the interface shows device-not-running.

To check this run the below command - The leases show: ipv4 static failed

viprexec -if ~/VDCLIST wicked ifstatus <interface name> |grep <interface name> -A 4

admin@node-a:~> viprexec -if ~/VDCLIST wicked ifstatus public.mgmt |grep public.mgmt -A 4
Output from host : 1XX.2XX.1.1
public.mgmt     device-not-running
      link:     #10, state up, mtu 1500
      type:     vlan public[1474], hwaddr a4:bf:01:1f:c3:ea
      config:   compat:suse:/etc/sysconfig/network/ifcfg-public.mgmt
      leases:   ipv4 static failed
Output from host : 1XX.2XX.1.2
public.mgmt     device-not-running
      link:     #10, state up, mtu 1500
      type:     vlan public[1474], hwaddr a4:bf:01:1f:c3:eb
      config:   compat:suse:/etc/sysconfig/network/ifcfg-public.mgmt
      leases:   ipv4 static failed
Output from host : 1XX.2XX.1.3
public.mgmt     device-not-running
      link:     #10, state up, mtu 1500
      type:     vlan public[1474], hwaddr a4:bf:01:1f:c3:ec
      config:   compat:suse:/etc/sysconfig/network/ifcfg-public.mgmt
      leases:   ipv4 static failed
Output from host : 1XX.2XX.1.4
public.mgmt     device-not-running
      link:     #10, state up, mtu 1500
      type:     vlan public[1474], hwaddr a4:bf:01:1f:c3:ed
      config:   compat:suse:/etc/sysconfig/network/ifcfg-public.mgmt
      leases:   ipv4 static failed

Static routes are configured on the public.mgmt interface.

Run below command to view static routes:   

getrackinfo -t

admin@node-a:~> getrackinfo -t
Static route list
=================
Node ID      Network            Netmask                  Gateway           Interface     
1                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt
1                  11.2.3.4             255.255.255.0     11.2.3.1            public.mgmt
2                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt
2                  11.2.3.4             255.255.255.0     11.2.3.1            public.mgmt
3                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt
3                  11.2.3.4             255.255.255.0     11.2.3.1            public.mgmt
4                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt
4                  11.2.3.4             255.255.255.0     11.2.3.1            public.mgmt

In above example, assume that 10.1.2.23 was the old routes that are a Secure Remote Services (SRS) Gateway, while the new is 11.2.3.4.

Now, it cannot reactivate the wicked interface with the old routes in place. The routes must be removed first.

Warning: Be sure to verify that no production traffic is running over this route before deleting. Deleting the wrong route may cause DU. Get permission to do this change prior to deleting any of the routes!

Once agreed upon, delete the old routes.

The routes can be removed one at a time or by creating a file with all the commands. The command is:    

sudo setrackinfo --static-route-del <node id> <destination> <netmask> <gateway> <interface>

Alternately, copy and paste directly from the output of getrackinfo -t

For example, create a file:    

vi delete-old-routes

Enter the list of routes to:    

sudo setrackinfo --static-route-del 1                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt
sudo setrackinfo --static-route-del 2                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt
sudo setrackinfo --static-route-del 3                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt
sudo setrackinfo --static-route-del 4                  10.1.2.23            255.255.255.224   10.1.2.1            public.mgmt

Save the file and make it executable:    

chmod +x delete-old-routes

Run the file:    

./delete-old-routes

Wicked interfaces should now change to up:

viprexec -if ~/VDCLIST wicked ifstatus <interface name> |grep <interface name> -A 1

viprexec -if ~/VDCLIST wicked ifstatus public.mgmt |grep public.mgmt -A 1
Output from host : 1XX.2XX.1.1
public.mgmt     up
Output from host : 1XX.2XX.1.2
public.mgmt     up
Output from host : 1XX.2XX.1.3
public.mgmt     up
Output from host : 1XX.2XX.1.4
public.mgmt     up