Avamar: How to make Avamar's dedicated Management interface routable

The use case for policy-based routing on Avamar Utility Node or a single node is when there is a separate management interface configured on the node.

Let us say the default route is configured for the backup network using bond0 interface and the management network is configured to use bond2 interface.
When a single default route is configured in the Linux OS for multiple network interfaces, the response to the traffic that is coming to the other interfaces is going back using the default route and default interface.
In our case, if bond2 receives the traffic, the return traffic goes out using default route (and bond0 interface). This may be a problem in many network configurations as the clients cannot properly communicate with the management interface.
In order to resolve this problem, the policy-based routing could be configured on the Avamar node. The OS on Avamar comes with iproute2 binaries and kernel compiled with the policy routing enabled.

This KB uses the following settings as an example. Change network, IP, interface, and so on, accordingly The name of the new routing table does not matter, however it must be consistent in all steps.

The name of the managed interface: bond2
The IP address of bond2: 10.217.21.20
Network: 10.217.21.0/24
Gateway: 10.217.21.254
The name of the routing table for the management interface: management

The configuration includes few steps and must be performed as root user. 

Notes: 

• Do not copy and paste the commands below and substitute all parameters with the correct values.
• Ensure that the routing file for the management interface (/etc/sysconfig/network/ifroute-bond2 in this example) does not exist. If it does exist, rename it by adding "x-" prefix to the file name. For example: Rename /etc/sysconfig/network/ifroute-bond2 to /etc/sysconfig/network/x-ifroute-bond2.

Procedure:

1. Create a new policy routing table entry.
 

echo "1 management" >> /etc/iproute2/rt_tables

2. Define the routing for bond2 in the newly created table.
 

ip route add 10.217.21.0/24 dev bond2 src 10.217.21.20 table management
ip route add default via 10.217.21.254 dev bond2 table management

3. Create a rule referencing how the OS should use this table. Again, change the IP address accordingly (from 10.217.21.20/32 to <some_other_IP>/32)
 

ip rule add from 10.217.21.20/32 table management
ip rule add to 10.217.21.20/32 table management

4. Commit the changes.
 

ip route flush cache

5. Verify that the policy routing rulesets have been created.
 

ip rule show

Sample output showing that both the management and main policies are in use: 

root@avamar:~/#: ip rule show
0:      from all lookup local
32764:  from all to 10.217.21.20 lookup management
32765:  from 10.217.21.20 lookup management
32766:  from all lookup main
32767:  from all lookup default

6. Make the changes permanent.

The SuSE Enterprise Linux configuration for Avamar does not include the NetworkManager. While it is possible to preserve routes, it would not be possible to preserve the rules using default configuration.
In order to make changes permanent, the entries should be added to a script that is being started on boot. The procedure is different for SLES 11 (Avamar 19.1 and earlier) and SLES 12 (Avamar 19.2 and newer). This is because SLES 11 uses SysV Init, while SLES 12 uses systemd.

The values in the following commands are examples, and they should be modified to include correct values.

a. SLES 11

Run the following commands:
 

echo "ip route add 10.217.21.0/24 dev bond2 src 10.217.21.20 table management" >> /etc/rc.d/rc.local
echo "ip route add default via 10.217.21.254 dev bond2 table management" >> /etc/rc.d/rc.local
echo "ip rule add from 10.217.21.20/32 table management" >> /etc/rc.d/rc.local
echo "ip rule add to 10.217.21.20/32 table management" >> /etc/rc.d/rc.local
echo "ip route flush cache" >> /etc/rc.d/rc.local

b. SLES 12

i. Run the following commands to create the script:
 

echo "/sbin/ip route add 10.217.21.0/24 dev bond2 src 10.217.21.20 table management" >> /usr/local/avamar/bin/policy-based-routing.sh
echo "/sbin/ip route add default via 10.217.21.254 dev bond2 table management" >> /usr/local/avamar/bin/policy-based-routing.sh
echo "/sbin/ip rule add from 10.217.21.20/32 table management" >> /usr/local/avamar/bin/policy-based-routing.sh
echo "/sbin/ip rule add to 10.217.21.20/32 table management" >> /usr/local/avamar/bin/policy-based-routing.sh
echo "/sbin/ip route flush cache" >> /usr/local/avamar/bin/policy-based-routing.sh
chmod +x /usr/local/avamar/bin/policy-based-routing.sh

ii. Use the vi editor and create the systemd service file.
 

vi /etc/systemd/system/policy-based-routing.service

The file content should be as follows:
 

[Unit]
Description = Policy based routing service unit file.
After = network.target

[Service]
ExecStart = /bin/bash /usr/local/avamar/bin/policy-based-routing.sh

[Install]
WantedBy = multi-user.target

iii. Enable the new service.
 

systemctl enable policy-based-routing.service

iv. Reload the systemctl daemon
 

systemctl daemon-reload

ip rule del to 10.217.21.20/32 table management
ip rule del from 10.217.21.20/32 table management
ip route del default via 10.217.21.254 dev bond2 table management
ip route del 10.217.21.0/24 dev bond2 src 10.217.21.20 table management
sed 's/1 management//g' /etc/iproute2/rt_tables
ip route flush cache

systemctl disable policy-based-routing.service
systemctl daemon-reload