NVP vProxy: SLES VM is recovered with incorrect network interface names
Summary: The NetWorker VMware Protection (NVP) solution is configured to backup SuSE Enterprise Linux (SLES) Virtual Machines (VM). After performing an image recovery of the VM, it is recovered with incorrect network interface names. ...
Symptoms
- A SuSE Enterprise Linux (SLES) Virtual Machine (VM) is deployed in VMware and protected using the NetWorker VMware Protection (NVP) solution.
- The SLES VM has multiple Network Interface Cards (NIC).
- The SLES VM is configured with a nonsequential Network Interface Card (NIC) naming convention. For example instead of eth0, eth1, eth2 (and so forth), the NICs are named eth0, eth1, eth4.
sles-client01:~ # ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:50:56:be:a4:8f brd ff:ff:ff:ff:ff:ff inet 192.168.9.120/24 brd 192.168.9.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:febe:a48f/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:50:56:be:8d:19 brd ff:ff:ff:ff:ff:ff inet 192.168.9.220/24 brd 192.168.9.255 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:febe:8d19/64 scope link valid_lft forever preferred_lft forever 4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:50:56:be:71:bd brd ff:ff:ff:ff:ff:ff inet 192.168.9.223/24 brd 192.168.9.255 scope global eth4 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:febe:71bd/64 scope link valid_lft forever preferred_lft forever sles-client01:~ # ls -l /sys/class/net total 0 lrwxrwxrwx 1 root root 0 Dec 4 12:22 eth0 -> ../../devices/pci0000:00/0000:00:16.0/0000:0b:00.0/net/eth0 lrwxrwxrwx 1 root root 0 Dec 4 12:22 eth1 -> ../../devices/pci0000:00/0000:00:16.1/0000:0c:00.0/net/eth1 lrwxrwxrwx 1 root root 0 Dec 4 12:22 eth4 -> ../../devices/pci0000:00/0000:00:17.0/0000:13:00.0/net/eth4 lrwxrwxrwx 1 root root 0 Dec 4 12:22 lo -> ../../devices/virtual/net/lo
- After performing a VM restore (VM revert, Image Restore, Instant Access Restore), the VM is recovered with an incorrect NIC naming convention:
sles-client01:~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:50:56:be:68:7e brd ff:ff:ff:ff:ff:ff
inet 192.168.9.120/24 brd 192.168.9.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:febe:687e/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:50:56:be:7c:c1 brd ff:ff:ff:ff:ff:ff
inet 192.168.9.220/24 brd 192.168.9.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:febe:7cc1/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:50:56:be:a2:fa brd ff:ff:ff:ff:ff:ff
sles-client01:~ # ls -l /sys/class/net
total 0
lrwxrwxrwx 1 root root 0 Dec 3 13:34 eth0 -> ../../devices/pci0000:00/0000:00:16.0/0000:0b:00.0/net/eth0
lrwxrwxrwx 1 root root 0 Dec 3 13:34 eth1 -> ../../devices/pci0000:00/0000:00:16.1/0000:0c:00.0/net/eth1
lrwxrwxrwx 1 root root 0 Dec 3 13:34 eth2 -> ../../devices/pci0000:00/0000:00:17.0/0000:13:00.0/net/eth2
lrwxrwxrwx 1 root root 0 Dec 3 13:34 lo -> ../../devices/virtual/net/lo
In this example the original VM has eth0, eth1, and eth4. After performing an Image restore, the recovered VM has eth2 instead of eth4. The eth2 NIC does not come online.
Cause
This problem is occurring outside of NetWorker/vProxy. NetWorker VMware Protection does not take into affect the settings/configurations of the file system or devices from within the operating system. During a VM image backup, a snapshot of the VM is created and the VM's configuration and disk files are backed up.
See SUSE KB: SLES Virtual Machine on VMware with three or more NICs does not keep the same NIC order | SUSE | Support Center
Resolution
Persistent NIC naming must be configured on the VM operating system to assign the NIC bus ID to the wanted device name.
The system GRUB biosdevname setting should also be disabled. The biosdevname feature is responsible for providing consistent and predictable network device names based on the BIOS settings (that is: eth0, eth1, eth2, and so fort).
This can be done to correct the networking on a VM after it has been recovered. Also, if the original VM is still available, the correction can be applied to it as well. Any restore operations of the VM using backups made after this change are expected to recover with the persistent interface names configured on the original VM.
Additional Information
The following steps were performed on a SLES 12 SP5 VM.
1. Create a VMware snapshot of the VM before making any changes.
sudo su -
lspci command:
lspci | grep -i ethernet
Example:
sles-client01:~ # lspci | grep -i ethernet 0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01) 0c:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01) 13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
ls -l /sys/class/net
sles-client01:~ # ls -l /sys/class/net total 0 lrwxrwxrwx 1 root root 0 Dec 4 12:22 eth0 -> ../../devices/pci0000:00/0000:00:16.0/0000:0b:00.0/net/eth0 lrwxrwxrwx 1 root root 0 Dec 4 12:22 eth1 -> ../../devices/pci0000:00/0000:00:16.1/0000:0c:00.0/net/eth1 lrwxrwxrwx 1 root root 0 Dec 4 12:22 eth4 -> ../../devices/pci0000:00/0000:00:17.0/0000:13:00.0/net/eth4 lrwxrwxrwx 1 root root 0 Dec 4 12:22 lo -> ../../devices/virtual/net/lo
udev persistent net rule file:
vi /etc/udev/rules.d/70-persistent-net.rules
udev rules exist under /etc/udev/rules.d. There could be other rules which may conflict with the new rule. If no other rules exist, proceed. If other rules exist, check to make sure that there are no other network naming related rules configured. Consult with your Linux system administrator regarding any rules configured on the system.
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:##:##.##", NAME="NIC_NAME1" SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:##:##.##", NAME="NIC_NAME2" SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:##:##.##", NAME="NIC_NAME3"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:0b:00.0", NAME="eth0" SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:0c:00.0", NAME="eth1" SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:13:00.0", NAME="eth4"
lspci command.
biosdevname in the GRUB default settings.
cp /etc/default/grub /etc/default/grub.bak ; vi /etc/default/grub
/etc/default/grub.bak), then opens the original for editing. If any issues are observed, you can revert to the default file or remove the setting shown below.
... biosdevname=0"
GRUB_CMDLINE_LINUX="... biosdevname=0"7. Reload the grub configuration:
grub2-mkconfig -o /boot/grub2/grub.cfg8. Reboot the VM and confirm that it comes up with the correct interface names, and that no issues are observed accessing the system or its applications.