Avamar: Linux FLR mount fails for ext4 VMs with METADATA_CSUM enabled

Linux FLR Mount Failure Symptoms

The following symptoms are observed when Linux FLR mount fails for ext4 VMs with METADATA_CSUM enabled:

• ext4 features such as sparse_supper2(3.16), metdata_csum(3.18), encrypt(4.1), project(4.5) are enabled
• Unable to Perform FLR for ext4 VMs

MountPoint logs show messages like:

[MP - DEBUG] 2019/02/25 09:56:08 TID=140458315512595 MID=245: sCmdline = /bin/mount -t ext4 /dev/loop1 /tmp/MntPnt5D1P0
[MP - DEBUG] 2019/02/25 09:56:08 TID=140458315512595 MID=246: sLine = mount: wrong fs type, bad option, bad superblock on /dev/loop1,
[MP - DEBUG] 2019/02/25 09:56:08 TID=140458315512595 MID=247: sLine = missing codepage or helper program, or other error
[MP - DEBUG] 2019/02/25 09:56:08 TID=140458315512595 MID=248: sLine =
[MP - DEBUG] 2019/02/25 09:56:08 TID=140458315512595 MID=249: sLine = In some cases useful info is found in syslog - try
[MP - DEBUG] 2019/02/25 09:56:08 TID=140458315512595 MID=250: sLine = dmesg | tail or so.
[MP - WARN] 2019/02/25 09:56:08 TID=140458315512595 MID=251: HLPR_Mount Failed
[MP - FUNC] 2019/02/25 09:56:08 TID=140458315512595 MID=252: HLPR_Mount LEAVE
[MP - DEBUG] 2019/02/25 09:56:08 TID=31507 MID=253: HLPR_Mount Failed: /dev/loop1
[MP - FUNC] 2019/02/25 09:56:08 TID=31507 MID=254: MbrGptCustomVmdkMount::MountVolumes LEAVE
[MP - ERROR] 2019/02/25 09:56:08 TID=31507 MID=255: MountVolumes Failed. Ret: 1048586

Root Cause of the Issue

The current proxy OS SLES 12 SP1 does not support certain configurations, which leads to the FLR limitation. These configurations include:

• Ext4 sparse_supper2(3.16)
• metdata_csum(3.18)
• Encrypt(4.1)
• Project(4.5)

These limitations are the primary cause of the Linux FLR mount failure for ext4 VMs with METADATA_CSUM enabled.

Resolving Linux FLR Mount Fails for ext4 VMs with METADATA_CSUM Enabled.

A good way to troubleshoot this issue is to try a manual mount of the vmdk on proxy. Take a snapshot of the original VM and then go to proxy -> edit settings -> new device -> add existing device -> select the vmdk -> attach as independent nonpersistent (read-only). If the customer is not comfortable doing this with a live VM, you can clone the VM and try.

Go to proxy -> scan the scsi -> create a directory using mkdir and try to mount the disk. For example:

aveproxy02:~ # mkdir /mnt/13576626
aveproxy02:~ #
aveproxy02:~ # mount /dev/sdc /mnt/13576626
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error

In some cases useful info is found in syslog - try
dmesg | tail or so.
aveproxy02:~ #

To check which feature the proxy is finding faulty, on the proxy try running:

 dmesg | tail

In certain proxies, dmesg does not show mount information in dmesg. In that case, on the destination VM run the following command, for example:

tune2fs -l /dev/sda1

Under file system features, you could see the features enabled on that partition. To turn off the metadata_csum feature, ensure that the FS passes the file system check. Then, using tune2fs, for example:

tune2fs -O ^metadata_csum