Avamar: VMware FLR for BTRFS Linux VMs Fails

A File Level Recovery (FLR) browse fails for SLES12SP5 BTRFS image backups on proxy 19.3 and earlier.

From the logs: /usr/local/avamarclient/bin/MountPoint/logs/

[MP - ERROR] 2021/04/15 07:34:23 TID=140166257708113 MID=218: ChangeBtrfsUuid Failed
[MP - FUNC] 2021/04/15 07:34:23 TID=140166257708113 MID=219:BtrfsMount::ChangeBtrfsUuid LEAVE
[MP - ERROR] 2021/04/15 07:34:23 TID=3153 MID=220: ChangeBtrfsUuid to /dev/loop3 Failed
[MP - FUNC] 2021/04/15 07:34:23 TID=3153 MID=221: BtrfsVmdkMount::MountDisks LEAVE
[MP - ERROR] 2021/04/15 07:34:23 TID=3153 MID=222: MountDisks Failed. Ret: 1048597
[MP - FUNC] 2021/04/15 07:34:23 TID=3153 MID=223: IVmdkMount::MountVmdk LEAVE
[MP - ERROR] 2021/04/15 07:34:23 TID=3153 MID=224: MountVmdk Failed. Ret: 1048597

The Linux Virtual Machine (VM) has BTRFS file system with Multi Device Disk:

VM-LIN>btrfs filesystem show

Label: 'ROOT'  uuid: 67a1ed99-1171-4b77-a0b0-XXXXXXXX
        Total devices 2 FS bytes used 120.76GiB
        devid    1 size 127.80GiB used 123.91GiB path /dev/sda3
        devid    2 size 20.00GiB used 1.16GiB path /dev/sdd

Label: none  uuid: 4f6fc552-9fb4-49ce-a248-YYYYYYYY
        Total devices 2 FS bytes used 29.53GiB
        devid    1 size 32.00GiB used 26.52GiB path /dev/sdb1
        devid    2 size 16.00GiB used 5.00GiB path /dev/sdb2
 
Label: none  uuid: 6f66a0a7-233d-4fb6-a23c-XXXXXXXXXX
        Total devices 1 FS bytes used 129.49GiB
        devid    1 size 160.00GiB used 146.02GiB path /dev/sdc

The FLR process takes a long time to change each BTRFS uuid using "btrfstune -f ":

[MP - FUNC] 2021/05/18 13:28:15 TID=140733193388507 MID=197: BtrfsMount::ChangeBtrfsUuid ENTER
[MP - DEBUG] 2021/05/18 13:28:15 TID=140733193388507 MID=198: sCmdline = /sbin/btrfstune -f -u /dev/loop3
..
[MP - DEBUG] 2021/05/18 13:32:53 TID=140733193388507 MID=200: sLine = New fsid: 2c1b273b-c01f-4cb8-8bf3-8a420f7b7416
[MP - DEBUG] 2021/05/18 13:32:53 TID=140733193388507 MID=201: sLine = Set superblock flag CHANGING_FSID
[MP - DEBUG] 2021/05/18 13:32:53 TID=140733193388507 MID=202: sLine = Change fsid in extents
[MP - DEBUG] 2021/05/18 13:32:53 TID=140733193388507 MID=203: sLine = Change fsid on devices
[MP - DEBUG] 2021/05/18 13:32:53 TID=140733193388507 MID=204: sLine = Clear superblock flag CHANGING_FSID
[MP - DEBUG] 2021/05/18 13:32:53 TID=140733193388507 MID=205: sLine = Fsid change finished

On Proxy 19.3 and older: The FLR process hangs and fails.

On Proxy 19.4: The new proxy OS (SLES12SP5) allows FLR to complete the mount but gives slow performance.

• FLR may take a long time to complete and can fill up filesystem if the VM has one or more btrfs filesystems with high metadata.

Short term Solution:

Add extra disk space on the Proxy,

• This prevents the Proxy from running out of disk space and allows the FLR to proceed. 
• However, the process will still be slow.

Long Term Solution:

A future proxy version addresses this issue.