Avamar: How to perform a Linux VMware File Level restore without using root user using sudo

Background on Avamar FLR:   
Avamar's File level restore is a agent less recovery solution that can be used to restore one or more guest file from a virtual machine image backup. The mechanism used to facilitate the file level restore is VMware tools and vSphere API guest operation.functions. The virtual machine image backup mounts the virtual disk (.vmdk) on the Avamar proxy during the browse. After files are select for restore they will be copied transfer file from proxy to target VM.    

FLR mandatory requirements:    

1. Target  virtual machine must be powered on.
2. Target virtual machine must have VMware tools installed and service must be running during restore operations. 
3. The Avamar administrator or Avamar backup/restore operation/user must have valid credentials on target guest (requirement of VMware guest operations SDK) and this credentials must be entered at start of the file level  restore operations.
4. The target guest user must have privileges to write location or operation will fail. For windows virtual machine, the restore VM user also must have privileges to write without UAC elevation, for Linux virtual machines, the restore VM must be able to write without sudo. 

FLR optional requirements: Required for Faster HTTPS transfers method. 

1. Target virtual machines must have TCP 443 access to the proxy virtual machines.
2. For Linux virtual machines, a modern wget application must be installed inside the virtual machine prior to restore operation.  

If operational requirements are not meet, the following warnings are seen prior to submitting job:   

If target VM cannot make a HTTPS connection to proxy:    



Inside the proxy: /usr/local/avamarclient/bin/logs/VSphereWebService.log, see wget non zero exit code 

[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398040: sGuestExeFullFilename = /usr/bin/wget --no-check-certificate --spider -t 1 -T 10 https://182proxy.burlington.lab:443/test
[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398041: sUnicodeExeDir = /usr/bin
[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398042: sExeFullPath = /usr/bin/wget
[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398043: sParameters = --no-check-certificate --spider -t 1 -T 10 https://182proxy.burlington.lab:443/test

[VWS - FUNC]  2020/02/24 15:56:2 TID=3662 MID=254389: VirtualMachineWs::CheckProcessStatus ENTER
[VWS - DEBUG] 2020/02/24 15:56:2 TID=3662 MID=254390: ullPid = 3053
[VWS - WARN]  2020/02/24 15:56:2 TID=140664473914958 MID=254428: CheckProcessStatus Succeeded
[VWS - DEBUG] 2020/02/24 15:56:2  TID=3662 MID=254441: StartProgramInGuest Succeeded. Ret: 4

Or
  
Linux without wget:     



Inside the proxy: /usr/local/avamarclient/bin/logs/VSphereWebService.log, see:   

[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398040: sGuestExeFullFilename = /usr/bin/wget --no-check-certificate --spider -t 1 -T 10 https://182proxy.burlington.lab:443/test
[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398041: sUnicodeExeDir = /usr/bin
[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398042: sExeFullPath = /usr/bin/wget
[VWS - DEBUG] 2020/02/24 15:56:20 TID=2403 MID=398043: sParameters = --no-check-certificate --spider -t 1 -T 10 https://182proxy.burlington.lab:443/test
[VWS - ERROR] 2020/02/24 15:56:20 TID=140363826202979 MID=398068: StartProgramInGuest Error: SOAP 1.1 fault: "":ServerFaultCode [no subcode]
"File /usr/bin/wget was not found"
Detail: <FileNotFoundFault xmlns="urn:vim25" xsi:type="FileNotFound"><file>/usr/bin/wget</file></FileNotFoundFault>

Or

[VWS - DEBUG] 2020/02/24 15:34:22 TID=2397 MID=395783: StartProgramInGuest Succeeded. Ret: 1

FLR Optional Feature: Restore Access Control List ( ACL) 

By default, this feature is not enabled.   
Result:    
Without ACL feature (default):    
All restored files from File Level Restore feature will be owned by the VM guest credentials user entered during FLR operations, and previous file permissions will not be set.

With ACL feature enabled:    
If user wants the restored files to have the original creation file ownership and permissions set on they restored file, they must enable this optional feature.



Result:    
During the restore operations, new files will be created as the FLR VM users, but at end of a restore, ACL workflow will run and set ownership and file permission back to original state. 

Inside the proxy: /usr/local/avamarclient/bin/logs/VSphereWebService.log, see:    

[VWS - FUNC] 2020/03/03 03:42:29 TID=140664473914958 MID=254066: VirtualMachineWs::ExecuteRestoreAcl ENTER
[VWS - FUNC] 2020/03/03 03:42:29 TID=140664473914958 MID=254067: VirtualMachineWs::ExecuteRestoreAcl RestoreAcl Input file created
[VWS - FUNC] 2020/03/03 03:42:29 TID=140664473914958 MID=254068: VirtualMachineWs::ExecuteRestoreAcl Input File transfer /usr/local/avamarclient/bin/7fc01f28-c871-4835-948d-2d8a14bd1d8b.xml to /tmp/7fc01f28-c871-4835-948d-2d8a14bd1d8b.xml
[VWS - FUNC] 2020/03/03 03:42:32 TID=140664473914958 MID=254140: VirtualMachineWs::ExecuteRestoreAcl Exe File transfer /usr/local/avamarclient/bin/RestoreAcl to /tmp/7fc01f28-c871-4835-948d-2d8a14bd1d8b.exe
[VWS - DEBUG] 2020/03/03 03:42:40 TID=3662 MID=254349: sGuestExeFullFilename = /tmp/7fc01f28-c871-4835-948d-2d8a14bd1d8b.exe /tmp/7fc01f28-c871-4835-948d-2d8a14bd1d8b.xml 0001
[VWS - FUNC] 2020/03/03 03:42:42 TID=3662 MID=254389: VirtualMachineWs::CheckProcessStatus ENTER
[VWS - DEBUG] 2020/03/03 03:42:42 TID=3662 MID=254390: ullPid = 3053
[VWS - WARN] 2020/03/03 03:42:47 TID=140664473914958 MID=254428: CheckProcessStatus Succeeded
[VWS - DEBUG] 2020/03/03 03:42:47 TID=3662 MID=254441: StartProgramInGuest Succeeded. Ret: 0
[VWS - FUNC] 2020/03/03 03:42:47 TID=140664473914958 MID=254519: VirtualMachineWs::ExecuteRestoreAcl LEAVE

In some Linux distributions like Ubuntu or Photon Os, the OS root users is not setup or configured, by default. Instead, Linux administrators rely on the sudo command to run elevation operations.  

However, due to VMware tools and VMware SDK, most operations (VM_LoginInGuest, CopyFileFromHostToGuest, and ListFilesInGuest) is not possible for VMware tools to execute sudo command. 

When restore operations are done with a normal user, the operation may fail due to insufficient target guest privileges.

For environments where Linux root user is disabled, not configured or due to security concerns cannot be shared, consider the follow resolution steps:     

Option 1. Restore to alternative location as non root user. 
Example:    
Source file:  /root/test  (restricted folder/file)
Destination file:  /home/flruser/ (not restricted location) 
VM Guest creations:  "flruser"  

Result:    
During FLR browse, the virtual disks are mounted to Avamar proxy and displayed in the user interface, this includes all folders and files, browse /root folder and select file1.txt, then choose 'restore everything to a different location' and either browse to or type in /home/flruser/ as the Absolution Destination.



Result:   
root/test file will be restored to /home/flruser/test 
Note, file will have FLR guest users permission. 

Option 2. Restore to alternative location with Restore ACL option using sudo feature. 
In Avamar 18.2, install proxy hotfix 318791 (or later)
In Avamar 19.2, install proxy hotfix 318791 (or later) 

These hotfixes add new 'enablesudouserrestore' feature that by default, is disabled.

Instruction to enable sudo feature:
On an Avamar proxy:

1. On the proxy, install hotfix above
2. Edit /usr/local/avamarclient/bin/config.xml, add/edit line (<enablesudouserrestore>1</enablesudouserrestore>) to the file
3. Restart FLR by command: service vmwareflr restart

On Every Linux virtual machine where the FLR feature will be on:    

1. Create dedicated FLR user:  example flruser.  For added security, shell can be disabled for this account
2. Copy /usr/local/avamarclient/bin/RestoreAcl binary from Avamar proxy to target VM using fixed location /usr/bin/RestoreACL and set permission to 755 using chmod

3. Modify /etc/sudoers to allow FLR user from step 1 to excute RestoreACL binary without password. 

Option 3: Restore to original location using sudo wget method with ACL. 

Note: 

• This option is only available when optional network HTTPS is available and when restore selection has >5 Mb or 10 files selected.
• Using Avamar GUI to browse live target VM locations with restricted folders location may fail, but explicitly typing in the 'Absolution Destination' will still be functional. 

1. Perform steps in Option 2 on proxy to enable ACL restore.
2. On Avamar proxy, edit /usr/local/avamarclient/bin/wget_linux_script.template 

3. Restart FLR by command: service vmwareflr restart
4. On every Linux virtual machine which FLR restore to original location will be used, modify /etc/sudoers to allow FLR user from step 1 to execute wget binary without password.