NVP vProxy: Virtual Machine FLR Consistently Failing With "Could not get the directory content…"
Summary: The File Level Recovery (FLR) for Windows Virtual Machine is consistently failing. The FLR mount is successful, but the browse fails for 3-4 folders with the message "Could not get the directory content unable to retrieve file list from VM (path) Access denied." ...
Symptoms
The NetWorker VMware Protection (NVP) solution is used to backup VMware Virtual Machines (VMs).
File Level Restore (FLR) mount operations complete successfully for a Windows VM; however, the browse operation fails to list directory contents reporting "Access Denied."
The vProxy applaince's /opt/emc/vproxy/runtime/logs/vflrd/mount-e28###a6-2##2-4##e-b##8-b900####2bd7.log reports:
YYYY-MM-DDTHH:MM:SSZ INFO: [29] ====End of output from agent mount. YYYY-MM-DDTHH:MM:SSZ INFO: [29] Successfully mounted VMDKs onto VM 'my_vmname' (vm-####) filesystem hierarchy at 'C:\Program Files (x86)\EMC\vProxy FLR Agent\flr\mountpoints\FLR65####027'. YYYY-MM-DDTHH:MM:SSZ INFO: [29] End of mount request processing; mountpoint is 'C:\Program Files (x86)\EMC\vProxy FLR Agent\flr\mountpoints\FLR65####027' (no error) YYYY-MM-DDTHH:MM:SSZ INFO: [29] Disconnected from session on vCenter 'myvcenter.com'. YYYY-MM-DDTHH:MM:SSZ INFO: [29] Adding FlrBrowseSessions session 'cb###4c0-7##0-4##9-b##1-711#####a983'
The vProxy applaince's /opt/emc/vproxy/runtime/logs/vflrd/browse-cb0###c0-7##0-4##9-b##1-7116#####983.log reports:
YYYY-MM-DDTHH:MM:SSZ ERROR: [29] FLR:browse_files: Unable to perform FLR Agent operation 'browse_files' on VM IP Address: ##.##.##.## (vm-####). YYYY-MM-DDTHH:MM:SSZ ERROR: [29] Unable to retrieve file list from ##.##.##.## (vm-####) path 'F:\DFS-SHARE\Administration\MY_FOLDER' offset 0. YYYY-MM-DDTHH:MM:SSZ ERROR: [29] Could not get directory contents: Unable to retrieve file list from ##.##.##.## (vm-####) path '#:\DFS-SHARE\Administration\MY_FOLDER' offset 0. Unable to perform FLR Agent operation 'browse_files' on VM ##.##.##.## (vm-####). Cannot stat 'C:\Program Files (x86)\EMC\vProxy FLR Agent\flr\mountpoints\FLR65####027\02-#$\DFS-SHARE\Administration\MY_FOLDER': CreateFile C:\Program Files (x86)\EMC\vProxy FLR Agent\flr\mountpoints\FLR65####027\02-F$\DFS-SHARE\Administration\EXE_ADMIN: Access is denied. YYYY-MM-DDTHH:MM:SSZ ERROR: [29] FLR:browse_files: Cannot stat 'C:\Program Files (x86)\EMC\vProxy FLR Agent\flr\mountpoints\FLR65####027\02-#$\DFS-SHARE\Administration\MY_FOLDER': CreateFile C:\Program Files (x86)\EMC\vProxy FLR Agent\flr\mountpoints\FLR65####027\02-#$\DFS-SHARE\Administration\MY_FOLDER: Access is denied.
Cause
The VMs that failed for FLR with the above symptoms are using Microsoft Distributed File System (DFS).
Microsoft DFS is a Windows file system feature that enables users to create a namespace of shared directories that are physically distributed across a network. DFS allows users to logically organize distributed directories, providing centralized access to files in various locations.
To know more about Microsoft DFS, see Microsoft article DFS Namespaces overview.
Resolution
Workaround:
- Initiate File Level Recovery (FLR) from NetWorker Management Console (NMC) UI.
- Mount File Level Recovery (FLR) in NetWorker Management Console (NMC) UI Recovery Page Title: Mount the save set for file level recovery
- Leave the File Level Recovery (FLR) Mountpoint in mounted state in NetWorker Management Console (NMC) UI.
- Use Windows Explorer to copy and paste the directory content on the Distributed File System (DFS) Target Virtual Machine (VM).
Additional Information
If you are observing this error and Windows DFS is not used, see: NVP-vProxy: Windows VM FLR Fails With "Error while browsing: 200: Error received from vProxy "could not get directory contents:"
An alternate FLR solution is outlined in: NVP vProxy: How To Perform A File Level Recovery When FLR Is Failing or Not Supported By the VM