Avamar Client: IDPA Linux Client Image Backup Completing with Exception due to Quiescing problem
Summary: Linux Image Backup Clients failing with Backup Completed with exception Error code 10020
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
The Linux image backup for the IDPA system is "Completing with Exception" due to the quiescing problem.
This example the vm was the internal ACM vm that comes with IDPA solution
From the vmware linux image log of the backup the following can be seen:
avvcbimage Info <40726>: VM Config Info: VMX version 4(F), ctk enabled(T), fault tolerant (F), Guest OS SUSE Linux Enterprise 12 (64-bit), Cloud UUID = NoUUID. version=9
avvcbimage Info <40727>: VM Guest Info: GuestOS Family = linuxGuest, Guest Full Name = SUSE Linux Enterprise 12 (64-bit), Guest OS State = running, Tools Running = guestToolsRunning, Tools Version Status = guestToolsUnmanaged
avvcbimage Info <42650>: Creating snapshot 'Avamar-<epoch_time><client_id>', quiesceFS=1
avvcbimage Info <41136>: SnapshotManager: createSnapshot for vmMoref = vm-21
avvcbimage Info <42182>: SnapshotManager: POST to /snapshotmanager/vm/vm-21/action/createSnapshot, payload({
"snapshotName": "Avamar-<epoch_time><client_id>",
"snapshotDesc": "Created by Avamar workorder Management_VM_Backup-Management_VM_Backup-<epoch > on <date>",
"quiesceFileSystem": "true"
...
avvcbimage Warning <19733>: vSphere Task failed (quiesce, snapshot error=45): 'An error occurred while saving the snapshot: Failed to quiesce the virtual machine.'.
avvcbimage Error <17775>: Snapshot 'Avamar-<epoch_time><client_id> ' creation for VM '[vsanDatastore] DataProtection-ACM/DataProtection-ACM.vmx' task creation encountered a quiesce problem
avvcbimage Warning <40710>: The VM could not be quiesced prior to snapshot creation and this backup will not be used as a base for subsequent CBT backups if successful.
avvcbimage Warning <40711>: Consider using the pre-freeze and post-thaw scripts to facilitate VM quiescing
avvcbimage Warning <41146>: This error might caused by the pre-freeze script returning a non-zero error code.
...
After 1st snapshot task fails, the backup proceeds by taking second snapshot but with quiesce feature disabled.
avvcbimage Info <14631>: VM Disk Consolidation for '[vsanDatastore] DataProtection-ACM/DataProtection-ACM.vmx' task still in progress, sleep for 3 sec
avvcbimage Info <42650>: Creating snapshot 'Avamar-<epoch_time><client_id>', quiesceFS=0
avvcbimage Info <41136>: SnapshotManager: createSnapshot for vmMoref = vm-21
avvcbimage Info <42182>: SnapshotManager: POST to /snapshotmanager/vm/vm-21/action/createSnapshot, payload({
"snapshotName": "Avamar-<epoch_time><client_id>",
"snapshotDesc": "Created by Avamar workorder Management_VM_Backup-Management_VM_Backup-<epoch> on <date>",
"quiesceFileSystem": "false"
The backup finishes but shows as completed with exception ( code 10020)
In the workorder log (.alg) the following is seen:
2019-04-03 00:12:42 Management_VM_Backup-Management_VM_Backup-<epoch>Warning <0000>: Workorder "Management_VM_Backup-Management_VM_Backup-epoch" non-zero exit status 'code 170: completed with errors, client log should be examined'
Cause
1. To see what is causing the quiescing to fail, set the debug logging for the vmtools.
Following steps in https://kb.vmware.com/s/article/1007873 to set the debug logging on the guest operating system and then restart vmware tools.
2. Do a new backup to recreate quiesce failure.
3. Review debug tools logs ( quiesce information should be in the vmsvc log and have a [vmbackup] prefix ).
In this example we can the issue was was caused by a partially deleted /var/log/brs directory. This directory could not be fully removed due background process still had directory open. This partially deleted folder will prevent Linux filesystem freeze with ioctls functions.
[ debug] [vmbackup] Submitted backup start task.
[ debug] [vmbackup] *** VmBackupSyncDriverStart
[ debug] [vmsvc] SyncDriver: Skipping remote filesystem, name=systemd-1, mntpt=/proc/sys/fs/binfmt_misc.
[ debug] [vmsvc] SyncDriver: Calling backend 0.
[ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
[ debug] [vmsvc] SyncDriver: opening path '/var/lib/named/proc'.
[ debug] [vmsvc] SyncDriver: freezing path '/var/lib/named/proc'.
[ debug] [vmsvc] SyncDriver: freeze on '/var/lib/named/proc' returned: 95 (Operation not supported)
[ debug] [vmsvc] SyncDriver: opening path '/var/log/logbundles'.
[ debug] [vmsvc] SyncDriver: freezing path '/var/log/logbundles'.
[ debug] [vmsvc] SyncDriver: successfully froze '/var/log/logbundles'.
[ debug] [vmsvc] SyncDriver: opening path '/var/log/brs (deleted)'.
[ debug] [vmsvc] SyncDriver: failed to open '/var/log/brs (deleted)': 2 (No such file or directory)
[ warning] [vmbackup] Error freezing filesystems.
Following steps in https://kb.vmware.com/s/article/1007873 to set the debug logging on the guest operating system and then restart vmware tools.
2. Do a new backup to recreate quiesce failure.
3. Review debug tools logs ( quiesce information should be in the vmsvc log and have a [vmbackup] prefix ).
In this example we can the issue was was caused by a partially deleted /var/log/brs directory. This directory could not be fully removed due background process still had directory open. This partially deleted folder will prevent Linux filesystem freeze with ioctls functions.
[ debug] [vmbackup] Submitted backup start task.
[ debug] [vmbackup] *** VmBackupSyncDriverStart
[ debug] [vmsvc] SyncDriver: Skipping remote filesystem, name=systemd-1, mntpt=/proc/sys/fs/binfmt_misc.
[ debug] [vmsvc] SyncDriver: Calling backend 0.
[ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
[ debug] [vmsvc] SyncDriver: opening path '/var/lib/named/proc'.
[ debug] [vmsvc] SyncDriver: freezing path '/var/lib/named/proc'.
[ debug] [vmsvc] SyncDriver: freeze on '/var/lib/named/proc' returned: 95 (Operation not supported)
[ debug] [vmsvc] SyncDriver: opening path '/var/log/logbundles'.
[ debug] [vmsvc] SyncDriver: freezing path '/var/log/logbundles'.
[ debug] [vmsvc] SyncDriver: successfully froze '/var/log/logbundles'.
[ debug] [vmsvc] SyncDriver: opening path '/var/log/brs (deleted)'.
[ debug] [vmsvc] SyncDriver: failed to open '/var/log/brs (deleted)': 2 (No such file or directory)
[ warning] [vmbackup] Error freezing filesystems.
Resolution
Closer examination of the /var/log/brs the directory with can see problems opening this directory from other commands. like lsof and stat also show issue.
IDPA-ACM:/var/log # lsof|grep brs
lsof: WARNING: can't stat() xfs file system /var/log/brs (deleted)
Output information may be incomplete.
IDPA-ACM:/var/log # ls -al
drwxr-xr-x 0 root root 6 Feb 27 2018 brs
The ACM Virtual Machine was rebooted to kill background acm process. After reboot the /var/log/brs directory was recreated by ACM process and the backup was able to complete without further quiesce issues
IDPA-ACM:/var/log # ls -al
drwxr-xr-x 2 root root 6 Apr 15 10:30 brs
IDPA-ACM:/var/log # lsof|grep brs
lsof: WARNING: can't stat() xfs file system /var/log/brs (deleted)
Output information may be incomplete.
IDPA-ACM:/var/log # ls -al
drwxr-xr-x 0 root root 6 Feb 27 2018 brs
The ACM Virtual Machine was rebooted to kill background acm process. After reboot the /var/log/brs directory was recreated by ACM process and the backup was able to complete without further quiesce issues
IDPA-ACM:/var/log # ls -al
drwxr-xr-x 2 root root 6 Apr 15 10:30 brs
Additional Information
Ensure to follow the above KB to remove debug logging for the vmtools on the client and restart the vmtools service.
Affected Products
Avamar ClientProducts
Avamar Client, PowerProtect Data Protection SoftwareArticle Properties
Article Number: 000056249
Article Type: Solution
Last Modified: 11 Oct 2024
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.