Avamar: VMware clients fail with 10007 error

Summary: Avamar VMware clients may fail with 10007 error "Process 4018 (/usr/local/avamarclient/bin/avtar) finished Signal 9 exit - killed" due to lack of memory available on the proxy.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

VMware image backups fail with error code 10007 and avtar logs end suddenly without an error. 
avtar Info <8688>: Status 2016-05-22 05:32:31, 110.6 GB (194.6 MB, 0.17% new) 233MB 2% CPU 7277985792 input bytes, VMFiles/2/virtdisk-flat.vmdk
avtar Info <10698>: - Changeblock #92 offset=80951115776, bytes=268435456
avtar Info <10698>: - Changeblock #184 offset=85034139648, bytes=131072
avtar Info <8688>: Status 2016-05-22 05:47:38, 115.8 GB (194.6 MB, 0.17% new) 233MB 2% CPU 11380670464 input bytes, VMFiles/2/virtdisk-flat.vmdk
avtar Info <10698>: - Changeblock #276 offset=89156222976, bytes=262144
avtar Info <10698>: - Changeblock #368 offset=91305017344, bytes=131072
avtar Info <10698>: - Changeblock #460 offset=93791191040, bytes=2359296
avtar Info <8688>: Status 2016-05-22 06:02:45, 126.8 GB (194.6 MB, 0.16% new) 239MB 2% CPU 15121465344 input bytes, VMFiles/2/virtdisk-flat.vmdk
avtar Info <10698>: - Changeblock #552 offset=97889288192, bytes=655360
avtar Info <8688>: Status 2016-05-22 06:17:49, 131.5 GB (194.6 MB, 0.16% new) 249MB 2% CPU 19517751296 input bytes, VMFiles/2/virtdisk-flat.vmdk
<end of log>
 
The avvcbimage log shows the backup failing due to a 'Signal 9 exit - Killed' issue. 
avvcbimage Error <0000>: [IMG0010] PAX stream append ([vmclient] vmclient/vmclient01.vmdk) at sector offset 200669952 length 128 sectors=65536 failed (32)
avvcbimage Info <6688>: Process 4018 (/usr/local/avamarclient/bin/avtar) finished (Signal 9 exit - Killed)
avvcbimage Warning <6690>: CTL workorder "backup_5pm-1463868002597_c8a07604f6a5f0c5dcdfc871a5b3f5a8d4d77441" non-zero exit status 'Signal 9 exit - Killed'
avvcbimage Info <9772>: Starting graceful (staged) termination, PAX stream append failed (wrap-up stage)
avvcbimage Info <40654>: isExitOK()=157
avvcbimage Info <16022>: Cancel detected(miscellaneous error), isExitOK(0).
avvcbimage Info <9746>: bytes submitted was 102743015424
avvcbimage Error <0000>: [IMG0010] pax_container::endfile(virtdisk-flat.vmdk,102743015424) returned problem:32
avvcbimage Error <0000>: [IMG0010] pax_container::enddir returned problem:32
avvcbimage Info <40654>: isExitOK()=157
avvcbimage Info <16022>: Cancel detected(miscellaneous error), isExitOK(0).
avvcbimage Info <12032>: backup ([vmclient] vmclient/vmclient01.vmdk) terminated 135168.00 MB
avvcbimage Info <40654>: isExitOK()=157
avvcbimage Error <0000>: [IMG1003] Backup of [vmclient] vmclient/vmclient01.vmdk failed

avvcbimage Error <0000>: [IMG0011] Avtar exited with 'Signal 9 exit - Killed'
avvcbimage Info <16038>: Final summary, cancelled/aborted 1, snapview 0, exitcode 157: miscellaneous error

Cause

Issue Description

The /var/log/messages file on the Avamar proxy hosts shows that the Linux Out of Memory (OOM) killer is terminating Avamar processes (such as avtar, avagent, and avvcbimage) and VMware processes during the backup window. When these Avamar processes are killed, VMware image backups fail.

Verification

To identify OOM-killer activity, run the following command on the proxy:
grep "oom-killer" /var/log/messages
Example output:
proxy1 kernel: [4772630.893189] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

proxy1 kernel: [4773061.607740] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

proxy1 kernel: [4774722.062882] avtar.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

proxy1 kernel: [4775034.371673] avagent.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

proxy1 kernel: [4775229.334239] avvcbimage invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0

proxy1 kernel: [4778545.598976] vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
 
These entries confirm that the Avamar VMware proxy is under memory pressure and invoking the OOM killer during backup operations.
 

Root Cause

The Avamar VMware proxy is undersized in terms of allocated memory. In this example, the proxy has approximately 1 GB of total memory, which is insufficient for image-level backups.
 

Memory Verification

To check the available memory on the proxy, run:
free -m
Example output:
              total    used    free  shared  buffers  cached

Mem:            990     219     770       0       21      117

-/+ buffers/cache:       80     909

Swap:           139      44      95
 
This output confirms that the proxy has limited memory available, leading to frequent OOM-killer activity and VMware image backup failures.

Resolution

Increase the memory allocated to the Avamar proxy servers.
 
After increasing the memory, reboot the Avamar proxies. Rebooting is recommended because the Linux OOM killer may have terminated other processes due to previous memory pressure.

Affected Products

Avamar, Avamar Client for VMware

Products

Avamar
Article Properties
Article Number: 000059044
Article Type: Solution
Last Modified: 10 Jul 2026
Version:  8
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.