Avamar: /data01 partition fills up due to large files in /usr/local/avamar/var/mc/server_tmp and /usr/local/avamar/var/client directories
Summary: This article describes an issue where the Avamar Server /data01 partition fills up due to large files under mc/server_tmp and /usr/local/avamar/var/client.
Symptoms
This issue can cause abnormal consumption of the Operating System (OS) file system space on an Avamar Utility Node or single node grid.
The /data01 partition fills up due to many large, old files remaining under the /usr/local/avamar/var/mc/server_tmp and /usr/local/avamar/var/client directories.
For example:
The output below shows that the /usr/local/avamar/var/mc/server_tmp directory contains 2757 .tgz files, each typically sized 66 MB and dating from March 2015 to May 2012:
File List:
root@utility:/data01/avamar/var/mc/server_tmp/#: ls -lh | grep .tgz
total 124G
-rw-r--r-- 1 admin admin 568 May 29 2012 logfiles_1338330318448.tgz
-rw-r--r-- 1 admin admin 13K May 30 2012 logfiles_1338405771641.tgz
-rw-r--r-- 1 admin admin 27K May 31 2012 logfiles_1338492171660.tgz
-rw-r--r-- 1 admin admin 37K Jun 1 2012 logfiles_1338578571654.tgz
...
-rw------- 1 admin admin 66M Mar 5 08:00 logfiles_1425567601020.tgz
-rw------- 1 admin admin 66M Mar 5 08:01 logfiles_1425567636597.tgz
-rw------- 1 admin admin 66M Mar 5 08:01 logfiles_1425567671397.tgz
-rw------- 1 admin admin 66M Mar 6 08:00 logfiles_1425654001017.tgz
-rw------- 1 admin admin 66M Mar 6 08:01 logfiles_1425654034980.tgz
-rw------- 1 admin admin 66M Mar 6 08:01 logfiles_1425654066354.tgz
File Count:
root@utility:/data01/avamar/var/mc/server_tmp/#: ls -lah *.tgz | wc -l
2757
Similar behavior is seen in the /usr/local/avamar/var/client directory:
Disk Usage of the various directories:
root@utility:/data01/avamar/var/mc/server_tmp/#: du . --max-depth=1 -h
41M ./replicate#MOD-1424203497509
117M ./replicate#MOD-1425398832493
68M ./replicate#Default_Replication_Schedule-clientA-1425672000054
120M ./replicate#MOD-1424970951485
68M ./replicate#Default_Replication_Schedule-clients-1425672000043
84M ./replicate#MOD-1415133503160
83M ./replicate#MOD-1405628619144
76M ./replicate#MOD-1397661154157
84M ./replicate#MOD-1410385621358
84M ./replicate#MOD-1409689173400
82M ./replicate#MOD-1404848813520
80M ./replicate#MOD-1403122780236
85M ./replicate#MOD-1412609411124
85M ./replicate#Default_Replication_Schedule-client.company.com-1416686400025
84M ./replicate#MOD-1414680883852
84M ./replicate#MOD-1422031065131
84M ./replicate#Default_Replication_Schedule-client.company.com-1416168000034
84M ./replicate#Default_Replication_Schedule-client.company.com-1415649600090
84M ./replicate#Default_Replication_Schedule-client.company.com-1415736000026
83M ./replicate#MOD-1405023714683
84M ./replicate#MOD-1409689187100
83M ./replicate#MOD-1407444765227
85M ./replicate#Default_Replication_Schedule-client.customer.com-1417204800027
84M ./replicate#Default_Replication_Schedule-client.customer.com-1416254400119
84M ./replicate#MOD-1421863201463
79M ./replicate#MOD-1415914397772
84M ./replicate#MOD-1421963374539
82M ./replicate#MOD-1406913347993
85M ./replicate#Default_Replication_Schedule-client.company.com-1412967600091
85M ./replicate#MOD-1412027453487
83M ./replicate#MOD-1408995300732
84M ./replicate#MOD-1415919766438
84M ./replicate#Default_Replication_Schedule-client.company.com-1415217600043
85M ./replicate#Default_Replication_Schedule-client.company.com-1416772800076
85M ./replicate#Default_Replication_Schedule-client.company.com-1417723200042
82M ./replicate#MOD-1406928734963
85M ./replicate#Default_Replication_Schedule-client.company.com-1417291200088
84M ./replicate#Default_Replication_Schedule-client.company.com-1421870400078
85M ./replicate#Default_Replication_Schedule-client.company.com-1422475200057
85M ./replicate#Default_Replication_Schedule-client.company.com-1422388800083
85M ./replicate#Default_Replication_Schedule-client.company.com-1422907200043
84M ./replicate#MOD-1411414931158
83M ./replicate#Default_Replication_Schedule-client.company.com-1409079600053
0 ./replicate#Default_Replication_Schedule-client.company.com-1422993600018
84M ./replicate#Default_Replication_Schedule-client.company.com-1415390400099
82M ./replicate#MOD-1407960092776
82M ./replicate#Default_Replication_Schedule-client.company.com-1407783600074
84M ./replicate#Default_Replication_Schedule-client.company.com-1409684400085
83M ./replicate#MOD-1409160327852
84M ./replicate#Default_Replication_Schedule-client.company.com-1414695600099
85M ./replicate#Default_Replication_Schedule-client.company.com-1416859200015
85M ./replicate#Default_Replication_Schedule-client.company.com-1422561600041
85M ./replicate#MOD-1412693930999
Cause
The contents of /usr/local/avamar/var/mc/server_tmp/ and /usr/local/avamar/var/client/ are not subject to any retention or rotation policies.
Due to their large size, they can consume a great deal of space in the /data01 partition.
As /data01 approaches capacity, Avamar eventually becomes unusable.
Resolution
Create a Service Request with the Dell Technologies Avamar Support team to have excess log files removed and bring the system back into production.