Avamar: Linux client backup size is misleading due to '/var/log/lastlog' and sparse file behavior

The behavior described in this article can impact Avamar backup size reporting.
This is of interest where we must report on the amount of data protected in an Avamar backup. An example would be where a service provider invoices end users for the amount of client.

The reported size of a Linux client backup where /var/log/lastlog is included may be larger than the total disk space available to the client.

Example:
A Linux client is configured to back up only the /var/log directory which contains 39 MB of data.

root@linuxclient:~/#: du -hs /var/log
39M /var/log

The /var/log directory and the backup contain 'lastlog',  

root@linuxclient:/var/log/#: ls -ltrhs /var/log/lastlog
48K -rw-rw-r-- 1 root root 272G Apr 30 11:46 /var/log/lastlog

Note: lastlog is consuming 48 K of space on disk, whereas the regular reported size is 272 GB. This is sparse file. On completion of the backup, avtar claims to have backed-up 272 GB of data.
 

2015-04-30 12:08:09 avtar Info <5163>: Backup complete, wrapping-up session with Server
2015-04-30 12:08:10 avtar Info <5156>: Backup #494 timestamp
2015-04-30 12:21:58, 131 files, 16 directories, 272.0 GB (5 files, 6.830 KB, 0.00% new)
2015-04-30 12:08:10 avtar Info <7539>: Label "MOD-1430395268242", scheduled to expire after 06/29/15 (2015-06-29 12:01:07 UTC), none backup
2015-04-30 12:08:10 avtar Info <6083>: Backed-up 272.0 GB in 20.83 minutes: 783 GB/hour (377 files/hour)

The reported value for the amount of data that is backed up is larger than the size of the Linux file system.

root@linuxclient:~/#: df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 7.9G 2.4G 5.1G 32% /
/dev/sda1 122M 13M 103M 12% /boot
none 3.0G 0 3.0G 0%
/dev/shm /dev/sda3 1.5G 125M 1.3G 9% /var

The file /var/log/lastlog changes whenever a user (human or otherwise) logs in to the system. Avtar must process the entire file due to the sparse file handling limitations described in the article Avamar and sparse files.

Workaround 1: Exclude /var/log/lastlog from the backup. 

If the user must back up lastlog for auditing purposes, consider creating a scheduled daily crontab to send the output of lastlog to an output file,
For example,

lastlog > /var/log/lastlog_$(date +%d%m%Y).log

The regular file lastlog_<date> would get backed up but the original, sparse lastlog would not.


Workaround 2: Reduce the range of userIDs and shrink 'lastlog'.

Consider if such a high user ID is required. Is it likely to consume all the userIDs in the given range?
If a lower user id is equally functional, the following could be useful; 

• Edit /etc/passwd to ensure that the range of userIDs is as small as possible.
• Rename the lastlog file and then re-create it using touch lastlog. Set the ownership and permissions to be the same as the original file.
• Connect to the system with a user to update the newly created lastlog file.

root@linuxclient:/var/log/#: ls -ltrhs | grep lastlog
36K -rw-r----- 1 root tty 272G Apr 28 09:44 lastlog.old
8.0K -rw-r----- 1 root tty 143K Apr 28 09:50 lastlog

Note #1:
From a performance perspective, a large sparse file takes significant amount of time be backed up. Due to the nature of 'lastlog', the file is likely to get modified and must be fully processed every day. For more information about sparse files, see Avamar and sparse files.

Note #2:
The file /var/log/lastlog is a sparse file whose size depends on the range of userIDs which exists in the /etc/passwd file.

In the example provided above, /etc/passwd originally had userIDs which went as high as 502.  

The file was sparse, but moderately so. The reported file size was small.

root@linuxclient:/var/log/#: ls -ltrhs /var/log/lastlog
16K -rw-rw-r-- 1 root root 143K Apr 30 11:26 /var/log/lastlog

Adding a user with a large userID causes the size of lastlog to greatly increase in reported size. As illustrated below, the higher the userID, the more sparse (and the higher reported size) of lastlog.

Add a user with a high userID.

root@linuxclient:/var/log/#: useradd sparsetest -u 999999 

User sparsetest created.

root@linuxclient:/var/log/#: tail -1 /etc/passwd sparsetest:x:999999:999999::/home/sparsetest:/bin/bash

The lastlog file increases in reported size.

root@linuxclient:/var/log/#: ls -ltrhs /var/log/lastlog
32K -rw-rw-r-- 1 root root 279M Apr 30 11:34 /var/log/lastlog

Add a user with an even higher userID.

root@linuxclient:/var/log/#: useradd sparsetest2 -u 999999999

The lastlog file becomes even more sparse.

root@linuxclient:/var/log/#: ls -ltrhs /var/log/lastlog
48K -rw-rw-r-- 1 root root 272G Apr 30 11:46 /var/log/lastlog