Unsolved
This post is more than 5 years old
132 Posts
0
2095
Avamar backup "stuck/hung" on last directory?
Has anyone ever seen an Avamar file system backup get "stuck/hung" on the last directory of the last drive of the client file system?
FWIW, I checked and this is not a "new" directory, and it has backed up successfully before - in fact, this particular group has backed up successfully without issue other than it taking longer than other clients (one, because it has 25 million files, and two, because its performance is half of the other file system clients for some still unknown reason).
All comments/feedback appreciated - thanks.
pkumawat
8 Posts
0
February 12th, 2018 00:00
Hi,
Is it client based backup or Image backup?
From your statement, I feel you are doing client based backup. If backup is getting stuck at the end, it may be related to backup cache.
Check the logs and search for "cache" keywords. You can see logs for f_cache and p_cache. if you read it carefully you can find if your cache is properly sized or not.
Regards.
Pawan Kumawat
Avamar Exorcist
462 Posts
0
February 12th, 2018 07:00
By 'stuck' do you mean the log shows that avtar is processing one particular file until the backup reaches a "Timed out - end" or does the backup hang in some unexpected way?
If the former, I'd suggest opening a SR with Support and including the backup log
If the backup is hitting a "Timed Out - End" you should focus on the overall performance of the client. Typical ranges we see vary from 0.5 to 3 million files/hour with the vast majority being around 1 million. For a client with 25 million files you're going to need much faster than average performance for it to complete the backup within a regular, daily, window
I'd recommend opening an SR and providing the information in
KB 303720 - Information required by Support when investigating Avamar client backup performance issues
connolly
132 Posts
0
February 12th, 2018 16:00
The backup in question has overtime enabled, so it never reached any "Timed Out - End" state. It was "hung" in a somewhat unique state.
From what I could tell, it appeared that the actual backup, at least in terms of the files it was scanning, was "at the end of the directory/file list and it "should have been" wrapping up. There was a message along the lines of "writing to container.cdsf" after the last activity status line that had shown actual progress.
What was interesting was what was after that - additional status lines, repeated over and over again, listing the last directory in the overall directory structure of the client file system as specified in the dataset.
Relative to how the log looked, it appeared that the client was ready to "wrap up" the backup, but never received an "OK, I'm ready to wrap up too" message from the Avamar server.
I will consider logging an SR but we had something similar in the context of a much larger issue with the Avamar node itself today, and Support is working that issue, which may be related, so for now, I'm focusing on that.
connolly
132 Posts
0
February 12th, 2018 16:00
Client based backup, with Avamar v7.5, so using the dynamic paged caching.
Not outside the realm of possibility that it might have something to do with the caching, but not relative to any space issues in terms of allowing the cache file to grow - there was more than adequate space on the volume in question.
However, the cache file had already grown to almost 40GB and was still growing - not sure if there is any ceiling or plateau for the demand page cache size.
pkumawat
8 Posts
0
February 15th, 2018 21:00
Not sure if yours is same issue. I found one of my windows client backup was not finishing after completing 500+GBs. I was not getting any meaningful errors logs. So we restarted the client server. Now backup is running fine.
You mentioned that file cache size is almost 40 GB and still growing. Thats normal with demand paging. Did you check backup activity logs to find out the number of pages in f_cache2.dat? There is method to limit the the size of f_cache2.dat but it will cause backup to run longer.
Regards,
Pawan