Regularly getting Volume marked full when cloning to DataDomain
Hi,
all of our Networker Policies include a step for cloning the completed backups to a second Data Domain. Primary System is also a DataDomain. We're using DDBoost over LAN. FC is not in use.
We're getting "Volume marked full" and "Verify Label on unload" messages for the Volume mounted in the DD used as clone target. Triggering networker to wait for another volume in the clone pool.
But the usage of the DD pools is constantly between 50-60 %. So they we're never full. Even when Networker terminates the affected nsrclone jobs, subsequently running clones from following policies succeed without troubleshooting. If Networker doesn't terminate the affected job, it won't help to reset the full flag on the DD clone volume. Networker still waits for another volume in the pool, leaving all the currently running nsrclone jobs stuck indefinately. We then have to terminate them by "kill -9". Then the volume wait event goes away leading to the same result as when Networker had terminated them itself.
Can this somehow be the result of concurrency between clone jobs trying to duplicate the same savesets? But why we're then getting the volume marked full messages?
I am just guessing now but this seems to be a timing issue. It could also well be a 'compatibility issue' among NW and DDOS versions. The daemon.raw file should contain more information which might help you.
But I would get Dell/EMC support involved to solve this issue.
I'm seeing this behaviour since using networker 18. Now we're on v19.2.1. DDOS is 6.2.0.30.
Already openend a SR. But as usual it's very difficult to even make the 1st level understand what the problem is. They're suggesting impractical workarounds.
daemon.raw contains only the messages I've already described. No further information about the root cause of the volume marked full incidents. I guess we have to implement some sort of debug logging for getting that. But without proper filtering and guidance one can bring the backup to a complete halt.
This kind of behavior can be seen if there are many errors related to the respective drive. Can you create a new device and start using it for a few days? Also, make sure that the DD has a stable connection.
bingo.1
1 Rookie
•
2430 Posts
487
0
Posted June 17th, 2020 00:00
I am just guessing now but this seems to be a timing issue. It could also well be a 'compatibility issue' among NW and DDOS versions. The daemon.raw file should contain more information which might help you.
But I would get Dell/EMC support involved to solve this issue.