This post is more than 5 years old
1 Rookie
•
124 Posts
0
16355
Recyclable SAve sets not clearing out
Hi
I have noticed that my recyclable save sets are not being cleaned up even with nsrim -X
It sometimes goes higher but it never gets lower than 180 on my index volume.
I have checked to set scan off and weather is is set for scan needed. What is confusing is sometimes it goes up to 267 recyclable when i run a nsrim -X it will delete but never go below 180 is this a setting somewhere?
INDEX.001: 2702 GB used, 6132 save sets, appendable, 5952 recoverable save sets, 180 recyclable save sets.
nmc2
268 Posts
0
March 17th, 2015 03:00
This might happen, if you have any incremental or level backup dependency with previous backups.
Till all depended incremental/level backup gets into recyclable mode, these save sets wont get removed.
ble1
2 Intern
2 Intern
•
14.3K Posts
1
March 16th, 2015 15:00
Do you have scan flag set or not? You can see that via mminfo -avot -q scan -r volume.
As for recyclable save sets, check mminfo -avot -q volume=INDEX.001 -r 'name,copies,savetime(20),ssretent(20),level,ssflags,sumflags,clflags' and see if you get anything strange. If scan flags is not set (you should also have warning in daemon.raw about it whenever nsrim runs cleanup on volume) and if there no dependency on save set by any other one, you should see ssid removed. You can use this one volume as test by running nsrim -X -V INDEX.001. If these ssids would still not go away and you feel absolute sure they should, you can remove them manually.
What I have seen on AIX based servers, but mostly on HPUX, was that in 8.0.3.x nsrim would skip removal operations if device had read operations on it. Rather scary if you run replication with ddboost. Then in 8.0.4.x that was fixed and nsrim become a bit more agressive, but now replication was queued (normally only replication on shortly limited groups was affected like log backups which would run eg. every 1h). Not sure yet how this works in 8.1.x and 8.2.x as behavior also depends on number of clients which came as surprise to me (I noticed number of clients on HPUX server made difference in a way how speedy nsrim in and nsrmmdbd operations were which makes no sense at all as we are talking about number of clients and not number of ssids in mdb). So I guess, your mileage may vary also based on NW version and OS on which it runs.
Davidtgnome
66 Posts
0
June 2nd, 2015 07:00
Is there a way to determine WHY a saveset that is recyclable isn't clearing out. I've had a case open for 3 weeks now and they want me to do it manually. I'm talking savesets that SHOULD have expired over a year ago, and I know that we have more recent full backups of the client.
nsrim -X doesn't remove them, neither does nsrstage -C....
AIX 7.1 and Networker 8.2.0
ble1
2 Intern
2 Intern
•
14.3K Posts
0
June 2nd, 2015 12:00
8.2.0.what? 8.2.0.X has many Xs and there is SP1 with 3 subsequent patches. Anyway, I would so something like this:
mminfo -avot -N 'save set name' -c -r 'savetime(20),ssflags,sumflags,ssid,cloneid,level,ssbrowse(20),ssretent(20),clretent(20),ssflags,sumflags,clflags,volume'
I would first inspect the sequence to make sure nothing in dependency is there. Next, check on which volume it is. If this is disk device, make sure no scan flag is set for device. It is rather unlikely as you would have other ssids affected then, but worth checking. Anyway, if nothing is found, it is strange indeed - then (trying) to remove it manually would make sense. If you really really wish to know why, you can run debug and run nsrim for that specific client and specific volume where ssid is and get info why it is not processes (you may wish to check with support if you need debug binaries for it or placing nsrd into debug mode should be enough - however it will be intensive on all other ops).
ble1
2 Intern
2 Intern
•
14.3K Posts
0
June 2nd, 2015 13:00
Well, it certainly does for me
So, your problem might be platform or whatever specific. With that said, you should do more investigation, but if this is the only case or just one ssid then I see no point of discussing it - especially since you are not on latest patch level. If I were you, I would first patch the server. Next, I would do all necessary checks and lastly I would do debug check to see what is really going on. I had something like this in the past with 8.0.x, but there it was an issue which has been fixed since and it was bunch of ssids (it was related to concurrent cloning and cleanup running against same device which was not handled properly). I currently run 8.2.1.2+ and haven't seen anything like it (but bare in mind that on at least one occasion I have seen similar issue to yours which higher releases didn't fix, but those ssids made with higher release acted fine so I had to manually remove those done with previous version - it might be that same code bug is present in your release in which case newer version should fix it).
Davidtgnome
66 Posts
0
June 2nd, 2015 13:00
8.2.0.4. There are no obvious dependencies. nsrmm -d will remove the saveset... Usually...
I've had a case open for 3 weeks. I'm trying to convince EMC that their software REALLY should delete recyclable savesets.
Davidtgnome
66 Posts
0
June 2nd, 2015 17:00
I apologize if I gave the impression it was only happening to one saveset. It is happening to ALL of the savesets. Quite literally thousands of savesets.
Unless we can identify a specific bug that is fixed between 8.2.0.4 and the latest version 8.2.1.3 , then I will not upgrade because my last upgrade only proved that upgrading introduces bugs. 8.1.1.1 to 8.2.0.4. None of the release notes indicate such a bug is fixed. It is almost working now, my point was to let /u/boerbokrib know that yes others are having this problem. Since cleaning old savesets is a core part of the functionality of Networker, there is very much a point in discussing it further. I intend to do so with my engineer, and update this thread in the unlikely event someone from EMC actually fixes something.
ble1
2 Intern
2 Intern
•
14.3K Posts
0
June 3rd, 2015 02:00
Well, without doing tests and providing data it is all guess work. For example, I was earlier referring to:
187916 (NW158511) - nsrim -X does not delete expired saveset on DDBOOST devices
This is fixed in 8.1.1.5, but I do not have information about specifics here so I do not know if this is an issue only with 8.1.x or other code brunches are affected (and if so, is fix documented, committed and so on). Your support shoud know this. Semi-related, it might be that code fixes ssids written after this has been patched while earlier ones need manual removal (bold guess and perhaps too much pessimistic).
We also do not know if you device(s) have scan flag posted. You can do something like mminfo -av -q scan -r volume to see that. 8.1.x also had an issue where concurrent ssid manipulation operations on the same volume didn't work well which has been fixed in SP3 - not sure if that was ever an issue on 8.2.x or on which platform (for example, I can say that I have seen this on AIX 8.1.1.5 based, but on 8.2.1.x Linux based this is not an issue from what I can see). So, there is plenty what you can do and check and I agree support could do the same. Perhaps you can give them link to this forum thread if they do not know what to do next
ble1
2 Intern
2 Intern
•
14.3K Posts
0
June 3rd, 2015 04:00
Command works - it only means it didn't find any match which is good as it means no volume has scan flag set
Can you show us one save set (eg. /tmp) for one client which has this issue in following format:
mminfo -avot -q client= -N ' -r 'name,group,client,clientid,savetime(20),ssflags,sumflags,ssid,cloneid,level,ssbrowse(20),ssretent(20),clretent(20),ssflags,sumflags,clflags,volume'
Davidtgnome
66 Posts
0
June 3rd, 2015 04:00
The bug referenced is supposed to be resolved in versions after 8.1.1.5 if you run nsrim -X -v [volumename] while the volume is not being read from or written to. I'm running the command on a weekly basis now, and it's still leaving savesets on the ddboost devices. Oddly enough if you do a saveset report on the media page, it shows all of the savesets still being there, but if you right click on the disk volume and look at the savesets, only a few of them are still there.
We ran the fix for the bug over a month ago, and most of my retention times are a month. So even if it were fixed, those savesets would be recycling on their own.
your mminfo command to check if scan is needed doesn't actually work. 6095:mminfo: no matches found for the query
So we don't know if that's because it's not marked as scan needed, or if it's because there is a syntactical issue. The command reference manual seems to think it should work.
mminfo -m | grep -v T00 shows: (eliminating tapes which all start with the syntax T00
state volume written (%) expires read mounts capacity
ArchiveLogs.001 2364 GB 100% 07/03/15 23 TB 74 0 KB
CECArchiveLogsClone.001 6874 GB 100% 07/03/15 982 GB 59 0 KB
CECBootStrap.001 50 GB 100% 07/03/15 602 MB 24 0 KB
DBExports3YDDClone.001 20 TB 100% 06/03/18 25 GB 50 0 KB
Mac.001 41 TB 100% 07/03/15 615 GB 5 0 KB
NDMP.001 84 TB 100% 07/02/15 0 KB 7 0 KB
OracleDB.001 79 TB 100% 05/07/16 111 TB 82 0 KB
Special.001 786 MB 100% 08/13/15 26 GB 43 0 KB
dd1backups.001 575 TB 100% 05/02/16 400 TB 56 0 KB
So none of the flags are set. However support has indicated this shouldn't be an issue. and I don't know from this manual what the state would be if scan were needed.
The problem is we've checked all of the things in the forum so far. They have come up with nothing. Keep trying though.
Davidtgnome
66 Posts
0
June 3rd, 2015 05:00
THAT line was from Notepad... I've got 6280 savesets that should have expired by yesterday. Some have flags set to E, but several had flags set to E when NSRIM -X was last run, and they are still here.
Davidtgnome
66 Posts
0
June 3rd, 2015 05:00
Sure. This is a full saveset from 4/24/15 which should have expired on 5/25/15.
This one is an incremental whose dependent full was manually expired (and why I hate manually expiring savesets) from 4/14 which should have expired on 5/14. NSRIM was run on the volume on 5/22 and 5/29 at 13:30 hours.
Davidtgnome
66 Posts
0
June 3rd, 2015 05:00
Oh the forum really doesn't like lines pasted from excel
ble1
2 Intern
2 Intern
•
14.3K Posts
0
June 3rd, 2015 05:00
I think it has to be bug; only question is if this is one of those previously known and fixed in recent patches or not (I had similar with 8.0.3.x which has been fixed by 8.0.4.x and was caused by removal and cloning running at the same time so removal would just skip). Support needs to collect data and see debug output, but they might ask you to use latest patch level first (as you are n 8.2 code tree already, it makes sense to use latest patch level).
crazyrov
4 Operator
4 Operator
•
1.3K Posts
0
June 3rd, 2015 07:00
davidtgnome, there might be a possibility that the recycle mode on the volume is set to manual, can you changing it to auto and then run a nsrim -X volume again ?
To change the vole to recycle auto, Go to Media -> volumes -> right click on the volume in question -> select recycle -> select Auto and then click OK.