This post is more than 5 years old
1 Rookie
•
14 Posts
0
10564
Policy Hung, Will not stop without stopping services
I recently upgraded my backup servers to Networker 9.1.1.5 from 9.1.1.4. Since the upgrade I have noticed that sometimes the policy completes but does not automatically stop itself, kinda like it is hung. This happens on all the Networker servers not just one. If I try to manually stop the policy it does not stop, I have to stop the Networker services to get the policy to stop. I added a timeout to workflow as well but it didn't help. Any suggestions?
12/29/2017 9:00:06 PM Action backup traditional 'Backup' has initialized as 'backup action job' with job id 832897
12/29/2017 9:00:10 PM fmssspvideosrv:C:\ level=full
12/29/2017 9:00:10 PM fmssspvideosrv:DISASTER_RECOVERY:\ level=full
12/29/2017 9:00:10 PM fmssiocngp:All level=full
12/29/2017 9:00:10 PM romeniagara:All level=full
12/29/2017 9:00:10 PM fmsspiint:All level=full
12/29/2017 9:00:10 PM romequadlogic:All level=full
12/29/2017 9:00:10 PM fmsssims:All level=full
12/29/2017 9:00:10 PM fmsseadvisor.ndc.nasa.gov:All level=full
12/29/2017 9:00:10 PM fmsspi:C:\ level=full
12/29/2017 9:00:10 PM fmsspi:DISASTER_RECOVERY:\ level=full
12/29/2017 9:00:10 PM romerbmware:All level=full
12/29/2017 9:00:10 PM romesummit:All level=full
12/29/2017 9:00:10 PM fmssanomalert:All level=full
12/29/2017 9:00:11 PM romestruxure:All level=full
12/29/2017 9:00:11 PM fmsspidb:C:\ level=full
12/29/2017 9:00:11 PM fmsspidb:DISASTER_RECOVERY:\ level=full
12/29/2017 9:00:11 PM fmssidm:All level=full
12/29/2017 9:00:11 PM fmssspsql:C:\ level=full
12/29/2017 9:00:11 PM fmssspsql:DISASTER_RECOVERY:\ level=full
12/29/2017 9:00:11 PM fmssspbackup.ndc.nasa.gov:C:\ level=full
12/29/2017 9:00:11 PM fmssspbackup.ndc.nasa.gov:DISASTER_RECOVERY:\ level=full
12/29/2017 9:00:11 PM romesql:C:\ level=full
12/29/2017 9:00:11 PM romesql:D:\ level=full
12/29/2017 9:00:11 PM romesql:DISASTER_RECOVERY:\ level=full
12/29/2017 9:00:11 PM fmssmissl:All level=full
12/29/2017 9:00:11 PM Action backup traditional will run up to 5 jobs in parallel
12/29/2017 9:00:11 PM Policy 'Backup', workflow 'Production 1', action 'Backup', group 'Production 1'.
12/29/2017 9:00:11 PM Starting action backup traditional 'Backup', which has 18 clients.
12/29/2017 9:00:11 PM fmssspvideosrv:savefs started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssspvideosrv -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:11 PM fmssiocngp:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssiocngp -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM romeniagara:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romeniagara -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM fmsspiint:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsspiint -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM romequadlogic:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romequadlogic -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM fmsssims:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsssims -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM fmsseadvisor.ndc.nasa.gov:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsseadvisor.ndc.nasa.gov -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM fmsspi:savefs started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsspi -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:11 PM romerbmware:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romerbmware -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM romesummit:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romesummit -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM fmssanomalert:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssanomalert -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM romestruxure:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romestruxure -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM fmsspidb:savefs started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsspidb -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:11 PM fmssidm:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssidm -g "Production 1" -p -l full -R -v
12/29/2017 9:00:11 PM fmssspsql:savefs started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssspsql -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:11 PM fmssspbackup.ndc.nasa.gov:savefs started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssspbackup.ndc.nasa.gov -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:11 PM romesql:savefs started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romesql -g "Production 1" -p -l full -R -v -F "C:\\" "D:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:11 PM fmssmissl:All started
12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssmissl -g "Production 1" -p -l full -R -v
12/29/2017 9:00:16 PM Group Production 1 waiting for 18 jobs (0 awaiting restart) to complete.
12/29/2017 9:00:40 PM romequadlogic:C:\
12/29/2017 9:00:40 PM level=full, vers=pools, p=4
12/29/2017 9:00:40 PM romequadlogic:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:00:40 PM level=full, vers=pools, p=4
12/29/2017 9:00:40 PM romequadlogic:DISASTER_RECOVERY:\
12/29/2017 9:00:40 PM level=full, vers=pools, p=4
12/29/2017 9:00:40 PM romequadlogic:All succeeded.
12/29/2017 9:00:40 PM Production 1 romequadlogic:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832902.log for command output
12/29/2017 9:00:40 PM Probe result returned DISASTER_RECOVERY save set for client romequadlogic.
12/29/2017 9:00:40 PM romequadlogic:pseudo_saveset started
12/29/2017 9:00:40 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romequadlogic -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:43 PM fmsspiint:C:\
12/29/2017 9:00:43 PM level=full, vers=pools, p=4
12/29/2017 9:00:43 PM fmsspiint:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:00:43 PM level=full, vers=pools, p=4
12/29/2017 9:00:43 PM fmsspiint:DISASTER_RECOVERY:\
12/29/2017 9:00:43 PM level=full, vers=pools, p=4
12/29/2017 9:00:43 PM fmsspiint:All succeeded.
12/29/2017 9:00:43 PM Production 1 fmsspiint:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832901.log for command output
12/29/2017 9:00:43 PM Probe result returned DISASTER_RECOVERY save set for client fmsspiint.
12/29/2017 9:00:43 PM fmsspiint:pseudo_saveset started
12/29/2017 9:00:43 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspiint -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:50 PM fmssspvideosrv:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:00:50 PM level=full, vers=pools, p=4
12/29/2017 9:00:50 PM fmssspvideosrv:C:\
12/29/2017 9:00:50 PM level=full, vers=pools, p=4
12/29/2017 9:00:50 PM fmssspvideosrv:DISASTER_RECOVERY:\
12/29/2017 9:00:50 PM level=full, vers=pools, p=4
12/29/2017 9:00:50 PM fmssspvideosrv:savefs succeeded.
12/29/2017 9:00:50 PM Production 1 fmssspvideosrv:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832898.log for command output
12/29/2017 9:00:50 PM Probe result returned DISASTER_RECOVERY save set for client fmssspvideosrv.
12/29/2017 9:00:50 PM fmssspvideosrv:pseudo_saveset started
12/29/2017 9:00:50 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssspvideosrv -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:52 PM fmsssims:C:\
12/29/2017 9:00:52 PM level=full, vers=pools, p=4
12/29/2017 9:00:52 PM fmsssims:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:00:52 PM level=full, vers=pools, p=4
12/29/2017 9:00:52 PM fmsssims:DISASTER_RECOVERY:\
12/29/2017 9:00:52 PM level=full, vers=pools, p=4
12/29/2017 9:00:52 PM fmsssims:All succeeded.
12/29/2017 9:00:52 PM Production 1 fmsssims:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832903.log for command output
12/29/2017 9:00:52 PM Probe result returned DISASTER_RECOVERY save set for client fmsssims.
12/29/2017 9:00:52 PM fmsssims:pseudo_saveset started
12/29/2017 9:00:52 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsssims -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:54 PM romeniagara:C:\
12/29/2017 9:00:54 PM level=full, vers=pools, p=4
12/29/2017 9:00:54 PM romeniagara:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:00:54 PM level=full, vers=pools, p=4
12/29/2017 9:00:54 PM romeniagara:DISASTER_RECOVERY:\
12/29/2017 9:00:54 PM level=full, vers=pools, p=4
12/29/2017 9:00:54 PM romeniagara:All succeeded.
12/29/2017 9:00:54 PM Production 1 romeniagara:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832900.log for command output
12/29/2017 9:00:54 PM Probe result returned DISASTER_RECOVERY save set for client romeniagara.
12/29/2017 9:00:54 PM romeniagara:pseudo_saveset started
12/29/2017 9:00:54 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romeniagara -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:C:\
12/29/2017 9:00:55 PM level=full, vers=pools, p=4
12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:00:55 PM level=full, vers=pools, p=4
12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:DISASTER_RECOVERY:\
12/29/2017 9:00:55 PM level=full, vers=pools, p=4
12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:All succeeded.
12/29/2017 9:00:55 PM Production 1 fmsseadvisor.ndc.nasa.gov:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832904.log for command output
12/29/2017 9:00:55 PM Probe result returned DISASTER_RECOVERY save set for client fmsseadvisor.ndc.nasa.gov.
12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:pseudo_saveset started
12/29/2017 9:00:55 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsseadvisor.ndc.nasa.gov -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:00 PM fmsspi:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:00 PM level=full, vers=pools, p=4
12/29/2017 9:01:00 PM fmsspi:C:\
12/29/2017 9:01:00 PM level=full, vers=pools, p=4
12/29/2017 9:01:00 PM fmsspi:DISASTER_RECOVERY:\
12/29/2017 9:01:00 PM level=full, vers=pools, p=4
12/29/2017 9:01:00 PM fmsspi:savefs succeeded.
12/29/2017 9:01:00 PM Production 1 fmsspi:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832905.log for command output
12/29/2017 9:01:00 PM Probe result returned DISASTER_RECOVERY save set for client fmsspi.
12/29/2017 9:01:00 PM fmsspi:pseudo_saveset started
12/29/2017 9:01:00 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspi -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:08 PM fmssanomalert:C:\
12/29/2017 9:01:08 PM level=full, vers=pools, p=4
12/29/2017 9:01:08 PM fmssanomalert:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:08 PM level=full, vers=pools, p=4
12/29/2017 9:01:08 PM fmssanomalert:DISASTER_RECOVERY:\
12/29/2017 9:01:08 PM level=full, vers=pools, p=4
12/29/2017 9:01:08 PM fmssanomalert:All succeeded.
12/29/2017 9:01:08 PM Production 1 fmssanomalert:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832908.log for command output
12/29/2017 9:01:08 PM Probe result returned DISASTER_RECOVERY save set for client fmssanomalert.
12/29/2017 9:01:08 PM fmssanomalert:pseudo_saveset started
12/29/2017 9:01:08 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssanomalert -a DIRECT_ACCESS=No -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:13 PM romesummit:C:\
12/29/2017 9:01:13 PM level=full, vers=pools, p=4
12/29/2017 9:01:13 PM romesummit:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:13 PM level=full, vers=pools, p=4
12/29/2017 9:01:13 PM romesummit:DISASTER_RECOVERY:\
12/29/2017 9:01:13 PM level=full, vers=pools, p=4
12/29/2017 9:01:13 PM romesummit:All succeeded.
12/29/2017 9:01:13 PM Production 1 romesummit:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832907.log for command output
12/29/2017 9:01:13 PM Probe result returned DISASTER_RECOVERY save set for client romesummit.
12/29/2017 9:01:13 PM romesummit:pseudo_saveset started
12/29/2017 9:01:13 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romesummit -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:15 PM fmssiocngp:C:\
12/29/2017 9:01:15 PM level=full, vers=pools, p=4
12/29/2017 9:01:15 PM fmssiocngp:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:15 PM level=full, vers=pools, p=4
12/29/2017 9:01:15 PM fmssiocngp:DISASTER_RECOVERY:\
12/29/2017 9:01:15 PM level=full, vers=pools, p=4
12/29/2017 9:01:15 PM fmssiocngp:All succeeded.
12/29/2017 9:01:15 PM Production 1 fmssiocngp:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832899.log for command output
12/29/2017 9:01:15 PM Probe result returned DISASTER_RECOVERY save set for client fmssiocngp.
12/29/2017 9:01:15 PM fmssiocngp:pseudo_saveset started
12/29/2017 9:01:15 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssiocngp -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:16 PM romerbmware:C:\
12/29/2017 9:01:16 PM level=full, vers=pools, p=4
12/29/2017 9:01:16 PM romerbmware:E:\
12/29/2017 9:01:16 PM level=full, vers=pools, p=4
12/29/2017 9:01:16 PM romerbmware:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:16 PM level=full, vers=pools, p=4
12/29/2017 9:01:16 PM romerbmware:DISASTER_RECOVERY:\
12/29/2017 9:01:16 PM level=full, vers=pools, p=4
12/29/2017 9:01:16 PM romerbmware:All succeeded.
12/29/2017 9:01:16 PM Production 1 romerbmware:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832906.log for command output
12/29/2017 9:01:16 PM Probe result returned DISASTER_RECOVERY save set for client romerbmware.
12/29/2017 9:01:16 PM romerbmware:pseudo_saveset started
12/29/2017 9:01:16 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romerbmware -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "E:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:21 PM romestruxure:C:\
12/29/2017 9:01:21 PM level=full, vers=pools, p=4
12/29/2017 9:01:21 PM romestruxure:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:21 PM level=full, vers=pools, p=4
12/29/2017 9:01:21 PM romestruxure:DISASTER_RECOVERY:\
12/29/2017 9:01:21 PM level=full, vers=pools, p=4
12/29/2017 9:01:21 PM romestruxure:All succeeded.
12/29/2017 9:01:21 PM Production 1 romestruxure:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832909.log for command output
12/29/2017 9:01:21 PM Probe result returned DISASTER_RECOVERY save set for client romestruxure.
12/29/2017 9:01:21 PM romestruxure:pseudo_saveset started
12/29/2017 9:01:21 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romestruxure -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:22 PM fmsspidb:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:22 PM level=full, vers=pools, p=4
12/29/2017 9:01:22 PM fmsspidb:C:\
12/29/2017 9:01:22 PM level=full, vers=pools, p=4
12/29/2017 9:01:22 PM fmsspidb:DISASTER_RECOVERY:\
12/29/2017 9:01:22 PM level=full, vers=pools, p=4
12/29/2017 9:01:22 PM fmsspidb:savefs succeeded.
12/29/2017 9:01:22 PM Production 1 fmsspidb:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832910.log for command output
12/29/2017 9:01:22 PM Probe result returned DISASTER_RECOVERY save set for client fmsspidb.
12/29/2017 9:01:22 PM fmsspidb:pseudo_saveset started
12/29/2017 9:01:22 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspidb -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:24 PM fmssidm:C:\
12/29/2017 9:01:24 PM level=full, vers=pools, p=4
12/29/2017 9:01:24 PM fmssidm:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:24 PM level=full, vers=pools, p=4
12/29/2017 9:01:24 PM fmssidm:DISASTER_RECOVERY:\
12/29/2017 9:01:24 PM level=full, vers=pools, p=4
12/29/2017 9:01:24 PM fmssidm:All succeeded.
12/29/2017 9:01:24 PM Production 1 fmssidm:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832911.log for command output
12/29/2017 9:01:24 PM Probe result returned DISASTER_RECOVERY save set for client fmssidm.
12/29/2017 9:01:24 PM fmssidm:pseudo_saveset started
12/29/2017 9:01:24 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssidm -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:27 PM fmssspsql:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:27 PM level=full, vers=pools, p=4
12/29/2017 9:01:27 PM fmssspsql:C:\
12/29/2017 9:01:27 PM level=full, vers=pools, p=4
12/29/2017 9:01:27 PM fmssspsql:DISASTER_RECOVERY:\
12/29/2017 9:01:27 PM level=full, vers=pools, p=4
12/29/2017 9:01:27 PM fmssspsql:savefs succeeded.
12/29/2017 9:01:27 PM Production 1 fmssspsql:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832912.log for command output
12/29/2017 9:01:27 PM Probe result returned DISASTER_RECOVERY save set for client fmssspsql.
12/29/2017 9:01:27 PM fmssspsql:pseudo_saveset started
12/29/2017 9:01:27 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssspsql -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:31 PM level=full, vers=pools, p=12
12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:C:\
12/29/2017 9:01:31 PM level=full, vers=pools, p=12
12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:DISASTER_RECOVERY:\
12/29/2017 9:01:31 PM level=full, vers=pools, p=12
12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:savefs succeeded.
12/29/2017 9:01:31 PM Production 1 fmssspbackup.ndc.nasa.gov:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832913.log for command output
12/29/2017 9:01:31 PM Probe result returned DISASTER_RECOVERY save set for client fmssspbackup.ndc.nasa.gov.
12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:pseudo_saveset started
12/29/2017 9:01:31 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssspbackup.ndc.nasa.gov -a DIRECT_ACCESS=No -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_REC
12/29/2017 9:01:35 PM romesql:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:35 PM level=full, vers=pools, p=4
12/29/2017 9:01:35 PM romesql:C:\
12/29/2017 9:01:35 PM level=full, vers=pools, p=4
12/29/2017 9:01:35 PM romesql:D:\
12/29/2017 9:01:35 PM level=full, vers=pools, p=4
12/29/2017 9:01:35 PM romesql:DISASTER_RECOVERY:\
12/29/2017 9:01:35 PM level=full, vers=pools, p=4
12/29/2017 9:01:35 PM romesql:savefs succeeded.
12/29/2017 9:01:35 PM Production 1 romesql:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832914.log for command output
12/29/2017 9:01:35 PM Probe result returned DISASTER_RECOVERY save set for client romesql.
12/29/2017 9:01:35 PM romesql:pseudo_saveset started
12/29/2017 9:01:35 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romesql -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "D:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:42 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{4FA49422-3F22-11E2-B398-806E6F6E6963}\' in client 'romequadlogic'.
12/29/2017 9:01:43 PM fmssmissl:C:\
12/29/2017 9:01:43 PM level=full, vers=pools, p=4
12/29/2017 9:01:43 PM fmssmissl:WINDOWS ROLES AND FEATURES:\
12/29/2017 9:01:43 PM level=full, vers=pools, p=4
12/29/2017 9:01:43 PM fmssmissl:DISASTER_RECOVERY:\
12/29/2017 9:01:43 PM level=full, vers=pools, p=4
12/29/2017 9:01:43 PM fmssmissl:All succeeded.
12/29/2017 9:01:43 PM Production 1 fmssmissl:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832915.log for command output
12/29/2017 9:01:43 PM Probe result returned DISASTER_RECOVERY save set for client fmssmissl.
12/29/2017 9:01:43 PM fmssmissl:pseudo_saveset started
12/29/2017 9:01:43 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssmissl -a DIRECT_ACCESS=No -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/29/2017 9:01:53 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmsssims'.
12/29/2017 9:01:54 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{63B2148B-1CE1-11E4-80B4-806E6F6E6963}\' in client 'fmsspiint'.
12/29/2017 9:01:59 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{77247CDD-5928-11E2-8B2B-806E6F6E6963}\' in client 'fmssspvideosrv'.
12/29/2017 9:02:11 PM Group Production 1 waiting for 22 jobs (0 awaiting restart) to complete.
12/29/2017 9:33:52 PM fmsspiint:C:\ succeeded.
12/29/2017 9:36:03 PM fmsspiint:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 9:36:05 PM fmsspiint:DISASTER_RECOVERY:\ succeeded.
12/29/2017 9:36:06 PM fmsspiint:pseudo_saveset succeeded.
12/29/2017 9:36:06 PM Production 1 fmsspiint:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832917.log for command output
12/29/2017 9:36:11 PM Group Production 1 waiting for 20 jobs (0 awaiting restart) to complete.
12/29/2017 9:38:42 PM romequadlogic:C:\ succeeded.
12/29/2017 9:38:46 PM romequadlogic:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 9:38:49 PM romequadlogic:DISASTER_RECOVERY:\ succeeded.
12/29/2017 9:38:50 PM romequadlogic:pseudo_saveset succeeded.
12/29/2017 9:38:50 PM Production 1 romequadlogic:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832916.log for command output
12/29/2017 9:38:55 PM Group Production 1 waiting for 18 jobs (0 awaiting restart) to complete.
12/29/2017 9:39:07 PM fmsssims:C:\ succeeded.
12/29/2017 9:39:12 PM fmsssims:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 9:39:14 PM Group Production 1 waiting for 19 jobs (0 awaiting restart) to complete.
12/29/2017 9:39:16 PM fmsssims:DISASTER_RECOVERY:\ succeeded.
12/29/2017 9:39:18 PM fmsssims:pseudo_saveset succeeded.
12/29/2017 9:39:18 PM Production 1 fmsssims:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832919.log for command output
12/29/2017 9:39:23 PM Group Production 1 waiting for 17 jobs (0 awaiting restart) to complete.
12/29/2017 9:39:44 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmsseadvisor.ndc.nasa.gov'.
12/29/2017 9:39:52 PM Group Production 1 waiting for 18 jobs (0 awaiting restart) to complete.
12/29/2017 9:44:15 PM fmssspvideosrv:C:\ succeeded.
12/29/2017 9:44:19 PM fmssspvideosrv:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 9:44:22 PM fmssspvideosrv:DISASTER_RECOVERY:\ succeeded.
12/29/2017 9:44:27 PM Group Production 1 waiting for 17 jobs (0 awaiting restart) to complete.
12/29/2017 9:44:32 PM fmssspvideosrv:pseudo_saveset succeeded.
12/29/2017 9:44:32 PM Production 1 fmssspvideosrv:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832918.log for command output
12/29/2017 9:44:37 PM Group Production 1 waiting for 16 jobs (0 awaiting restart) to complete.
12/29/2017 9:45:00 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{ADD03291-1441-11E4-80B4-806E6F6E6963}\' in client 'fmsspi'.
12/29/2017 9:45:07 PM Group Production 1 waiting for 17 jobs (0 awaiting restart) to complete.
12/29/2017 10:09:26 PM romeniagara:C:\ succeeded.
12/29/2017 10:09:40 PM romeniagara:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 10:09:43 PM romeniagara:DISASTER_RECOVERY:\ succeeded.
12/29/2017 10:09:44 PM romeniagara:pseudo_saveset succeeded.
12/29/2017 10:09:44 PM Production 1 romeniagara:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832920.log for command output
12/29/2017 10:09:47 PM Group Production 1 waiting for 15 jobs (0 awaiting restart) to complete.
12/29/2017 10:10:18 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{02595869-7A2F-11E4-80D8-806E6F6E6963}\' in client 'fmssanomalert'.
12/29/2017 10:10:24 PM Group Production 1 waiting for 16 jobs (0 awaiting restart) to complete.
12/29/2017 10:26:43 PM fmsseadvisor.ndc.nasa.gov:C:\ succeeded.
12/29/2017 10:26:48 PM fmsseadvisor.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 10:26:51 PM fmsseadvisor.ndc.nasa.gov:DISASTER_RECOVERY:\ succeeded.
12/29/2017 10:26:53 PM fmsseadvisor.ndc.nasa.gov:pseudo_saveset succeeded.
12/29/2017 10:26:53 PM Production 1 fmsseadvisor.ndc.nasa.gov:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832921.log for command output
12/29/2017 10:27:03 PM Group Production 1 waiting for 14 jobs (0 awaiting restart) to complete.
12/29/2017 10:27:15 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{2E5DBABF-FB20-11E2-A04D-806E6F6E6963}\' in client 'romesummit'.
12/29/2017 10:27:18 PM Group Production 1 waiting for 15 jobs (0 awaiting restart) to complete.
12/29/2017 10:28:28 PM fmsspi:C:\ succeeded.
12/29/2017 10:28:33 PM fmsspi:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 10:28:36 PM fmsspi:DISASTER_RECOVERY:\ succeeded.
12/29/2017 10:28:38 PM fmsspi:pseudo_saveset succeeded.
12/29/2017 10:28:38 PM Production 1 fmsspi:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832922.log for command output
12/29/2017 10:28:43 PM Group Production 1 waiting for 13 jobs (0 awaiting restart) to complete.
12/29/2017 10:29:49 PM Group Production 1 waiting for 14 jobs (0 awaiting restart) to complete.
12/29/2017 11:19:57 PM fmssanomalert:C:\ succeeded.
12/29/2017 11:20:02 PM fmssanomalert:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 11:20:06 PM fmssanomalert:DISASTER_RECOVERY:\ succeeded.
12/29/2017 11:20:08 PM fmssanomalert:pseudo_saveset succeeded.
12/29/2017 11:20:08 PM Production 1 fmssanomalert:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832923.log for command output
12/29/2017 11:20:16 PM Group Production 1 waiting for 12 jobs (0 awaiting restart) to complete.
12/29/2017 11:20:27 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{8363C2E3-7831-11E0-A0EA-806E6F6E6963}\' in client 'romerbmware'.
12/29/2017 11:20:36 PM Group Production 1 waiting for 14 jobs (0 awaiting restart) to complete.
12/29/2017 11:22:15 PM romesummit:C:\ succeeded.
12/29/2017 11:22:20 PM romesummit:WINDOWS ROLES AND FEATURES:\ succeeded.
12/29/2017 11:22:22 PM romesummit:DISASTER_RECOVERY:\ succeeded.
12/29/2017 11:22:24 PM romesummit:pseudo_saveset succeeded.
12/29/2017 11:22:24 PM Production 1 romesummit:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832924.log for command output
12/29/2017 11:22:29 PM Group Production 1 waiting for 12 jobs (0 awaiting restart) to complete.
12/29/2017 11:28:11 PM romerbmware:E:\ succeeded.
12/29/2017 11:28:18 PM Group Production 1 waiting for 11 jobs (0 awaiting restart) to complete.
12/29/2017 11:28:40 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{4FA49422-3F22-11E2-B398-806E6F6E6963}\' in client 'romestruxure'.
12/29/2017 11:28:50 PM Group Production 1 waiting for 12 jobs (0 awaiting restart) to complete.
12/30/2017 12:06:59 AM fmssiocngp:C:\ succeeded.
12/30/2017 12:07:13 AM fmssiocngp:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 12:07:16 AM fmssiocngp:DISASTER_RECOVERY:\ succeeded.
12/30/2017 12:07:20 AM fmssiocngp:pseudo_saveset succeeded.
12/30/2017 12:07:20 AM Production 1 fmssiocngp:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832925.log for command output
12/30/2017 12:07:23 AM Group Production 1 waiting for 10 jobs (0 awaiting restart) to complete.
12/30/2017 12:07:56 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{9218E71E-1441-11E4-80B4-806E6F6E6963}\' in client 'fmsspidb'.
12/30/2017 12:08:03 AM Group Production 1 waiting for 11 jobs (0 awaiting restart) to complete.
12/30/2017 12:19:11 AM romerbmware:C:\ succeeded.
12/30/2017 12:19:14 AM romerbmware:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 12:19:18 AM romerbmware:DISASTER_RECOVERY:\ succeeded.
12/30/2017 12:19:23 AM Group Production 1 waiting for 10 jobs (0 awaiting restart) to complete.
12/30/2017 12:19:23 AM romerbmware:pseudo_saveset succeeded.
12/30/2017 12:19:23 AM Production 1 romerbmware:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832926.log for command output
12/30/2017 12:19:25 AM Group Production 1 waiting for 9 jobs (0 awaiting restart) to complete.
12/30/2017 12:19:48 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmssidm'.
12/30/2017 12:19:54 AM Group Production 1 waiting for 10 jobs (0 awaiting restart) to complete.
12/30/2017 1:20:42 AM romestruxure:C:\ succeeded.
12/30/2017 1:20:47 AM romestruxure:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 1:20:47 AM romestruxure:DISASTER_RECOVERY:\ succeeded.
12/30/2017 1:20:47 AM romestruxure:pseudo_saveset succeeded.
12/30/2017 1:20:47 AM Production 1 romestruxure:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832927.log for command output
12/30/2017 1:20:52 AM Group Production 1 waiting for 8 jobs (0 awaiting restart) to complete.
12/30/2017 1:21:43 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmssspsql'.
12/30/2017 1:21:47 AM Group Production 1 waiting for 9 jobs (0 awaiting restart) to complete.
12/30/2017 1:35:01 AM fmssidm:C:\ succeeded.
12/30/2017 1:35:06 AM fmssidm:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 1:35:09 AM fmssidm:DISASTER_RECOVERY:\ succeeded.
12/30/2017 1:35:11 AM fmssidm:pseudo_saveset succeeded.
12/30/2017 1:35:11 AM Production 1 fmssidm:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832929.log for command output
12/30/2017 1:35:17 AM Group Production 1 waiting for 7 jobs (0 awaiting restart) to complete.
12/30/2017 1:35:41 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{5FE1070D-4273-11E4-80B4-806E6F6E6963}\' in client 'fmssspbackup.ndc.nasa.gov'.
12/30/2017 1:35:47 AM Group Production 1 waiting for 8 jobs (0 awaiting restart) to complete.
12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.gov:C:\ succeeded.
12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.gov:DISASTER_RECOVERY:\ succeeded.
12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.gov:pseudo_saveset succeeded.
12/30/2017 1:55:19 AM Production 1 fmssspbackup.ndc.nasa.gov:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832931.log for command output
12/30/2017 1:55:24 AM Group Production 1 waiting for 6 jobs (0 awaiting restart) to complete.
12/30/2017 1:55:31 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{2D3FAA29-E7DD-11E1-BC04-806E6F6E6963}\' in client 'romesql'.
12/30/2017 1:55:44 AM Group Production 1 waiting for 8 jobs (0 awaiting restart) to complete.
12/30/2017 2:08:54 AM romesql:D:\ succeeded.
12/30/2017 2:08:59 AM Group Production 1 waiting for 7 jobs (0 awaiting restart) to complete.
12/30/2017 2:57:58 AM fmssspsql:C:\ succeeded.
12/30/2017 2:58:14 AM fmssspsql:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 2:58:18 AM fmssspsql:DISASTER_RECOVERY:\ succeeded.
12/30/2017 2:58:20 AM fmssspsql:pseudo_saveset succeeded.
12/30/2017 2:58:20 AM Production 1 fmssspsql:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832930.log for command output
12/30/2017 2:58:25 AM Group Production 1 waiting for 5 jobs (0 awaiting restart) to complete.
12/30/2017 2:59:03 AM Group Production 1 waiting for 6 jobs (0 awaiting restart) to complete.
12/30/2017 3:45:54 AM fmssmissl:C:\ succeeded.
12/30/2017 3:45:54 AM fmssmissl:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 3:45:54 AM fmssmissl:DISASTER_RECOVERY:\ succeeded.
12/30/2017 3:45:54 AM fmssmissl:pseudo_saveset succeeded.
12/30/2017 3:45:54 AM Production 1 fmssmissl:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832935.log for command output
12/30/2017 3:45:59 AM Group Production 1 waiting for 4 jobs (0 awaiting restart) to complete.
12/30/2017 4:05:00 AM romesql:C:\ succeeded.
12/30/2017 4:05:06 AM romesql:WINDOWS ROLES AND FEATURES:\ succeeded.
12/30/2017 4:05:13 AM romesql:DISASTER_RECOVERY:\ succeeded.
12/30/2017 4:05:15 AM romesql:pseudo_saveset succeeded.
12/30/2017 4:05:15 AM Production 1 romesql:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832932.log for command output
12/30/2017 4:05:20 AM Group Production 1 waiting for 2 jobs (0 awaiting restart) to complete.
12/30/2017 5:01:30 AM fmsspidb:C:\ succeeded.
12/30/2017 5:01:35 AM Job 833117 for client fmsspidb exited with return code 1
12/30/2017 5:01:35 AM Job 833117 host: fmsspidb savepoint: WINDOWS ROLES AND FEATURES:\ had ERROR indication(s) at completion
12/30/2017 5:01:35 AM fmsspidb:WINDOWS ROLES AND FEATURES:\ failed.
12/30/2017 5:01:35 AM fmsspidb:WINDOWS ROLES AND FEATURES:\ will retry 1 more time(s).
12/30/2017 5:01:35 AM fmsspidb:WINDOWS ROLES AND FEATURES:\ next retry in 1 seconds.
12/30/2017 5:01:35 AM Job 832928 with command 'save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspidb -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"' for client fmsspidb exited with return code 22
12/30/2017 5:01:35 AM fmsspidb:pseudo_saveset never started.
12/30/2017 5:01:35 AM fmsspidb:pseudo_saveset will retry 1 more time(s).
12/30/2017 5:01:35 AM fmsspidb:pseudo_saveset next retry in 1 seconds.
12/30/2017 5:01:35 AM Production 1 fmsspidb:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832928.log for command output
12/30/2017 5:01:36 AM Group Production 1 waiting for 1 jobs (1 awaiting restart) to complete.
12/30/2017 5:01:36 AM fmsspidb:pseudo_saveset started
12/30/2017 5:01:36 AM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspidb -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"
12/30/2017 5:01:41 AM Group Production 1 waiting for 1 jobs (0 awaiting restart) to complete.
12/30/2017 5:02:05 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{9218E71E-1441-11E4-80B4-806E6F6E6963}\' in client 'fmsspidb'.
12/30/2017 5:02:06 AM Group Production 1 waiting for 2 jobs (0 awaiting restart) to complete.
12/30/2017 12:54:15 PM Invalid argument
12/30/2017 12:54:15 PM Invalid argument
12/30/2017 12:54:15 PM fmsspidb:pseudo_saveset succeeded.
12/30/2017 12:54:15 PM Production 1 fmsspidb:pseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\833118.log for command output
12/30/2017 12:54:20 PM Group Production 1 waiting for 3 jobs (0 awaiting restart) to complete.
Davidtgnome
66 Posts
1
January 8th, 2018 07:00
Take a look at Escalation 30437.
The symptoms I was seeing were similar, the backups may or may not have completed, however the savegrp and nsrworkflow processes were hanging which caused both inconsistent data with regards to successes or failure in the NMC and resulted in the next scheduled backup for the workflow not kicking off..
According to the notes the route cause was when a failed job was restarting, it wasn't resetting the internal state of the job and as a result state change messages for that job were being ignored.
There is a binary for 9.1.1.5 for savegrp and the rumor is the fix will be included with 9.1.1.6. I began seeing this problem in 9.0.1.8, and I fought with engineering for close to 2 months to get them to recognize and then fix it.
aartyn4800
1 Rookie
1 Rookie
•
14 Posts
1
January 2nd, 2018 08:00
Hi Bingo,
The screenshot was made while the workflow was still showing active. I have not restarted the services yet due to another workflow still running.
Disregard the log, I posted the wrong log, it is from a different workflow that is having the same issue.
I will following the steps you provided,
- Manually stop the workflow from the GUI
BE PATIENT - it might take some minutes until the workflow icon will show the status 'aborted'.
- Restart the workflow from the GUI
Especially if the 'Failed' list is empty, it will only take a few seconds for the workflow to come up as 'succeeded'.
Thanks,
Aaron
bingo.1
2.4K Posts
1
January 2nd, 2018 08:00
Such a huge logfile is pretty useless - i doubt someone will read it at all.
I also wonder when you made the screenshot - while the workflow was still active of after you restarted the NW services? - At the client or the NW server itself?
I also have a slight problem with 'hanging' NW workflows on NW 9.1.1.5 - it happens constantly on a very small number of our clients. We are still looking for the origin but there is an easy way to recover:
- Manually stop the workflow from the GUI
BE PATIENT - it might take some minutes until the workflow icon will show the status 'aborted'.
- Restart the workflow from the GUI
Especially if the 'Failed' list is empty, it will only take a few seconds for the workflow to come up as 'succeeded'.
It might be as easy as that.
Unfortunately 'ERROR(50)' is so unspecific that it will not really help.
aartyn4800
1 Rookie
1 Rookie
•
14 Posts
1
January 2nd, 2018 12:00
I wasn't ever able to get the workflow to stop via the NMC. But I was able to get the workflow stopped by opening the windows task manager and finding the nsrworkflow.exe and killing it.
Thanks,
Aaron
aartyn4800
1 Rookie
1 Rookie
•
14 Posts
0
January 8th, 2018 10:00
Thanks for the update! I will keep an eye out for the 9.1.1.6
Jalli Raj
17 Posts
0
January 9th, 2018 12:00
Our customer had same issue . The Savegrp goes to infinite loop and never exits.. The fix is now availble with Engineering , its a very simple savegrp binary swap.. I believe I am the first customer to do it and it worked for us ... Yes it will be rolled out in 9.1.1.6
aartyn4800
1 Rookie
1 Rookie
•
14 Posts
0
January 11th, 2018 06:00
Good to hear, thank you for the update!
Aegh
1 Rookie
1 Rookie
•
81 Posts
0
January 16th, 2018 19:00
I have the same issue, can you share the link for this escalation?
Davidtgnome
66 Posts
0
January 17th, 2018 06:00
If I could I would, however I've asked my engineer for the link 3 times, to troubleshoot new problems in my environment, and he has yet to provide it. He always provides the number and the brief summary.