aartyn4800
Phosphor

Policy Hung, Will not stop without stopping services

Jump to solution

I recently upgraded my backup servers to Networker 9.1.1.5 from 9.1.1.4. Since the upgrade I have noticed that sometimes the policy completes but does not automatically stop itself, kinda like it is hung. This happens on all the Networker servers not just one. If I try to manually stop the policy it does not stop, I have to stop the Networker services to get the policy to stop. I added a timeout to workflow as well but it didn't help. Any suggestions?01022018.PNG.png

12/29/2017 9:00:06 PM Action backup traditional 'Backup' has initialized as 'backup action job' with job id 832897

12/29/2017 9:00:10 PM fmssspvideosrv:C:\                        level=full

12/29/2017 9:00:10 PM fmssspvideosrvSmiley Very HappyISASTER_RECOVERY:\        level=full

12/29/2017 9:00:10 PM fmssiocngp:All                            level=full

12/29/2017 9:00:10 PM romeniagara:All                           level=full

12/29/2017 9:00:10 PM fmsspiint:All                             level=full

12/29/2017 9:00:10 PM romequadlogic:All                         level=full

12/29/2017 9:00:10 PM fmsssims:All                              level=full

12/29/2017 9:00:10 PM fmsseadvisor.ndc.nasa.gov:All             level=full

12/29/2017 9:00:10 PM fmsspi:C:\                                level=full

12/29/2017 9:00:10 PM fmsspiSmiley Very HappyISASTER_RECOVERY:\                level=full

12/29/2017 9:00:10 PM romerbmware:All                           level=full

12/29/2017 9:00:10 PM romesummit:All                            level=full

12/29/2017 9:00:10 PM fmssanomalert:All                         level=full

12/29/2017 9:00:11 PM romestruxure:All                          level=full

12/29/2017 9:00:11 PM fmsspidb:C:\                              level=full

12/29/2017 9:00:11 PM fmsspidbSmiley Very HappyISASTER_RECOVERY:\              level=full

12/29/2017 9:00:11 PM fmssidm:All                               level=full

12/29/2017 9:00:11 PM fmssspsql:C:\                             level=full

12/29/2017 9:00:11 PM fmssspsqlSmiley Very HappyISASTER_RECOVERY:\             level=full

12/29/2017 9:00:11 PM fmssspbackup.ndc.nasa.gov:C:\             level=full

12/29/2017 9:00:11 PM fmssspbackup.ndc.nasa.govSmiley Very HappyISASTER_RECOVERY:\ level=full

12/29/2017 9:00:11 PM romesql:C:\                               level=full

12/29/2017 9:00:11 PM romesqlSmiley Very Happy:\                               level=full

12/29/2017 9:00:11 PM romesqlSmiley Very HappyISASTER_RECOVERY:\               level=full

12/29/2017 9:00:11 PM fmssmissl:All                             level=full

12/29/2017 9:00:11 PM Action backup traditional will run up to 5 jobs in parallel

12/29/2017 9:00:11 PM Policy 'Backup', workflow 'Production 1', action 'Backup', group 'Production 1'.

12/29/2017 9:00:11 PM Starting action backup traditional 'Backup', which has 18 clients.

12/29/2017 9:00:11 PM fmssspvideosrv:savefs                                      started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssspvideosrv -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:11 PM fmssiocngp:All                                             started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssiocngp -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM romeniagara:All                                            started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romeniagara -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM fmsspiint:All                                              started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsspiint -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM romequadlogic:All                                          started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romequadlogic -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM fmsssims:All                                               started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsssims -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM fmsseadvisor.ndc.nasa.gov:All                              started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsseadvisor.ndc.nasa.gov -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM fmsspi:savefs                                              started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsspi -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:11 PM romerbmware:All                                            started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romerbmware -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM romesummit:All                                             started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romesummit -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM fmssanomalert:All                                          started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssanomalert -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM romestruxure:All                                           started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romestruxure -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM fmsspidb:savefs                                            started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmsspidb -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:11 PM fmssidm:All                                                started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssidm -g "Production 1" -p -l full -R -v

12/29/2017 9:00:11 PM fmssspsql:savefs                                           started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssspsql -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:11 PM fmssspbackup.ndc.nasa.gov:savefs                           started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssspbackup.ndc.nasa.gov -g "Production 1" -p -l full -R -v -F "C:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:11 PM romesql:savefs                                             started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c romesql -g "Production 1" -p -l full -R -v -F "C:\\" "D:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:11 PM fmssmissl:All                                              started

12/29/2017 9:00:11 PM savefs -s fmssspbackup.ndc.nasa.gov -c fmssmissl -g "Production 1" -p -l full -R -v

12/29/2017 9:00:16 PM Group Production 1 waiting for 18 jobs (0 awaiting restart) to complete.

12/29/2017 9:00:40 PM romequadlogic:C:\                    

12/29/2017 9:00:40 PM level=full, vers=pools, p=4

12/29/2017 9:00:40 PM romequadlogic:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:00:40 PM level=full, vers=pools, p=4

12/29/2017 9:00:40 PM romequadlogicSmiley Very HappyISASTER_RECOVERY:\    

12/29/2017 9:00:40 PM level=full, vers=pools, p=4

12/29/2017 9:00:40 PM romequadlogic:All succeeded.

12/29/2017 9:00:40 PM Production 1 romequadlogic:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832902.log for command output

12/29/2017 9:00:40 PM Probe result returned DISASTER_RECOVERY save set for client romequadlogic.

12/29/2017 9:00:40 PM romequadlogicSmiley Tongueseudo_saveset                               started

12/29/2017 9:00:40 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romequadlogic -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:43 PM fmsspiint:C:\                        

12/29/2017 9:00:43 PM level=full, vers=pools, p=4

12/29/2017 9:00:43 PM fmsspiint:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:00:43 PM level=full, vers=pools, p=4

12/29/2017 9:00:43 PM fmsspiintSmiley Very HappyISASTER_RECOVERY:\        

12/29/2017 9:00:43 PM level=full, vers=pools, p=4

12/29/2017 9:00:43 PM fmsspiint:All succeeded.

12/29/2017 9:00:43 PM Production 1 fmsspiint:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832901.log for command output

12/29/2017 9:00:43 PM Probe result returned DISASTER_RECOVERY save set for client fmsspiint.

12/29/2017 9:00:43 PM fmsspiintSmiley Tongueseudo_saveset                                   started

12/29/2017 9:00:43 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspiint -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:50 PM fmssspvideosrv:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:00:50 PM level=full, vers=pools, p=4

12/29/2017 9:00:50 PM fmssspvideosrv:C:\                   

12/29/2017 9:00:50 PM level=full, vers=pools, p=4

12/29/2017 9:00:50 PM fmssspvideosrvSmiley Very HappyISASTER_RECOVERY:\   

12/29/2017 9:00:50 PM level=full, vers=pools, p=4

12/29/2017 9:00:50 PM fmssspvideosrv:savefs succeeded.

12/29/2017 9:00:50 PM Production 1 fmssspvideosrv:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832898.log for command output

12/29/2017 9:00:50 PM Probe result returned DISASTER_RECOVERY save set for client fmssspvideosrv.

12/29/2017 9:00:50 PM fmssspvideosrvSmiley Tongueseudo_saveset                              started

12/29/2017 9:00:50 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssspvideosrv -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:52 PM fmsssims:C:\                         

12/29/2017 9:00:52 PM level=full, vers=pools, p=4

12/29/2017 9:00:52 PM fmsssims:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:00:52 PM level=full, vers=pools, p=4

12/29/2017 9:00:52 PM fmsssimsSmiley Very HappyISASTER_RECOVERY:\         

12/29/2017 9:00:52 PM level=full, vers=pools, p=4

12/29/2017 9:00:52 PM fmsssims:All succeeded.

12/29/2017 9:00:52 PM Production 1 fmsssims:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832903.log for command output

12/29/2017 9:00:52 PM Probe result returned DISASTER_RECOVERY save set for client fmsssims.

12/29/2017 9:00:52 PM fmsssimsSmiley Tongueseudo_saveset                                    started

12/29/2017 9:00:52 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsssims -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:54 PM romeniagara:C:\                      

12/29/2017 9:00:54 PM level=full, vers=pools, p=4

12/29/2017 9:00:54 PM romeniagara:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:00:54 PM level=full, vers=pools, p=4

12/29/2017 9:00:54 PM romeniagaraSmiley Very HappyISASTER_RECOVERY:\      

12/29/2017 9:00:54 PM level=full, vers=pools, p=4

12/29/2017 9:00:54 PM romeniagara:All succeeded.

12/29/2017 9:00:54 PM Production 1 romeniagara:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832900.log for command output

12/29/2017 9:00:54 PM Probe result returned DISASTER_RECOVERY save set for client romeniagara.

12/29/2017 9:00:54 PM romeniagaraSmiley Tongueseudo_saveset                                 started

12/29/2017 9:00:54 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romeniagara -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:C:\        

12/29/2017 9:00:55 PM level=full, vers=pools, p=4

12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:00:55 PM level=full, vers=pools, p=4

12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.govSmiley Very HappyISASTER_RECOVERY:\

12/29/2017 9:00:55 PM level=full, vers=pools, p=4

12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.gov:All succeeded.

12/29/2017 9:00:55 PM Production 1 fmsseadvisor.ndc.nasa.gov:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832904.log for command output

12/29/2017 9:00:55 PM Probe result returned DISASTER_RECOVERY save set for client fmsseadvisor.ndc.nasa.gov.

12/29/2017 9:00:55 PM fmsseadvisor.ndc.nasa.govSmiley Tongueseudo_saveset                   started

12/29/2017 9:00:55 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsseadvisor.ndc.nasa.gov -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:00 PM fmsspi:WINDOWS ROLES AND FEATURES:\  

12/29/2017 9:01:00 PM level=full, vers=pools, p=4

12/29/2017 9:01:00 PM fmsspi:C:\                           

12/29/2017 9:01:00 PM level=full, vers=pools, p=4

12/29/2017 9:01:00 PM fmsspiSmiley Very HappyISASTER_RECOVERY:\           

12/29/2017 9:01:00 PM level=full, vers=pools, p=4

12/29/2017 9:01:00 PM fmsspi:savefs succeeded.

12/29/2017 9:01:00 PM Production 1 fmsspi:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832905.log for command output

12/29/2017 9:01:00 PM Probe result returned DISASTER_RECOVERY save set for client fmsspi.

12/29/2017 9:01:00 PM fmsspiSmiley Tongueseudo_saveset                                      started

12/29/2017 9:01:00 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspi -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:08 PM fmssanomalert:C:\                    

12/29/2017 9:01:08 PM level=full, vers=pools, p=4

12/29/2017 9:01:08 PM fmssanomalert:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:08 PM level=full, vers=pools, p=4

12/29/2017 9:01:08 PM fmssanomalertSmiley Very HappyISASTER_RECOVERY:\    

12/29/2017 9:01:08 PM level=full, vers=pools, p=4

12/29/2017 9:01:08 PM fmssanomalert:All succeeded.

12/29/2017 9:01:08 PM Production 1 fmssanomalert:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832908.log for command output

12/29/2017 9:01:08 PM Probe result returned DISASTER_RECOVERY save set for client fmssanomalert.

12/29/2017 9:01:08 PM fmssanomalertSmiley Tongueseudo_saveset                               started

12/29/2017 9:01:08 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssanomalert -a DIRECT_ACCESS=No -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:13 PM romesummit:C:\                       

12/29/2017 9:01:13 PM level=full, vers=pools, p=4

12/29/2017 9:01:13 PM romesummit:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:13 PM level=full, vers=pools, p=4

12/29/2017 9:01:13 PM romesummitSmiley Very HappyISASTER_RECOVERY:\       

12/29/2017 9:01:13 PM level=full, vers=pools, p=4

12/29/2017 9:01:13 PM romesummit:All succeeded.

12/29/2017 9:01:13 PM Production 1 romesummit:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832907.log for command output

12/29/2017 9:01:13 PM Probe result returned DISASTER_RECOVERY save set for client romesummit.

12/29/2017 9:01:13 PM romesummitSmiley Tongueseudo_saveset                                  started

12/29/2017 9:01:13 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romesummit -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:15 PM fmssiocngp:C:\                       

12/29/2017 9:01:15 PM level=full, vers=pools, p=4

12/29/2017 9:01:15 PM fmssiocngp:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:15 PM level=full, vers=pools, p=4

12/29/2017 9:01:15 PM fmssiocngpSmiley Very HappyISASTER_RECOVERY:\       

12/29/2017 9:01:15 PM level=full, vers=pools, p=4

12/29/2017 9:01:15 PM fmssiocngp:All succeeded.

12/29/2017 9:01:15 PM Production 1 fmssiocngp:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832899.log for command output

12/29/2017 9:01:15 PM Probe result returned DISASTER_RECOVERY save set for client fmssiocngp.

12/29/2017 9:01:15 PM fmssiocngpSmiley Tongueseudo_saveset                                  started

12/29/2017 9:01:15 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssiocngp -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:16 PM romerbmware:C:\                      

12/29/2017 9:01:16 PM level=full, vers=pools, p=4

12/29/2017 9:01:16 PM romerbmware:E:\                      

12/29/2017 9:01:16 PM level=full, vers=pools, p=4

12/29/2017 9:01:16 PM romerbmware:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:16 PM level=full, vers=pools, p=4

12/29/2017 9:01:16 PM romerbmwareSmiley Very HappyISASTER_RECOVERY:\      

12/29/2017 9:01:16 PM level=full, vers=pools, p=4

12/29/2017 9:01:16 PM romerbmware:All succeeded.

12/29/2017 9:01:16 PM Production 1 romerbmware:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832906.log for command output

12/29/2017 9:01:16 PM Probe result returned DISASTER_RECOVERY save set for client romerbmware.

12/29/2017 9:01:16 PM romerbmwareSmiley Tongueseudo_saveset                                 started

12/29/2017 9:01:16 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romerbmware -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "E:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:21 PM romestruxure:C:\                     

12/29/2017 9:01:21 PM level=full, vers=pools, p=4

12/29/2017 9:01:21 PM romestruxure:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:21 PM level=full, vers=pools, p=4

12/29/2017 9:01:21 PM romestruxureSmiley Very HappyISASTER_RECOVERY:\     

12/29/2017 9:01:21 PM level=full, vers=pools, p=4

12/29/2017 9:01:21 PM romestruxure:All succeeded.

12/29/2017 9:01:21 PM Production 1 romestruxure:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832909.log for command output

12/29/2017 9:01:21 PM Probe result returned DISASTER_RECOVERY save set for client romestruxure.

12/29/2017 9:01:21 PM romestruxureSmiley Tongueseudo_saveset                                started

12/29/2017 9:01:21 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romestruxure -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:22 PM fmsspidb:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:22 PM level=full, vers=pools, p=4

12/29/2017 9:01:22 PM fmsspidb:C:\                         

12/29/2017 9:01:22 PM level=full, vers=pools, p=4

12/29/2017 9:01:22 PM fmsspidbSmiley Very HappyISASTER_RECOVERY:\         

12/29/2017 9:01:22 PM level=full, vers=pools, p=4

12/29/2017 9:01:22 PM fmsspidb:savefs succeeded.

12/29/2017 9:01:22 PM Production 1 fmsspidb:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832910.log for command output

12/29/2017 9:01:22 PM Probe result returned DISASTER_RECOVERY save set for client fmsspidb.

12/29/2017 9:01:22 PM fmsspidbSmiley Tongueseudo_saveset                                    started

12/29/2017 9:01:22 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspidb -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:24 PM fmssidm:C:\                          

12/29/2017 9:01:24 PM level=full, vers=pools, p=4

12/29/2017 9:01:24 PM fmssidm:WINDOWS ROLES AND FEATURES:\ 

12/29/2017 9:01:24 PM level=full, vers=pools, p=4

12/29/2017 9:01:24 PM fmssidmSmiley Very HappyISASTER_RECOVERY:\          

12/29/2017 9:01:24 PM level=full, vers=pools, p=4

12/29/2017 9:01:24 PM fmssidm:All succeeded.

12/29/2017 9:01:24 PM Production 1 fmssidm:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832911.log for command output

12/29/2017 9:01:24 PM Probe result returned DISASTER_RECOVERY save set for client fmssidm.

12/29/2017 9:01:24 PM fmssidmSmiley Tongueseudo_saveset                                     started

12/29/2017 9:01:24 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssidm -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:27 PM fmssspsql:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:27 PM level=full, vers=pools, p=4

12/29/2017 9:01:27 PM fmssspsql:C:\                        

12/29/2017 9:01:27 PM level=full, vers=pools, p=4

12/29/2017 9:01:27 PM fmssspsqlSmiley Very HappyISASTER_RECOVERY:\        

12/29/2017 9:01:27 PM level=full, vers=pools, p=4

12/29/2017 9:01:27 PM fmssspsql:savefs succeeded.

12/29/2017 9:01:27 PM Production 1 fmssspsql:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832912.log for command output

12/29/2017 9:01:27 PM Probe result returned DISASTER_RECOVERY save set for client fmssspsql.

12/29/2017 9:01:27 PM fmssspsqlSmiley Tongueseudo_saveset                                   started

12/29/2017 9:01:27 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssspsql -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:31 PM level=full, vers=pools, p=12

12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:C:\        

12/29/2017 9:01:31 PM level=full, vers=pools, p=12

12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.govSmiley Very HappyISASTER_RECOVERY:\

12/29/2017 9:01:31 PM level=full, vers=pools, p=12

12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.gov:savefs succeeded.

12/29/2017 9:01:31 PM Production 1 fmssspbackup.ndc.nasa.gov:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832913.log for command output

12/29/2017 9:01:31 PM Probe result returned DISASTER_RECOVERY save set for client fmssspbackup.ndc.nasa.gov.

12/29/2017 9:01:31 PM fmssspbackup.ndc.nasa.govSmiley Tongueseudo_saveset                   started

12/29/2017 9:01:31 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssspbackup.ndc.nasa.gov -a DIRECT_ACCESS=No -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_REC

12/29/2017 9:01:35 PM romesql:WINDOWS ROLES AND FEATURES:\ 

12/29/2017 9:01:35 PM level=full, vers=pools, p=4

12/29/2017 9:01:35 PM romesql:C:\                          

12/29/2017 9:01:35 PM level=full, vers=pools, p=4

12/29/2017 9:01:35 PM romesqlSmiley Very Happy:\                          

12/29/2017 9:01:35 PM level=full, vers=pools, p=4

12/29/2017 9:01:35 PM romesqlSmiley Very HappyISASTER_RECOVERY:\          

12/29/2017 9:01:35 PM level=full, vers=pools, p=4

12/29/2017 9:01:35 PM romesql:savefs succeeded.

12/29/2017 9:01:35 PM Production 1 romesql:savefs See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832914.log for command output

12/29/2017 9:01:35 PM Probe result returned DISASTER_RECOVERY save set for client romesql.

12/29/2017 9:01:35 PM romesqlSmiley Tongueseudo_saveset                                     started

12/29/2017 9:01:35 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m romesql -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "D:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:42 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{4FA49422-3F22-11E2-B398-806E6F6E6963}\' in client 'romequadlogic'.

12/29/2017 9:01:43 PM fmssmissl:C:\                        

12/29/2017 9:01:43 PM level=full, vers=pools, p=4

12/29/2017 9:01:43 PM fmssmissl:WINDOWS ROLES AND FEATURES:\

12/29/2017 9:01:43 PM level=full, vers=pools, p=4

12/29/2017 9:01:43 PM fmssmisslSmiley Very HappyISASTER_RECOVERY:\        

12/29/2017 9:01:43 PM level=full, vers=pools, p=4

12/29/2017 9:01:43 PM fmssmissl:All succeeded.

12/29/2017 9:01:43 PM Production 1 fmssmissl:All See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832915.log for command output

12/29/2017 9:01:43 PM Probe result returned DISASTER_RECOVERY save set for client fmssmissl.

12/29/2017 9:01:43 PM fmssmisslSmiley Tongueseudo_saveset                                   started

12/29/2017 9:01:43 PM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmssmissl -a DIRECT_ACCESS=No -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/29/2017 9:01:53 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmsssims'.

12/29/2017 9:01:54 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{63B2148B-1CE1-11E4-80B4-806E6F6E6963}\' in client 'fmsspiint'.

12/29/2017 9:01:59 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{77247CDD-5928-11E2-8B2B-806E6F6E6963}\' in client 'fmssspvideosrv'.

12/29/2017 9:02:11 PM Group Production 1 waiting for 22 jobs (0 awaiting restart) to complete.

12/29/2017 9:33:52 PM fmsspiint:C:\ succeeded.

12/29/2017 9:36:03 PM fmsspiint:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 9:36:05 PM fmsspiintSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 9:36:06 PM fmsspiintSmiley Tongueseudo_saveset succeeded.

12/29/2017 9:36:06 PM Production 1 fmsspiintSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832917.log for command output

12/29/2017 9:36:11 PM Group Production 1 waiting for 20 jobs (0 awaiting restart) to complete.

12/29/2017 9:38:42 PM romequadlogic:C:\ succeeded.

12/29/2017 9:38:46 PM romequadlogic:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 9:38:49 PM romequadlogicSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 9:38:50 PM romequadlogicSmiley Tongueseudo_saveset succeeded.

12/29/2017 9:38:50 PM Production 1 romequadlogicSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832916.log for command output

12/29/2017 9:38:55 PM Group Production 1 waiting for 18 jobs (0 awaiting restart) to complete.

12/29/2017 9:39:07 PM fmsssims:C:\ succeeded.

12/29/2017 9:39:12 PM fmsssims:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 9:39:14 PM Group Production 1 waiting for 19 jobs (0 awaiting restart) to complete.

12/29/2017 9:39:16 PM fmsssimsSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 9:39:18 PM fmsssimsSmiley Tongueseudo_saveset succeeded.

12/29/2017 9:39:18 PM Production 1 fmsssimsSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832919.log for command output

12/29/2017 9:39:23 PM Group Production 1 waiting for 17 jobs (0 awaiting restart) to complete.

12/29/2017 9:39:44 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmsseadvisor.ndc.nasa.gov'.

12/29/2017 9:39:52 PM Group Production 1 waiting for 18 jobs (0 awaiting restart) to complete.

12/29/2017 9:44:15 PM fmssspvideosrv:C:\ succeeded.

12/29/2017 9:44:19 PM fmssspvideosrv:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 9:44:22 PM fmssspvideosrvSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 9:44:27 PM Group Production 1 waiting for 17 jobs (0 awaiting restart) to complete.

12/29/2017 9:44:32 PM fmssspvideosrvSmiley Tongueseudo_saveset succeeded.

12/29/2017 9:44:32 PM Production 1 fmssspvideosrvSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832918.log for command output

12/29/2017 9:44:37 PM Group Production 1 waiting for 16 jobs (0 awaiting restart) to complete.

12/29/2017 9:45:00 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{ADD03291-1441-11E4-80B4-806E6F6E6963}\' in client 'fmsspi'.

12/29/2017 9:45:07 PM Group Production 1 waiting for 17 jobs (0 awaiting restart) to complete.

12/29/2017 10:09:26 PM romeniagara:C:\ succeeded.

12/29/2017 10:09:40 PM romeniagara:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 10:09:43 PM romeniagaraSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 10:09:44 PM romeniagaraSmiley Tongueseudo_saveset succeeded.

12/29/2017 10:09:44 PM Production 1 romeniagaraSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832920.log for command output

12/29/2017 10:09:47 PM Group Production 1 waiting for 15 jobs (0 awaiting restart) to complete.

12/29/2017 10:10:18 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{02595869-7A2F-11E4-80D8-806E6F6E6963}\' in client 'fmssanomalert'.

12/29/2017 10:10:24 PM Group Production 1 waiting for 16 jobs (0 awaiting restart) to complete.

12/29/2017 10:26:43 PM fmsseadvisor.ndc.nasa.gov:C:\ succeeded.

12/29/2017 10:26:48 PM fmsseadvisor.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 10:26:51 PM fmsseadvisor.ndc.nasa.govSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 10:26:53 PM fmsseadvisor.ndc.nasa.govSmiley Tongueseudo_saveset succeeded.

12/29/2017 10:26:53 PM Production 1 fmsseadvisor.ndc.nasa.govSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832921.log for command output

12/29/2017 10:27:03 PM Group Production 1 waiting for 14 jobs (0 awaiting restart) to complete.

12/29/2017 10:27:15 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{2E5DBABF-FB20-11E2-A04D-806E6F6E6963}\' in client 'romesummit'.

12/29/2017 10:27:18 PM Group Production 1 waiting for 15 jobs (0 awaiting restart) to complete.

12/29/2017 10:28:28 PM fmsspi:C:\ succeeded.

12/29/2017 10:28:33 PM fmsspi:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 10:28:36 PM fmsspiSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 10:28:38 PM fmsspiSmiley Tongueseudo_saveset succeeded.

12/29/2017 10:28:38 PM Production 1 fmsspiSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832922.log for command output

12/29/2017 10:28:43 PM Group Production 1 waiting for 13 jobs (0 awaiting restart) to complete.

12/29/2017 10:29:49 PM Group Production 1 waiting for 14 jobs (0 awaiting restart) to complete.

12/29/2017 11:19:57 PM fmssanomalert:C:\ succeeded.

12/29/2017 11:20:02 PM fmssanomalert:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 11:20:06 PM fmssanomalertSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 11:20:08 PM fmssanomalertSmiley Tongueseudo_saveset succeeded.

12/29/2017 11:20:08 PM Production 1 fmssanomalertSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832923.log for command output

12/29/2017 11:20:16 PM Group Production 1 waiting for 12 jobs (0 awaiting restart) to complete.

12/29/2017 11:20:27 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{8363C2E3-7831-11E0-A0EA-806E6F6E6963}\' in client 'romerbmware'.

12/29/2017 11:20:36 PM Group Production 1 waiting for 14 jobs (0 awaiting restart) to complete.

12/29/2017 11:22:15 PM romesummit:C:\ succeeded.

12/29/2017 11:22:20 PM romesummit:WINDOWS ROLES AND FEATURES:\ succeeded.

12/29/2017 11:22:22 PM romesummitSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/29/2017 11:22:24 PM romesummitSmiley Tongueseudo_saveset succeeded.

12/29/2017 11:22:24 PM Production 1 romesummitSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832924.log for command output

12/29/2017 11:22:29 PM Group Production 1 waiting for 12 jobs (0 awaiting restart) to complete.

12/29/2017 11:28:11 PM romerbmware:E:\ succeeded.

12/29/2017 11:28:18 PM Group Production 1 waiting for 11 jobs (0 awaiting restart) to complete.

12/29/2017 11:28:40 PM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{4FA49422-3F22-11E2-B398-806E6F6E6963}\' in client 'romestruxure'.

12/29/2017 11:28:50 PM Group Production 1 waiting for 12 jobs (0 awaiting restart) to complete.

12/30/2017 12:06:59 AM fmssiocngp:C:\ succeeded.

12/30/2017 12:07:13 AM fmssiocngp:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 12:07:16 AM fmssiocngpSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 12:07:20 AM fmssiocngpSmiley Tongueseudo_saveset succeeded.

12/30/2017 12:07:20 AM Production 1 fmssiocngpSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832925.log for command output

12/30/2017 12:07:23 AM Group Production 1 waiting for 10 jobs (0 awaiting restart) to complete.

12/30/2017 12:07:56 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{9218E71E-1441-11E4-80B4-806E6F6E6963}\' in client 'fmsspidb'.

12/30/2017 12:08:03 AM Group Production 1 waiting for 11 jobs (0 awaiting restart) to complete.

12/30/2017 12:19:11 AM romerbmware:C:\ succeeded.

12/30/2017 12:19:14 AM romerbmware:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 12:19:18 AM romerbmwareSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 12:19:23 AM Group Production 1 waiting for 10 jobs (0 awaiting restart) to complete.

12/30/2017 12:19:23 AM romerbmwareSmiley Tongueseudo_saveset succeeded.

12/30/2017 12:19:23 AM Production 1 romerbmwareSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832926.log for command output

12/30/2017 12:19:25 AM Group Production 1 waiting for 9 jobs (0 awaiting restart) to complete.

12/30/2017 12:19:48 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmssidm'.

12/30/2017 12:19:54 AM Group Production 1 waiting for 10 jobs (0 awaiting restart) to complete.

12/30/2017 1:20:42 AM romestruxure:C:\ succeeded.

12/30/2017 1:20:47 AM romestruxure:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 1:20:47 AM romestruxureSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 1:20:47 AM romestruxureSmiley Tongueseudo_saveset succeeded.

12/30/2017 1:20:47 AM Production 1 romestruxureSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832927.log for command output

12/30/2017 1:20:52 AM Group Production 1 waiting for 8 jobs (0 awaiting restart) to complete.

12/30/2017 1:21:43 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{547BAEA3-2C73-11E4-80B4-806E6F6E6963}\' in client 'fmssspsql'.

12/30/2017 1:21:47 AM Group Production 1 waiting for 9 jobs (0 awaiting restart) to complete.

12/30/2017 1:35:01 AM fmssidm:C:\ succeeded.

12/30/2017 1:35:06 AM fmssidm:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 1:35:09 AM fmssidmSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 1:35:11 AM fmssidmSmiley Tongueseudo_saveset succeeded.

12/30/2017 1:35:11 AM Production 1 fmssidmSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832929.log for command output

12/30/2017 1:35:17 AM Group Production 1 waiting for 7 jobs (0 awaiting restart) to complete.

12/30/2017 1:35:41 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{5FE1070D-4273-11E4-80B4-806E6F6E6963}\' in client 'fmssspbackup.ndc.nasa.gov'.

12/30/2017 1:35:47 AM Group Production 1 waiting for 8 jobs (0 awaiting restart) to complete.

12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.gov:C:\ succeeded.

12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.gov:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.govSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 1:55:19 AM fmssspbackup.ndc.nasa.govSmiley Tongueseudo_saveset succeeded.

12/30/2017 1:55:19 AM Production 1 fmssspbackup.ndc.nasa.govSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832931.log for command output

12/30/2017 1:55:24 AM Group Production 1 waiting for 6 jobs (0 awaiting restart) to complete.

12/30/2017 1:55:31 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{2D3FAA29-E7DD-11E1-BC04-806E6F6E6963}\' in client 'romesql'.

12/30/2017 1:55:44 AM Group Production 1 waiting for 8 jobs (0 awaiting restart) to complete.

12/30/2017 2:08:54 AM romesqlSmiley Very Happy:\ succeeded.

12/30/2017 2:08:59 AM Group Production 1 waiting for 7 jobs (0 awaiting restart) to complete.

12/30/2017 2:57:58 AM fmssspsql:C:\ succeeded.

12/30/2017 2:58:14 AM fmssspsql:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 2:58:18 AM fmssspsqlSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 2:58:20 AM fmssspsqlSmiley Tongueseudo_saveset succeeded.

12/30/2017 2:58:20 AM Production 1 fmssspsqlSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832930.log for command output

12/30/2017 2:58:25 AM Group Production 1 waiting for 5 jobs (0 awaiting restart) to complete.

12/30/2017 2:59:03 AM Group Production 1 waiting for 6 jobs (0 awaiting restart) to complete.

12/30/2017 3:45:54 AM fmssmissl:C:\ succeeded.

12/30/2017 3:45:54 AM fmssmissl:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 3:45:54 AM fmssmisslSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 3:45:54 AM fmssmisslSmiley Tongueseudo_saveset succeeded.

12/30/2017 3:45:54 AM Production 1 fmssmisslSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832935.log for command output

12/30/2017 3:45:59 AM Group Production 1 waiting for 4 jobs (0 awaiting restart) to complete.

12/30/2017 4:05:00 AM romesql:C:\ succeeded.

12/30/2017 4:05:06 AM romesql:WINDOWS ROLES AND FEATURES:\ succeeded.

12/30/2017 4:05:13 AM romesqlSmiley Very HappyISASTER_RECOVERY:\ succeeded.

12/30/2017 4:05:15 AM romesqlSmiley Tongueseudo_saveset succeeded.

12/30/2017 4:05:15 AM Production 1 romesqlSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832932.log for command output

12/30/2017 4:05:20 AM Group Production 1 waiting for 2 jobs (0 awaiting restart) to complete.

12/30/2017 5:01:30 AM fmsspidb:C:\ succeeded.

12/30/2017 5:01:35 AM Job 833117 for client fmsspidb exited with return code 1

12/30/2017 5:01:35 AM Job 833117 host: fmsspidb savepoint: WINDOWS ROLES AND FEATURES:\ had ERROR indication(s) at completion

12/30/2017 5:01:35 AM fmsspidb:WINDOWS ROLES AND FEATURES:\ failed.

12/30/2017 5:01:35 AM fmsspidb:WINDOWS ROLES AND FEATURES:\ will retry 1 more time(s).

12/30/2017 5:01:35 AM fmsspidb:WINDOWS ROLES AND FEATURES:\ next retry in 1 seconds.

12/30/2017 5:01:35 AM Job 832928 with command 'save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspidb -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "C:\\" "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"' for client fmsspidb exited with return code 22

12/30/2017 5:01:35 AM fmsspidbSmiley Tongueseudo_saveset never started.

12/30/2017 5:01:35 AM fmsspidbSmiley Tongueseudo_saveset will retry 1 more time(s).

12/30/2017 5:01:35 AM fmsspidbSmiley Tongueseudo_saveset next retry in 1 seconds.

12/30/2017 5:01:35 AM Production 1 fmsspidbSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\832928.log for command output

12/30/2017 5:01:36 AM Group Production 1 waiting for 1 jobs (1 awaiting restart) to complete.

12/30/2017 5:01:36 AM fmsspidbSmiley Tongueseudo_saveset                                    started

12/30/2017 5:01:36 AM save -LL -s fmssspbackup.ndc.nasa.gov -g "Backup/Production 1/Backup/Production 1" -a "*policy action jobid=832897" -a "*policy name=Backup" -a "*policy workflow name=Production 1" -a "*policy action name=Backup" -y "Wed Dec 29 23:59:59 GMT-0500 2027" -w "Wed Dec 29 23:59:59 GMT-0500 2027" -m fmsspidb -b Default -o "\"RENAMED_DIRECTORIES:index_lookup=on;REQUESTED_LEVEL:level=full;\"" -l full -q -W 78 -N pseudo_saveset "WINDOWS ROLES AND FEATURES:\\" "DISASTER_RECOVERY:\\"

12/30/2017 5:01:41 AM Group Production 1 waiting for 1 jobs (0 awaiting restart) to complete.

12/30/2017 5:02:05 AM Unable to handle job monitor message: Unable to add the job to monitor list: No entry found for save set '\\?\VOLUME{9218E71E-1441-11E4-80B4-806E6F6E6963}\' in client 'fmsspidb'.

12/30/2017 5:02:06 AM Group Production 1 waiting for 2 jobs (0 awaiting restart) to complete.

12/30/2017 12:54:15 PM Invalid argument

12/30/2017 12:54:15 PM Invalid argument

12/30/2017 12:54:15 PM fmsspidbSmiley Tongueseudo_saveset succeeded.

12/30/2017 12:54:15 PM Production 1 fmsspidbSmiley Tongueseudo_saveset See the file C:\Program Files\EMC NetWorker\nsr\logs\policy\Backup\Production 1\Backup_832897_logs\833118.log for command output

12/30/2017 12:54:20 PM Group Production 1 waiting for 3 jobs (0 awaiting restart) to complete.

Labels (1)
Tags (2)
0 Kudos
9 Replies
bingo
Germanium

Re: Policy Hung, Will not stop without stopping services

Jump to solution

Such a huge logfile is pretty useless - i doubt someone will read it at all.

I also wonder when you made the screenshot - while the workflow was still active of after you restarted the NW services? - At the client or the NW server itself?

I also have a slight problem with 'hanging' NW workflows on NW 9.1.1.5 - it happens constantly on a very small number of our clients. We are still looking for the origin but there is an easy way to recover:

  -  Manually stop the workflow from the GUI

        BE PATIENT - it might take some minutes until the workflow icon will show the status 'aborted'.

  - Restart the workflow from the GUI

        Especially if the 'Failed' list is empty, it will only take a few seconds for the workflow to come up as 'succeeded'.

It might be as easy as that.

Unfortunately 'ERROR(50)' is so unspecific that it will not really help.

Highlighted
aartyn4800
Phosphor

Re: Policy Hung, Will not stop without stopping services

Jump to solution

Hi Bingo,

The screenshot was made while the workflow was still showing active. I have not restarted the services yet due to another workflow still running.

Disregard the log, I posted the wrong log, it is from a different workflow that is having the same issue.

I will following the steps you provided,

  -  Manually stop the workflow from the GUI

        BE PATIENT - it might take some minutes until the workflow icon will show the status 'aborted'.

  - Restart the workflow from the GUI

        Especially if the 'Failed' list is empty, it will only take a few seconds for the workflow to come up as 'succeeded'.

Thanks,

Aaron

aartyn4800
Phosphor

Re: Policy Hung, Will not stop without stopping services

Jump to solution

I wasn't ever able to get the workflow to stop via the NMC. But I was able to get the workflow stopped by opening the windows task manager and finding the nsrworkflow.exe and killing it.

Thanks,


Aaron

Re: Policy Hung, Will not stop without stopping services

Jump to solution

Take a look at Escalation 30437.

The symptoms I was seeing were similar, the backups may or may not have completed, however the savegrp and nsrworkflow processes were hanging which caused both inconsistent data with regards to successes or failure in the NMC and resulted in the next scheduled backup for the workflow not kicking off..

According to the notes the route cause was when a failed job was restarting, it wasn't resetting the internal state of the job and as a result state change messages for that job were being ignored.

There is a binary for 9.1.1.5 for savegrp and the rumor is the fix will be included with 9.1.1.6. I began seeing this problem in 9.0.1.8, and I fought with engineering for close to 2 months to get them to recognize and then fix it.

aartyn4800
Phosphor

Re: Policy Hung, Will not stop without stopping services

Jump to solution

Thanks for the update! I will keep an eye out for the 9.1.1.6

0 Kudos
Jalli Raj
Phosphor

Re: Policy Hung, Will not stop without stopping services

Jump to solution

Our customer had same issue . The Savegrp goes to infinite loop and never exits.. The fix is now availble with Engineering , its a very simple savegrp binary swap.. I believe I am the first customer to do it and it worked for us ... Yes it will be rolled out in 9.1.1.6

0 Kudos
aartyn4800
Phosphor

Re: Policy Hung, Will not stop without stopping services

Jump to solution

Good to hear, thank you for the update!

0 Kudos
Iman2011
Boron

Re: Policy Hung, Will not stop without stopping services

Jump to solution

I have the same issue, can you share the link for this escalation?

0 Kudos

Re: Policy Hung, Will not stop without stopping services

Jump to solution

If I could I would, however I've asked my engineer for the link 3 times, to troubleshoot new problems in my environment, and he has yet to provide it. He always provides the number and the brief summary.

0 Kudos