NetWorker: Azure VM Backup Workflow Fails at the End of the Backup on Pseudo_saveset

Azure VM snapshot backups complete; however, the workflow fails at the end of the backup.

The following error is observed in the backup action log:

• Linux: /nsr/logs/policy/POLICY_NAME/WOKFLOW_NAME/backup_JOBID.raw
• Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\POLICY_NAME\WOKFLOW_NAME\backup_JOBID.raw
• NetWorker: How to use nsr_render_log to render .raw log files

174900 MM/DD/YY HH:mm:SS  1 5 0 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR notice Step (5 of 5): Creating a pseudo_saveset job for all the configured clients.
174901 MM/DD/YY HH:mm:SS  1 5 0 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR notice Creating a save job for the save set 'pseudo_saveset' on the host 'NETWORKER_SERVER_NAME'.
174895 MM/DD/YY HH:mm:SS  1 5 0 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR notice Executing a 'pseudo_saveset' job on the host 'NETWORKER_SERVER_NAME'. This job is an anchor save set for the workflow, and will be completed at the end of the client's backup.
83643 MM/DD/YY HH:mm:SS  1 5 0 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR notice NETWORKER_SERVER_NAME:pseudo_saveset                             started
...
174289 MM/DD/YY HH:mm:SS  2 5 0 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR warning Aborting inactive job (1184047) NETWORKER_SERVER_NAME:pseudo_saveset. It has been inactive from 'DAY MONTH DATE HH:mm:SS YYYY'(1756325582) to 'DAY MONTH DATE HH:mm:SS YYYY'(17563DATE399) for 1817 seconds.
128139 MM/DD/YY HH:mm:SS  0 0 2 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR info NETWORKER_SERVER_NAME:pseudo_saveset aborted, inactivity timeout has been reached.
7224 MM/DD/YY HH:mm:SS  1 5 0 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR notice NETWORKER_SERVER_NAME:pseudo_saveset Termination request was sent to job 1184047 as requested; Reason given: Inactive
0491 MM/DD/YY HH:mm:SS  1 5 0 2442626880 17965 0 NETWORKER_SERVER_NAME savegrp NSR notice NETWORKER_SERVER_NAME:pseudo_saveset failed.

The backup action contains an Inactivity Timeout that the default value is 30 minutes. There is a default retry operation but in some instances the same timeout occurs or another condition causes a stoppage on the workflow.

The above error shows that the job was inactive for 1817 seconds (approximately 30 minutes). 

There are many conditions that could cause a job to reach an inactivity timeout. 

Increase the inactivity timeout or set it to zero to disable it temporarily. Establish a baseline for each workflow’s typical completion time before reenabling. Once benchmarks are in place, you can reintroduce appropriate timeout values to balance job completion with inactivity monitoring. 

If the problem is more consistent, or psuedo_saveset is failing or aborting due to other issues, consider the following:

1. The appropriate server parallelism setting is applied: NetWorker: Azure VM Snapshot Backups Failing "current save session has exceeded NetWorker server's parallelism"
2. Appropriate number of VMs and overall data throughput per Azure client. Azure VMs are backed up as a save set configured under a NetWorker client. The NetWorker Azure Snapshot Integration Guide Manuals & Documents states that the client parallelism must be set to a value of 1 through 4. This parallelism controls the number of disks that you can concurrently back up. 
   1. Ensure that enough Azure client resources are created to load balance VM backups in a workflow efficiently. Azure clients that contain many VMs or several large VMs may take a long time to complete the psuedo_saveset, increasing the possibility of inactivity timeout. The number of clients required is unique to the environment and factors in the number of Azure VMs, number of disks, size of disks, workflow scheduling, and number of target Data Domains.
   2. Avoid scheduling backups using the same Azure client resource to overlap in a manner that would initiate multiple nsrazure_save processes, especially for clients that contain many VMs, or several large Azure VMs.