NVP vProxy: Backups Fail with "The VM backup request is not valid"

The NetWorker VMware Protection integration is configured with the vProxy Appliance. The backups have been running successfully for some time, and they started intermittently failing with "The VM backup request is not valid." The backup action log shows it keeps retrying for a direct save to the NetWorker Storage node for an hour before it fails with the message.  

Example of one VM failure from the VMware Protection Policy backup action log:

MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR notice Action backup vmware-vproxy 'backup' has initialized as 'vproxysave job' with job id 2370140 
...
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR notice Saving the backup data in the pool 'Prod VM'. 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR notice Received the media management binding information on the host '[NETWORKER_STORAGE_NODE]'. 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR notice Connected to the nsrmmd process on the host '[NETWORKER_STORAGE_NODE]'. 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR warning Unable to set up the direct save with server '[NETWORKER_STORAGE_NODE]': retry needed. 
...
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR error Unable to process VM '[VM_NAME]': Request timed out before session established.  
...
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR warning Unable to set up the direct save with server '[NETWORKER_STORAGE_NODE]': The VM backup request is not valid.. 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR critical [VM_NAME]: Cannot start session with NetWorker server '[NETWORKER_STORAGE_NODE]': The VM backup request is not valid.. 
...
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrvproxy_save NSR notice Action backup vmware-vproxy 'backup' with job id 2370140 is exiting with status 'failed', exit code 1 

The deamon.raw shows that at the start of the workflow the NetWorker Server was generating warnings indicating the Data Domain reached the read stream count limit. During the VMware Protection Policy execution, the NetWorker Server daemon.raw shows:

MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrd NSR info Protection Policy Notice: Starting workflow '[POLICY_NAME]/[WORKFLOW_NAME]' 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrd NSR info Protection Policy Notice: Starting action 'backup' for workflow '[POLICY_NAME]/[WORKFLOW_NAME]' 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrd NSR warning current_read_stream (89) reach max_read_stream 50 for DDR '[DATA_DOMAIN]'. 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrd NSR warning current_read_stream (90) reach max_read_stream 50 for DDR '[DATA_DOMAIN]'. 
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrd NSR warning current_read_stream (90) reach max_read_stream 50 for DDR '[DATA_DOMAIN]'. 
...
MM/DD/YYYY HH:MM:SS [NETWORKER_SERVER] nsrd NSR info policy notice: action backup failed

See article NetWorker: How to use nsr_render_log

The NetWorker Storage node nsrmmd processes or the target Data Domain is being overloaded during the execution of the workflow. The exhausting of the Data Domain read stream is preventing the VMware Protection Policy backup action from starting the backup session.

Stagger the traditional client and VMware workflows in the environment to distribute the load throughout the day for the target Data Domain.