NetWorker: Parallel Save Streams (PSS) backup fails with error: RPC send operation failed, broken pipe

Configuring a Parallel Save Streams (PSS) client with many save sets (>5)

Around a dozen of save sets configured on a PSS client, backup often fails with the following error:

76677:save: RPC send operation failed; peer = <SN IP>:<Port number>, errno = Broken pipe
MM/DD/YY HH:mm:SS walker(): PSS emitter calling pss_abort() ...

PSS backups place a higher load on client resources, including CPU, memory, and disk I/O. This increased load delays some processes until resources become available.
In some situations, the save process loses connection to its corresponding nsrmmd (backup storage service that writes the backup data to disk)

This is an example for a local connection (the client is also a storage node) timeout.

76677:save: RPC send operation failed; peer = 127.0.0.1:65384, errno = Broken pipe
05/18/16 08:56:18.320899 walker(): PSS emitter calling pss_abort() ...

PSS Emitter_i aborting ...

pss_abort(): ENTER

PSS Emitter_i aborting ...

149696:save: PSS save point backup has timed out after 26 minutes of internal operation inactivity.
MM/DD/YY HH:mm:SS pre_read_peek_block_timeout_retry() (tid:65674344923136): PSS internal operation read pipe inactivity timeout after 25 minutes (actual 26 minutes), so aborting the backup
<Date and time> find_idle_emitter() (tid:65674344923136): ERROR: Could not get Emitter
Do MBS: Unable to find an idle emitter.
149699:save: PSS save point <saveset name> has encountered a critical error.

PSS Emitter_i aborting ...

PSS Emitter_i aborting ...

PSS backup timeout problem requires smoothing of the load, can also use a backup timeout variable to extend the timeout interval, suggested solutions include.

Option One:

Configure multiple clients for the same host and split the save sets up across the client resources. The number of save sets set per-client can vary and it may take some testing and tweaking until an appropriate amount is determined.
Split the clients up into different groups that are scheduled to run at different times under different Workflows.
Each workflow uses system resources for about an hour before new workflows share those resources. This scheduling prevents resource spikes and reduces wait times.

Option Two:

Use the PSS backup configuration variable "PSS:timeout_mins" to extend the timeout value of the PSS save process, this assures that the Parent save process allows more waiting time for its child save processes to respond:

1. Enable diagnostic mode on NetWorker Management Console (NMC) [View -> Diagnostic Mode]
2. Open client properties
3. Go to App & Modules tab
4. Enter the following value in "save operations" PSS:timeout_mins=60, then press ok
5. Check the backup success rate on the next run

• NetWorker: Troubleshooting High Memory or CPU usage by Process
• NetWorker: Best practices for networking configuration