NetWorker: Backups to Newly Added Data Domain Fail with "Too many save streams" Error

• Backups to the new Data Domain fail or stop responding.
• Backup sessions remain in Waiting for writable volume state.
• The following error is logged in the daemon logs of the NetWorker Server:
  • Linux: /nsr/logs/daemon.raw
  • Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw
  • NetWorker: How to use nsr_render_log to render .raw log files

nsrd GSS warning Cannot use DD device err: Too many save streams (0) on DDR 'DD_name'since that would cause the device to exceed the maximum DDR write stream counts (0)

• A new Data Domain is added on the local Storage Node.
• After successful creation of the NSR Data Domain resource, the following fields are not populated:
  • Max Write Streams
  • Max Read Streams
  • Capacity
  • Serial Number
  • DD OS Version

• Device creation, labeling, and mounting operations complete successfully.
• If a different Storage Node (not the backup server itself) is selected when adding the Data Domain:
  • Stream values, serial number, and DD OS version populate correctly.
  • Backups complete successfully.
• Required ports (111 and 2049) are confirmed open between the NetWorker server and the Data Domain.
  • nsrports -t DATA_DOMAIN_HOSTNAME -p PORT

The issue is caused by missing Max Write Streams information in the NSR Data Domain resource.

When this field (and related Data Domain attributes) is not populated, NetWorker defaults the maximum write streams value to 0, which results in backup failures with a Too many save streams error.

Follow the action plan below to remove and re-create the affected resources so that the Data Domain attributes are correctly populated.
 

Action Plan:

1. Open an elevated prompt on the NetWorker server.
2. Collect the bootstrap save set information and save it in another location: mminfo -B
3. Stop NetWorker Services

Windows: net stop nsrexecd
Linux: systemctl stop networker

4. Create a copy of the NetWorker Server Resource Database (NSRDB) folder.

Windows (Default): C:\Program Files\EMC Networker\nsr\res\nsrdb
Linux: /nsr/res/nsrdb

5. From an elevated prompt, go to the res folder:

Windows: cd "C:\Program Files\EMC Networker\nsr\res"
Linux: cd /nsr/res

6. Use the following nsradmin command to access the NSRDB in offline mode

nsradmin -d nsrdb

7. Delete the Affected Storage Node Resource

p type:NSR Storage Node
delete type:NSR Storage Node; name:<Storage_Node_Name>
delete? y
deleted resource id xxxxx

8. Delete the Data Domain Resource

p type:NSR Data Domain
delete type:NSR Data Domain; name:<Data_Domain_Name>
delete? y
deleted resource id xxxxx

9. Delete the Associated Device Resources

p type:NSR device
delete type:NSR device; name:<Device_Name>
delete? y
deleted resource id xxxxx

Repeat this step for all devices associated with the affected Data Domain.

10. Start NetWorker Services

11. Re-create Resources
    1. Log in to the NetWorker Management Console (NMC).
    2. Verify that the Storage Node is automatically re-created
    3. Readd the Data Domain system.
    4. Re-create the devices.
12. Confirm that each of the following settings is correctly populated in the Data Domain System properties in NetWorker.

• Max write read streams
• Serial number
• DD OS version

13. Trigger the backups and confirm successful completion.