NetWorker: Clone failure issue due to running out of space on the destination DD system

Summary: Encountered a clone failure issue due to running out of space on the destination DD system. In the DD logs, the error message "filecopy failed for... errno=28, errstr=No space left on device" indicates that the destination DD system has run out of space while attempting to copy the file Similarly, the log message "MSG-CM-00002: ... out of space" further confirms that the container set on the destination DD system has run out of space. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

1. Daemon logs show that a clone operation failed due to a termination request. 
5/2/2023 4:54:35 PM 0 0 0 3304 3300 0 nwserver.emc.com nsrd NSR info Error Alert: Clone operation failed with error: 'NSRCLONE stopped due to a termination request. 
5/2/2023 4:56:07 PM 0 0 0 3304 3300 0 nwserver.emc.com nsrd NSR info Error Alert: Clone operation failed with error: 'NSRCLONE stopped due to a termination request.
 
2. Clone logs show that there was an error performing the clone of client saveset. The file copy operation failed with an NFS I/O error. 
5/2/2023 4:07:38 PM Unable to perform clone of client saveset. Error: proxy_clone_rs failed.
5/2/2023 4:07:38 PM Failed to obtain filecopy status for ssid=2320332870, cloned 0 bytes out of 214485172228: Retrieving the status of a file copy operation failed [5009] ([13344] [16888] Tue May 02 16:07:38 2023 ddp_filecopy_status() failed, start_offset[0], length[0], Err: 5009-filecopy operation failed (nfs: I/O error)).
5/2/2023 4:07:38 PM Failed to obtain filecopy status for ssid=2991432354, cloned 0 bytes out of 386350972932: Retrieving the status of a file copy operation failed [5009] ([13344] [17300] Tue May 02 16:07:38 2023 ddp_filecopy_status() failed, start_offset[0], length[0], Err: 5009-filecopy operation failed (nfs: I/O error)).
 
3. Source DD logs indicate that there is no space left on the device and that the container set is out of space. 
05/02 16:07:37.563176 [7f182be641f0] filecopy failed for <file: task idx 214037, /data/col1/nwserver/BACKUPDEV01/15/57/653f07b8-00000006-c64d9d05-644d9d05-0b3b1500-1139205a, destination: 172.31.143.218>, errno=28, errstr=No space left on device
05/02 01:07:40.212021 [10bd5490] CRITICAL: MSG-CM-00002: /../vpart:/vol1/col1/cp1/cset: Container set 21fa5af1e27ac4be:c2a3fa5b19a4ba85 out of space.
05/02 01:07:40.212057 [10c4b110] repl: ctx 572:0 No space left on device
 
4. The destination dd system is running out of space and is unable to complete the operation. 
05/02 01:07:40.211978 [10c4b110] WARNING: MSG-REPL-00403: repl ctx 572:0: server error in send_segs: Out of space
05/02 01:07:40.212010 [10bd5490] DDR_SPACE: set for limit "write", space_id 0x4, full(before) 0x0, full(after) 0x4, full_count 1
05/02 01:07:40.212021 [10bd5490] CRITICAL: MSG-CM-00002: /../vpart:/vol1/col1/cp1/cset: Container set 21fa5af1e27a4abe:c2a3fa5b19a4ba85 out of space.
05/02 01:07:40.212057 [10c4b110] repl: ctx 572:0 No space left on device
 

 

Cause

The symptoms indicate a problem with the device's storage capacity.

The failure to perform the clone operation, along with the "No space left on device" error in the DD logs, suggests that there is insufficient space to perform the operation.

Resolution

  1. Delete unnecessary files: Start by identifying and deleting any unnecessary files or snapshots. You can use a file system management tool to locate and delete the files.
  2. Check the available space: Check the amount of available space in the file system using the "df -h" command.
  3. Identify the full container set: Identify the container set that is out of space by looking at the error message. The container set ID is listed in the message.
  4. Run cleaning To recover space, you must run a cleaning process on the container set that is out of space
  5. Verify space recovery after the cleaning process completes. Verify that space has been recovered by checking the available space using the "df -h" command.

Additional Information

  • It is recommended to review the usage of the destination DD GCCDDVEPOC file system by running the command "filesys show space" and checking the stream count usage using "ddboost stream show active all". This helps identify if the issue is related to running out of space or if there are any other issues that must be addressed.
  • If the usage of the destination DD file system is not the root cause, the next step would be to reproduce the NW clone error and check the source ddfs.info and destination ddfs.info logs for the latest filecopy error. This helps in identifying the exact cause of the error and take appropriate measures to resolve it.

Affected Products

NetWorker
Article Properties
Article Number: 000213160
Article Type: Solution
Last Modified: 08 Sept 2023
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.