Data Domain: Troubleshooting Slow Single File CIFS Restore Performance

This is the most important information to acknowledge, with and without a backup application a single file CIFS restore uses only one stream. 

The Data Domain performs this operation as a simple sequential file read dictated by the operating system's file transfer protocol (CIFS/Server Message Block (SMB)). 

Since the data is being read from a single mounted network share path using a standard protocol, it operates as a single logical connection or stream.

CIFS SMB uses a single stream for a single-file restore because the protocol enforces ordered writes through one file handle to maintain data integrity and consistency. 

This single stream restore operation per file is not a design limitation from the data domain but unique to the CIFS process itself. 

If you are using a backup application and want to verify with your vendor, then open a case with the vendor and verify stream usage as well as other configuration. 

Also, besides how the CIFS process works other components involved in the restore process are the following:

• Client-side
  Network Interface Card (NIC) configuration (speed, duplex, offloads)
  MTU settings
  CPU or memory utilization
• Network Path
  Switches (port speed, duplex, buffer utilization, QoS policies)
  Routers (routing policies, MTU, ACLs)
  Firewalls (packet inspection, rate limiting, SMB-specific filtering)
  Load balancers (if present)
  VLAN configuration (tagging, MTU consistency)
  SFPs/Transceivers (check for errors, mismatched speeds)
  Cables (damaged or low-quality cables causing CRC errors)
  Network congestion (monitor for high utilization or packet drops)
• Storage-side
  MTU settings on storage interfaces MTU must be consistent end-to-end (client → switch → router → firewall → storage). A mismatch anywhere can cause fragmentation or dropped packets, which kills CIFS performance.
  NIC bonding/LACP configuration
  Storage controller CPU or memory load
  Disk backend performance (write cache, tiering, latency)
• Intermediate Devices
  IDS/IPS systems/cloud strike antivirus (deep packet inspection can throttle SMB)
  WAN optimizers (if remote restore)
  VPN tunnels (encryption overhead, MTU mismatch)
• Physical Layer
  Patch panels and cabling integrity
  Fiber cleanliness and proper insertion for SFPs
• Other Factors
  Jumbo frame consistency (MTU across all hops)
  Network errors (CRC, drops, retransmissions)
  QoS or traffic shaping policies that deprioritize SMB traffic
• MTU Consistency Test - Verify end-to-end MTU using ping with -f and appropriate packet size.
  Iperf Throughput Test - Measure raw network bandwidth between client and Data Domain.
  NIC Speed or Duplex Settings - Confirm that both ends are at correct speed (1/10/25/40GbE) and full duplex.
  Network Switch Configuration - Check port speed, buffer utilization, and QoS policies.
  Router/Firewall Inspection - Ensure no packet inspection or rate limiting throttles SMB traffic.
  Cable and SFP Health - Inspect for CRC errors, damaged cables, or mismatched transceivers.
  Backup Application Load - Confirm no heavy jobs (full backups, deduplication tasks) running concurrently.
  Backup Job Priority - Ensure that the restore job is not throttled or running at low priority.
  Inline Processing Overhead - Disable compression, encryption, or deduplication during restore if possible
  Storage Controller Load - Check DD CPU or memory utilization and disk backend latency.
  QoS or Traffic Shaping Policies - Verify that SMB traffic is not deprioritized on the network.

Further troubleshooting can be done to check the health of CIFS restore. 

Verify if the restore is connecting to an IP address that is in the same subnet as the data domain. 

If you are uncertain, then verify if the data domain has gateways for the subnets of each of its IP addresses. 
Log in to the CLI and run the command to display data domain IP addresses. 

net show set

Then check the gateways. 

net route show gateways detailed 

It is recommended that gateways are added for all subnets. 

net route add gateway x.x.x.x 

• Check if the Maximum Transmission Unit (MTU) is consistent from the client performing the restore. 
• If the MTU test fails at 1472, then try a lower MTU such as 1372. 

WINDOWS:
If MTU is standard 1500 
C:> ping xx.xx.xx.xx -f -l 1472 
OR 
If you are using Jumbo  
C:> ping xx.xx.xx.xx -f -l 8972   

From DD side,
# net ping <IP address> count 2 packet-size 1472 path-mtu do
If you know the interface on the data domain that communicates with the restore you can specify it in the command. 
# net ping <IP address> count 2 packet-size 1472 path-mtu do interface ethXx (MTU size of 1500)
# net ping <IP address> count 2 packet-size 8972 path-mtu do interface ethXx (MTU size of 9000)  

• Checking MTU on the client ensures that its network interface matches the storage and network path settings, preventing packet fragmentation and restore performance issues.
• MTU is the largest packet size that can be transmitted over a network without fragmentation.

If you are unsure what the MTU size is on the window client, you can run the command. 

netsh interface ipv4 show subinterfaces

Or with PowerShell 

Get-NetIPInterface | Select-Object

If there is an issue with MTU, then contact your network team for further troubleshooting as the issue is not from the data domain side but within your network on your switches, routers, firewalls. 

Also what is important besides the ping is the milliseconds and round-trip time. If a ping results in high milliseconds it is assumed that the device is not on a LAN but a WAN. 

If on a WAN, then the further distance the higher the impact of throughput There is nothing support can do if distance is the issue, it is best to speak to your Internet service provider. 

If MTU is consistent and pings result in no packet loss, then next check bandwidth in the direction of the data domain to the client. 

You can compare both directions from data domain to client and from client to data domain to ensure that they are the same. 

Iperf measures the remaining bandwidth that is left over, so if you have a backup running then it measures the remaining bandwidth. 

If you want to measure the full bandwidth, then it is best to stop all jobs. 
Data Domain: Troubleshooting Network Performance Using iperf

If MTU is consistent and bandwidth is also measured to be accurate, then a windows explorer drag-and-drop test can be done. 

Examples go to the windows explorer folder where the file is and drag-and-drop it into the CIFS share drive on the Data Domain. 

If drag-and-drop is faster than the backup application, then it is possible there is an issue with the backup application. 

Below are commands that display statistics for CPU, network, and disk. 

CIFS commands:

system show stats view cifs interval 2

sysadmin@DD6900-2# system show stats view cifs interval 2

INTERVAL: 2 secs
"-" indicates that system is busy and unable to get recent data.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
CPU      |State     |CIFS                                                            |Protocol Aggr                 |Net        |Disk                |NVRAM
aggr aggr|          |                                                                |                              | aggr  aggr|                    |   aggr    aggr
busy  max|          |ops/s      in     out  read write create getattr setattr readdir|        ops/s data in data out|   in   out|   read   write busy|   read   write
   %    %|CDPVMSFIRL|         MB/s    MB/s ops/s ops/s  ops/s   ops/s   ops/s   ops/s|                 MB/s     MB/s| MB/s  MB/s|  KiB/s   KiB/s    %|  KiB/s   KiB/s
---- ---- ---------- ----- ------- ------- ----- ----- ------ ------- ------- ------- ------------- ------- -------- ----- ----- ------- ------- ---- ------- -------
   2    8        I       0       0       0     0     0      0       0       0       0             5    0.09     0.01  0.10  0.02    1649    1070    0       -       -
   6    7        I       0       0       0     0     0      0       0       0       0             0       0        0  0.09  0.02    1084     962    0       -       -
   7    7        I       0       0       0     0     0      0       0       0       0             0       0        0     0     0    1090    4385    0       0       5
   6    7        I       0       0       0     0     0      0       0       0       0             0       0        0     0     0    1087     312    0       0      39
   7    7        I       0       0       0     0     0      0       0       0       0             0       0        0     0     0    1077    1097    0       0      27
   7    7        I       0       0       0     0     0      0       0       0       0             3       0        0     0     0    1043    3285    0       0       4
   6    7        I       0       0       0     0     0      0       0       0       0             8       0        0     0     0    1043    1845    0       0      15

Below is an example output of the command to check for frame errors on data domain interfaces. 

If the frame counter is high, it is an indication that an SFP, Ethernet cable, or optical cable is faulty either on the data domain side or on the switch side. 

Contact your network team to replace cable and or SFP. 
Data Domain - Troubleshooting Network Frame errors

sysadmin@DD6900-2# sysadmin@DD6900-2# net show config
ethMa     Link encap:Ethernet  HWaddr 34:80:0D:05:A7:6C
          inet addr:10.60.36.127  Bcast:10.60.36.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1000  Metric:1
          RX packets:183498916 errors:0 dropped:10508 overruns:0 frame:0 <========
          TX packets:68813481 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:144299913892 (134.3 GiB)  TX bytes:7890283345 (7.3 GiB)
          Interrupt:152 Memory:a2060000-a207ffff

Next you want to check cleaning configuration on the data domain. 

The standard configuration is a 50% throttle scheduled on Tuesdays. 

If the throttle is set to higher than 50% in addition to scheduled multiple times per week, it can impact performance. 

You can also set the throttle to lower than 50% temporarily so that it frees up extra resources but it is not guaranteed to improve the restore.  

Below is an example of a command and clean configuration on data domain 6900. 

sysadmin@dd6900# filesys clean show config
        50 Percent Throttle
Filesystem cleaning is scheduled to run "Tue" at "0600".

If you are using a backup application for CIFS restores, check the following: 

Performance is limited to one logical stream per file when restoring a single file over CIFS/SMB. To avoid additional bottlenecks caused by the backup application, check the following:

• Ensure that no full backups, synthetic backups, or large multistream restores are running concurrently.
• Check for deduplication or compression jobs that consume CPU and I/O resources on the backup server. Confirm Restore Job Priority.
• Ensure the single-file restore job is not running with low priority or in a throttled resource group.
• If the application supports job prioritization, temporarily boost the restore job priority. Check for Resource Contention
• Validate that the backup server has sufficient CPU, memory, and disk I/O available.
• Look for queue delays or pending tasks in the backup application that could slow down the restore. Disable Inline Processing During Restore.
• Turn off inline deduplication, compression, or encryption for the restore job if possible.
• These features add overhead and can slow down single-stream CIFS transfers. Review Network Settings in Backup Application
• Confirm that the application is not enforcing bandwidth throttling or rate limits for restore jobs.
• Check if client-side multiplexing is disabled for CIFS restores (some apps allow multiple streams for multifile restores but not for single-file). Check Logging for Errors or Retries.
• Look for I/O retries, timeout errors, or oplock conflicts in the backup application logs
• Excessive retries indicate network or CIFS server issues that required troubleshooting. Pause Non-Essential Maintenance Tasks
• Stop indexing, catalog updates, or verification jobs during the restore window.
• These tasks can compete for disk and network resources.

If you have thoroughly health checked your switches, routers, backup application, firewall, bandwidth, network, host machine, backup application and still want Data Domain, then open a case and provide the information below. 

• What is the Backup Application and version? 
• What are the restore times? 
• What is the file size? 
• What is the Client Operating System?
• What is the hostname of the slow client?
• What is the IP address of the slow client?
• What is the IP address on the dd that is in use for the restore? 
• What is the filename of the restore job? 
• Have there been any recent changes in the environment? 
• What is the MTree being used for backups? 
• Is the restore traffic data travel over a WAN or Lan? 
  • If over a WAN what is the distance and locations of the client and data domain? 
• Does backup traffic travel over a VPN? 
• Do you have any IDS, IPS, or IDPS on your network?
• Are you using QoS?
• What is the MTU size of the network?
• Is the backup application restore data precompressed? 
• Is the backup application restore data encrypted? 
• Is the backup application restore data deduplicated?
• Is the Data Domain a DDVE in the cloud? Amazon, Google, Azure, Alibaba? 
• Is the client also in the cloud?      
• What type of restore job from the backup application is slow?