Data Domain: What is Multi Stream Restore (MSR) as available in DDOS 6.2 and later

MSR is enabled, by default, on applicable DDOS releases and DD/DDVE devices as enumerated above. It works without any need for configuration or tuning, and works transparently (neither the administrator nor the backup application need to do anything or change anything outside the DDOS for MSR to work).
  If MSR turns out to be supported but disabled by default for the DD, and it would benefit the particular workload, contact Dell Data Domain Support for assistance.

MSR only works for sequential reads for files larger than 8 GiB and only after at least 500 MiB of the file had been read to start with, which is the amount of data the internal heuristics need to process to determine if MSR is applicable to the ongoing read or not. The number of parallel read threads which a single external file read turns into depends on the DD/DDVE hardware, with values of 4 and 8 threads being typical.

These and other values are configurable to some extent, but only through prior consultation and analysis by Data Domain Support, on a case by case basis.

Multi Stream Restore (MSR) is a feature added to physical DDs as of DDOS 6.2, to on-premise DDVEs as of DDOS 7.0, and to off-premise Active Tier on Object Store (ATOS) DDVEs as of DDOS 7.2.

The purpose of this feature is to speed up read loads (both reads from Active tier, such as restores, and Cloud tier, for file recalls) for larger files, so that if the backup application only uses a single stream to read from the file, the DD internally reads the file by using several threads in parallel, thereby returning the data to the backup application faster than if the file was read through a single sequential process.

N/A

When a single-streamed read for a file lands in the DD FS process, the heuristics for MSR kick in, and wait for the read to proceed to see if:     

• The file is larger than the minimum (8 GiB by default)
• If at least 500 MiB of the file has been read sequentially (non-sequential reads/restores do not qualify for MSR)
• If the system is not loaded enough for MSR spawning additional sub-threads not putting performance at risk

If this is the case, the FS process internally create a number of streams (typically 4 or 8) for non-overlapping file offsets, which read from the file in parallel, so that the external stream (the one used by the backup application to request the file read from the DD) is fed with data sequentially, theoretically at a much faster speed than if the read occurred using a single internal stream.

During the lifetime of the external read, internal threads may complete reading from their pre-assigned offsets and move on to ones further in the file being read. Also, MSR continuously monitors for system load and read activity on the file, so that if either the file reads stop or the system load goes higher, it may tear down the internal threads, and leave the file read with the single external thread it would have had on a non MSR system.

There are no CLI sections in the GUI or stats printed in daily ASUPs for MSR, as these are per file read and short lived. A user may check the contents of the "ddfs.info" log file ("log view debug/ddfs.info" from the CLI)  for matches for "MSR" and "_msr" to see some chatter about MSR being used for individual files. Something similar to the below (and much additional chatter) when MSR kicks off for a file being read may be seen:     
 
Eventually MSR stops being used (file stopped being read, file closed) but it may also occur due to non-sequential reads received or even due to system load. In that case, something similar to the below may be seen in the logs: