Data Domain - Best practices for data streams sent to DD systems with DDOS 7.x.

Background on DDR Streams

In general, a stream in this context refers to the byte-stream associated with a sequential file access. Usually, we are referring to a large file being written for a backup or read for a restore (hence "write stream" and "read stream").

Additionally, there are two other types of streams referenced in discussion below:

• Managed File Replication stream -- associated with a file replication operation, also known as optdup (optimized duplication) in the Symantec NetBackup and BackupExec applications. This feature was introduced in DD OS release 4.7.

• MTree Replication stream -- associated with an MTree replication context, while MTree replication is in progress. This feature was introduced in DD OS release 5.1

Although the use cases for these replication streams are quite different from the normal backup and restore operations, the stream resource utilization within the DDR is roughly equivalent to a read stream (for a source context) or a write stream (for a destination context).

The Supported Stream Count table referenced in below, represent the recommended limits for concurrent read and write streams. Exceeding these limits results in performance degradation (throughput as well as compression). Internally, we refer to them as soft limits. Additionally, there is an internal hard limit that refers to the total number of streams that are supported by DDFS. The NFS server in DDFS recycles stream resources to remain under the hard limit. It is this recycling (or thrashing) of stream resources that is responsible for the steep performance decline when the hard limit is exceeded. The hard limit is 2x the soft limit. Stream thrashing will occur if the number of concurrent streams exceed the hard limit. Stream thrashing causes extremely low (around 1mb/sec) performance, and reverse compression. When the file system detects stream thrashing, it generates the "Too many streams open" alert message.

As for replication streams, the Supported Stream Count table referenced in below use the term 'Repl streams' to refer to all types of replication streams (where applicable).

For MTree replication, the values in the Supported Stream Count table referenced in below indicates the maximum number of contexts that can be created. This is an enforced limit that cannot be exceeded. In some cases, we allow more MTree replication contexts to be created than the read/write stream soft limit would dictate because replication utilization can be less demanding than backup and recover operations, and having the ability to oversubscribe beyond the soft limit offers greater replication density (e.g. in many-to-1 replication topologies).

For optdup (file replication) streams, the values in the Supported Stream Count table referenced in below are recommended limits, as described above for read and write streams counts.

Best practices for data streams sent to DD systems

For optimal performance, Dell EMC recommends limits on simultaneous streams between DD systems, and your backup servers.

A data stream, in the context of the following table, refers to a large byte stream associated with sequential file access, such as a write stream to a backup file or a read stream from a restore image. A Replication source or destination stream refers to a replication operation or a DD Boost file replication stream associated with a file replication operation.

Replication streams count shown graph applies to directory replication, ddboost replication and mtree replication.

Replication (MRepl, MFR) could fail with error "cannot connect, Unknown error, 5 (RPC: Timed out); 5029 rpc operation/function failure", when destination DD is rejecting connection to its TCP 2051 (Rejected connection attempt to port 2051 from client).