Skip to main content
Sign Out

Welcome to Dell

My Account
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
  • Manage your Dell EMC sites, products, and product-level contacts using Company Administration.
Sign In Create an Account Dell Financial Services Premier Sign In Partner Program Sign In
Contact Us

Your Dell.com Carts

PowerScale OneFS 9.8.0.0 Web Administration Guide

PDF

Dell PowerScale Datamover (SmartSync) overview

Dell PowerScale OneFS Datamover (also called SmartSync) enables you to transfer data between PowerScale clusters and S3 object stores (ECS, AWS) using the Datamover transfer engine that is embedded in OneFS. Datamover ensures that you have a consistent copy of your data on another PowerScale cluster or cloud platform. Datamover allows you to control the frequency of data transfers at scheduled times using policies. Similar to the SyncIQ module, you can transfer data at the directory level, while optionally excluding specific files and subdirectories from being transferred.

The embedded Datamover feature provides data replication for file and object deployments on-premises or in the cloud. Datamover enables file-to-file transfers between PowerScale clusters using RPC and file-to-object copy transfers to S3 (ECS, AWS) and Azure cloud systems.

This section provides an overview of the Datamover transfer engine. You can configure and administer Datamover using OneFS CLI commands. See the PowerScale OneFS CLI Administration Guide for details.

Datamover provides the following primary functions:

  • Data protection
  • Data repurposing (copy)
  • Data archive

Datamover provides a flexible execution model of push or pull data transfers between systems. While SyncIQ allows administrators to push data from a source to a target cluster, Datamover also allows for a target cluster to pull data from a source cluster, resulting in reduced throughput and CPU impacts on the source cluster.

  • Faster data transfers than SyncIQ
  • Snapshot locking
  • Separation between Datamover datasets and user snapshots prevents accidental deletion of snapshots during transfers
  • Scalable run-time engine
  • Dataset "reconnect" allows systems with identical datasets to reconnect for instant incremental backups during failover scenarios
  • Namespace contention avoidance
  • Batch operations for efficient small file transfers
  • Bulk operations to address file ID-mapping contention
  • Improved bench marking
  • Smart scheduling
  • CPU and bandwidth throttling
  • Centralized management of policies and jobs
  • Replication between multiple sets of clusters
  • NANO(A)N: Not-All-Nodes-On-(All)-Networks detection. Active accounts are monitored on-the-fly by each node.
    • Nodes with no accessibility to an account do not participate in a transfer
  • Improved error handling and graceful crash recovery to ensure checkpointing stability
  • Data recovery
    • You can restore from a dataset that you replicated to another cluster. For example, you can copy a dataset from an archive tier to a production tier as a one-time copy. That copy is read/write on arrival.
    • You can perform a one-time copy at any time to your archive tier or between any two clusters. A one-time copy provides the option to not make Datamover datasets, which means you can start using them read/write immediately.

File-to-file high-performance data transfers

  • Streamlined baseline and incremental file transfers
  • Namespace contention avoidance. Namespace creation is separated from data transfers
  • Batch transfers of small files, attributes, and data blocks
  • Asynchronous I/O backed by lightweight threads (fibers) allows for maximized parallel transfers

File-to-object content distribution "copy" format limitations

The following table lists current limitations in file-to-object transfers.

Table 1. Current LimitationsThe following table describes each limitation.
Limitation Description
ADS files Skipped when encountered
Hardlinks Not supported. An object is created for each link (hard links are not preserved)
Symlinks Skipped when encountered
Special files Skipped when encountered
Metadata Only the following POSIX attributes are copied: mode, UID, GID, atime, mtime, ctime
File name encoding Encodings are converted to UTF-8
Large files Errors are returned for files greater than the cloud provider's maximum object size
Sparse files Sparse sections are not preserved; they are written out fully as zeros
CloudPools Not supported
Compression in transit Not supported
Copy back from the cloud Not supported if the data was not created by Data Mover
Incremental transfers Not supported for file-to-object transfers. Only one-time copy to cloud oir copy back from cloud is supported

Licensing and credential requirements

  • Datamover must be hosted on all PowerScale clusters where transfers are planned.
  • For OneFS copy to cloud and copy back from cloud transfers, Datamover is installed on OneFS but not on the cloud systems.
  • Datamover waits for the administrator to commit the OneFS upgrade to OneFS 9.4.0.0.
  • You must have a current SyncIQ (ISI_LICENSING_SYNCIQ_v_2_0) license activated on PowerScale clusters before you can run Datamover between them.
    • Datamover is enabled when a SyncIQ license is activated and the certificates in the following table are configured.
    • If a SyncIQ license expires after configuring policies, the jobs will continue to run. Datamover Rest APIs will serve the GET and DELETE calls; PUT and POST calls will not be allowed.
  • A shared CA is the simplest configuration, however it is not a requirement to communicate with peer Datamover engines. Two systems trust each other if they have the CAs that signed each other's identity certificates.
  • Users must have the ISI_PRIV_DATAMOVER administrative (AIMA) privilege to configure the Datamover using the Rest APIs.
  • Inbound TCP port 7722 must be opened in firewalls.

Certificate requirements

The following Certificate Authorities (CA) and trust hierarchies are required.

Table 2. Certificate RequirementsThe following table describes each certificate requirement.
Requirement Description
TLS certificates
  • A mutually authenticated TLS handshake is required. Authorization, authentication, and encryption are provided by TLS certificates.
  • TLS certificates are always required for daemon startup and all communication between Datamover engines.
  • Encryption can be disabled, but authorization and authentication cannot be disabled.
Certificate Authorities (CA)
  • One or more Certificate Authorities (CA) are required on each Datamover system.
  • Dell recommends that customers use a new, Datamover-specific CA for signing Datamover identity certificates.
  • The CA that signs an identity certificate does not need to be installed on the system that the identity certificate is installed on. Two systems trust each other if they have the CAs that signed each other's identity certificates.
Identity certificates
  • The certificate that provides authentication of the identity claimed.
  • Exactly one identity certificate must exist on each Datamover system.
  • Identity certificates are signed by one of the CAs deployed on the systems that the system is going to communicate with.
Trust hierarchies
  • Two systems trust each other if they have the CAs that signed each other's identity certificates.
  • There is no concept of unidirectional trust—trust is entirely mutual.

Reference documentation

The OneFS CLI Administration Guide provides details for configuring and administering Datamover. The Datamover feature includes a full set of isi dm command line interface (CLI) commands and APIs in the PowerScale OneFS 9.4.0.0 CLI Command Reference and PowerScale OneFS 9.4.0.0 API Reference Guides.

You can find these guides under the Documentation tab on the PowerScale OneFS Support site.

Rate this content

Accurate
Useful
Easy to understand
Was this article helpful?
0/3000 characters
  Please provide ratings (1-5 stars).
  Please provide ratings (1-5 stars).
  Please provide ratings (1-5 stars).
  Please select whether the article was helpful or not.
  Comments cannot contain these special characters: <>()\