EMC Isilon OneFS SmartFlash

Isilon has leveraged the system memory and non-volatile memory in each node, and then OneFS 7.1.1 enable to use SSDs to increase cache capacity to offer extreme performance. This article provides an introduction to the EMC Isilon SmartFlash.

In traditional scale-up storage systems, cache associated with controller is only available to volumes on that device, however, OneFS uses up to three levels of read cache and an NVRAM/Write Coalescer cache. L2 cached data is available to any node on Isilon cluster, when the same content exists in the private caches of multiple nodes, this cached data is consistent across all instances. OneFS caching hierarchy are illustrated in the following diagram:

L1 Cache

• Prefetchesdata from remote nodes.
• Used by clients connected to that node.
• Data is prefetchedper file.
• Coherent across the cluster, but available only to clients on that node.
• Uses file-based addressing.

L2 Cache

• Contains data from disk drives on the local node.
• Data are used for remote node.
• Globally accessible from any node on cluster.
• Remote nodes access L2 cache via a lookup of the block address.
• No MESI invalidation necessary.

L3 Cache (option)

• Caches evicted L2 blocks on SSDs.
• Only nodes with SSDs can be configured for L3 cache.
• Provides cost effective way to extend main memory.
• Flash storage provides durable, persistent cache.
• If L3 drive fails, some cache will disappear.

OneFS SmartFlash

SmartFlash offer a cost-effective method of extending of main memory per node from gigabytes to terabytes. This allows clients to retain a larger working set of data in cache, before being forced to retrieve data from spinning disk via a HDD operation.

The following diagram illustrates how clients interact with the OneFS read cache infrastructure and the write coalesce. L1 cache still interacts with the L2 cache on any node it requires, and the L2 cache interacts with both the storage subsystem and L3 cache. L3 cache is stored on an SSD within the node and each node in the same node pool has L3 cache enabled. The diagram also illustrates a separate node pool where L3 cache is not enabled. This node pool either does not contain the required SSDs, or has L3 cache disabled, with the SSDs being used for a filesystem-based SmartPools SSD data or metadata strategy.

To maintain L3 cache persistence, the following steps are performed:

1.   L1 cache is checked. If not hit, L2 cache is consulted.

2.   If the data is found in memory, it’s done.

3.   If not in memory, L3 is then checked.

4.   If there’s an L3 hit and that item is near the end of the L3 FIFO, a flag is set on the block that caused it to be evicted to L3 again when it’s evicted out of L2.

Before attempting to use L3 policy on Isilon read, there are some rules to remember:

• It’s not possible that L3 cache and Metadata acceleration policies on the same node pool.
• L3 cache and Metadata policies can exist on different node pools.
• Customers can switch between L3 cache and Metadata policies, however the transition should spend some time to be recognized.
• When switching from Metadata to L3, the metadata copies need to be relocated to disk and L3 needs to be warmed.
• When switching from L3 to Metadata, L3 needs to be cleared and metadata maps copies to SSD.

Enabling SSDs for use as L3 Cache

Below demonstrates how to manage SmartCache in Isilon OneFS.

1. SmartCache should be running in the OneFS 7.1.1 at least.

2. Go to “File System Management” > “Storage Pools” > “Smartpools”, then setting “Enable L3 cache” for managing L3 cache by default on any new node pool with SSDs.

3. Go to “File System Management” > “Storage Pools” > “SmartpoolsSettings”, then setting “use SSDs as L3 Cache by default for new node pools” for a global setting to manage L3 cache by default for new node pools.

4. Using isi_cache_stat for monitoring L3 cache performance.

                                

Author: Jeffey