Faster Block Device Performance with NVMe PCIe SSD based dm-cache on RHEL 7

Faster Block Device Performance with NVMe PCIe SSD based dm-cache on RHEL 7

dm-cache (Device Mapper cache) is a device mapper target to improve performance of a block device (e.g., a spindle device ) HDD by dynamically migrating some of its data to a faster, smaller device (SSD). NVMe based PCIe SSD will act as the cache device.
For more details about the NVMe based PCIe SSD refer the blog in the link dm-cache can be configured using the user space tool set LVM2( logical volume manager tools).

Virtual cache created by dm-cache is made of three physical devices. (Refer Fig1).The Origin device is the slower destination storage device (HDD).The Cache device is used to store the user data block and the meta device stores the meta data like block placement, dirty flags and other internal data. In this case we use same device [Cache device] for both data blocks and metadata

Steps to setup and configure dm-cache using LVM tools
Figure 1: Steps to setup and configure dm-cache using LVM tools

Hardware Setup:

Origin Device [HDD] : /dev/sda1
Cache Device [NVMe based PCIe SSD] (Block/Meta): /dev/nvme0n1p1

Software Setup:

Operating system : Rhel 7
Tools : LVM2 user space tools for dm-cache configuration

Configuration Steps:

Following are the steps to configure the Virtual device dm-cache using LVM .

Each step output maps to block with numbering mentioned in Fig.1

1) Create 100 GB partition on the hard disk [Origin Device]

parted -a optimal /dev/sda mkpart primary 1 100G

This creates /dev/sda1

2) Create 10 GB partition on the NVMe PCIe SSD disk [Cache Device]

parted -a optimal /dev/nvme0n1 mkpart primary 1 10G

3) Create a Volume group (cache) with the Origin and Cache device

vgcreate cache /dev/sda1 /dev/nvme0n1p1

Check for the "cache" volume group by vgdisplay

[root@localhost ~]# vgdisplay

--- Volume group ---

VG Name cache

System ID

Format lvm2

Metadata Areas 2

Metadata Sequence No 7

VG Access read/write

VG Status resizable


Cur LV 2

Open LV 0

Max PV 0

Cur PV 2

Act PV 2

VG Size 103.36 GiB

PE Size 4.00 MiB

Total PE 26461

Alloc PE / Size 25399 / 99.21 GiB

Free PE / Size 1062 / 4.15 GiB

VG UUID Zd8dNe-6Kdt-7qgY-dmSN-8WHe-4wqQ-euM3Ql

4) Create Origin Device logical Volume (origin_device)

lvcreate -l 90%FREE -n origin_device cache /dev/sda1

5) Create a cache meta data logical Volume (cache_meta)

lvcreate -l 5%FREE -n cache_meta cache /dev/nvme0n1p1

6) Create a cache block data logical Volume (cache_block)

lvcreate -l 80%FREE -n cache_block cache /dev/nvme0n1p1

7) Create a cache pool logical volume by combining cache block and cache meta data logical volumes (combining step 5 and 6)

lvconvert --type cache-pool --poolmetadata cache/cache_meta cache/cache_block
NOTE : The created cache pool will have the same name as "cache_block" that is the name of cache block data logical volume created in step 6.

This is the behavior of the tool

Check for the "cache" volume group by lvdisplay

root@localhost ~]# lvdisplay

--- Logical volume ---

LV Path /dev/cache/cache_block

LV Name cache_block

VG Name cache

LV UUID kWYQxP-Jdlr-JdxE-aleB-JJpj-3rmw-Q0cojx

LV Write Access read/write

LV Creation host, time localhost.localdomain, 2014-06-28 09:05:32 -0400

LV Status available

# open 0

LV Size 5.07 GiB

Current LE 1297

Segments 1

Allocation inherit

Read ahead sectors auto

- currently set to 256

Block device 253:2

8) Create a cache logical volume by linking the cache pool (step 7) logical volume and origin_device logical volume (step 4)

lvconvert --type cache --cachepool cache/cache_block cache/origin_device

This creates the virtual cache device (dm-cache) orign_device as shown below as dm4

[root@localhost ~]# ls -l /dev/cache/origin_device

lrwxrwxrwx. 1 root root 7 Jun 28 09:13 /dev/cache/origin_device -> ../dm-4

[root@localhost ~]#

9) Format the Virtual cache device with a file system and use it

mkfs.ext4 /dev/cache/origin_device

Performance data

Performance run done using fio tool shows a great improvement of the read,write and read-write mix operations.

Table-1 shows the FIO tool output of performance difference and the improvement in percentage with the dmcache in place compared to the normal drive (rotational drive) and Fig 2.shows the corresponding graph plot.

The test is done with 1M block size for Random Read,Random Write and Random Read-Write mix (50%) on the Original hard drive(sda) and the drive with Dmcache configured

DEVICE Read Write Read-Write
Drive with dmcache 197 798 159
sda [original drive] 124 118 58
Improvement in % 58.87% 576.20% 174.13%
Table 1: Performance Improvement with Dmcache

Performance Improvement with Dmcache
Figure 2: Performance Improvement with Dmcache

Conclusion : Thus NVMe PCIe SSD based dm-cache improves the performance of legacy spindle based hard drives to a greater extent

Article ID: SLN312372

Last Date Modified: 08/14/2018 01:24 AM

Rate this article

Easy to understand
Was this article helpful?
Yes No
Send us feedback
Comments cannot contain these special characters: <>()\
Sorry, our feedback system is currently down. Please try again later.

Thank you for your feedback.