PowerEdge: Faster Block Device Performance with NVMe PCIe SSD-based dm-cache on RHEL

NVMe based PCIe SSDs can act as the cache device for dm-cache using the user space tool set Logical Volume Manager (LVM) tools.

The virtual cache created by dm-cache is made of three physical devices. (Refer Fig1.) The Origin device is the slower destination storage device (HDD). The Cache device is used to store the user data and the meta cache stores the metadata like block placement, dirty flags, and other internal data.
In this case, we use the same device [Cache device] for both data blocks and metadata.

 
Figure 1: Steps to setup and configure dm-cache using LVM tools.

Hardware Setup:

Origin Device [HDD]: /dev/sda1
Cache Device [NVMe based PCIe SSD] (Block/Meta): /dev/nvme0n1p1 

Software Setup:

Operating system: RHEL 7
Tools: LVM2 user space tools for dm-cache configuration

Configuration Steps:

Following are the steps to configure the Virtual device dm-cache using LVM.

Each step output maps to the block with numbering mentioned in Fig.1.

1. Create 100 GB partition on the hard disk [Origin Device]

   parted -a optimal /dev/sda mkpart primary 1 100G 

   Note: This creates /dev/sda1
2. Create a 10 GB partition on the NVMe PCIe SSD disk [Cache Device]

   parted -a optimal /dev/nvme0n1 mkpart primary 1 10G 

3. Create a Volume group (cache) with the Origin and Cache device

   vgcreate cache /dev/sda1  /dev/nvme0n1p1 

   1. Check for the "cache" volume group using vgdisplay

      [root@localhost ~]# vgdisplay
      
      --- Volume group ---
      VG Name               cache
      System ID
      Format                lvm2
      Metadata Areas        2
      Metadata Sequence No  7
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                2
      Open LV               0
      Max PV                0
      Cur PV                2
      Act PV                2
      VG Size               103.36 GiB
      PE Size               4.00 MiB
      Total PE              26461
      Alloc PE / Size       25399 / 99.21 GiB
      Free  PE / Size       1062 / 4.15 GiB
      VG UUID               Zd8dNe-6Kdt-7qgY-dmSN-8WHe-4wqQ-euM3Ql 

4. Create the Origin Device logical Volume (origin_device)

   lvcreate -l 90%FREE -n origin_device cache /dev/sda1 

5. Create a cache meta data logical Volume (cache_meta)

   lvcreate -l 5%FREE -n cache_meta cache /dev/nvme0n1p1 

6. Create a cache block data logical Volume (cache_block)

   lvcreate -l 80%FREE -n cache_block cache /dev/nvme0n1p1 

7. Create a cache pool logical volume by combining cache block and cache meta data logical volumes (combining step 5 and 6)

   lvconvert --type cache-pool --poolmetadata cache/cache_meta cache/cache_block 

   Note: The created cache pool has the same name as "cache_block" that is the name of the cache block data logical volume created in step 6.

   1. This is the behavior of the tool. Check for the "cache" volume group using lvdisplay

      root@localhost ~]# lvdisplay
      --- Logical volume ---
      LV Path                /dev/cache/cache_block
      LV Name                cache_block
      VG Name                cache
      LV UUID                kWYQxP-Jdlr-JdxE-aleB-JJpj-3rmw-Q0cojx
      LV Write Access        read/write
      LV Creation host, time localhost.localdomain, 2014-06-28 09:05:32 -0400
      LV Status              available
      # open                 0
      LV Size                5.07 GiB
      Current LE             1297
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
      - currently set to     256
      Block device           253:2 

8. Create a cache logical volume by linking the cache pool (step 7) logical volume and origin_device logical volume (step 4)

   lvconvert --type cache --cachepool cache/cache_block cache/origin_device 

   1. This creates the virtual cache device (dm-cache) orign_device as shown below as dm4.

      [root@localhost ~]# ls -l /dev/cache/origin_device
      lrwxrwxrwx. 1 root root 7 Jun 28 09:13 /dev/cache/origin_device -> ../dm-4 

9. Format the Virtual cache device with a file system and use it.

   mkfs.ext4 /dev/cache/origin_device 

Performance Data:

Performance tests done using the FIO tool shows a great improvement of the read, write, and read-write mix operations.

Table-1 shows the FIO tool output of performance difference and the improvement in percentage with the dmcache in place compared to the normal drive (rotational drive) and Fig 2 shows the corresponding graph plot.

The test is done with 1M block size for Random Read, Random Write, and Random Read-Write mix (50%) on the Original hard drive(sda) and the drive with dmcache configured.
 

Table 1: Performance Improvement with dm-cache

 
Figure 2: Performance Improvement with dm-cache.

Conclusion: NVMe PCIe SSD-based dm-cache improves the performance of legacy spindle-based hard drives to a greater extent.