PowerFlex: K8s CSI Driver for PowerFlex v1.2.1 has a low inode on xfs format

A customer uses the CSI Driver for PowerFlex version 1.2.1 to format a PowerFlex volume as an XFS file system type.

Symptoms

The inode size becomes 256 bytes (isize=256) when creating an XFS file system from a 2TB PowerFlex volume by the CSI Driver version 1.2.1. Also, CRC is disabled (crc=0), while it was isize=512 before.

The issue was not present in the CSI Driver for PowerFlex version 1.2.0.

The inode size becomes 512 bytes (isize=512) and CRC is enabled (crc=1) when creating an XFS file system from a 2TB PowerFlex volume by the mkfs.xfs Linux command with the default option: 

cf. man page for mkfs.xfs: mkfs.xfs(8) — Linux manual page  (External Link)

Size=value (isize)

The inode size is specified either as a value in bytes with size= or as the number fitting in a file system block with perblock=.

The minimum (and default) value is 256 bytes without crc, 512 bytes with crc enabled.

crc=value is used to create a file system which maintains and checks CRC information in all metadata objects on a disk.

The value is either 0 to disable the feature, or 1 to enable the use of CRCs. By default, mkfs.xfs enables metadata CRCs.

Impact: Users cannot have a volume with more than 400 million files due to a lack of free inodes.

The root cause is not on the PowerFlex software side but on the CSI Driver side.

With the CSI driver, a Go library called gofsutil is internally used to interact with a file system format operation, and there is a side effect of forcing formatting with crc=0 in that library: https://github.com/dell/gofsutil/blob/v1.4.0/gofsutil_mount_linux.go#L133  (External Link)

The issue introduces the Universal Base Image (UBI) version change with the gofsutil for CSI Driver for PowerFlex version 1.2.1: https://github.com/dell/csi-powerflex/compare/v1.2.0…v1.2.1#diff-d947c64a65fbadc3496bff6bf4f3dd823c7300954d9072ddbd3f54513c2766b5R7  (External Link)

mkfs.xfs command examples:

mkfs.xfs -i size=256 is not allowed; it is required by disabling "crc":

$ mkfs.xfs -i size=256 /dev/scinib
Minimum inode size for CRCs is 512 bytes

The mkfs.xfs -m crc=0 option gets an xfs file system with isize=256 and crc=0, which is equivalent to CSI driver v1.2.1:

$ xfs_info /dev/scinib
meta-data=/dev/scinib            isize=256    agcount=4, agsize=134217728 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=536870912, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=262144, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

The mkfs.xfs default option gets an xfs file system with isize=512 and crc=1, which are expected metadata values in general:

$ xfs_info /dev/scinib
meta-data=/dev/scinib            isize=512    agcount=4, agsize=134217728 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=536870912, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=262144, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

This is not a PowerFlex software issue.

CSI Driver case handling notes:

• Typically, the ViPR/SRM L2 team is the first contact point for CSI-related issues.
• The PowerFlex L2 (and L3) team can possibly be engaged on a case-by-case basis for collaboration from the PowerFlex software perspective.
• Refer to PowerFlex: Requirements for Kubernetes CSI Escalation

References for CSI Driver for PowerFlex:

• csi-powerflex (CSI Driver for PowerFlex): https://github.com/dell/csi-powerflex (External Link)
• gofsutil (A Go library for filesystem-related operations such as mount, format, so on): https://github.com/dell/gofsutil (External Link)