SC Storage Customer Notification: Linux Large IO Sizes Can Affect iSCSI Volume Availability

Alert:

These changes are outlined below.

Oracle Linux 6/7:

ELSA-2015-3098 - Unbreakable Enterprise kernel security and bugfix update

http://linux.oracle.com/errata/ELSA-2015-3098.html

Release Date: 2015-11-13

[3.8.13-100.el6uek/3.8.13-100.el7uek]

block: remove artifical max_hw_sectors cap (Christoph Hellwig) [Orabug: 21511611]

Oracle Linux 5:

ELBA-2015-3076 - Unbreakable Enterprise kernel bug fix update

http://linux.oracle.com/errata/ELBA-2015-3076.html

Release Date: 2015-08-27

[2.6.39-400.260.1.el5uek]

block: remove artifical max_hw_sectors cap (Joe Jin) [Orabug: 21455630]

Overview:

Each block device that exists on a Linux system is allocated a queue directory (located at /sys/block/xxx/queue/). This directory contains a number of configuration files. Details of these files can be located at:

https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt

The two parameters that are relevant to this CSTA:

max_hw_sectors_kb

The maximum number of kilobytes supported in a single data transfer.

This parameter is read-only and provided by the target storage device through SCSI inquiry response. The SCSI Target subsystem within Storage Center Operating System (SCOS) currently responds with a zero. This indicates no limit, so the server sets the value to the maximum allowed within that linux kernel version (32MB).

max_sectors_kb

The maximum number of kilobytes that the block layer will allow for a filesystem request.

This value is read-write. This value must be smaller than or equal to the max_hw_sectors_kb value.

The changes to the Oracle Linux kernel that were introduced in kernels 2.6.39-400.260.1.el5uek, 3.8.13-100.el6uek, and 3.8.13-100.el7uek changed the way that max_sectors_kb is calculated. In previous kernel releases, the value was always set to 512KB. After this change, the value of max_sectors_kb is set to the value of max_hw_sectors_kb. This essentially increased the maximum size of a single IO transfer to a given block device from 512KB to 32MB.

This change exposed a problem with the Storage Center 10Gb iSCSI driver for IO sizes greater than ~8MB. The server process producing the large IO may hang or the volume may become unavailable due to the large IO size. In addition, the 10Gb iSCSI driver issue may cause the controller to consume memory required by other Storage Center subsystems, which could lead to a larger impact.

Workaround:

In order to prevent this problem, the max_sectors_kb for each block device must be changed dynamically on the Linux server. Below are sample commands for a multipath block device connected to a Linux system:

#mpath1:

echo 512 > /sys/block/dm-6/queue/max_sectors_kb

echo 512 > /sys/block/sdc/queue/max_sectors_kb

echo 512 > /sys/block/sde/queue/max_sectors_kb

echo 512 > /sys/block/sdk/queue/max_sectors_kb

echo 512 > /sys/block/sdm/queue/max_sectors_kb

These changes are not persistent through reboots. An init script can be implemented to run when a given server boots that will modify this parameter for all iSCSI block devices. Please consult a Linux administrator or Oracle support on how best to implement this persistence for each customer environment.

The following script (udev rule) will set the max_sectors_kb of every iSCSI attached Compellent volume to 512KB. Please use this script at your own discretion as no guarantees or warranties are implied. It is recommended to test the script, but it should work without issue.

Create the file 99-cml-iscsi.rules in the directory /etc/udev/rules.d/ with the following contents:

[root@localhost /]# cat /etc/udev/rules.d/99-cml-iscsi.rules

# Modify max_sectors_kb for Dell Storage Center iSCSI mapped volumes

ACTION=="add",\

ENV =="*-iscsi-*compellent*",\

RUN+="/bin/sh -c 'echo 512 > /sys$DEVPATH/queue/max_sectors_kb'"

Affected versions:

All SCOS versions are impacted that utilize 10Gb iSCSI interfaces. Fibre Channel, 1 Gb iSCSI, and FCoE are not affected.

Resolution:

The max_sectors_kb issue has been resolved in SCOS versions 7.01.12, 7.02.01, and 7.03.01 and higher.

Changes are available in SCOS 6.07.03 and higher to address the memory consumption issue with the 10Gb iSCSI driver.