The PowerFlex scini driver (SDC) reports memory allocation errors (blkIO_HandleChecksumOnIoStart)

Scenario
SDC installed on Linux
Volumes mapped to the SDC are allocated from a storage pool configured to use checksum.
The system experiences memory shortage or memory fragmentation.

Symptoms
Messages similar to the following appear in the syslog, usually: /var/log/messages

Dec 12 15:36:09 server002 kernel: kworker/16:2H: page allocation failure: order:8, mode:0x104020 
Dec 12 15:36:09 server002 kernel: CPU: 16 PID: 11460 Comm: kworker/16:2H Tainted: P OE ------------ 3.10.0-514.21.1.el7.x86_64 #1 
Dec 12 15:36:09 server002 kernel: Hardware name: HP ProLiant DL560 Gen9/ProLiant DL560 Gen9, BIOS P85 01/04/2016 
Dec 12 15:36:09 server002 kernel: Workqueue: kblockd blk_delay_work 
Dec 12 15:36:09 server002 kernel: 0000000000104020 0000000054aac964 ffff886aecfd37c0 ffffffff81686e13 
Dec 12 15:36:09 server002 kernel: ffff886aecfd3850 ffffffff81187090 ffff88607ffda000 0000000000000010 
Dec 12 15:36:09 server002 kernel: ffffffffffffff00 0010402000000000 0022522000000000 0000000054aac964 
Dec 12 15:36:09 server002 kernel: Call Trace: 
Dec 12 15:36:09 server002 kernel: [<ffffffff81686e13>] dump_stack+0x19/0x1b 
Dec 12 15:36:09 server002 kernel: [<ffffffff81187090>] warn_alloc_failed+0x110/0x180 
Dec 12 15:36:09 server002 kernel: [<ffffffff816829a7>] __alloc_pages_slowpath+0x6b7/0x725 Dec 12 15:36:09 server002 kernel: [<ffffffff8118b645>] __alloc_pages_nodemask+0x405/0x420 Dec 12 15:36:09 server002 kernel: [<ffffffff811cf7fa>] alloc_pages_current+0xaa/0x170 
Dec 12 15:36:09 server002 kernel: [<ffffffff81185f6e>] __get_free_pages+0xe/0x50 
Dec 12 15:36:09 server002 kernel: [<ffffffff811db09e>] kmalloc_order_trace+0x2e/0xa0 
Dec 12 15:36:09 server002 kernel: [<ffffffff811dd871>] __kmalloc+0x221/0x240 
Dec 12 15:36:09 server002 kernel: [<ffffffff811de7c3>] ? kmem_cache_alloc+0x193/0x1e0 
Dec 12 15:36:09 server002 kernel: [<ffffffffa07b697a>] blkIO_OnChecksumIoStart+0x5a/0x550 [scini] 
Dec 12 15:36:09 server002 kernel: [<ffffffffa07ed587>] ? mosMCAllocator_Alloc+0x57/0xa0 [scini] 
Dec 12 15:36:09 server002 kernel: [<ffffffffa07b3ac1>] blkIO_HandleChecksumOnIoStart+0xc1/0x220 [scini] 
Dec 12 15:36:09 server002 kernel: [<ffffffffa07b7ea8>] mapVolIO_Do_CK+0x398/0x1e80 [scini] Dec 12 15:36:09 server002 kernel: [<ffffffffa07b4306>] blkIO_Do_CK+0x266/0xa00 [scini] 
Dec 12 15:36:09 server002 kernel: [<ffffffffa07e9838>] mosMit_EntryWrapper+0x128/0x1b0 [scini] 
Dec 12 15:36:09 server002 kernel: [<ffffffffa07e9a2c>] mosMit_StartNow+0x7c/0x90 [scini] 
Dec 12 15:36:09 server002 kernel: [<ffffffffa07b4ea6>] blkIO_Request+0x356/0x730 [scini] 
Dec 12 15:36:09 server002 kernel: [<ffffffff810cf27e>] ? dequeue_task_fair+0x41e/0x660 
Dec 12 15:36:09 server002 kernel: [<ffffffff812eb7e3>] __blk_run_queue+0x33/0x40 
Dec 12 15:36:09 server002 kernel: [<ffffffff812ed915>] blk_delay_work+0x25/0x60 
Dec 12 15:36:09 server002 kernel: [<ffffffff810a845b>] process_one_work+0x17b/0x470 
Dec 12 15:36:09 server002 kernel: [<ffffffff810a9296>] worker_thread+0x126/0x410 
Dec 12 15:36:09 server002 kernel: [<ffffffff810a9170>] ? rescuer_thread+0x460/0x460 
Dec 12 15:36:09 server002 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 
Dec 12 15:36:09 server002 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Dec 12 15:36:09 server002 kernel: [<ffffffff81697418>] ret_from_fork+0x58/0x90 
Dec 12 15:36:09 server002 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140

The messages begin with: page allocation failure
The stack trace includes: blkIO_HandleChecksumOnIoStart

Impact
The ScaleIO-internal checksum mechanism may not verify some I/O operations.

When the checksum feature is enabled on a storage pool, I/O operations on volumes allocated from the storage pool have a checksum included with the network messages between SDC and SDS.
To calculate these checksum values, the SDC dynamically allocates memory.
If the host is suffering from memory shortage or memory fragmentation, the memory allocation required for the checksum operation may fail.

When the memory allocation fails, the SDC cannot calculate the checksum, and does not attach it to the network message.
When the SDS receives a message from the SDC, which does not include a checksum, it performs the I/O operation, without verifying the contents of the message on its side.

To conclude, the memory allocation failures have no effect on applications' I/O operations or ScaleIO performance.

If possible, ensure that the host has enough free unfragmented memory.

Impacted versions
All ScaleIO versions - work as designed

Fixed in version
N/A