PowerFlex SDC 在将新 SDS 添加到群集后出现 IO 错误

Summary: 将新 SDS 添加到 PowerFlex 群集并将数据重新平衡到新 SDS 后，某些 SDC 显示 I/O 错误

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

- SDS 已添加到群集（来自 MDM 事件）

2022-03-07 18:28:07.080 MDM_CLI_CONF_COMMAND_RECEIVED INFO     	 Command add_sds received, User: 'admin'. [4576114] protection_domain with ID a876543f00000000, fault_set N/A, 

New SDS name: sds12, Hostnames: 172.1.0.12, Port: 7072
2022-03-07 18:28:07.324 CLI_COMMAND_SUCCEEDED         INFO     	 Command add_sds succeeded. [4576114] ID: 7d120650000002f

 - SDC 日志显示尝试连接到新 SDS 时的内核堆栈跟踪。当 SDC 尝试连接到新 SDS 时，这些可能会显示多次：

  
022-03-07T18:29:02.880040-05:00 sdc32 kernel: net_sched: page allocation failure: order:4, mode:0x104020
2022-03-07T18:29:02.880170-05:00 sdc32 kernel: CPU: 16 PID: 5662 Comm: net_sched Kdump: loaded Tainted: P           OE  ------------   3.10.0-1062.el7.x86_64 #1
2022-03-07T18:29:02.880242-05:00 sdc32 kernel: Hardware name: Dell Inc. PowerEdge R740xd/5XXXXX, BIOS 2.4.8 11/26/2019
2022-03-07T18:29:02.880355-05:00 sdc32 kernel: Call Trace:
2022-03-07T18:29:02.880453-05:00 sdc32 kernel: [<ffffffffbb179262>] dump_stack+0x19/0x1b
2022-03-07T18:29:02.880553-05:00 sdc32 kernel: [<ffffffffbabc23d0>] warn_alloc_failed+0x110/0x180
2022-03-07T18:29:02.880623-05:00 sdc32 kernel: [<ffffffffbb1747fc>] __alloc_pages_slowpath+0x6b6/0x724
2022-03-07T18:29:02.880709-05:00 sdc32 kernel: [<ffffffffc09c021e>] ? netAddress_ToStrCopy+0x2de/0x520 [scini]
2022-03-07T18:29:02.880787-05:00 sdc32 kernel: [<ffffffffbabc6b84>] __alloc_pages_nodemask+0x404/0x420
2022-03-07T18:29:02.880851-05:00 sdc32 kernel: [<ffffffffbac14c68>] alloc_pages_current+0x98/0x110
2022-03-07T18:29:02.880922-05:00 sdc32 kernel: [<ffffffffbabc117e>] __get_free_pages+0xe/0x40
2022-03-07T18:29:02.880995-05:00 sdc32 kernel: [<ffffffffbac2052e>] kmalloc_order_trace+0x2e/0xa0
2022-03-07T18:29:02.881053-05:00 sdc32 kernel: [<ffffffffbac24571>] __kmalloc+0x211/0x230
2022-03-07T18:29:02.881110-05:00 sdc32 kernel: [<ffffffffc0988274>] mapClass_AllocAndInitObj+0x44/0x140 [scini]
2022-03-07T18:29:02.881167-05:00 sdc32 kernel: [<ffffffffc098959e>] mapClass_UpdateAll+0x40e/0x9b0 [scini]
2022-03-07T18:29:02.881229-05:00 sdc32 kernel: [<ffffffffc09c9994>] ? mosMitSchedThrd_CurThrdOurs+0x64/0x90 [scini]
2022-03-07T18:29:02.881298-05:00 sdc32 kernel: [<ffffffffc099aa07>] mapMdm_HandleObjUpdate_CK+0x327/0x730 [scini]
2022-03-07T18:29:02.881360-05:00 sdc32 kernel: [<ffffffffc099c070>] ? mapMdm_SendUpdateReq_CK+0xf0/0xed0 [scini]
2022-03-07T18:29:02.881419-05:00 sdc32 kernel: [<ffffffffc099c4b0>] mapMdm_SendUpdateReq_CK+0x530/0xed0 [scini]
2022-03-07T18:29:02.881480-05:00 sdc32 kernel: [<ffffffffc09a704f>] ? netSock_DoIO+0xef/0x7d0 [scini]
2022-03-07T18:29:02.881543-05:00 sdc32 kernel: [<ffffffffc09aaf47>] ? netChanThrottler_TryToWakeupWaiterOfCond+0x27/0x100 [scini]
2022-03-07T18:29:02.881606-05:00 sdc32 kernel: [<ffffffffc09ab118>] ? netChanThrottler_TryToWakeupWaiter+0x58/0x80 [scini]
2022-03-07T18:29:02.881660-05:00 sdc32 kernel: [<ffffffffc095cc80>] ? netChan_SendReq_CK+0x90/0xa70 [scini]
2022-03-07T18:29:02.881716-05:00 sdc32 kernel: [<ffffffffc095cdd0>] netChan_SendReq_CK+0x1e0/0xa70 [scini]
2022-03-07T18:29:02.881775-05:00 sdc32 kernel: [<ffffffffc09699c5>] netCon_SendReq_CK+0x175/0x590 [scini]
2022-03-07T18:29:02.881835-05:00 sdc32 kernel: [<ffffffffc0963cf7>] ? netRPC_SendDone_CK+0x47/0xab0 [scini]
2022-03-07T18:29:02.881890-05:00 sdc32 kernel: [<ffffffffc0963dd3>] netRPC_SendDone_CK+0x123/0xab0 [scini]
2022-03-07T18:29:02.881959-05:00 sdc32 kernel: [<ffffffffc09c9c24>] mosMit_RunWithTLS+0x54/0x60 [scini]
2022-03-07T18:29:02.882017-05:00 sdc32 kernel: [<ffffffffc09cbb92>] mosMitSchedThrd_ThrdEntry+0x1a2/0x500 [scini]
2022-03-07T18:29:02.882071-05:00 sdc32 kernel: [<ffffffffc09c82f0>] ? mosTicks_DestroyEnvSpecific+0x10/0x10 [scini]
2022-03-07T18:29:02.882126-05:00 sdc32 kernel: [<ffffffffc09c8310>] mosOsThrd_Entry+0x20/0x50 [scini]
2022-03-07T18:29:02.882185-05:00 sdc32 kernel: [<ffffffffbaac50d1>] kthread+0xd1/0xe0
2022-03-07T18:29:02.882240-05:00 sdc32 kernel: [<ffffffffbaac5000>] ? insert_kthread_work+0x40/0x40
2022-03-07T18:29:02.882294-05:00 sdc32 kernel: [<ffffffffbb18bd37>] ret_from_fork_nospec_begin+0x21/0x21
2022-03-07T18:29:02.882351-05:00 sdc32 kernel: [<ffffffffbaac5000>] ? insert_kthread_work+0x40/0x40
2022-03-07T18:29:02.891431-05:00 sdc32 kernel: ScaleIO mapClass_AllocAndInitObj:1301 :Error: Failed to allocate memory 38480.Cannot process MDM response


 - SDC 日志随后显示 I/O 错误：
 

2022-03-07T18:49:03.652009-05:00 sdc32 kernel: ScaleIO mapMultiHead_UpdateInPlace:561 :Warning: Invalid primaryTgtIdx 47 at 2. Multi-head ID dd990042.
2022-03-07T18:49:03.652130-05:00 sdc32 kernel: ScaleIO mapClass_UpdateAll:614 :Error: Object ffff948c7f350000 (class multi_head) failed to update in place.status NO_RESOURCES (67)
2022-03-07T18:49:04.662144-05:00 sdc32 kernel: ScaleIO mapVolIO_ReportIOErrorIfNeeded:491 :[67641433832] IO-ERROR Type READ. comb: 6ecc80210154. offsetInComb 14536800. SizeInLB 256. SDS_ID 7dd0d24700000033. Comb Gen 21c1. Head Gen 222c. StartLB 5fb1ac860.
2022-03-07T18:49:04.662268-05:00 sdc32 kernel: blk_update_request: I/O error, dev scinic, sector 25687672672


 

影响

某些 SDC 上出现 I/O 错误



 

Cause

运行 SDC 的主机作系统的连续内存区块在 64 KB 范围内。“page allocation failure： order：4“消息告诉我们这是一个不可用的 64 KB 区块。SDC 要求作系统分配内存以创建新插槽并连接到新添加的 SDS，但这是失败的原因。如果 SDC 无法连接到新 SDS，并且数据重新平衡到新 SDS，则会导致 SDC 主机上出现 IO 错误。

Resolution

通过将新的 SDS 置于即时维护模式，可以立即停止 IO 错误。SDC 可以从其余 SDS 获取其数据。当 SDC 内存问题得到解决时，使 SDS 退出维护。 

这里有几件事要做，在行动计划中都应该考虑：

  1. 如果需要确定的方法来修复它，则重新启动 SDC 主机可修复该问题。它会重新初始化作系统，之前存在的内存碎片将消失
     

  2. 如果无法重新启动，请使用以下命令。它会尝试尽可能多地回收和压缩碎片化内存。这不能保证有效，并且可能会在回收/压缩内存碎片的几秒钟内中断当前应用程序： 

    sync; echo 1 > /proc/sys/vm/drop_caches; echo 1 > /proc/sys/vm/compact_memory



    使用“cat /proc/buddyinfo”检查内存使用情况，查看上述命令是否有帮助。在运行上述命令之前和之后使用它，看看它是否高于零。我们查看的是“Node 0， zone Normal”行中的第四列，这是可用内存块的 64 KB 连续计数。在此实例中，它显示在运行上述命令以回收/压缩之前：

    sdc32:~ # cat /proc/buddyinfo
Node 0, zone      DMA        1        0       0      1      2      1      1      0      1      1      2
Node 0, zone    DMA32        9        5       7      5      9      6      8      6      7      5    295
Node 0, zone   Normal   909921  1263646  202478      0<===  0      0      0      0      0      0      0
Node 1, zone   Normal	 39085	      0	      0      0      0      0      0      0      0      0      0

     

  3. 使用“sysctl -a”检查内核上的内存过量使用设置。如果内核使用默认值，如下所示：
    vm.overcommit_memory = 0
    vm.overcommit_ratio = 50

    与作系统供应商讨论更改这些参数将是有利的，以便在重新启动后为作系统提供更多内存以供使用，并避免将来出现此问题：
    vm.overcommit_memory = 2
    vm.overcommit_ratio = 98
     
  4. 如果这成为作系统内存碎片的常规模式，则可能需要升级 Linux 内核

 

受影响的版本

所有 PowerFlex SDC 都可能受到作系统内存碎片问题的影响。
 

Affected Products

PowerFlex Software
Article Properties
Article Number: 000197199
Article Type: Solution
Last Modified: 09 Jun 2025
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.