Avamar：Isilon 9.1.0 NDMP 备份停止响应，并由对等节点重置连接

在 NDMP作期间观察到备份失败。

使用 NDMP 备份 Isilon 9.1.0 群集时，观察到以下症状：

• NDMP 备份开始，但在达到特定字节偏移后停止响应。
• 备份作业最终失败，并报告错误消息“Connection reset by peer”（连接被对等节点重置）。
• 系统日志包含类似于以下内容的条目，指示群集节点上的低内存状况

2021-09-07T13:59:26-07:00 <3.3> [NODE-2] isi_ndmp_d[30159]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.
2021-09-07T16:35:15-07:00 <3.3> [NODE-1] isi_ndmp_d[32728]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.
2021-09-08T08:16:08-07:00 <3.3> [NODE-3] isi_ndmp_d[64816]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.

导致 NDMP 备份失败的底层问题

Isilon 软件缺陷导致内存耗尽

由于软件缺陷导致群集节点耗尽可用内存，运行版本 9.1.0 的 Isilon 群集上的 NDMP 备份进程停止响应。内存耗尽时，NDMP 守护程序无法分配所需的读取器缓冲区，从而导致错误“连接被对等方重置”和备份终止。

• 版本 9.1.0 的 Isilon 代码库中存在缺陷，导致节点在 NDMP 文件传输期间达到内存不足的情况。
• 内存不足时，NDMP 守护程序会记录以下错误消息：

ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.

• 在备份的数据传输阶段触发内存不足情况，导致备份在特定字节偏移处停止并最终失败。

缺陷的证据

验证命令在系统日志中显示已安装的 RUP 以及是否存在低内存错误。

# Verify installed RUP
isi upgrade patches list

# isi upgrade patches list
Patch Name                              Description                     Status
---------------------------------------------------------------------------
9.1.0.9_GA-RUP_2021-07_PSP-1248         Multiple Userspace and Kernel Fixes  Installed
9.X_GA-TOOLS_2021-06_PSP-1306           Support Tools                     Installed
HealthCheck_9.1.0_2021-06               [9.1.0 UHC 2021-06] HealthCheck   RUP Installed
---------------------------------------------------------------------------
Total: 3

# Search for low‑memory errors in logs across the cluster
isi_for_array -QX "grep -h 'isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition' /var/log/messages" | sort | tail -40

2021-09-07T13:59:26-07:00 <3.3> RED...(id2) isi_ndmp_d[30159]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.
2021-09-07T16:35:15-07:00 <3.3> RED...(id1) isi_ndmp_d[32728]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.
... (additional similar entries)

• 多个节点上的重复日志条目可确认内存耗尽是群集范围的软件问题。

验证已安装的 RUP 和修补程序级别

确认 Isilon 群集上的当前 RUP 版本。

# Run as root
isi upgrade patches list

输出示例：

REDACTED_ISILON_NODE-1# isi upgrade patches list
Patch Name                              Description                         Status
---------------------------------------------------------------------------
9.1.0.9_GA-RUP_2021-07_PSP-1248          Multiple Userspace and Kernel Fixes Installed
9.X_GA-TOOLS_2021-06_PSP-1306            Support Tools                        Installed
HealthCheck_9.1.0_2021-06               [9.1.0 UHC 2021-06] HealthCheck RUP Installed
---------------------------------------------------------------------------
Total: 3

检查低内存错误

在群集日志中搜索特定的 NDMP 内存不足消息。

isi_for_array -QX "grep -h 'isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition' /var/log/messages" | sort | tail -40

典型日志条目：

2021-09-07T13:59:26-07:00 <3.3> REDACTED_ISILON_NODE-2(id2) isi_ndmp_d[30159]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.
2021-09-07T16:35:15-07:00 <3.3> REDACTED_ISILON_NODE-1(id1) isi_ndmp_d[32728]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later.
...

应用解决方法（临时）

• 向 Dell EMC Isilon 支持部门创建支持案例。
• 在应用永久修复之前，请求官方解决方法以缓解内存不足的情况。

永久修复 — 升级 Isilon 代码

• 从 Dell EMC 支持门户下载包含版本 9.1.0.10 的最新 RUP。
• 计划维护窗口以应用升级。
• 在所有群集节点上安装新 RUP：

# Example installation command (run on each node)
isi upgrade apply -p /path/to/9.1.0.10_RUP.pkg

• 升级后，验证新的修补程序级别：

isi upgrade patches list

升级后验证

• 运行新的 NDMP 备份作业并监视进度。
• 确认备份已完成，并且没有出现“Connection reset by peer”错误。
• 从“检查低内存错误”部分重新运行日志搜索命令;不应出现新条目。