Metro Node:升级到 8.0.x 后,元数据备份将停止工作
Summary: 本文讨论升级到 8.0.x 代码后,元数据备份变得无法作的问题。本文提供了还原元数据备份功能的解决方法步骤。
Symptoms
戴尔受影响的硬件:
Metro Node mn114
Metro Node mn215
Metro Node-Local/Metro
戴尔受影响的软件:
Metro node OS 8.0.0.0.0.267
Metro node OS 8.0.0.1.0.21
Metro node OS 8.0.1.0.0.220
受影响的变更活动:
升级到 Metro Node OS 8.0.x 后
问题:
-
如果您主要使用 CIFS/NFS 来写入备份,则应使用
ndu pre-check命令为 Metro Node 配置中的每个群集报告以下错误:Cluster-1 的示例:
VPlexcli:/> ndu pre-check Warning: During the NDU process, multiple directors will be offline for a portion of the time. This is non-disruptive but is dependent on a host-based multipathing solution being installed, configured, and operating on all connected hosts. ================================================================================ Performing NDU pre-checks ================================================================================ Verify NDU is not in progress.. OK Verify that the directors have been running continuously for 15 days.. OK Verify director communication status.. OK . . . Verify meta-volume backup configuration.. ERROR . . . ================================================================================ Errors (x errors found) ================================================================================ cluster-1 Metadata backups are NOT created according to schedule Last backup: Mon Aug 19 00:00:00 UTC 20xx Current time: Fri Dec 13 03:41:33 UTC 20xx There has been no metadata backup for 116 day(s) Run 'metadatabackup local' on cluster-1
Cluster-2 的示例:
VPlexcli:/> ndu pre-check Warning: During the NDU process, multiple directors will be offline for a portion of the time. This is non-disruptive but is dependent on a host-based multipathing solution being installed, configured, and operating on all connected hosts. ================================================================================ Performing NDU pre-checks ================================================================================ Verify NDU is not in progress.. OK Verify that the directors have been running continuously for 15 days.. OK Verify director communication status.. OK . . . Verify meta-volume backup configuration.. ERROR . . . ================================================================================ Errors (x errors found) ================================================================================ cluster-2 Metadata backups are NOT created according to schedule Last backup: Sat Mar 16 01:30:00 UTC 20xx Current time: Fri Dec 13 03:41:33 UTC 20xx There has been no metadata backup for 272 day(s) Run 'metadatabackup local' on cluster-2
-
当命令
ll ~system-volumes命令运行时,元数据备份卷日期将反映之前的日期。在下面的示例中,元数据备份停止在 Metro 环境中的两个群集上工作:
VPlexcli:/> ll ~system-volumes /clusters/cluster-1/system-volumes: Name Volume Type Operational Health Active Ready Geometry Component Block Block Capacity Slots --------------------------------------- -------------- Status State ------ ----- -------- Count Count Size -------- ----- --------------------------------------- -------------- ----------- ------ ------ ----- -------- --------- -------- ----- -------- ----- meta_C1_xxxxxx meta-volume ok ok true true raid-1 2 20971264 4K 80G 64000 meta_C1_xxxxxxx_backup_20xx-11-21_01-30 meta-volume ok ok false true raid-1 1 20971264 4K 80G 64000 \------------/ date and time the last backup was run /clusters/cluster-2/system-volumes: Name Volume Type Operational Health Active Ready Geometry Component Block Block Capacity Slots --------------------------------------- -------------- Status State ------ ----- -------- Count Count Size -------- ----- --------------------------------------- -------------- ----------- ------ ------ ----- -------- --------- -------- ----- -------- ----- meta_C2_xxxxxx meta-volume ok ok true true raid-1 2 20971264 4K 80G 64000 meta_C2_xxxxxxx_backup_20xx-11-20_12-43 meta-volume ok ok false true raid-1 1 20971264 4K 80G 64000 \------------/ date and time the last backup was run
症状:
- 元数据备份在 Metro 环境中 的两个 群集上停止工作。
- 元数据备份停止在 Metro 环境中 的任 一群集上工作
- 元数据备份在 本地 群集中停止工作
Cause
在计划的每日元数据备份期间,服务“daily_metadata_backup.service”偶尔会在 director-1-1-A、director-2-1-A 或两者上卡在激活状态。
Resolution
永久性解决方案:
Metro Node 工程部门正在调查此问题。当修复程序可用时,本文将进行更新。
解决办法:
-
要检查服务“daily_metadata_backup.service”的状态,请在 Shell 提示符处运行以下命令:
sudo systemctl status daily_metadata_backup.service在 A 节点上,例如 director-1-1-A 或 director-2-1-A。检查并确认“Active: activating (start)”属性存在,并且该属性的运行时间超过一分钟。如果是,这意味着此服务卡在该特定 A 节点上。以下示例显示,director-1-1-A 和 director-2-1-A 都存在服务“daily_metadata_backup.service”属性“Active: activating (start)”,并且运行时间超过一分钟,这意味着此服务卡在这些节点上,如下所示。
群集-1:
service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.service ● daily_metadata_backup.service - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.service; static) Active: activating (start) since Sat 2024-10-xx 01:30:18 UTC; 1 month 3 days ago <--------------------------- TriggeredBy: ● daily_metadata_backup.timer Main PID: 22553 (daily_metadata_) Tasks: 1 CGroup: /system.slice/daily_metadata_backup.service └─22553 /usr/bin/python3 /opt/dell/vplex/sbin/daily_metadata_backup.py Oct xx 01:30:18 director-2-1-a systemd[1]: Starting metronode automated daily metadata backups... . . . <truncated>群集 2:
service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.service ● daily_metadata_backup.service - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.service; static) Active: activating (start) since Sat 2024-10-xx 01:30:18 UTC; 1 month 2 days ago <--------------------------- TriggeredBy: ● daily_metadata_backup.timer Main PID: 22553 (daily_metadata_) Tasks: 1 CGroup: /system.slice/daily_metadata_backup.service └─22553 /usr/bin/python3 /opt/dell/vplex/sbin/daily_metadata_backup.py Oct xx 01:30:18 director-1-1-a systemd[1]: Starting metronode automated daily metadata backups... . . . <truncated> -
接下来,要检查 A 节点上服务“daily_metadata_backup.timer”的状态,例如 director-1-1-A、director-2-1-A,请运行命令
sudo systemctl status daily_metadata_backup.timer并确认“触发器:”属性显示为“不适用”。如果是,这意味着此服务卡在该特定 A 节点上。以下示例显示,director-1-1-A 和 director-2-1-A 的服务“daily_metadata_backup.timer”属性“Trigger:”显示为“n/a”,这意味着此服务卡在这些节点上。
群集-1:
service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.timer ● daily_metadata_backup.timer - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d └─daily_backup.conf Active: active (running) since Wed 2024-11-20 12:46:10 UTC; 18h ago Trigger: n/a <<<<<<<<<<<< Triggers: ● daily_metadata_backup.service Nov 20 12:46:10 director-1-1-a systemd[1]: Started metronode automated daily metadata backups. service@director-1-1-a:~>
群集 2:
service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.timer ● daily_metadata_backup.timer - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d └─daily_backup.conf Active: active (running) since Wed 2024-11-xx 12:46:10 UTC; 18h ago Trigger: n/a >>>>>>>>>>>>>>>>>>>>>>> Triggers: ● daily_metadata_backup.service Nov xx 12:46:10 director-2-1-a systemd[1]: Started metronode automated daily metadata backups. service@director-2-1-a:~>
-
确认是哪个节点(或者可能是两个节点)将提到的两个服务卡住后,停止服务“daily_metadata_backup.service”和“daily_metadata_backup.timer”,然后启动“daily_metadata_backup.timer”服务以解决这种情况,并让元数据备份开始运行。
提醒:请勿使用 “restart”命令选项。在下面的示例中,由于两个 A 节点都受到影响,因此服务将按如下所示停止和启动:
sudo systemctl stop daily_metadata_backup.service
sudo systemctl stop daily_metadata_backup.timer
sudo systemctl start daily_metadata_backup.timer
-
运行以下命令以检查状态以确认它不再卡住,如下所示:
以下示例显示了运行“daily_metadata_backup.service”的状态命令,以检查表示服务确实未运行的“Active: inactive (dead)”行是否在等待元数据的下一个备份周期时为“inactive (dead)”:
service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.service ● daily_metadata_backup.service - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.service; static) Active: inactive (dead) since Fri 2024-11-22 21:07:41 UTC; 1min 49s ago >>>>>>>>>>>> TriggeredBy: ● daily_metadata_backup.timer Process: 9183 ExecStart=/opt/dell/vplex/sbin/daily_metadata_backup.py (code=exited, status=0/SUCCESS) Main PID: 9183 (code=exited, status=0/SUCCESS) Nov 22 21:07:36 director-2-1-a systemd[1]: Starting metronode automated daily metadata backups... Nov 22 21:07:41 director-2-1-a systemd[1]: daily_metadata_backup.service: Succeeded. Nov 22 21:07:41 director-2-1-a systemd[1]: Finished metronode automated daily metadata backups. service@director-2-1-a:~>service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.service ● daily_metadata_backup.service - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.service; static) Active: inactive (dead) since Fri 2024-11-22 21:07:41 UTC; 1min 49s ago >>>>>>>>>>>> TriggeredBy: ● daily_metadata_backup.timer Process: 9183 ExecStart=/opt/dell/vplex/sbin/daily_metadata_backup.py (code=exited, status=0/SUCCESS) Main PID: 9183 (code=exited, status=0/SUCCESS) Nov 22 21:07:36 director-1-1-a systemd[1]: Starting metronode automated daily metadata backups... Nov 22 21:07:41 director-1-1-a systemd[1]: daily_metadata_backup.service: Succeeded. Nov 22 21:07:41 director-1-1-a systemd[1]: Finished metronode automated daily metadata backups. service@director-2-1-a:~>下面的示例显示服务“daily_metadata_backup.timer”应为“active(waiting)”,并且“Trigger”应设置为当前或现在日期,这表示服务现在按预期运行:
service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.timer ● daily_metadata_backup.timer - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d └─daily_backup.conf Active: active (waiting) since Fri 2024-11-22 21:09:24 UTC; 14s ago >>>>>>>>>>> Trigger: Sat 2024-11-23 01:30:00 UTC; 4h 20min left >>>>>>>>>>> Triggers: ● daily_metadata_backup.service Nov 22 21:09:24 director-2-1-a systemd[1]: Started metronode automated daily metadata backups. service@director-2-1-a:~>service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.timer ● daily_metadata_backup.timer - metronode automated daily metadata backups Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d └─daily_backup.conf Active: active (waiting) since Fri 2024-11-22 21:09:24 UTC; 14s ago >>>>>>>>>>> Trigger: Sat 2024-11-23 01:30:00 UTC; 4h 20min left >>>>>>>>>>> Triggers: ● daily_metadata_backup.service Nov 22 21:09:24 director-1-1-a systemd[1]: Started metronode automated daily metadata backups. service@director-2-1-a:~> -
通过运行以下命令,等待并监视下一次元数据备份完成
ll ~system-volumes命令确认问题已解决,并且元数据备份成功进行,如下所示。示例:
VPlexcli:/> ll ~system-volumes /clusters/cluster-1/system-volumes: Name Volume Type Operational Health Active Ready Geometry Component Block Block Capacity Slots --------------------------------------- -------------- Status State ------ ----- -------- Count Count Size -------- ----- --------------------------------------- -------------- ----------- ------ ------ ----- -------- --------- -------- ----- -------- ----- meta_C1_xxxxxx meta-volume ok ok true true raid-1 2 20971264 4K 80G 64000 meta_C1_xxxxxxx_backup_2024-11-23_01-30 meta-volume ok ok false true raid-1 1 20971264 4K 80G 64000 meta_C1_4UQT429_backup_2024-11-24_01-30 meta-volume ok ok false true raid-1 1 20971264 4K 80G 64000 /clusters/cluster-2/system-volumes: Name Volume Type Operational Health Active Ready Geometry Component Block Block Capacity Slots --------------------------------------- -------------- Status State ------ ----- -------- Count Count Size -------- ----- --------------------------------------- -------------- ----------- ------ ------ ----- -------- --------- -------- ----- -------- ----- meta_C2_xxxxxx meta-volume ok ok true true raid-1 2 20971264 4K 80G 64000 meta_C2_xxxxxxx_backup_2024-11-23_12-43 meta-volume ok ok false true raid-1 1 20971264 4K 80G 64000 meta_C2_xxxxxxx_backup_2024-11-24_12-43 meta-volume ok ok false true raid-1 1 20971264 4K 80G 64000