Metro Node:升级到 8.0.x 后,元数据备份将停止工作

Summary: 本文讨论升级到 8.0.x 代码后,元数据备份变得无法作的问题。本文提供了还原元数据备份功能的解决方法步骤。

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Symptoms

戴尔受影响的硬件:
Metro Node mn114
Metro Node mn215
Metro Node-Local/Metro

戴尔受影响的软件:
Metro node OS 8.0.0.0.0.267
Metro node OS 8.0.0.1.0.21
Metro node OS 8.0.1.0.0.220

受影响的变更活动:
升级到 Metro Node OS 8.0.x 后

问题:

  1. 如果您主要使用 CIFS/NFS 来写入备份,则应使用 ndu pre-check 命令为 Metro Node 配置中的每个群集报告以下错误:

    Cluster-1 的示例:

    VPlexcli:/> ndu pre-check
    Warning:
    During the NDU process, multiple directors will be offline for a portion of the time.
    This is non-disruptive but is dependent on a host-based multipathing solution being
    installed, configured, and operating on all connected hosts.
    ================================================================================
    Performing NDU pre-checks
    ================================================================================
    Verify NDU is not in progress..                                            OK
    Verify that the directors have been running continuously for 15 days..     OK
    Verify director communication status..                                     OK
    . . .
    Verify meta-volume backup configuration..                                  ERROR
    . . .
    ================================================================================
    Errors (x errors found)
    ================================================================================
    cluster-1
         Metadata backups are NOT created according to schedule
         Last backup: Mon Aug 19 00:00:00 UTC 20xx
         Current time: Fri Dec 13 03:41:33 UTC 20xx
         There has been no metadata backup for 116 day(s)
         Run 'metadatabackup local' on cluster-1

    Cluster-2 的示例:

    VPlexcli:/> ndu pre-check
    Warning:
    During the NDU process, multiple directors will be offline for a portion of the time.
    This is non-disruptive but is dependent on a host-based multipathing solution being
    installed, configured, and operating on all connected hosts.
    ================================================================================
    Performing NDU pre-checks
    ================================================================================
    Verify NDU is not in progress..                                            OK
    Verify that the directors have been running continuously for 15 days..     OK
    Verify director communication status..                                     OK
    . . .
    Verify meta-volume backup configuration..                                  ERROR
    . . .
    ================================================================================
    Errors (x errors found)
    ================================================================================
    cluster-2
         Metadata backups are NOT created according to schedule
         Last backup: Sat Mar 16 01:30:00 UTC 20xx
         Current time: Fri Dec 13 03:41:33 UTC 20xx
         There has been no metadata backup for 272 day(s)
         Run 'metadatabackup local' on cluster-2
  2. 当命令 ll ~system-volumes 命令运行时,元数据备份卷日期将反映之前的日期。

    在下面的示例中,元数据备份停止在 Metro 环境中的两个群集上工作:

    VPlexcli:/> ll ~system-volumes
    /clusters/cluster-1/system-volumes:
    
    Name                                     Volume Type     Operational  Health  Active  Ready  Geometry  Component  Block     Block  Capacity  Slots
    
    ---------------------------------------  --------------  Status       State   ------  -----  --------  Count      Count     Size   --------  -----
    
    ---------------------------------------  --------------  -----------  ------  ------  -----  --------  ---------  --------  -----  --------  -----
    
    meta_C1_xxxxxx                           meta-volume     ok           ok      true    true   raid-1    2          20971264  4K     80G       64000
    
    meta_C1_xxxxxxx_backup_20xx-11-21_01-30  meta-volume     ok           ok      false   true   raid-1    1          20971264  4K     80G       64000
                            \------------/ date and time the last backup was run 
    
    /clusters/cluster-2/system-volumes:
    
    Name                                     Volume Type     Operational  Health  Active  Ready  Geometry  Component  Block     Block  Capacity  Slots
    
    ---------------------------------------  --------------  Status       State   ------  -----  --------  Count      Count     Size   --------  -----
    
    ---------------------------------------  --------------  -----------  ------  ------  -----  --------  ---------  --------  -----  --------  -----
    
    meta_C2_xxxxxx                           meta-volume     ok           ok      true    true   raid-1    2          20971264  4K     80G       64000
    
    meta_C2_xxxxxxx_backup_20xx-11-20_12-43  meta-volume     ok           ok      false   true   raid-1    1          20971264  4K     80G       64000
                           \------------/ date and time the last backup was run

症状:

  • 元数据备份在 Metro 环境中 的两个 群集上停止工作。
  • 元数据备份停止在 Metro 环境中 的任 一群集上工作
  • 元数据备份在 本地 群集中停止工作

 

Cause

在计划的每日元数据备份期间,服务“daily_metadata_backup.service”偶尔会在 director-1-1-A、director-2-1-A 或两者上卡在激活状态。

 

Resolution

永久性解决方案:

Metro Node 工程部门正在调查此问题。当修复程序可用时,本文将进行更新。

解决办法:

  1. 要检查服务“daily_metadata_backup.service”的状态,请在 Shell 提示符处运行以下命令: sudo systemctl status daily_metadata_backup.service 在 A 节点上,例如 director-1-1-A 或 director-2-1-A。检查并确认“Active: activating (start)”属性存在,并且该属性的运行时间超过一分钟。如果是,这意味着此服务卡在该特定 A 节点上。

    以下示例显示,director-1-1-A 和 director-2-1-A 都存在服务“daily_metadata_backup.service”属性“Active: activating (start)”,并且运行时间超过一分钟,这意味着此服务卡在这些节点上,如下所示。

    群集-1:

    service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.service
    ● daily_metadata_backup.service - metronode automated daily metadata backups
         Loaded: loaded (/etc/systemd/system/daily_metadata_backup.service; static)
         Active: activating (start) since Sat 2024-10-xx 01:30:18 UTC; 1 month 3 days ago   <---------------------------
    TriggeredBy: ● daily_metadata_backup.timer
       Main PID: 22553 (daily_metadata_)
          Tasks: 1
         CGroup: /system.slice/daily_metadata_backup.service
                 └─22553 /usr/bin/python3 /opt/dell/vplex/sbin/daily_metadata_backup.py
    Oct xx 01:30:18 director-2-1-a systemd[1]: Starting metronode automated daily metadata backups...
    .
    .
    .
    <truncated>

    群集 2:

    service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.service
    ● daily_metadata_backup.service - metronode automated daily metadata backups
         Loaded: loaded (/etc/systemd/system/daily_metadata_backup.service; static)
         Active: activating (start) since Sat 2024-10-xx 01:30:18 UTC; 1 month 2 days ago   <---------------------------
    TriggeredBy: ● daily_metadata_backup.timer
       Main PID: 22553 (daily_metadata_)
          Tasks: 1
         CGroup: /system.slice/daily_metadata_backup.service
                 └─22553 /usr/bin/python3 /opt/dell/vplex/sbin/daily_metadata_backup.py
    Oct xx 01:30:18 director-1-1-a systemd[1]: Starting metronode automated daily metadata backups...
    .
    .
    .
    <truncated>
  2. 接下来,要检查 A 节点上服务“daily_metadata_backup.timer”的状态,例如 director-1-1-A、director-2-1-A,请运行命令 sudo systemctl status daily_metadata_backup.timer 并确认“触发器:”属性显示为“不适用”。如果是,这意味着此服务卡在该特定 A 节点上。

    以下示例显示,director-1-1-A 和 director-2-1-A 的服务“daily_metadata_backup.timer”属性“Trigger:”显示为“n/a”,这意味着此服务卡在这些节点上。

    群集-1:

    service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.timer
    ● daily_metadata_backup.timer - metronode automated daily metadata backups
    Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d
    └─daily_backup.conf
    Active: active (running) since Wed 2024-11-20 12:46:10 UTC; 18h ago
    Trigger: n/a                       <<<<<<<<<<<<
    Triggers: ● daily_metadata_backup.service
    Nov 20 12:46:10 director-1-1-a systemd[1]: Started metronode automated daily metadata backups.
    service@director-1-1-a:~>

    群集 2:

    service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.timer
    ● daily_metadata_backup.timer - metronode automated daily metadata backups
    Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d
    └─daily_backup.conf
    Active: active (running) since Wed 2024-11-xx 12:46:10 UTC; 18h ago
    Trigger: n/a                           >>>>>>>>>>>>>>>>>>>>>>>
    Triggers: ● daily_metadata_backup.service
    Nov xx 12:46:10 director-2-1-a systemd[1]: Started metronode automated daily metadata backups.
    service@director-2-1-a:~>
  3. 确认是哪个节点(或者可能是两个节点)将提到的两个服务卡住后,停止服务“daily_metadata_backup.service”和“daily_metadata_backup.timer”,然后启动“daily_metadata_backup.timer”服务以解决这种情况,并让元数据备份开始运行。

    提醒:请勿使用 “restart”命令选项。

    在下面的示例中,由于两个 A 节点都受到影响,因此服务将按如下所示停止和启动:

    sudo systemctl stop daily_metadata_backup.service
    sudo systemctl stop daily_metadata_backup.timer
    sudo systemctl start daily_metadata_backup.timer
  4. 运行以下命令以检查状态以确认它不再卡住,如下所示:

    以下示例显示了运行“daily_metadata_backup.service”的状态命令,以检查表示服务确实未运行的“Active: inactive (dead)”行是否在等待元数据的下一个备份周期时为“inactive (dead)”:

    service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.service
    ● daily_metadata_backup.service - metronode automated daily metadata backups
         Loaded:  loaded (/etc/systemd/system/daily_metadata_backup.service; static)
         Active: inactive (dead) since Fri 2024-11-22 21:07:41 UTC; 1min 49s ago          >>>>>>>>>>>>
    TriggeredBy: ● daily_metadata_backup.timer
        Process: 9183 ExecStart=/opt/dell/vplex/sbin/daily_metadata_backup.py (code=exited, status=0/SUCCESS)
       Main PID: 9183 (code=exited, status=0/SUCCESS)
    Nov 22 21:07:36 director-2-1-a systemd[1]: Starting metronode automated daily metadata backups...
    Nov 22 21:07:41 director-2-1-a systemd[1]: daily_metadata_backup.service: Succeeded.
    Nov 22 21:07:41 director-2-1-a systemd[1]: Finished metronode automated daily metadata backups.
    service@director-2-1-a:~>
    service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.service
    ● daily_metadata_backup.service - metronode automated daily metadata backups
         Loaded:  loaded (/etc/systemd/system/daily_metadata_backup.service; static)
         Active: inactive (dead) since Fri 2024-11-22 21:07:41 UTC; 1min 49s ago          >>>>>>>>>>>>
    TriggeredBy: ● daily_metadata_backup.timer
        Process: 9183 ExecStart=/opt/dell/vplex/sbin/daily_metadata_backup.py (code=exited, status=0/SUCCESS)
       Main PID: 9183 (code=exited, status=0/SUCCESS)
    Nov 22 21:07:36 director-1-1-a systemd[1]: Starting metronode automated daily metadata backups...
    Nov 22 21:07:41 director-1-1-a systemd[1]: daily_metadata_backup.service: Succeeded.
    Nov 22 21:07:41 director-1-1-a systemd[1]: Finished metronode automated daily metadata backups.
    service@director-2-1-a:~>

    下面的示例显示服务“daily_metadata_backup.timer”应为“active(waiting)”,并且“Trigger”应设置为当前或现在日期,这表示服务现在按预期运行:

    service@director-2-1-a:~> sudo systemctl status daily_metadata_backup.timer
    ● daily_metadata_backup.timer - metronode automated daily metadata backups
         Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled)
        Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d
                 └─daily_backup.conf
         Active: active (waiting) since Fri 2024-11-22 21:09:24 UTC; 14s ago   >>>>>>>>>>>
        Trigger: Sat 2024-11-23 01:30:00 UTC; 4h 20min left   >>>>>>>>>>>
       Triggers: ● daily_metadata_backup.service
    Nov 22 21:09:24 director-2-1-a systemd[1]: Started metronode automated daily metadata backups.
    service@director-2-1-a:~>
    service@director-1-1-a:~> sudo systemctl status daily_metadata_backup.timer
    ● daily_metadata_backup.timer - metronode automated daily metadata backups
         Loaded: loaded (/etc/systemd/system/daily_metadata_backup.timer; enabled; vendor preset: disabled)
        Drop-In: /etc/systemd/system/daily_metadata_backup.timer.d
                 └─daily_backup.conf
         Active: active (waiting) since Fri 2024-11-22 21:09:24 UTC; 14s ago   >>>>>>>>>>>
        Trigger: Sat 2024-11-23 01:30:00 UTC; 4h 20min left   >>>>>>>>>>>
       Triggers: ● daily_metadata_backup.service
    Nov 22 21:09:24 director-1-1-a systemd[1]: Started metronode automated daily metadata backups.
    service@director-2-1-a:~>
  5. 通过运行以下命令,等待并监视下一次元数据备份完成 ll ~system-volumes 命令确认问题已解决,并且元数据备份成功进行,如下所示。

    示例:

    VPlexcli:/> ll ~system-volumes
    /clusters/cluster-1/system-volumes:
    
    Name                                     Volume Type     Operational  Health  Active  Ready  Geometry  Component  Block     Block  Capacity  Slots
    
    ---------------------------------------  --------------  Status       State   ------  -----  --------  Count      Count     Size   --------  -----
    
    ---------------------------------------  --------------  -----------  ------  ------  -----  --------  ---------  --------  -----  --------  -----
    
    meta_C1_xxxxxx                           meta-volume     ok           ok      true    true   raid-1    2          20971264  4K     80G       64000
    meta_C1_xxxxxxx_backup_2024-11-23_01-30  meta-volume     ok           ok      false   true   raid-1    1          20971264  4K     80G       64000
    meta_C1_4UQT429_backup_2024-11-24_01-30  meta-volume     ok           ok      false   true   raid-1    1          20971264  4K     80G       64000
    
    /clusters/cluster-2/system-volumes:
    
    Name                                     Volume Type     Operational  Health  Active  Ready  Geometry  Component  Block     Block  Capacity  Slots
    
    ---------------------------------------  --------------  Status       State   ------  -----  --------  Count      Count     Size   --------  -----
    
    ---------------------------------------  --------------  -----------  ------  ------  -----  --------  ---------  --------  -----  --------  -----
    
    meta_C2_xxxxxx                           meta-volume     ok           ok      true    true   raid-1    2          20971264  4K     80G       64000
    meta_C2_xxxxxxx_backup_2024-11-23_12-43  meta-volume     ok           ok      false   true   raid-1    1          20971264  4K     80G       64000
    meta_C2_xxxxxxx_backup_2024-11-24_12-43  meta-volume     ok           ok      false   true   raid-1    1          20971264  4K     80G       64000

 

Affected Products

metro node

Products

metro node mn-114, metro node mn-215
Article Properties
Article Number: 000264665
Article Type: Solution
Last Modified: 22 Apr 2025
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.