PowerScale: SED nodes fail to boot with SED errors after upgrading to OneFS 9.5 from a version older than 9.2

Summary: When SED nodes are upgraded directly to OneFS 9.5 from a OneFS version older than 9.2, the nodes fail to boot completely, and the drives show in a SED_ERROR state.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

When SED nodes are upgraded directly to OneFS 9.5 from a OneFS version older than 9.2, the nodes fail to boot completely, and the drives show in a SED_ERROR state. isi_sed shows that drive authentication failed:

example-1# /usr/bin/isi_hwtools/isi_sed drive da3
ERROR: Key Manager error: error=1=ISI_KM_INIT_FAILED
WARNING: Unable to open Key Manager connection: No connection to the Key Manager
ERROR: Key Manager error: Key 5000C500CB360D98.current: error=34=SED_RESULT_NO_KM_CONNECTION
WARNING: Attempt to get current drive key returned No connection to the Key Manager
ERROR: Key Manager error: Key 5000C500CB360D98.future: error=34=SED_RESULT_NO_KM_CONNECTION
WARNING: Attempt to get future drive key returned No connection to the Key Manager
ERROR: Key Manager error: Key 5000C500CB360D98.change: error=34=SED_RESULT_NO_KM_CONNECTION
WARNING: Attempt to get change drive key returned No connection to the Key Manager
Drive Status: Bay  A0: WWN 5000c500cb360d98
Drive Model SEAGATE STMFSKF2CLAR4000, SN ZC0A00A0
FIPS Compliance: FIPS 140-2 (0x32), Level 2 (0x32), H/W: STMFSKF2CLAR4000, F/W: BV01, Module: Seagate Secure TCG Enterprise SSC Self-Encrypting Drives FIPS 140 Module
Key      Key      MSID     Drive            Drive
Exists   Works    Works    State            Status
=======  =======  =======  ===============  ===============
Error    Unknown  No       AUTH_FAILED      AUTH_FAILED
Owned by another node
Note: Future Key Exists: Error, Works: Unknown
Note: Change Key Exists: Error, Works: Unknown
Bay  A0: Dev da3, SEAGATE STMFSKF2CLAR4000, SN ZC1E03Y3, WWN 5000c500cb360d98: drive_display: 
Success
 
isi_km_d.log shows SQLite errors:
  
023-05-05T12:22:20.188716-05:00 <3.3> example-7(id7) isi_km_d[36798]: [0x803309000]key_mgr: [SEDS] initprovider failed with sqlite error: Sqlite error: SQL logic error (1) - Failed to prepare statement: 'SELECT version, passphrase_mode, kmip_uid, kmip_index, kmip_host, kmip_port, kmip_meta, kmip_cacert, kmip_clientcert FROM seds_config WHERE id = ?'.

Cause

Changes to support the KM Rekey feature in OneFS 9.5.0 assume the existence of a keystore database table that was added in 9.2.0. If the cluster is upgraded from a version older than OneFS 9.2 directly to OneFS 9.5, the key manager cannot read the keystore database correctly because the table in question is missing.

Resolution

To prevent this issue, customers upgrading to OneFS 9.5 from a version of OneFS older than 9.2 should first upgrade to an intermediate version of either OneFS 9.2 or 9.4. Once this upgrade is complete, the cluster can then be safely upgraded to OneFS 9.5.

If the issue has already occurred and the cluster is in the state described in the symptoms section, you must contact support immediately and reference this article.

Affected Products

Isilon A200, Isilon A2000, Isilon F800, Isilon F810, Isilon H400, Isilon H500, Isilon H5600, Isilon H600, Isilon HD400, Isilon NL410, Isilon S210, Isilon X210, Isilon X410, PowerScale Archive A300, PowerScale Archive A3000, PowerScale F200 , PowerScale F600, PowerScale F900, PowerScale Hybrid H700, PowerScale Hybrid H7000 ...
Article Properties
Article Number: 000213188
Article Type: Solution
Last Modified: 22 Jun 2023
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.