PowerScale: Upgrade Fails With Remove-lsa-nthashes.py

powerscale-46# isi_upgrade_logs
Warning: Insufficient data for Node LNN :[2]
8  2026-03-31T16:59:18  /usr/sbin/isi_upgrade_agent_d  Debug  Queueing up hook script: /usr/share/upgrade/event-actions/upgrade-post-reboot-final/remove-lsa-nthashes.py
8  2026-03-31T17:00:19  /usr/sbin/isi_upgrade_agent_d  Warn  Hook Failed [/usr/share/upgrade/event-actions/upgrade-post-reboot-final/remove-lsa-nthashes.py]
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Warn : Upgrade may already be in progress... create failed : File Name : /b/mnt/src/isilon/lib/isi_upgrade/Lock.cpp : Line Number : 111
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Traceback (most recent call last):
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform    File "/usr/share/upgrade/event-actions/upgrade-post-reboot-final/remove-lsa-nthashes.py", line 57, in <module>
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform      u.clear_old_nthash()
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  isi.auth_util.Error: The authentication provider was created in a zone that makes it unmodifiable
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  During handling of the above exception, another exception occurred:
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Traceback (most recent call last):
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform    File "/usr/share/upgrade/event-actions/upgrade-post-reboot-final/remove-lsa-nthashes.py", line 61, in <module>
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform      upgrade_helper.log_error(
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  AttributeError: 'HookOnceFromVersionHelper' object has no attribute 'log_error'
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Checking relevance...
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Is this the first node to run the hook?
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  No. Script not relevant, exiting.
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Checking relevance...
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Is this the first node to run the hook?
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Yes.
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Are we upgrading from a relevant version?
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Yes.
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  We are relevant.
8  2026-03-31T17:00:19  remove-lsa-nthashes.py  Inform  Starting deletion of NTHashes from all Local providers

The hook is unable to run the script due to lsa-local-provider:System in nonsystem zones.

Engineering is aware of the issue and is working on a permanent fix. In order to work around the error, lsa-local-provider:System must be temporarily removed from nonsystem zones. To check for this, on the CLI run:

powerscale-46# isi zone zones list -v
             
                       Name: System   <<<<<<<<< Default, do not edit this one
                       Path: /ifs
                   Groupnet: groupnet0
              Map Untrusted: 
             Auth Providers: lsa-local-provider:System, lsa-file-provider:System
               NetBIOS Name: 
         User Mapping Rules: -
       Home Directory Umask: 0077
         Skeleton Directory: /usr/share/skel
         Cache Entry Expiry: 4H
Negative Cache Entry Expiry: 1m
                    Zone ID: 1

                       Name: eyeglass   <<<<<<nonsystem zone
                       Path: /ifs/data/
                   Groupnet: groupnet0
              Map Untrusted:
             Auth Providers: lsa-local-provider:System, lsa-local-provider:eyeglass   <<<<<<<<<<< lsa System needs to be removed 
               NetBIOS Name:
         User Mapping Rules: -
       Home Directory Umask: 0077
         Skeleton Directory: /usr/share/skel
         Cache Entry Expiry: 4H
Negative Cache Entry Expiry: 1m
                    Zone ID: 6

Remove the provider with this command:

powerscale-46# isi zone zones modify <non-system-zone> --remove-auth-providers=lsa-local-provider:System

Run the command for each nonsystem zone, then retry the last action on the error nodes:

powerscale-46# isi upgrade view

Upgrade Status:

Current Upgrade Activity: OneFS upgrade
   Cluster Upgrade State: error
                          (see output of `isi upgrade nodes list` to know which node failed)
   Upgrade Process State: Stopped
      Upgrade Start Time: 2026-03-31T10:29:20.362000
      Current OS Version: 9.7.1.5_build(4)style(5)
      Upgrade OS Version: 9.10.1.7_build(4)style(5)
        Percent Complete: -

Nodes Progress:

     Total Cluster Nodes: 36
       Nodes On Older OS: 30
          Nodes Upgraded: 0
Nodes Transitioning/Down: 6

If you encounter an error, you can run the command isi_upgrade_logs to gather more information. If you would like to retry a failed action on the required nodes, use the command isi upgrade cluster retry-last-action --nodes=<LNN>.  

If you want to roll back the upgrade, use the command isi upgrade cluster rollback.

LNN                                                        Version   Status
-----------------------------------------------------------------------------------
25,30                                                      Unknown   non-responsive
26,31,33-60                                                9.7.1.5   upgrading
27                                                         9.10.1.7  upgrading
28-29,32                                                   9.10.1.7  error

In this case, the command would be:

powerscale-46# isi upgrade cluster retry-last-action --nodes=28-29,32

If the upgrade continues, lsa-local-provider:System can be added back to the zones:

powerscale-46# isi zone zones modify <non-system-zone> --add-auth-providers=lsa-local-provider:System

If it errors again and it is not obvious what this error is, contact Dell Technical Support.