Isilon: Process isi_drive_repurpose_d of service isi_drive_repurpose_d has failed to restart after multiple attempts

Process isi_drive_repurpose_d of service isi_drive_repurpose_d has failed to restart after multiple attempts

2019-12-02T23:57:42+01:00 <1.4> ge1fs02-1(id6) isi_drive_repurpose_d[26205]: Exception: 7: Traceback (most recent call last): File "/usr/bin/isi_drive_repurpose_d", line 1416, in <module> main() File "/usr/bin/isi_drive_repurpose_d", line 124, in main run_as_daemon() File "/usr/bin/isi_drive_repurpose_d", line 738, in run_as_daemon run_uncaught() File "/usr/bin/isi_drive_repurpose_d", line 788, in run_uncaught made_progress = try_resume() File "/usr/bin/isi_drive_repurpose_d", line 697, in try_resume resume_revert(wk) File "/usr/bin/isi_drive_repurpose_d", line 1261, in resume_revert lnns = set([ devid_to_lnn[dd[0]] for dd in dds ]) KeyError: 7

# cat /ifs/.ifsvar/modules/drive_repurpose_d/state.json

# cat local/ifsvar_modules.tar/modules/drive_repurpose_d/state.json

{"queue": [], "active": {"start": 1565276407.05864, "kind": "revertl3", "deviddrives": [{"devid": 7, "drives": [15]}], "stage": 0}, "version": 1, "history": []}

kolhem@durhamelvis:/logviewer/ETAT_DE_GENEVE_co_Nouvelle_Hotel_de_Police/2019-12-04-001$ cat ge1fs02-*/etcifs.tar/ifs/array.xml | grep -i array_id | sort | uniq
      <array_id>10</array_id>
      <array_id>11</array_id>
      <array_id>6</array_id>
      <array_id>8</array_id>
      <array_id>9</array_id>

pipeline-1# isi_for_array 'cat /etc/ifs/array.xml | grep array_id' | sort | uniq
pipeline-1:       <array_id>1</array_id>
pipeline-1:       <array_id>2</array_id>
pipeline-2:       <array_id>1</array_id>
pipeline-2:       <array_id>2</array_id>

# mkdir /ifs/data/Isilon_Support/<sr_number>/
# cp /ifs/.ifsvar/modules/drive_repurpose_d/state.json /ifs/data/Isilon_Support/<sr_number>/

# isi services -a isi_drive_repurpose_d disable

# rm /ifs/.ifsvar/modules/drive_repurpose_d/state.json

# isi services -a isi_drive_repurpose_d enable

# isi_for_array -s 'ps -auwx | grep -i isi_drive_repurpose_d | grep -v grep'