PowerScale: JWT and PAPI timeouts on 9.5 and higher PowerScale OneFS

This applies to OneFS 9.5 and higher for impacts; but some still older versions of OneFS can sometimes see these conditions.
 Some of the commands in this KB will not work on versions of OneFS older than 9.6 (not 9.5) and are not likely to be backported. Consider upgrading to modern target code levels.

As OneFS evolves, the underlying API systems are routinely updated. Beginning in 9.5, certain workflows and use cases with high levels of frequent or concurrent API/web authentication sessions may see higher rates of connection, PAPI, API, and similar timeouts that may overlap with JWT-related processes coring. JWT is one of the systems involved in management of CSRF tokens for API authentication as required by our modern security model.

The cores, if present, appear like this:

Powerscale-OneFS-27# isi_for_array -s ls -lah /var/crash/ | grep -i jwt
Powerscale-OneFS-12: -rw-rw----   1 root  daemon     167M Jul 14 23:39 jwt.core.gz
Powerscale-OneFS-13: -rw-rw----   1 root  daemon     214M Jul 15 12:19 jwt.core.gz
Powerscale-OneFS-17: -rw-rw----   1 root  daemon     181M Jul 15 14:03 jwt.core.gz
Powerscale-OneFS-21: -rw-rw----   1 root  daemon     172M Jul 24 14:20 jwt.core.gz
Powerscale-OneFS-22: -rw-rw----   1 root  daemon      37M Jul 24 14:23 jwt.core.gz
Powerscale-OneFS-27: -rw-rw----   1 root  daemon     165M Jul 24 10:08 jwt.core.gz
Powerscale-OneFS-28: -rw-rw----   1 root  daemon     181M Jul 24 13:35 jwt.core.gz
Powerscale-OneFS-29: -rw-rw----   1 root  daemon     181M Jul 24 13:10 jwt.core.gz

Which may include similar log errors like this:

$ cat Powerscale-OneFS-*/varlog.tar/log/jwt.log | grep 2024-07 | less

2024-07-15T13:00:22.459910-05:00 <30.5> Powerscale-OneFS-66(id71) jwt[83869]: Logging started
2024-07-15T13:25:54.388061-05:00 <30.3> Powerscale-OneFS-66(id71) jwt[83869]: [jwt] Failed to update active session user index. errno: 12 - Cannot allocate memory
2024-07-15T13:25:54.495084-05:00 <30.3> Powerscale-OneFS-66(id71) jwt[83869]: [jwt] Failed to add session: 0801b74c-1abf-4b23-89a0-1e9daf429559 to active session user index identity: SID:S-1-22-1-0, zone: 1. STATUS_INVALID_PARAMETER (0xC000000D)
2024-07-15T14:05:56.869968-05:00 <30.3> Powerscale-OneFS-66(id71) jwt[83869]: [jwt] Failed to update active session user index. errno: 12 - Cannot allocate memory
2024-07-15T14:05:57.115920-05:00 <30.3> Powerscale-OneFS-66(id71) jwt[83869]: [jwt] Failed to add session: 08c173f1-60de-472d-a89d-1d6af2462a28 to active session user index identity: SID:S-1-22-1-0, zone: 1. STATUS_INVALID_PARAMETER (0xC000000D)

2024-07-15T14:28:28.441603-05:00 <30.5> Powerscale-OneFS-66(id71) jwt[2077]: Logging started
2024-07-15T14:52:08.794040-05:00 <30.3> Powerscale-OneFS-66(id71) jwt[2077]: [jwt] Failed to update active session user index. errno: 12 - Cannot allocate memory
2024-07-15T14:52:09.043575-05:00 <30.3> Powerscale-OneFS-66 syslogd: last message repeated 1 times
2024-07-15T14:52:09.070500-05:00 <30.3> Powerscale-OneFS-66(id71) jwt[2077]: [jwt] Failed to add session: a1164a8f-cd68-45bf-bc07-5d7beefe7ba3 to active session user index identity: SID:S-1-22-1-0, zone: 1. STATUS_INVALID_PARAMETER (0xC000000D)

The issue may not be limited to JWT cores, however. The issues may be present on 9.5 or earlier but the workarounds and fixes are available on 9.6 OneFS and higher.
The workarounds in this KB on 9.6 and higher OneFS may provide relief to related OneFS API-related timeout issues.

These workarounds are applicable only to OneFS 9.6 and higher.
 

On lower versions that are not end of life, please contact Support directly. There may be options to pursue, but you must either upgrade OneFS, or work with the API/3rd party vendor to either more efficiently distribute your API-based workload across more nodes and over more time.


Addressing vmemory limits for JWT:

The current vmlimit for the JWT process is 1 GB. We can set it to 2 GB and check if it helps. Displaying the 1GB limit for JWT:

9610-1# limits -P $(pgrep jwt)  -B
Resource limits (current):
  cputime              infinity secs
  filesize             infinity kB
  datasize             33554432 kB
  stacksize              524288 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked           131072 kB
  maxprocesses             7390
  openfiles               16384
  sbsize               infinity bytes
  vmemoryuse            1048576 kB  <----
  pseudo-terminals     infinity
  swapuse              infinity kB
  kqueues              infinity
  umtxp                infinity

To set it to 2 GB instead:
 

# isi_for_array -s 'limits -P $(pgrep jwt)  -v 2g'

To confirm the new limit:

# isi_for_array -s  'limits -P $(pgrep jwt) -B' | grep vmemoryuse
9610-1:   vmemoryuse            2097152 kB
9610-2:   vmemoryuse            2097152 kB
9610-3:   vmemoryuse            2097152 kB
9610-4:   vmemoryuse            2097152 kB
9610-5:   vmemoryuse            2097152 kB
9610-6:   vmemoryuse            2097152 kB

You can add it to cron to make it permanent if those results are satisfactory using the information in Isilon How to edit crontab

To cron this, you will want to set this command to run at some reasonable interval to account for it to be reset after any possible cluster or node reboot, such as a cron to run every few hours:

isi_for_array -s 'limits -P $(pgrep jwt) -v 2g'

Engage your Dell System Engineering with any questions on implementing cron. If you encounter technical difficulties with that afterward, please engage Technical Support.



Addressing PAPI concurrent limits:

Another factor that may impact your operations is the number of concurrent allowed PAPI/API sessions. Some modern third-party vendors may send a veritable flood of activity towards the storage platform for API requests, which combined with internal operations of our own that use API can sometimes cause the maximum number of sessions to be exceeded. The default maximum limit is 80 with a minimum floor of 40:

9610-1# isi_gconfig -t papi | grep -i child
proc.auto_configure_child_limit (bool) = true
proc.child_limit (int) = 80
proc.child_limit_ceiling (int) = 80
proc.child_limit_floor (int) = 40
proc.child_limit_extension (int) = 0

The limit and ceiling can be increased thus for testing/remediation:

9610-1# isi_gconfig -t papi proc.child_limit=100; isi_gconfig -t papi proc.child_limit_ceiling=100

9610-1# isi_gconfig -t papi | grep child
proc.auto_configure_child_limit (bool) = true
proc.child_limit (int) = 100
proc.child_limit_ceiling (int) = 100
proc.child_limit_floor (int) = 40
proc.child_limit_extension (int) = 0

Keep in mind that the above gconfigs are a global setting for all nodes; there is no local option.

If you increase this, there may be unexpected impacts such as increased cost to related systems or theoretical performance impacts unique to a given environment, as no two storage clusters are likely to have identical performance profiles. Such considerations are normally impossible to forecast ahead of time and only reactively.

Contact your SE or Support with questions on this.

If you are seeing API/PAPI errors like or similar to this, but without the explicit JWT terminology or verbiage, please review this KB as well:

PowerScale OneFS 9.5/9.7.1 and beyond Commands timing out with "OneFS API is temporarily unavailable" or "Too many open files with cluster"