ECS: resourcesvc bouncing from excessive memory consumption

Recent updates to ILM policies
 
The resourcesvc is constantly restarting:

admin@node1:~> svc_node services -sr
svc_node v1.1.0 (svc_tools v1.6.6)                 Started 2019-11-20 18:32:20


Total Restarts on each node (all services) from 7 hours ago to now:

Node          # Service Restarts
--------------------------------

169.254.1.1                    1
169.254.1.2                   84
169.254.1.3                    1
169.254.1.4                   43
169.254.1.5                   51
169.254.1.6                   72
169.254.1.7                    0
169.254.1.8                   13

Aggregate restarts on nodes 'all':

Time              blbsvc  cm      crdsvc  datahd  dtq     evntsv  georcv  metrng  objctl  portal  rm      rsrcsv  sr      ss      ssm     stat    vnest
--------------------------------------------------------------------------------------------------------------------------------------------------------

2019-11-20 11:xx  -       -       -       -       -       -       -       -       -       -       -       38      -       -       -       -       -
2019-11-20 12:xx  -       -       -       -       -       -       -       -       -       -       -       36      -       -       -       -       -
2019-11-20 13:xx  -       -       -       -       -       -       -       -       -       -       -       30      -       -       -       -       -

RR or RT DTs bouncing between ready and unready

admin@Rack1Node1:~> svc_dt check

svc_dt v1.0.20 (svc_tools v1.6.6)                 Started 2019-11-20 19:32:48

Date                     Total DT       Unknown #      Unready #      Check type     Time since check

2019-11-20 19:28:01      1920           0              58             AutoCheck      4m 47s
2019-11-20 19:22:31      1920           2              46             AutoCheck      10m 17s
2019-11-20 19:16:41      1920           0              0              AutoCheck      16m 7s
2019-11-20 19:11:26      1920           0              16             AutoCheck      21m 22s
                           BR1     BR2     CT1     CT2     ET0     LS0     MA0     MR0     OB0     PR1     PR2     RR0     RT0     SS1     SS2     TT0
Date                     Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|Unk Unr|

2019-11-20 19:28:01      0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   58 |0   0  |0   0  |0   0  |
2019-11-20 19:22:31      0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |0   0  |2   46 |0   0  |0   0  |0   0  |

The resourcesvc log show similar memory issues:

169.254.1.1 2019-11-15T20:03:16,427 [Thread-16] ERROR ProcessMonitor.java (line 264) heap memory usage (99%) exceeded threshold (85%)
169.254.1.1 2019-11-15T20:03:16,428 [Thread-16] ERROR ServiceMemoryListener.java (line 47) WSCritical. Memory usage threshold exceeded. usedMemory=1028612296, percentageUsed=99.09949521251302

Client experiencing loading UI pages 

Check service restarts over /var/log/warn in object-main container.
# svc_exec -c "grep restart /var/log/warn | tail -2"

admin@ecsnode1:~>  svc_exec -c "grep restart /var/log/warn | tail -2"
svc_exec v1.0.6 (svc_tools v2.11.1)                 Started 2023-04-27 18:01:41

Output from node: r1n1 (object-main)                  retval: 0
2023-04-27T17:56:56.028973+00:00 gf01rsso833v-pub01 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T17:59:18.281755+00:00 gf01rsso833v-pub01 monitor: Process /opt/storageos/bin/resourcesvc is restarting.

Output from node: r1n2 (object-main)                  retval: 0
2023-04-27T17:44:17.243145+00:00 gf01rsso833v-pub02 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T17:51:58.026469+00:00 gf01rsso833v-pub02 monitor: Process /opt/storageos/bin/resourcesvc is restarting.

Output from node: r1n3 (object-main)                  retval: 0
2023-04-27T17:49:30.865189+00:00 gf01rsso833v-pub03 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T17:57:22.368574+00:00 gf01rsso833v-pub03 monitor: Process /opt/storageos/bin/resourcesvc is restarting.

Output from node: r1n4 (object-main)                  retval: 0
2023-04-27T17:40:11.327713+00:00 gf01rsso833v-pub04 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T17:59:24.566502+00:00 gf01rsso833v-pub04 monitor: Process /opt/storageos/bin/resourcesvc is restarting.

Output from node: r1n5 (object-main)                  retval: 0
2023-04-27T17:50:35.589292+00:00 gf01rsso833v-pub05 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T18:01:09.344803+00:00 gf01rsso833v-pub05 monitor: Process /opt/storageos/bin/resourcesvc is restarting.

Output from node: r1n6 (object-main)                  retval: 0
2023-04-27T17:39:39.480222+00:00 gf01rsso833v-pub06 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T17:51:49.229058+00:00 gf01rsso833v-pub06 monitor: Process /opt/storageos/bin/resourcesvc is restarting.

Output from node: r1n7 (object-main)                  retval: 0
2023-04-27T17:58:08.383501+00:00 gf01rsso833v-pub07 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T18:00:00.346869+00:00 gf01rsso833v-pub07 monitor: Process /opt/storageos/bin/resourcesvc is restarting.

Output from node: r1n8 (object-main)                  retval: 0
2023-04-27T17:41:11.263187+00:00 gf01rsso833v-pub08 monitor: Process /opt/storageos/bin/resourcesvc is restarting.
2023-04-27T18:00:10.438252+00:00 gf01rsso833v-pub08 monitor: Process /opt/storageos/bin/resourcesvc is restarting.