Issues seen with REST API clients and Embedded unisphere

Symptoms of this issue vary,

Too many calls from the Rest API client can result in the below issues 

• Gaps in performance data in the Unisphere metrics dashboards,
• Unisphere periodically displaying that an array is unregistered on the GUI.
• Services on the embedded container going offline in particular PR_Storstpd.

To troubleshoot these symptoms an examination of the Rest API log is needed.

This is not enabled by default to enable this , go to unisphere > support> modify server logging > enable RestApi logs > check this tickbox.

This cannot be done from the array , must be enabled by Customer from their embedded unisphere

See below Screenshots for path


 


 

Once this is enabled a file under the /server0 folder from an emcgrab from the vapp manger or smas.logs will have a file generated called restapi.



 

These issues can be caused by not adhering to best practices as below from the Unisphere for PowerMax install guide. (see resolution)

Another cause might be that versioned endpoints are not being used as below 

Supported:
/92/sloprovisioning/symmetrix/

Not supported:
/sloprovisioning/symmetrix/

Also another factor to consider is if there are repeated API calls with MS count in comparison to the average see below 

2021-06-30 11:10:55,510 INFO  [em.bp.RESTAPIUSAGE] (default task-46) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.xx.x.xxx    GET    314 ms    200    null    null    null
2021-06-30 11:11:35,937 INFO  [em.bp.RESTAPIUSAGE] (default task-43) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.xx.xx.xx    GET    299 ms    200    null    null    null
2021-06-30 11:11:35,938 INFO  [em.bp.RESTAPIUSAGE] (default task-37) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.x.xx.xx    GET    327 ms    200    null    null    null
2021-06-30 11:11:35,939 INFO  [em.bp.RESTAPIUSAGE] (default task-19) 2021-06-30 10:55:55,533 INFO  [em.bp.RESTAPIUSAGE] (default task-32) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.xx.xx.xx   GET    320 ms    200    null    null    null
2021-06-30 10:55:55,534 INFO  [em.bp.RESTAPIUSAGE] (default task-10) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.xx.xx.xx    GET    320 ms    200    null    null    null
2021-06-30 10:56:35,795 INFO  [em.bp.RESTAPIUSAGE] (default task-14) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.xx.xx.xx   GET    40584 ms    200    null    null    null
2021-06-30 10:56:35,795 INFO  [em.bp.RESTAPIUSAGE] (default task-15) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.xx.xx.xx   GET    40584 ms    200    null    null    null
2021-06-30 10:56:35,798 INFO  [em.bp.RESTAPIUSAGE] (default task-34) /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}    UNKNOWN/10.xx.xx.xx  GET    40454 ms    200    null    null


The above example is also not supported as it is not a versioned endpoint see /sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}

this should be 
92/sloprovisioning/symmetrix/{symmetrixId}/volume/{volumeId}

Ensure that best practices are being followed with REST API calls - see Unisphere for PowerMax install guide 9.2.1 and below 

Unisphere for PowerMax supports up to 10 concurrent users, with some performance degradation occurring with more than 5
storage admin users.
Note the following recommendations on REST API client limits, which must be adhered to in order to achieve optimal
performance:
● Each REST API session counts as a concurrent user while the REST calls are running.
● A maximum of 5 REST API clients can connect to the Unisphere server concurrently.
● A maximum of 50 GET calls can be made to the Unisphere server per second. These can be made up of performance REST
API GET calls and non-performance REST API GET calls.
● A maximum of 5 non-performance POST or PUT calls can be run against the Unisphere server concurrently.
● It is recommended that you use bulk REST API calls, where possible.
NOTE: If the recommendation above are not followed, it might result in performance degradation of the Unisphere for PowerMax server.

Ensure versioned endpoints are also being used