NetWorker REST API: HTTP GET requests are intermittently timing out
Summary: The NetWorker representational state transfer (REST) application program interface (API) is used to request information (HTTP GET) from a NetWorker Server in the environment. The REST API queries intermittently times out, and the restapi.log shows an internal server error message due to a Socket Timeout. ...
Symptoms
YYYY-MM-DD HH:MM:SS INFO [https-jsse-nio-9090-exec-26] c.e.n.w.WebApiResponse - Response status Method: 'GET', URI:'v3/global/backups', Status '200'
YYYY-MM-DD HH:MM:SS INFO [https-jsse-nio-9090-exec-26] c.e.n.w.WebApiResponse - Response status Method: 'GET', URI:'v3/global/backups', Status '500'
...
YYYY-MM-DD HH:MM:SS ERROR [https-jsse-nio-9090-exec-11] c.e.n.w.WebApiExceptionMapper - Status 'Internal Server Error', msg: java.io.IOException: Broken pipe
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:333) ~[catalina.jar:9.0.10]
A curl command to the NetWorker Server REST API shows:
[root@NetworkerServer]# curl -X GET -H "Content-Type: application/json" "https://mynsr.mydomain.com:9090/nwrestapi/v3/global/backups" -u "administrator" -k -1
Enter host password for user 'administrator':
curl: (28) Operation timed out after 300406 milliseconds with 0 out of 0 bytes received
Cause
The REST API calls "v3/global/backups" and "v3/global/jobs" alone is a resource-intensive query and can cause the request to timeout in larger environments.
The NetWorker REST API services are hosted on an Apache Tomcat Java servlet container which is run on a Java Virtual Machine (JVM). The REST API uses JVM memory to process every request, and it has been shown that not all the memory is released after a resource-intensive query. The increase in JVM memory utilization may eventually lead to other less resource-intensive REST API requests to timeout.
Resolution
Avoid using the resource intense requests to retrieve information from the NetWorker Server. For example, the "v3/global/backups" or "v3/global/clients/CLIENT_NUMBER/backups" with a defined time range can be used instead of using the global "v3/global/backups" to retrieve all the backups.
For example, restrict the results for saveTime between any specified time range. An example of a query for jobs which completed within the last 24 hours:
../global/backups example:
https://NETWORKER_SERVER_ADDRESS:9090/nwrestapi/v3/global/backups?q=saveTime:["START" TO "END"]
https://nsr.amer.lan:9090/nwrestapi/v3/global/backups?q=saveTime:["2024-07-24T00:00:01" TO "2024-07-24T23:59:59"]
../global/clients/client_resourceId_number/backups
https://NETWORKER_SERVER_ADDRESS:9090/nwrestapi/v3/global/clients/client_resourceID_number/backups?q=saveTime:["START" TO "END"]
https://nsr.amer.lan:9090/nwrestapi/v3/global/clients/87.0.90.20.0.0.0.0.196.80.99.102.192.168.9.150/backups?q=saveTime:["2024-07-24T00:00:01" TO "2024-07-24T23:59:59"]
../global/jobs:
https://NETWORKER_SERVER_ADDRESS:9090/nwrestapi/v3/global/jobs?q=endTime:["START" TO "END"]
https://nsr.amer.lan:9090/nwrestapi/v3/global/jobs?q=endTime:["2024-07-29T00:00:01" TO "2024-07-29T23:59:59"]
If less resource-intensive REST API requests are timing out, then the NetWorker Server services may be restarted to release the existing JVM memory usage.