NetWorker: High Memory Usage Observed by 19.12+ NMC GSTD Process

The NetWorker Management Console (NMC) server is 19.12.0.0 -> 19.14.0.1.
The NMC server may have been upgraded from a previous release or configured as a new/fresh deployment on these versions.
The NMC server service/daemon (gstd) consumes memory over time, to the point where all available system memory is consumed.

This was identified as a defect impacting NetWorker versions:

• 19.12.0.x
• 19.13.0.x 
• 19.14.0.0 -> 19.14.0.1

This may be observed on both Windows and Linux platforms.

This issue will be addressed in NetWorker 19.14.0.2 through code fix NETWORKER-134465.

Workaround:

The only available workaround at this time is to routinely restart the NMC gstd service before it consumes system memory. 
The following examples show how to automate a 24 hour restart that occurs at 11PM (NMC server time). This interval may differ depending on the NMC server's configuration and overall NetWorker environment size. Restarting the NMC server service does not impact any on-going NetWorker protection operations. The following approaches may be used to automate this task using OS tools.

Windows:
Use Windows Task Scheduler to create a task that restarts gstd at a defined interval. From an Administrator Command/PowerShell prompt, run:

schtasks /Create /TN "Restart GSTD Service" /TR "cmd.exe /c net stop gstd & timeout /t 10 /nobreak >nul & net start gstd" /SC DAILY /ST 23:00 /RU SYSTEM

Alternatively the task can be created from the Windows Task Schedule UI.

The following commands can be used to verify, test, and remove the task

:: Verify the task
schtasks /Query /TN "Restart GSTD Service" /V /FO LIST

:: Run it immediately for testing
schtasks /Run /TN "Restart GSTD Service"

Essentially, the task stops the gstd service (if running), waits 10 seconds, then starts gstd. If gstd is not running, the task will attempt to start gstd. 

Once 19.14.0.2 is available and the task is no longer needed you can either delete if from the Task Scheduler UI or using:

schtasks /Delete /TN "Restart GSTD Service" /F

Linux:

Use cron to schedule a systemctl restart of the gst service.

1. From a root/elevated prompt, run: crontab -e
2. Choose the text editor of your choice.
3. In the crontab file enter:
   

   0 23 * * * systemctl restart gst

4. Ensure that crontab is running/enabled. The service may appear as cron or crond depending on OS vendor and version:
   

   # Check which cron service is installed
   systemctl list-unit-files | grep -E 'cron|crond'
   
   # Enable the service if it is not already enabled
   systemctl enable --now <cron-servic-name>

Once 19.14.0.2 is available and the task is no longer needed you can remove the restart entry from crontab.

Alternatively, the gstd service can be stopped/disabled until a fix is available. Use the NetWorker Web User Interface (NWUI) to manage the NetWorker server instead, see: NetWorker Management Web UI: How to Install