Skip to main content
Contact Us

NVP-vProxy: Backups intermittently fail with "-500: Panic occurred: runtime error"

Summary: The vProxy appliance backups fail with a panic runtime error when the /data01 partition has reached 100% capacity.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

The NetWorker VMware Protection (NVP) integration is configured with the vProxy Appliance. The virtual machine (VM) backups fail and the NetWorker workflow log shows: 
MM/DD/YYYY HH:MM:SS [VM_NAME]: Unable to start backup on vProxy '[VPROXY_NAME]': Received an HTTP code: 500, libCURL message: "", vProxy message: "Error received from vProxy ="-500: Panic occurred: runtime error: invalid memory address or nil pointer dereference". ", url: "https://[VPROXY_NAME]:9090/api/v1/BackupVmSessions", body: "{"Config":{"SessionId":"","LogTag":"@(#) Build number: 288","IdleTimeout":300,"LogLevel":"TRACE","AcceptedLanguage":"en","TransportModeRequested":"hotadd","Parallelism":1,"UseCbt":true,"AutoEnableCbt":true,"AutoRepairCbt":true,"VimServerRef":{"HostName":"[VCENTER_NAME]","UserName":"S001891","UserPassword":"****","TcpPort":0},"VmSpec":{"Name":"[VM_NAME]","VmMoref":"vm-152712","VirtualDisks":[{"Label":"Hard disk 1","Key":2000}],"CustomFieldName":"Last EMC vProxy Backup","CustomFieldValue":"Backup Server=[NSR_NAME], Policy=[POLICY_NAME], Workflow=[WORKFLOW_NAME], Action=[ACTION_NAME], JobId=490597"},"SnapshotSpec":{"Name":"NetWorker Backup Snapshot","Description":"Snapshot is created as part of protecting this VM by EMC NetWorker VProxy.","Quiesce":false,"DumpMemory":false,"RemoveAll":false,"Consolidate":false},"TargetSpec":{"DeviceType":"DataDomain","HostName":"[DATADOMAIN_NAME]","UserName":"boostadmin","UserPassword":"****","BackupMode":"VSS","BackupPath":"[NSR_NAME]/[DEVICE_NAME]//[NSR_NAME]/[DEVICE_NAME]/05/47/e03a0faf-00000006-9122ca5e-5b22ca5e-5380e9f4-822ca7d4","PreviousBackupPath":"/[NSR_NAME]/[DEVICE_NAME]/75/82/7f12c16b-00000006-2720fcc2-5b20fcc2-40eae9f4-822ca7d4","CurrentBackupPath":"/[NSR_NAME]/[DEVICE_NAME]/active/e03a0faf-00000006-9122ca5e-5b22ca5e-5380e9f4-822ca7d4"}}}".

Cause

The /data01 file system is full on the vProxy appliance. The storage is consumed by the backup session logs, which reside in the "/opt/emc/vproxy/runtime/" (resides on /data01) directory and have a default retention of one month. The vProxy is unable to write session logs to start a backup.  
myvproxy:/data01/runtime/logs # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda4 17G 2.7G 14G 18% /
devtmpfs 3.9G 8.0K 3.9G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 8.8M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda4 17G 2.7G 14G 18% /.snapshots
/dev/sda4 17G 2.7G 14G 18% /var/crash
/dev/sda4 17G 2.7G 14G 18% /var/tmp
/dev/sda4 17G 2.7G 14G 18% /usr/local
/dev/sda4 17G 2.7G 14G 18% /var/spool
/dev/sda4 17G 2.7G 14G 18% /var/lib/pgsql
/dev/sda4 17G 2.7G 14G 18% /var/opt
/dev/sda4 17G 2.7G 14G 18% /var/lib/mailman
/dev/sda4 17G 2.7G 14G 18% /var/lib/named
/dev/sda4 17G 2.7G 14G 18% /srv
/dev/sda4 17G 2.7G 14G 18% /opt
/dev/sda4 17G 2.7G 14G 18% /home
/dev/sda4 17G 2.7G 14G 18% /tmp
/dev/sda4 17G 2.7G 14G 18% /var/log
/dev/sdb1 82G 82G 20K 100% /data01
/dev/sda2 95M 73M 18M 81% /boot
/dev/sda4 17G 2.7G 14G 18% /boot/grub2/x86_64-efi
/dev/sda4 17G 2.7G 14G 18% /boot/grub2/i386-pc
/dev/sdb2 16G 33M 16G 1% /data01/logs

Resolution

The vProxy appliance can be redeployed to clear the existing logs. The vProxy does not contain any information required for performing recoveries of data which has already been backed up. 

Additional Information

Space can be reclaimed on the vProxy's /data01 mountpoint by deleting logs from: /data01/runtime/logs/recycle/

Here is an example procedure to create a cron job that deletes the logs that are 14 days old:

  • Log in to the vProxy over SSH as admin, then switch to root: sudo su - root
  • Create a crontab job for the root user with: crontab -e
  • In the text editor, insert the following line to delete the session logs older than 14 days every day at 6 AM:
* 6 * * * find /opt/emc/vproxy/runtime/logs/recycle/vbackupd -type f -mtime +14 -delete
  • View the contents of the new root user cron job with: crontab -l
  • The number of days to retain logs may be modified to keep the file system from reaching 100%.
NOTE: These changes are not carried over during a vProxy upgrade.
Article Properties
Article Number: 000033993
Article Type: Solution
Last Modified: 09 Oct 2025
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.