PPDM: When viewing the ServerDR Manage Backups from the user interface returns a 500 Internal Server Error
Summary: This article describes when viewing the ServerDR Manage Backups from the user interface returns a 500 Internal Server Error.
Symptoms
Issue (impact on customer):
When viewing the ServerDR > Manage Backups from the PowerProtect Data Manager UI, the UI returns a 500 Internal Server Error.
Symptoms found in the logs:
In the "/var/log/brs/serverdr/serverdr.log," we see that the browse of serverdr backups returns manifests, but it takes over 5 minutes to read the manifest data.
2024-07-17T14:04:26.500Z INFO [] [https-jsse-nio-8456-exec-10] [][][][TRACE_ID:ab408aff17ab0467][] [c.e.d.e.c.s.c.s.i.ManifestServiceImpl.getBackupManifestFromPaths(683)] - Extracting all manifests from [ "/data01/server_backups/srv0235_1a09b6dc-7dd1-4900-839e-6e8143998198", "/data01/server_backups/_1a09b6dc-7dd1-4900-839e-6e8143998198" ] 2024-07-17T14:04:26.538Z INFO [] [https-jsse-nio-8456-exec-10] [][][][TRACE_ID:ab408aff17ab0467][] [c.e.d.e.c.s.c.s.i.ManifestServiceImpl.getManifestPaths(654)] - Manifest paths retrieved 11 And then it takes 6 minutes to retrieve the 11 manifest objects. 2024-07-17T14:10:59.539Z INFO [] [https-jsse-nio-8456-exec-10] [][][][TRACE_ID:ab408aff17ab0467][] [c.e.d.e.c.s.c.s.i.ManifestServiceImpl.getBackupManifestFromPaths(693)] - Retrieved 11 BackupManifest objects
IN ADDITION to the logs above, if a user runs the following command on the PowerProtect Data Manager appliance.
ps -ef | grep boostfs
A user gets a response similar to:
admin 25778 1 0 Jul26 ? 00:06:51 /opt/emc/boostfs/bin/boostfs mount -d 10.241.216.52 -s SysDR_sky8 -o local-user-security=false -o allow-others=true /data01/server_backups -l lockbox_path/lockbox_name
From this response, the IP address used in the mount command is NOT in the IFGROUPs defined on the DDS.
Impacted Configuration and Settings:
This KB applies to PowerProtect Data Manager Appliances writing the ServerDR backups to a Data Domain System (DDS) with IFGROUPs enabled.
The IP address being used by the PowerProtect Data Manager appliance to mount the SysDR serverDR storage unit is NOT in the IFGROUP definition.
Impact on PowerProtect Data Manager:
Attempts to connect to the boostfs mount are instantaneous. HOWEVER, attempts to read data from the boostfs mount take 3-5 minutes during the initial attempt. Afterwards, reading data may be cached and return quickly on secondary and subsequent attempts.
Cause
If an IFGROUP is configured on the DD, the client will first try to connect to the interface on the IFGROUP's least busy interface. When it receives an RPC error, it falls back to the connecting IP for the ddboost mount.
When the IPs are on different subnets, it is trying to switch between the interfaces and failing to do so.
Resolution
Workaround: We added an IFGROUP for the PowerProtect Data Manager as the client, which only has the management interface of the DD and everything is working fine now.