NVE: Upgrade Fails, reporting "Configure NVE server protection policy" Failed

You are upgrading the NVE appliance which fails at 92% stating:

Configure NVE server protection policy (42 of 44)

The NVE's /data01/avamar/repo/temp/*/tmp/workflow.log reports:

YYYY-MM-DD HH:MM:SS (-0500) 47393356618400 ERROR: "nsradmin -s 127.0.0.1 -i /space/avamar/repo/temp/NETWORKER_VERSION/tmp/nsr-config.txt", exit status=1 (error)
YYYY-MM-DD HH:MM:SS (-0500) 47393356618400 INFO: post: "nsradmin -s 127.0.0.1 -i /space/avamar/repo/temp/NETWORKER_VERSION/tmp/nsr-config.txt", exit status=1 (error)
...
YYYY-MM-DD HH:MM:SS (-0500) 47393356618400 INFO: Report home message: YYYY-MM-DD HH:MM:SS|NSR_HOSTNAME|unknown|NveUpgrade|NETWORKER_VERSION|/space/avamar/repo/temp/NETWORKER_VERSION/tmp|ConfigureServerProtectionPolicy|Configure NVE server protection policy (54 of 57)|ERROR: ConfigureServerProtectionPolicy::configure_group   - operation failed
YYYY-MM-DD HH:MM:SS (-0500) 47393356618400 ERROR:  - operation failed

Symptoms:

• NetWorker server is accessible through the NetWorker Management Console (NMC) or NetWorker Web User Interface (NWUI).
• NetWorker protection operations (backup, restore, clone) are functional on the upgraded NetWorker code.

Impact:

• The NVE upgrade process is stuck. It is not possible to apply additional updates until the current upgrade process has been completed.

Cause 1: The default Server Protection configuration exists, but default resources, such as the "NVE Server Backup" group is assigned to another workflow.
See: NetWorker Virtual Edition: NVE Upgrade Fails at 92% "Configure NVE Server Protection Policy" 

Cause 2: The default Server Protection policy is corrupted.

1. Stop services on the NVE:

nsr_shutdown
systemctl stop gst
systemctl stop nwui

ps -ef | grep "nsr\|gst\|nwui"

cp -R /nsr/res/nsrdb /nsr/res/nsrdb_$(date -I)

4. Enter nsradmin visual mode offline and delete the corrupted "Server Protection" policy:

nsradmin -d /nsr/res/nsrdb
visual

D. You are prompted to confirm the Deletion. With Yes highlighted, press Enter.

5. Create a text file /tmp/serverprotection.txt containing the default Server Protection Policy parameters:

create type: NSR Protection Policy; name: Server Protection;
comment: Default policy for server that includes server backup and maintenance;
start now: ;
restart now: ;
stop now: ;
override parameters: ;
operational information: ;
policy definition: \
"{\"policyName\":\"Server Protection\",\"policyComment\":\"Default policy for \
server that includes server backup and maintenance\",\"policyProtectionPeriod\"\
:\"\",\"policySummaryNotification\":{\"policyCompletionNotificationExecuteOn\"\
:\"completion\",\"policyCompletionNotificationAction\":\"nsrlog -f policy_noti\
fications.log\"},\"policyWorkflows\":[{\"workflowName\":\"Server backup\",\"sy\
nthesisRoot\":[\"NSR group/Server backup\",\"NSR Snapshot Policy/Server backup\
\"],\"workflowActions\":[{\"actionName\":\"Server db backup\",\"actionSpecific\
\":{\"actions\":{\"actionType\":\"server backup\",\"asbDestinationPool\":\"Def\
ault\",\"asbDestinationStorageNode\":\"nsrserverhost\",\"asbPerformBootstrap\"\
:true,\"asbPerformCFI\":true,\"asbRetentionPeriod\":\"2 Weeks\"}},\"actionSche\
dulePeriod\":\"month\",\"actionScheduleActivity\":[\"full\",\"1\",\"1\",\"1\",\
\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\
\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\
\"1\"],\"actionComment\":\"Perform server database backup that is required for\
 disaster recovery\",\"actionCompletionNotification\":{\"policyCompletionNotif\
icationExecuteOn\":\"ignore\",\"policyCompletionNotificationAction\":\"\"},\"a\
ctionConcurrent\":false,\"actionDrivenBy\":\"\",\"actionEnabled\":true,\"actio\
nFailureImpact\":\"continue\",\"actionHardLimit\":\"00:00\",\"actionInactivity\
Timeout\":30,\"actionParallelism\":0,\"actionRetries\":1,\"actionRetryDelay\":\
30,\"actionScheduleOverride\":[],\"actionSoftLimit\":\"00:00\"},{\"actionName\"\
:\"Expiration\",\"actionSpecific\":{\"actions\":{\"actionType\":\"expire\"}},\"\
actionSchedulePeriod\":\"week\",\"actionScheduleActivity\":[\"exec\",\"exec\",\
\"exec\",\"exec\",\"exec\",\"exec\",\"exec\"],\"actionComment\":\"Expire the s\
avesets\",\"actionCompletionNotification\":{\"policyCompletionNotificationExec\
uteOn\":\"ignore\",\"policyCompletionNotificationAction\":\"\"},\"actionConcur\
rent\":false,\"actionDrivenBy\":\"Server db backup\",\"actionEnabled\":true,\"\
actionFailureImpact\":\"continue\",\"actionHardLimit\":\"00:00\",\"actionInact\
ivityTimeout\":30,\"actionParallelism\":0,\"actionRetries\":1,\"actionRetryDel\
ay\":30,\"actionSoftLimit\":\"00:00\"},{\"actionName\":\"clone\",\"actionSpeci\
fic\":{\"actions\":{\"actionType\":\"clone\",\"acBrowsePeriod\":\"1 Months\",\"\
acChronologicalOrder\":false,\"acDDRetentionLockTime\":\"\",\"acDeleteSource\"\
:false,\"acDestinationPool\":\"Default Clone\",\"acDestinationStorageNode\":\"\
nsrserverhost\",\"acEnableDDRetentionLock\":false,\"acRetentionPeriod\":\"1 Mo\
nths\",\"acSourceStorageNode\":\"nsrserverhost\"}},\"actionSchedulePeriod\":\"\
week\",\"actionScheduleActivity\":[\"exec\",\"exec\",\"exec\",\"exec\",\"exec\"\
,\"exec\",\"exec\"],\"actionComment\":\"\",\"actionCompletionNotification\":{\"\
policyCompletionNotificationExecuteOn\":\"ignore\",\"policyCompletionNotificat\
ionAction\":\"\"},\"actionConcurrent\":false,\"actionDrivenBy\":\"Server db ba\
ckup\",\"actionEnabled\":false,\"actionFailureImpact\":\"continue\",\"actionHa\
rdLimit\":\"00:00\",\"actionInactivityTimeout\":30,\"actionParallelism\":0,\"a\
ctionRetries\":1,\"actionRetryDelay\":1,\"actionScheduleOverride\":[],\"action\
SoftLimit\":\"00:00\"}],\"workflowAutostartEnabled\":true,\"workflowComment\":\
\"Perform server backup\",\"workflowCompletionNotification\":{\"policyCompleti\
onNotificationExecuteOn\":\"ignore\",\"policyCompletionNotificationAction\":\"\
\"},\"workflowEnabled\":true,\"workflowEndtime\":\"10:00\",\"workflowGroups\":\
[\"Server Protection\"],\"workflowInterval\":\"24:00\",\"workflowNextstart\":\"\
2025-01-17T10:00:00-0500\",\"workflowRestartWindow\":\"12:00\",\"workflowStart\
time\":\"10:00\"},{\"workflowName\":\"NVE Server Backup\",\"synthesisRoot\":[\"\
NSR group/NVE Server Backup\",\"NSR Snapshot Policy/NVE Server Backup\"],\"wor\
kflowActions\":[{\"actionName\":\"NVE Backup\",\"actionSpecific\":{\"actions\"\
:{\"actionType\":\"backup\",\"actionBackupSubtypeSpecific\":{\"backupSubtypes\"\
:{\"abBackupSubtype\":\"traditional\",\"abtDestinationPool\":\"Default\",\"abF\
orceBackupLevel\":\"\",\"abtBrowsePeriod\":\"1 Months\",\"abtDDRetentionLockTi\
me\":\"\",\"abtEnableCfiBackup\":false,\"abtEnableDDRetentionLock\":false,\"ab\
tEstimate\":false,\"abtFailBackupOnTwinFailure\":false,\"abtFileInactivityAler\
tThreshold\":0,\"abtFileInactivityThreshold\":0,\"abtRevertToFullWhenSynthetic\
FullFails\":true,\"abtTimestampFormat\":\"none\",\"abtVerifySyntheticFull\":tr\
ue}},\"abDestinationStorageNode\":[\"nsrserverhost\"],\"abRetentionPeriod\":\"\
1 Months\",\"abOverrideRetentionPeriod\":false,\"abOverrideBackupSchedule\":fa\
lse,\"abClientOverridesBehavior\":\"clientCanOverride\",\"abSuccessThreshold\"\
:\"success\"}},\"actionSchedulePeriod\":\"week\",\"actionScheduleActivity\":[\"\
full\",\"full\",\"full\",\"full\",\"full\",\"full\",\"full\"],\"actionComment\"\
:\"\",\"actionCompletionNotification\":{\"policyCompletionNotificationExecuteO\
n\":\"ignore\",\"policyCompletionNotificationAction\":\"\"},\"actionConcurrent\
\":false,\"actionDrivenBy\":\"\",\"actionEnabled\":true,\"actionFailureImpact\"\
:\"continue\",\"actionHardLimit\":\"00:00\",\"actionInactivityTimeout\":30,\"a\
ctionParallelism\":100,\"actionRetries\":1,\"actionRetryDelay\":30,\"actionSch\
eduleOverride\":[],\"actionSoftLimit\":\"00:00\"},{\"actionName\":\"clone\",\"\
actionSpecific\":{\"actions\":{\"actionType\":\"clone\",\"acBrowsePeriod\":\"1\
 Months\",\"acChronologicalOrder\":false,\"acDDRetentionLockTime\":\"\",\"acDe\
leteSource\":false,\"acDestinationPool\":\"Default Clone\",\"acDestinationStor\
ageNode\":\"nsrserverhost\",\"acEnableDDRetentionLock\":false,\"acRetentionPer\
iod\":\"1 Months\",\"acSourceStorageNode\":\"nsrserverhost\"}},\"actionSchedul\
ePeriod\":\"week\",\"actionScheduleActivity\":[\"exec\",\"exec\",\"exec\",\"ex\
ec\",\"exec\",\"exec\",\"exec\"],\"actionComment\":\"\",\"actionCompletionNoti\
fication\":{\"policyCompletionNotificationExecuteOn\":\"ignore\",\"policyCompl\
etionNotificationAction\":\"\"},\"actionConcurrent\":false,\"actionDrivenBy\":\
\"NVE Backup\",\"actionEnabled\":false,\"actionFailureImpact\":\"continue\",\"\
actionHardLimit\":\"00:00\",\"actionInactivityTimeout\":30,\"actionParallelism\
\":0,\"actionRetries\":1,\"actionRetryDelay\":1,\"actionScheduleOverride\":[],\
\"actionSoftLimit\":\"00:00\"}],\"workflowAutostartEnabled\":true,\"workflowCo\
mment\":\"Perform NVE server backup\",\"workflowCompletionNotification\":{\"po\
licyCompletionNotificationExecuteOn\":\"ignore\",\"policyCompletionNotificatio\
nAction\":\"\"},\"workflowEnabled\":true,\"workflowEndtime\":\"14:05\",\"workf\
lowGroups\":[\"NVE Server\"],\"workflowInterval\":\"24:00\",\"workflowNextstar\
t\":\"2025-01-17T14:05:00-0500\",\"workflowRestartWindow\":\"12:00\",\"workflo\
wStarttime\":\"14:05\"},{\"workflowName\":\"NMC server backup\",\"synthesisRoo\
t\":[\"NSR group/NMC server backup\",\"NSR Snapshot Policy/NMC server backup\"\
],\"workflowActions\":[{\"actionName\":\"NMC server backup\",\"actionSpecific\"\
:{\"actions\":{\"actionType\":\"backup\",\"actionBackupSubtypeSpecific\":{\"ba\
ckupSubtypes\":{\"abBackupSubtype\":\"traditional\",\"abtDestinationPool\":\"D\
efault\",\"abForceBackupLevel\":\"\",\"abtBrowsePeriod\":\"2 Weeks\",\"abtDDRe\
tentionLockTime\":\"\",\"abtEnableCfiBackup\":false,\"abtEnableDDRetentionLock\
\":false,\"abtEstimate\":false,\"abtFailBackupOnTwinFailure\":false,\"abtFileI\
nactivityAlertThreshold\":0,\"abtFileInactivityThreshold\":0,\"abtRevertToFull\
WhenSyntheticFullFails\":true,\"abtTimestampFormat\":\"none\",\"abtVerifySynth\
eticFull\":true}},\"abDestinationStorageNode\":[\"nsrserverhost\"],\"abRetenti\
onPeriod\":\"2 Weeks\",\"abOverrideRetentionPeriod\":false,\"abOverrideBackupS\
chedule\":false,\"abClientOverridesBehavior\":\"clientCanOverride\",\"abSucces\
sThreshold\":\"success\"}},\"actionSchedulePeriod\":\"week\",\"actionScheduleA\
ctivity\":[\"full\",\"full\",\"full\",\"full\",\"full\",\"full\",\"full\"],\"a\
ctionComment\":\"\",\"actionCompletionNotification\":{\"policyCompletionNotifi\
cationExecuteOn\":\"ignore\",\"policyCompletionNotificationAction\":\"\"},\"ac\
tionConcurrent\":false,\"actionDrivenBy\":\"\",\"actionEnabled\":true,\"action\
FailureImpact\":\"continue\",\"actionHardLimit\":\"00:00\",\"actionInactivityT\
imeout\":30,\"actionParallelism\":100,\"actionRetries\":1,\"actionRetryDelay\"\
:30,\"actionScheduleOverride\":[],\"actionSoftLimit\":\"00:00\"},{\"actionName\
\":\"clone\",\"actionSpecific\":{\"actions\":{\"actionType\":\"clone\",\"acBro\
wsePeriod\":\"1 Months\",\"acChronologicalOrder\":false,\"acDDRetentionLockTim\
e\":\"\",\"acDeleteSource\":false,\"acDestinationPool\":\"Default Clone\",\"ac\
DestinationStorageNode\":\"nsrserverhost\",\"acEnableDDRetentionLock\":false,\"\
acRetentionPeriod\":\"1 Months\",\"acSourceStorageNode\":\"nsrserverhost\"}},\"\
actionSchedulePeriod\":\"week\",\"actionScheduleActivity\":[\"exec\",\"exec\",\
\"exec\",\"exec\",\"exec\",\"exec\",\"exec\"],\"actionComment\":\"\",\"actionC\
ompletionNotification\":{\"policyCompletionNotificationExecuteOn\":\"ignore\",\
\"policyCompletionNotificationAction\":\"\"},\"actionConcurrent\":false,\"acti\
onDrivenBy\":\"NMC server backup\",\"actionEnabled\":false,\"actionFailureImpa\
ct\":\"continue\",\"actionHardLimit\":\"00:00\",\"actionInactivityTimeout\":30\
,\"actionParallelism\":0,\"actionRetries\":1,\"actionRetryDelay\":1,\"actionSc\
heduleOverride\":[],\"actionSoftLimit\":\"00:00\"}],\"workflowAutostartEnabled\
\":true,\"workflowComment\":\"Perform NMC database backup\",\"workflowCompleti\
onNotification\":{\"policyCompletionNotificationExecuteOn\":\"ignore\",\"polic\
yCompletionNotificationAction\":\"\"},\"workflowEnabled\":true,\"workflowEndti\
me\":\"14:00\",\"workflowGroups\":[\"NMC server\"],\"workflowInterval\":\"24:0\
0\",\"workflowNextstart\":\"2025-01-17T14:00:00-0500\",\"workflowRestartWindow\
\":\"12:00\",\"workflowStarttime\":\"14:00\"}]}";
Notification action: nsrlog -f policy_notifications.log;
Notification execute on: completion;
manual saves: No;
hostname: NSR_HOSTNAME;
administrator: "user=root,host=NSR_HOSTNAME",\
"user=administrator,host=NSR_HOSTNAME",\
"user=system,host=NSR_HOSTNAME",\
"user=nsrnmc,host=NSR_HOSTNAME",\
nsrnmc@NSR_HOSTNAME;

6. Import the configuration file into the nsrdb: 

nsradmin -d /nsr/res/nsrdb -i /tmp/serverprotection.txt

nve:~ # nsradmin -d /nsr/res/nsrdb -i /tmp/serverprotection.txt
173680:nsradmin: RPC client handle: Connection refused.
The host address '::1' is unknown.
172088:nsradmin: Unable to create the connection to host '::1' at port number 7938.
173677:nsradmin: Check whether the client services are running on the host '::1'.
173680:nsradmin: RPC client handle: Connection refused.
The host address '127.0.0.1' is unknown.
172088:nsradmin: Unable to create the connection to host '127.0.0.1' at port number 7938.
173677:nsradmin: Check whether the client services are running on the host '127.0.0.1'.
42503:nsradmin: RPC cannot make a procedure call: Create failed: Remote system error - Connection refused
189067:nsradmin: Policy 'Server Protection' was successfully created.

7. Start NetWorker services:

systemctl start networker
systemctl start gst
systemctl start nwui

8. Log in to the NMC or NetWorker Web User Interface (NWUI).
9. Reconfigure the default Server Protection workflows to use the target pools configured in your environment.
10. Log in to the avinstaller user interface as root: https://NVE_HOSTNAME
11. Click "Monitor" on the ongoing upgrade and select "Retry Current Task" the upgrade process should complete.
12. Start each of the workflows in the Server Protection policy and confirm that they are successful.

NetWorker Virtual Edition: Upgrade or Install Failure Log Locations and Collection