Dell Networking SONiC update partial configuration lost after the reboot

Summary: This article explains about Dell SONiC configuration loss during the upgrade process and a workaround.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

 

 If we have pending config changes to be saved into the startup-config, then we must pay attention when we save the configuration as these two slightly similar actions have different outcomes.

This sequence order will correctly reflect all the changes after the reboot.

     Sonic# copy running-configuration startup-configuration
     Sonic# image install http://SONiC_EntStd_4.1.1.bin
     Sonic# reboot
 

This sequence order will cause any nonsaved configuration changes into the startup-configuration to be lost after the reboot.

     Sonic# image install http://SONiC_EntStd_4.1.1.bin
     Sonic# copy running-configuration startup-configuration
     Sonic# reboot   

We will discuss why this happens in the next section.

Cause

When, we issue image install http://<DELL-SONIC-INSTALL-FILE.bin> to upgrade the following process happens.

- The startup_config /etc/sonic/config_db.json is migrated into /host/old_config/config_db.json applying any required schema change in the config for the new version.
- During the switch, reboot this configuration stored in /host/old_config/config_db.json is transferred to /etc/sonic/config_db.json after changes related to the new version are applied.
 

Hence any pending changes saved after we run image install http://<DELL-SONIC-INSTALL-FILE.bin> will not be reflected into /host/old_config/config_db.json. In consequence this configuration change will be lost during the reboot. 


 

 

Resolution

There could be a few workarounds once you land into this situation, two of the easiest would be:
 

Option 1

 Boot the switch into the old SONiC version, delete the new version, and reinstall the new version again:

Sonic# show image list
Sonic# image set-default <Old-SONIC version>
Sonic# reboot

When the switch comes back with the old version:

Sonic# show image list
Sonic# image remove <new-SONiC-version>
Sonic# image install <http://new-SONIC version>
Sonic# reboot


NOTE: It is assumed the configuration was saved after image install command, successfully performed and before reboot.

Sample output

admin@sonic:~$ sonic-cli
sonic# show image list
Current: SONiC-OS-4.0.6-Enterprise_Standard
Next: SONiC-OS-4.0.6-Enterprise_Standard
Available: 
SONiC-OS-4.0.3-Enterprise_Base
SONiC-OS-4.0.6-Enterprise_Standard
sonic# image set-default SONiC-OS-4.0.3-Enterprise_Base
sonic# reboot

reboot in process .....
Waiting for the reboot operation to complete
11, 32, 15, 00068001, 19, 00068000,

When the switch comes back:

sonic login: admin
Password: 
Last login: Wed Oct  4 10:10:30 UTC 2023 on ttyS0
Linux sonic 4.19.0-9-2-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64
You are on
  ____   ___  _   _ _  ____
 / ___| / _ \| \ | (_)/ ___|
 \___ \| | | |  \| | | |
  ___) | |_| | |\  | | |___
 |____/ \___/|_| \_|_|\____|

-- Software for Open Networking in the Cloud --

Unauthorized access and/or use are prohibited.
All access and/or use are subject to monitoring.

Help:    http://azure.github.io/SONiC/

admin@sonic:~$ sonic-cli
sonic# show image list
Current: SONiC-OS-4.0.3-Enterprise_Base
Next: SONiC-OS-4.0.3-Enterprise_Base
Available: 
SONiC-OS-4.0.3-Enterprise_Base
SONiC-OS-4.0.6-Enterprise_Standard
sonic# image remove SONiC-OS-4.0.6-Enterprise_Standard
Remove image SONiC-OS-4.0.6-Enterprise_Standard? [y/N]:y
sonic# show image list
Current: SONiC-OS-4.0.3-Enterprise_Base
Next: SONiC-OS-4.0.3-Enterprise_Base
Available: 
SONiC-OS-4.0.3-Enterprise_Base
sonic# image install http://10.24.30.105/software/Dell%20EMC/Networ
king/SONiC_EntStd_4.0.6.bin
%Info: Check 'show image status' for image install progress.
sonic# show image status
-----------------------------------------------------------
Global operation status  : GLOBAL_STATE_SUCCESS
-----------------------------------------------------------
File operation status    : TRANSFER_STATE_SUCCESS
File size(bytes)         : 1055807512
File transfer bytes      : 1055807512
File progress            : 100%
Transfer start time      : 2023-10-04 10:17:40+0000
Transfer end time        : 2023-10-04 10:17:49+0000
-----------------------------------------------------------
Install operation status : INSTALL_STATE_SUCCESS
Install start time       : 2023-10-04 10:17:49+0000
Install end time         : 2023-10-04 10:18:26+0000
sonic# show image list
Current: SONiC-OS-4.0.3-Enterprise_Base
Next: SONiC-OS-4.0.6-Enterprise_Standard
Available: 
SONiC-OS-4.0.3-Enterprise_Base
SONiC-OS-4.0.6-Enterprise_Standard
sonic# reboot

 

Option 2

Apply the missing configuration in the new SONiC OS. 

NOTE: If we attempt to copy the config_db.json from the old partition into the new partition, we may land into other issues as the config file’s schema may have changed between the versions.

Additional Information

This behavior was validating in both 4.0.x upgrades in branch and 4.1.x upgrades in the same branch. Expected to remain for any upgrades.

Affected Products

Enterprise SONiC Distribution, PowerSwitch E3200-ON Series, Dell EMC Networking N3200-ON, PowerSwitch S5212F-ON, PowerSwitch S5224F-ON, PowerSwitch S5232F-ON, PowerSwitch S5248F-ON, PowerSwitch S5296F-ON, PowerSwitch Z9264F-ON, PowerSwitch Z9332F-ON , PowerSwitch Z9432F-ON ...
Article Properties
Article Number: 000218166
Article Type: Solution
Last Modified: 11 Oct 2023
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.