Dell NativeEdge: Upgrade from v3.0 to v3.1 fails with 0x01902004 error

When users upgrade NativeEdge Orchestrator from v3.0 to v3.1, 0x01902004 error might be observed. Users can find the following failure message in the upgrade script output. 

OK. Connection can be established!
Splitting upgrade bundle into chunks: /home/user/baobab/.resources/upload/eoBundle_v3.1.0.0.tar.gz Starting to upload the upgrade bundle: /home/user/baobab/.resources/upload/eoBundle_v3.1.0.0.tar.gz
Did not get the expected response while uploading chunk 1418. Received response: no healthy upstream. Error code: 0x019020004 cleaning up temp files

The upgrade service was reporting readiness probe failures from time to time, causing the upgrade service to be marked as not ready and not accept traffic. The result is the failure during chunk upload. 

The NativeEdge Orchestrator upgrade service monitors NATS to see if it is available to function properly. Prior to v3.1 upgrade, the service uses the /varz endpoint of NATS that has a relatively larger payload. When this call takes more time than expected, the upgrade service marks itself as not ready causing the problem. 

Note

• This issue only occurs when upgrade from v3.0.0.0 to v3.1.0.0. 
• Upgrade from v3.1.0.0 to future releases should not observe this issue. 

Rebooting the NativeEdge Orchestrator virtual machine. It is observed in the testing that rebooting the virtual machine resolves the issue most of the time.

If rebooting the virtual machine does not resolve the issue, engage Dell support for further instructions.