Recover environment variables missing post-reboot in InsightIQ 6.x.
Summary: Certain critical environment variables (for example, IAM_CPUS, RDP_MEMORY, and so forth) may intermittently be unset or appear blank after upgrading InsightIQ from version 5.x to 6.x.
Symptoms
Upgrading InsightIQ from version 5.x to 6.x or during reboots on a 6.x setup such as those triggered by a system-shutdown or system-reboot, may cause some critical environment variables (for example, IAM_CPUS, RDP_MEMORY) to intermittently be unset or appear blank.
This issue typically manifests during execution of upgrade scripts (from earlier to later 6.x versions) or system-level scripts. Errors related to missing environment variables are displayed on the console and InsightIQ service containers fail to initialize.
Error messages such as: WARN[0000] The "RDP_CPUS" variable is not set. Defaulting to a blank string.
WARN[0000] The "DC_MEMORY" variable is not set. Defaulting to a blank string.
Cause
Systems upgraded from 5.x to 6.x have multiple execution calls in the crontab which causes the scripts to run concurrently when the system comes up after reboot. This causes a race condition where some environment variables are abruptly deleted. This issue is addressed in the 6.1.0 release.
Resolution
Open a Support case if you suspect you are experiencing this issue to get the file stated in the resolution. This allows verification that you are truly experiencing this issue as there are similar issues that the below action plan does not address.
When opening a case, take the information from article Isilon: How to gather InsightIQ logs (login as a registered Dell Support user may be required to view) and any PuTTY output that shows the error.
- Download the
restore-env-vars.tar.gz.
- Untar the file using
tar xvzf restore-env-vars.tar.gz. cd restore-env-varschmod 777 main.sh utils.sh./main.sh- This restores the environment variables for the InsightIQ 6.x.- Log out from the current terminal and log in again.
- After applying the fix, proceed with the upgrade from the 6.x earlier version to the 6.x later version, or with system scripts like
system-shutdown,system-rebootorbash restart.shto restart the InsightIQ services.