VxRail: Manager Remediate Apache Log4Shell Vulnerability (CVE-2021-44228, CVE-2021-45046)

The Apache Software Foundation has published information about a critical Apache Log4j Library Remote Code Execution Vulnerability issue that is known as Log4Shell as per the GitHub Advisory Database (also detailed in CVE-2021-44228, CVE-2021-45046, and CVE-2021-4104). VxRail Manager is exposed to the issue outlined in the vulnerability.
 

For more on these CVEs, see the following articles:

• Apache Log4j Security Vulnerabilities
• CVE-2021-45046 (Log4j 2.15)
• CVE-2021-4104 (Log4j 1.2)

An additional issue was discovered in the previous script which may result in an impacted file being restored on VxRail Manager from a system archive. This issue has also been addressed in this release.

If you used any previous scripts that were provided with this article, then download the latest script (1.1.2) and run it on VxRail Manager to ensure that you have the complete fix.
 

Requirements and Scope

This scope of what is covered by the remediation steps in this article are:

• This article is applicable to VxRail Manager in VxRail 4.5.x, 4.7.x, and 7.0.x releases along with VxRail Manager in VCF 3.x and 4.x releases.
• The script and remediation steps provided remediate the vulnerability in VxRail Manager appliance VM only.
• Other components outside VxRail Manager such as vCenter Server Appliance (vCSA), NSX, and so forth must be mitigated separately, and are not in this script.
• Also, the script does not remediate any applications or services running inside VMs which may be exposed to the vulnerability. Dell Technologies recommends all customers check with their application or software vendors for services running in VMs to ensure that they are not impacted.

Links to impacted VMware products and potential workarounds are detailed in the following VMware VMSA article:

• VMware Security Advisory VMSA-2021-0028

VMware provides a script to automate the remediation in the vCenter Server Appliance in the following article:

• Python script to automate the workaround steps of VMSA-2021-0028 vulnerability on vCenter Server Appliance (87088)

Attached to this article is file fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip which contains the script for VxRail Manager only.
 

VxRail releases with fix included

This issue has is resolved in the following VxRail software releases:

• VxRail Package Software 7.0.320
• VxRail Appliance Software 4.7.541
• VxRail Appliance Software 4.5.471

It is recommended to upgrade to a VxRail software release which includes the fix.
The script is recommended for customers who are unable to upgrade immediately.

• VxRail: Upgrading vCenter to 7.0 U3c or Later Fails Due to Potential Driver Conflict

Additional Considerations for Legacy Patching Scripts:

If you have previously run an earlier updating script (older than fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip) on the VxRail Manager, be aware of the following:

• • That script creates backup copies of original jar files with a ".bak" extension and keeps them in the original library directories.

  • These .bak files may cause service startup issues after upgrading to VxRail 8.0.380 or later, as they will be mistakenly loaded along with active files in the same directory.

Follow the steps below to relocate legacy .bak files and ensure normal system operation:

1. Check for existing .bak files
   • find /mystic/connectors -name "log4j-core*.bak"
2. If .bak files are found, move them to a safe location:
   • mkdir -p /tmp/log4jbak
   • find /mystic/connectors -name "log4j-core*.bak" -type f -exec mv {} /tmp/log4jbak/ \;
3. Check.bak files not existing in /mystic/connectors folder, then restart service:

• • Scenario A — If you have NOT yet performed the LCM upgrade:
    • Proceed with the LCM upgrade. The upgrade process will automatically restart all services.

• • Scenario B — If you have ALREADY completed the LCM upgrade and are experiencing VC plugin failure:
    • Restart VxRail Manager:
      • systemctl restart vmware-marvin
      • service runjars restart

Remediation steps

To remediate against the issues perform the following steps:

1. Download the fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip file that is attached to this article.
2. Upload the .zip file to VxRail Manager using mystic user over SCP (WinSCP is an example of an SCP client that can be used).
3. Log in to the VxRail Manager VM console or SSH using the mystic user. 
4. Change the directory to where you uploaded the .zip file and extract it using the unzip command:

   mystic@vxrm:~> unzip fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip
   Archive:  fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip
     inflating: fixlog4j-CVE-2021-44228-CVE-2021-45046.sh

5. Make the script executable using the chmod command:

   mystic@vxrm:~> chmod +x fixlog4j-CVE-2021-44228-CVE-2021-45046.sh

6. Log in as the VxRail Manager root user using the su command:

   mystic@vxrm:~> su -
   Password:

7. Ensure that you are in the same directory where you extracted the script package:

   vxrm:~ # cd /home/mystic
   vxrm:/home/mystic #

8. Run the script:

   vxrm:/home/mystic # ./fixlog4j-CVE-2021-44228-CVE-2021-45046.sh

   Example script output:

   Stop MARVIN and runjars service before patching the system
   
   /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar is affected by CVE-2021-44228 and CVE-2021-45046, need to apply patch
   patching /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar
   Successfully patched /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar
   
   /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar is affected by CVE-2021-44228 and CVE-2021-45046, need to apply patch
   patching /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar
   Successfully patched /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar
   
   To ensure there is no reload behavior, we need to pack the .war file as well.
   looks like /usr/lib/vmware-marvin/marvind/webapps/ROOT.war contains the bad log4j-core library WEB-INF/lib/log4j-core-2.13.0.jar
   Archive:  /usr/lib/vmware-marvin/marvind/webapps/ROOT.war
     inflating: WEB-INF/lib/log4j-core-2.13.0.jar
   Patching WEB-INF/lib/log4j-core-2.13.0.jar in /usr/lib/vmware-marvin/marvind/webapps/ROOT.war
   Repack /usr/lib/vmware-marvin/marvind/webapps/ROOT.war
   updating: WEB-INF/lib/log4j-core-2.13.0.jar (deflated 11%)
   Clean up the ROOT folder...
   
   Always apply a reboot of MARVIN and runjars services
   restart MARVIN
   MARVIN restart successfully
   restart runjars
   runjars restart successfully

Wait at least 10 minutes if you plan to perform any of the manual validation steps below.

There are several different versions of the lib4j-core library depending on the release of VxRail Manager.

The script has been designed to correctly remediate VxRail Manager regardless of the version of lib4j-core which is included with that version of VxRail Manager.

The above output from running the script may show different files being updated depending on the version of lib4j-core included.
 

Links to VMware articles covering workarounds to their products and fixes is found in VxRail: Information on Log4Shell (CVE-2021-44228/CVE-2021-45046/CVE-2021-4104) and VxRail Environments
 

Validation Steps

In order to remediate the issue, the script removes the JndiLookup.class file from the lib4j-core-* jar files.

A jar file is a Java packaging format for including multiple class, metadata, and other java programs in a single file. It is similar in concept to a .zip file and is based on the .zip format. The script execution validates that each jar file has been successfully updated.

To perform a manual validation that the script has worked, you can check if the log4j-core-* jar files present on VxRail Manager still contain the impacted JndiLookup.class file. If it has worked, you should see no output to the below commands which confirms the impacted JndiLookup.class file is no longer present in the jar file.
 

 Validation with automated command 

The following command can be run on VxRail Manager to scan for all log4j-core-xxxx.jar files present on VxRail Manager and check to see if they contain the impacted JndiLookup.class file:

vxrm:/home/mystic # for myfile in `find / -name log4j-core*jar -print |grep -v log4jbak`; do echo $myfile; unzip -l $myfile | grep JndiLookup.class; done

Sample output (updated system):

/mystic/connectors/eservice/lib/log4j-core-2.13.0.jar
/mystic/connectors/cluster/lib/log4j-core-2.13.0.jar
/usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.0.jar

In the above example the JndiLookup.class file is not present in jar, thus the script worked, and verification is successful.

Below is a Sample output of an impacted system which must be updated:

/mystic/connectors/eservice/lib/log4j-core-2.13.3.jar
     2892  2020-05-10 12:08   org/apache/logging/log4j/core/lookup/JndiLookup.class
/mystic/connectors/cluster/lib/log4j-core-2.13.3.jar
     2892  2020-05-10 12:08   org/apache/logging/log4j/core/lookup/JndiLookup.class
/usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.3.jar
     2892  2020-05-10 12:08   org/apache/logging/log4j/core/lookup/JndiLookup.class

In the above example, the impacted JndiLookup.class file is still present in the log4j-core-2.13.3.jar jar file.

Below is Sample output of a system which has a fixed or updated log4j-core library where seeing the JndiLookup.class file can be ignored:

/usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.17.1.jar
     3158  2021-12-27 17:30   org/apache/logging/log4j/core/lookup/JndiLookup.class
/mystic/connectors/eservice/lib/log4j-core-2.17.1.jar
     3158  2021-12-27 17:30   org/apache/logging/log4j/core/lookup/JndiLookup.class
/mystic/connectors/cluster/lib/log4j-core-2.17.1.jar
     3158  2021-12-27 17:30   org/apache/logging/log4j/core/lookup/JndiLookup.class

In the above example you can see the JndiLookup.class file in the output but the fix for the issue is in log4j-core-2.17.1.jar.

 

Validation with manual check of each file

To quickly identify any log4j-core-xxxx.jar files present on VxRail Manager, run the following command (this also formats the output into a usable command):

vxrm:/home/mystic # find / -name log4j-core*jar -print |grep -v log4jbak | awk '{print("unzip -l " $1 "|grep JndiLookup.class")}'

Sample output:

unzip -l /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
unzip -l /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
unzip -l /usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.0.jar|grep JndiLookup.class

From the sample output above, run each command manually to see if they detect the impacted JndiLookup.class file:

vxrm:/home/mystic # unzip -l /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
vxrm:/home/mystic #

vxrm:/home/mystic # unzip -l /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
vxrm:/home/mystic #

vxrm:/home/mystic # unzip -l /usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
vxrm:/home/mystic #

In the above example, the JndiLookup.class file is not present in jar, thus the script worked, and verification is successful.
 

vxrm:/home/mystic # unzip -l /mystic/connectors/cluster/lib/log4j-core-2.4.1.jar |grep JndiLookup.class
2576 2015-10-08 17:50 org/apache/logging/log4j/core/lookup/JndiLookup.class

In the above example, the impacted JndiLookup.class file is still present in the log4j-core-2.4.1.jar jar file.