ECS: ECS Solution to Apache Log4j Remote Code Execution Vulnerability
Summary: Apache Log4j security vulnerability
Symptoms
CVE Identifier CVE-2021-44228
CVE Identifier CVE_2021-45046
Apache Publication: Apache Log4j Remote Code Execution
Cause
Apache Log4j security vulnerability
Resolution
Who should run this procedure?
Dell requests that customers perform this procedure of upgrading xDoctor, and installing the patch. This is the quickest and safest method as it avoids prolonged exposure to this Apache vulnerability. All the steps are detailed in this KB. There is also a video guide which can be followed to accompany this KB which resides at below link.
Video: Apache-Log4j
Impact of procedure:
Expect possible I/O timeouts while services are restarted. Applications accessing the cluster must be able to handle the I/O timeout. A maintenance window is advised when performing this procedure.
Time taken for the activity (approximately):
An approximately 7 minute delay is set by default per node between service restarts. The number of nodes in a Virtual Data Center (VDC) multiplied by 7 minutes + 60 minutes for preparation, DT stabilization, and post checks needed.
Examples:
A 48 node VDC system can take approximately 6.5 hours:
7.5 Minutes X 48 (Number of VDC nodes) + 30 minutes (preparation) = 6.5 hours or 390 minutes
An eight node VDC system can take approximately 1.5 hours:
7.5 Minutes X 8 (Number of VDC nodes) + 30 minutes (preparation) = 1.5 hours or 90 minutes
Frequently Asked Questions (FAQ):
Q: Is the patch part of the xDoctor release?
A: The patch install script is part of xDoctor release 4.8-79.1 and higher. Instructions for the download of xDoctor and execution of patch install are in the resolution steps.
Q: Can I update multiple VDCs in parallel?
A: No. Patch 1 VDC at a time.
Q: Can I apply this patch on ECS running code version 3.2.x or earlier?
A: No, this patch is applicable only to ECS versions 3.3.x - 3.6.x. Open a service request to schedule an upgrade for earlier versions.
Q: If I upgrade ECS after running this procedure, do I rerun the procedure post upgrade?
A: No, if upgrading to a code version specified in DSA-2021-273 which has the permanent fix. Yes, if upgrading to a code version not specified in this same DSA.
Q: Does the patch require reapplication on a system where it was previously installed after a node replacement, reimage, or expansion?
A: No, if VDC is at the code version that is specified in DSA-2021-273. Yes, if doing any of these actions against a VDC running a code version not specified in this same DSA. Where a patch is required for these scenarios, the Dell engineer in question contacts you to inform that an update is required.
Q: What user should you be logged in as to run all commands in this KB?
A: Admin
Q: Does svc_patch have to be run on all racks or with a specialized MACHINES file where multiple racks in a VDC?
A: No, it auto-detects if multiple racks exist and updates all nodes on all racks on that VDC.
Q: I notice that the target xDoctor release is now 4.8-79.1 and not 4.8-79.0. Why?
A: xDoctor releases occur frequently so it is always recommended to upgrade to the highest released version. If however you have previously run Apache fix using 4.8-79.0, then system is fully protected against the vulnerability, and does not have to be rerun.
Resolution Summary:
- Upgrade your ECS xDoctor software to version 4.8.-79.1 or later
- Run Prechecks.
- Apply the system patch with the svc_patch tool included with xDoctor.
- Confirm that the fix has been applied.
- Troubleshooting.
Resolution Steps:
1. Upgrade your ECS xDoctor software to the latest Version available.
-
Check the xDoctor version running on your system. If the version is 4.8-79.1 or later, move to step 2 "Run Prechecks." If not, proceed with the steps below.
Command:
# sudo xdoctor --version
Example:
admin@node1:~> sudo xdoctor --version 4.8-79.1
- Log in to the Support Site, connect directly to the download link, search for xDoctor using the keyword search, and click the xDoctor RPM link to download. To view the Release notes, follow ReleaseNotes, select Manuals and documents from the sidebar from where they should be available for download.
- Once the RPM is downloaded, use any remote SCP program to upload the file to the /home/admin directory on the first ECS node.
- Once the upload is complete, SSH to the first node of the ECS system using admin.
-
Upgrade xDoctor on all the nodes with the newly distributed version.
Command:
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-79.1.noarch.rpm
Example:
admin@node1:~> sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-79.1.noarch.rpm 2021-12-20 12:06:11,358: xDoctor_4.8-78.2 - INFO : xDoctor Upgrader Instance (2:FTP_SFTP) 2021-12-20 12:06:11,358: xDoctor_4.8-78.2 - INFO : Local Upgrade (/home/admin/xDoctor4ECS-4.8-79.1.noarch.rpm) 2021-12-20 12:06:11,392: xDoctor_4.8-78.2 - INFO : Current Installed xDoctor version is 4.8-78.2 2021-12-20 12:06:11,429: xDoctor_4.8-78.2 - INFO : Requested package version is 4.8-79.1 2021-12-20 12:06:11,430: xDoctor_4.8-78.2 - INFO : Updating xDoctor RPM Package (RPM) 2021-12-20 12:06:11,482: xDoctor_4.8-78.2 - INFO : - Distribute package 2021-12-20 12:06:12,099: xDoctor_4.8-78.2 - INFO : - Install new rpm package 2021-12-20 12:06:37,829: xDoctor_4.8-78.2 - INFO : xDoctor successfully updated to version 4.8-79.1
-
If the environment is a multirack VDC, the new xDoctor package must be installed on the first node of each rack. To identify these rack primaries, run the below command. In this instance, there are four racks and four rack primaries highlighted.
-
Command:
# svc_exec -m "ip address show private.4 |grep -w inet"
Example:
admin@ecsnode1~> svc_exec -m "ip address show private.4 |grep -w inet" svc_exec v1.0.2 (svc_tools v2.1.0) Started 2021-12-20 14:03:33 Output from node: r1n1 retval: 0 inet 169.254.1.1/16 brd 169.254.255.255 scope global private.4 Output from node: r2n1 retval: 0 inet 169.254.2.1/16 brd 169.254.255.255 scope global private.4 Output from node: r3n1 retval: 0 inet 169.254.3.1/16 brd 169.254.255.255 scope global private.4 Output from node: r4n1 retval: 0 inet 169.254.4.1/16 brd 169.254.255.255 scope global private.4 -
Copy the package from the first node of the system (R1N1) to the other rack primaries per below:
Example:
admin@ecsnode1~> scp xDoctor4ECS-4.8-79.1.noarch.rpm 169.254.2.1:/home/admin/ xDoctor4ECS-4.8-79.1.noarch.rpm 100% 32MB 31.9MB/s 00:00 admin@ecsnode1~> scp xDoctor4ECS-4.8-79.1.noarch.rpm 169.254.3.1:/home/admin/ xDoctor4ECS-4.8-79.1.noarch.rpm 100% 32MB 31.9MB/s 00:00 admin@ecsnode1~> scp xDoctor4ECS-4.8-79.1.noarch.rpm 169.254.4.1:/home/admin/ xDoctor4ECS-4.8-79.1.noarch.rpm 100% 32MB 31.9MB/s 00:00 admin@ecsnode1~>
-
Per step e above, run the same xDoctor install command on each of the above rack primaries that are identified previously.
Command:
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-79.1.noarch.rpm
-
2. Run Prechecks.
-
Use the svc_dt command to check if DTs are stable. DTs are stable if "Unready #" column shows 0. If yes, go to the next check. If no, wait 15 minutes and check again. If DTs have not stabilized, open a service request with the ECS support team.
Command:
# svc_dt check -b
Example:
admin@node1:~> svc_dt check -b svc_dt v1.0.25 (svc_tools v2.0.2) Started 2021-12-16 16:44:51 Date Total DT Unknown # Unready # RIS Fail # Dump Fail # Check type Time since check Check successful 2021-12-16 16:43:44 2432 0 0 0 0 AutoCheck 1m 7s True 2021-12-16 16:42:33 2432 0 0 0 0 AutoCheck 2m 18s True 2021-12-16 16:41:23 2432 0 0 0 0 AutoCheck 3m 28s True 2021-12-16 16:40:13 2432 0 0 0 0 AutoCheck 4m 38s True 2021-12-16 16:39:02 2432 0 0 0 0 AutoCheck 5m 49s True 2021-12-16 16:37:52 2432 0 0 0 0 AutoCheck 6m 59s True 2021-12-16 16:36:42 2432 0 0 0 0 AutoCheck 8m 9s True 2021-12-16 16:35:31 2432 0 0 0 0 AutoCheck 9m 20s True 2021-12-16 16:34:21 2432 0 0 0 0 AutoCheck 10m 30s True 2021-12-16 16:33:11 2432 0 0 0 0 AutoCheck 11m 40s True
-
Use the svc_patch command to validate that all nodes are online. If yes, go to the next step. If no, investigate the reason, bring it back online, and run the check again. If a node cannot be brought online, open a service request with the ECS support team to investigate.
Command:
# /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch status
Example:
admin@node1:~> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch status svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online DONE Checking Installed Patches and Dependencies DONE Patches/releases currently installed: [ None detected ] Patches that need to be installed: CVE-2021-44228_log4j-fix_3.3.x-3.6.2 (PatchID: 3298) Files that need to be installed: /opt/storageos/lib/log4j-core-2.5.jar (from CVE-2021-44228_log4j-fix_3.3.x-3.6.2) The following services need to be restarted: ALL
3. Apply the system patch with the svc_patch tool included with xDoctor.
-
Run the svc_patch command, type "y" and press Enter key when prompted to install the patch. The command can run on any ECS node.
Commands:
# screen -S patchinstall
# unset TMOUT
# /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch install
Example:
Note: There is a prompt to proceed in the output below.admin@node1:~> screen -S patchinstall admin@node1:~> unset TMOUT admin@node1:~> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch install svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online DONE Checking Installed Patches and Dependencies DONE Patches/releases currently installed: [ None detected ] Patches that will be installed: CVE-2021-44228_log4j-fix_3.3.x-3.6.2 (PatchID: 3298) Files that will be installed: /opt/storageos/lib/log4j-core-2.5.jar (from CVE-2021-44228_log4j-fix_3.3.x-3.6.2) The following services will be restarted: ALL Patch Type: Standalone Number of nodes: 8 Number of seconds to wait between restarting node services: 450 Check DT status between node service restarts: true Do you wish to continue (y/n)?y [...Truncated Output of each node Distributing files and restarting services...] Distributing files to node 1xx.xxx.xx.xx Distributing patch installer to node '1xx.xxx.xx.xx' Restarting services on 1xx.xxx.xx.xx Restarting all services Waiting 180 seconds for services to stabilize... [...Truncated Output of each node Distributing files and restarting services...] Stopping ViPR services..done Services status 3: stat georeceiver eventsvc blobsvc dataheadsvc blobsvc-perf blobsvc-fi resourcesvc resourcesvc-perf resourcesvc-fi rm cm ssm objcontrolsvc metering sr storageserver nvmeengine nvmetargetviewer dtquery dtsm vnest coordinatorsvc ecsportalsvc transformsvc Setting up SSL certificates ...done Starting ViPR services..done Waiting 300 seconds for services to stabilize...DONE Patching complete. admin@node1:~>
-
Exit session Screen when updating is completed per the above output.
Example:
admin@node1:/> exit logout [screen is terminating] admin@node1:/>
Note: If you accidentally close the PuTTY session while execution is in progress, reattach by logging back into the same node and run the below command:Command:
# screen -ls
admin@node 1:~> screen -ls There is a screen on: 114475.pts-0.ecs-n3 (Detached) 1 Socket in /var/run/uscreens/S-admin.Reattach to Detached session from previous output.
admin@node1:~> screen -r 114475.pts-0.ecs-n3
4. Confirm that the fix has been applied.
-
The output below is from a system where the fix has been applied.
Command:
# /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch status
Example:
admin@node1:/> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch status svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online DONE Checking Installed Patches and Dependencies DONE Patches/releases currently installed: CVE-2021-44228_log4j-fix_3.3.x-3.6.2 (PatchID: 3298) Fixes for Log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046 Patches that need to be installed: No files need to be installed. The following services need to be restarted: No services need to be restarted. -
The output below is from a system where the fix has not been applied.
Example:
admin@node1:/> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch status svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online DONE Checking Installed Patches and Dependencies DONE Patches/releases currently installed: [ None detected ] Patches that need to be installed: CVE-2021-44228_log4j-fix_3.3.x-3.6.2 (PatchID: 3298) Files that need to be installed: /opt/storageos/lib/log4j-core-2.5.jar (from CVE-2021-44228_log4j-fix_3.3.x-3.6.2) The following services need to be restarted: ALL
Troubleshooting:
-
DT stabilization taking too long
-
If DT stabilization is taking more time than default 7.5 minutes, svc_patch application prompts to either continue or discontinue the patch process.
Example:
admin@node1:~> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch install [...Truncated Output of each node Distributing files and restarting services...] Restarting services on 1xx.xx.xx.xx Restarting all services Waiting 180 seconds for services to stabilize...DONE Waiting for DTs to come online ERROR: DT Check failed. DTs did not come ready or could not be checked after several passes. Do you wish to continue anyway (y/n)?
-
Open a PuTTY session on another node and run svc_dt command to check DTs in "Unready #" column. If there are no "0" values, wait 15 minutes and run the check again. Return to the session with svc_patch when there are no unready DTs. Answer "y" and continue. If svc_dt continues to list values in "Unready #" DTs, open a service request with the ECS support team.
Command:
# svc_dt check -b
Example:
admin@node1:~> svc_dt check -b svc_dt v1.0.25 (svc_tools v2.0.2) Started 2021-12-15 17:18:52 Date Total DT Unknown # Unready # RIS Fail # Dump Fail # Check type Time since check Check successful 2021-12-15 17:17:54 1920 0 0 0 0 AutoCheck 0m 58s True 2021-12-15 17:16:44 1920 0 0 0 0 AutoCheck 2m 8s True 2021-12-15 17:16:10 1920 0 0 0 0 Manual Check 2m 42s True 2021-12-15 17:15:34 1920 0 0 0 0 AutoCheck 3m 18s True 2021-12-15 17:14:24 1920 0 0 0 0 AutoCheck 4m 28s True 2021-12-15 17:13:13 1920 0 0 0 0 AutoCheck 5m 39s True 2021-12-15 17:12:03 1920 0 0 0 0 AutoCheck 6m 49s True 2021-12-15 17:10:53 1920 0 0 0 0 AutoCheck 7m 59s True 2021-12-15 17:09:43 1920 0 0 0 0 AutoCheck 9m 9s True 2021-12-15 17:08:32 1920 0 0 0 0 AutoCheck 10m 20s True
-
-
All services are not restarted on all nodes because not run in screen and the PuTTY session ends prematurely.
Example: Services restarted on four out of six nodes after logging back in. See nodes 5 and 6 highlighted below.
admin@node1:~> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch status svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online DONE Checking Installed Patches and Dependencies DONE All nodes currently do not have the same patches installed. Patches/releases currently installed: 169.254.1.1: CVE-2021-44228_45046_log4j-fix 169.254.1.2: CVE-2021-44228_45046_log4j-fix 169.254.1.3: CVE-2021-44228_45046_log4j-fix 169.254.1.4: CVE-2021-44228_45046_log4j-fix 169.254.1.5: CVE-2021-44228_45046_log4j-fix 169.254.1.6: CVE-2021-44228_45046_log4j-fix Patches that need to be installed: 169.254.1.1: <None> 169.254.1.2: <None> 169.254.1.3: <None> 169.254.1.4: <None> 169.254.1.5: <None> 169.254.1.6: <None> Files that need to be installed: 169.254.1.1: <None> 169.254.1.2: <None> 169.254.1.3: <None> 169.254.1.4: <None> 169.254.1.5: <None> 169.254.1.6: <None> Services that need to be restarted: 169.254.1.1: <None> 169.254.1.2: <None> 169.254.1.3: <None> 169.254.1.4: <None> 169.254.1.5: ALL 169.254.1.6: ALL admin@ecsnode1:~>Resolution:
Run the procedure again and the remaining nodes that were originally missed, get their services restarted. The original nodes where services had restarted are untouched.admin@node1:~> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch install svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online DONE Checking Installed Patches and Dependencies DONE All nodes currently do not have the same patches installed. Patches/releases currently installed: 169.254.1.1: CVE-2021-44228_45046_log4j-fix 169.254.1.2: CVE-2021-44228_45046_log4j-fix 169.254.1.3: CVE-2021-44228_45046_log4j-fix 169.254.1.4: CVE-2021-44228_45046_log4j-fix 169.254.1.5: CVE-2021-44228_45046_log4j-fix 169.254.1.6: CVE-2021-44228_45046_log4j-fix Patches that will be installed: 169.254.1.1: <None> 169.254.1.2: <None> 169.254.1.3: <None> 169.254.1.4: <None> 169.254.1.5: <None> 169.254.1.6: <None> Files that will be installed: 169.254.1.1: <None> 169.254.1.2: <None> 169.254.1.3: <None> 169.254.1.4: <None> 169.254.1.5: <None> 169.254.1.6: <None> Services that will be restarted: 169.254.1.1: <None> 169.254.1.2: <None> 169.254.1.3: <None> 169.254.1.4: <None> 169.254.1.5: ALL 169.254.1.6: ALL Patch Type: Standalone Number of nodes: 8 Number of seconds to wait between restarting node services: 450 Check DT status between node service restarts: true Do you wish to continue (y/n)?y No files to install on 169.254.1.1 Distributing patch installer to node '169.254.1.1' No files to install on 169.254.1.2 Distributing patch installer to node '169.254.1.2' No files to install on 169.254.1.3 Distributing patch installer to node '169.254.1.3' No files to install on 169.254.1.4 Distributing patch installer to node '169.254.1.4' No files to install on 169.254.1.5 Distributing patch installer to node '169.254.1.5' No files to install on 169.254.1.6 Distributing patch installer to node '169.254.1.6' No services to restart on 169.254.1.1 No services to restart on 169.254.1.2 No services to restart on 169.254.1.3 No services to restart on 169.254.1.4 Restarting services on 169.254.1.5 Restarting all services Waiting 450 seconds for services to stabilize...DONE Waiting for DTs to come online Restarting services on 169.254.1.6 Restarting all services Waiting 450 seconds for services to stabilize...DONE Waiting for DTs to come online Patching complete. admin@ecsnode1:~> -
Failed to add the host to the list of known hosts while applying the patch.
Example:
svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online FAILED ERROR: Could not execute commands on the object-main container on 169.254.x.x Output was 'Failed to add the host to the list of known hosts (/home/admin/.ssh/known_hosts). :patchtest:' Patching is unable to continue with unreachable nodes. To proceed: - Resolve problems accessing node(s) from this one. - Manually pass a MACHINES file containing the list of working nodes to patch (not recommended). - Contact your next level of support for other options or assistance.
Resolution:
The reason could be that the user of the file /home/admin/.ssh/known_hosts was root which should be admin by default.Example:
admin@node1:~> ls -l /home/admin/.ssh/known_hosts -rw------- 1 root root 1802 Jul 23 2019 /home/admin/.ssh/known_hosts admin@ecs:~>
To fix the issue from another PuTTY session, log in to the reported node or nodes and change the user to admin on the nodes where it is present as root user using below command on all the reported nodes:
Command:
# sudo chown admin:users /home/admin/.ssh/known_hosts
Example:
admin@node1:~> sudo chown admin:users /home/admin/.ssh/known_hosts
Now rerun the svc_patch command again and it should pass.
admin@node1:~> /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch install
-
Could not run commands on the object-main container on 169.254.x.x due to incorrect host key in /home/admin/.ssh/known_hosts.
Example:
svc_patch Version 2.9.1 Verifying patch bundle consistency DONE Detecting nodes in current VDC DONE Reading in patch details (1 of 2) DONE Reading in patch details (2 of 2) DONE Validating nodes are online FAILED ERROR: Could not execute commands on the object-main container on 169.254.x.x Output was '@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:RcwOsFj7zPA5p5kSeYovF4UlZTm125nLVeCL1zCqOzc. Please contact your system administrator. Add correct host key in /home/admin/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/admin/.ssh/known_hosts:14 You can use following command to remove the offending key: ssh-keygen -R 169.254.x.x -f /home/admin/.ssh/known_hosts Password authentication is disabled to avoid man-in-the-middle attacks. Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks. :patchtest:' Patching is unable to continue with unreachable nodes. To proceed: - Resolve problems accessing node(s) from this one. - Manually pass a MACHINES file containing the list of working nodes to patch (not recommended). - Contact your next level of support for other options or assistance.
Resolution:
Contact ECS support for a resolution. -
When using xDoctor version 4.8-85.0 release to apply this patch, you may get an alert outlining that the md5sum did not match for svc_base.py:
# /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/svc_patch status svc_patch Version 2.9.1 Verifying patch bundle consistency FAILED Patch bundle onsistency check failed - md5sums for one or more files in the patch bundle were invalid, or files were not found. svc_patch will attempt to validate files in the patch using MD5SUMS.bundle, which is bundled with the patch. Output from md5sum was: ./lib/libs/svc_base.py: FAILED md5sum: WARNING: 1 computed checksum did NOT match
Resolution:
Run the below commands before applying the patch to update the md5sum:# sudo sed -i '/svc_base.py/d' /opt/emc/xdoctor/patches/CVE-2021-44228_45046_log4j-fix/MD5SUMS.bundle # sudo sed -i '/MD5SUMS.bundle/d' /opt/emc/xdoctor/.xdr_chksum