ECS: How to Prevent Host Header Injection on ECS 3.8.0 and later to Avoid Management API 403 Errors and Inaccessible Grafana UI

Summary: Starting with ECS 3.8.0.x, there is a security issue addressed regarding host header injection (See Release notes for STORAGE-27164).

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

What is Host Header Injection?

  • The application is trusting the HTTP Host Header supplied by the user and using the user-supplied host header to redirect to another page. 
  • An attacker could take advantage of it by changing the HTTP Host header to a malicious hostname and redirecting the user to a malicious web page. 
Note: This only impacts Management API calls through a proxy or a load balancer. It does not impact data operations.


What changes are being implemented starting ECS 3.8.0.x?

As part of the fix, a new set of APIs is implemented that allows setting or unsetting the list of allowed server names. Host header value of incoming requests is checked against that list. The request is rejected if the host header value is not present in that list.

Below is the new set of REST APIs introduced for registering and managing the list of accepted servers.
 
Resource Name Description
GET /acceptedservernames/ Gets the list of accepted management server names
POST /acceptedservernames/ Creates or replaces the list of accepted management server names
PUT /acceptedservernames/ Update the list of accepted management server names.
DELETE /acceptedservernames/ Deletes the list of accepted management server names

 

In what circumstances are you affected?

Use-cases to implement the solution of configuring the list of accepted servers on ECS:

Upgrading From Upgrading To or Installing On Remediation
ECS 3.7 and earlier ECS 3.8.0.x
  • Implementing the solution is mandatory if you are using a proxy or a load balancer to send Management API calls to ECS.
  • Not implementing the solution means you receive Management API 403 errors and the Grafana UI is inaccessible.
ECS 3.7 and earlier ECS 3.8.1.x or later
  • Implementing the solution is optional if you are using a proxy or a load balancer to send Management API calls to ECS.
  • Not implementing the solution means you remain vulnerable to this vulnerability, and ECS accepts all Management API calls. 
ECS 3.8.0.x ECS 3.8.1.x or later
  • If you have already implemented the solution on ECS 3.8.0, then the solution remains in place on ECS 3.8.1 or newer.
  • You have the option to revert the solution to allow all IPs to make Management API calls.
  • If you choose to revert the solution, you remain vulnerable to this vulnerability, and ECS accepts all Management API calls. 



What indicates you have encountered a problem?

  • All the management login APIs will fail with 403-forbidden after new install or upgrade to 3.8.0.x from earlier 3.8 versions.
  • The Grafana dashboard may also be inaccessible. 


What errors do the logs show on ECS?

"GET /login HTTP/1.1" 403 error is seen in /opt/storageos/logs/nginx/access.log and /opt/storageos/logs/objcontrolsvc.log on nodes API calls are made against.

 

Resolution Summary (After upgrade to 3.8.0.x or later)

  1. Upgrade your ECS xDoctor software to latest version.
  2. Add the list of external IPs.
  3. There is no requirement to register external servers making Management API calls directly to the ECS nodes directly, or using the ECS nodes' IPs, as the connection would be successful already. The below procedure is only required if the connection is passed through a proxy server or a Load Balancer. Therefore all proxy and or Load Balancers must be added to the Management Server Names Accept List.


1-xDoctor Upgrade

  1. Upgrade your ECS xDoctor software to latest Version available.
  1. Check the xDoctor version running on your system. If the version is 4.8-87.0 or later, move to step "Add list of external IPs." If not, proceed with the steps below.
Command:  
# sudo xdoctor --version
Example: 
admin@node1:~> sudo xdoctor --version
4.8-87.0
  1. Sign to the Dell Support Site, connect directly to this download ECS Download location, search for xDoctor using the keyword search bar, and click xDoctor 4.8-87.0 RPM link to download. To view the Release notes, connect directly to the ECS Product Page, select Manuals and Documents from the sidebar from where they should be available for download.
  2. Once the RPM is downloaded, use any remote SCP program to upload the file to the /home/admin directory on the first ECS node.
  3. Once the upload is complete, SSH to the first node of the ECS system using admin.
  4. Upgrade xDoctor on all the nodes with the newly distributed version.  
Command: 
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-87.0.noarch.rpm
Example: 
admin@ecs-n1:~> sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-87.0.noarch.rpm
2022-07-04 07:41:49,209: xDoctor_4.8-83.0 - INFO    : xDoctor Upgrader Instance (1:SFTP_ONLY)
2022-07-04 07:41:49,210: xDoctor_4.8-83.0 - INFO    : Local Upgrade (/home/admin/xDoctor4ECS-4.8-84.0.noarch.rpm)
2022-07-04 07:41:49,226: xDoctor_4.8-83.0 - INFO    : Current Installed xDoctor version is 4.8-86.0
2022-07-04 07:41:49,242: xDoctor_4.8-86.0 - INFO    : Requested package version is 4.8-84.0
2022-07-04 07:41:49,242: xDoctor_4.8-86.0 - INFO    : Updating xDoctor RPM Package (RPM)
2022-07-04 07:41:49,293: xDoctor_4.8-86.0 - INFO    :  - Distribute package
2022-07-04 07:41:50,759: xDoctor_4.8-86.0 - INFO    :  - Install new rpm package
2022-07-04 07:42:04,401: xDoctor_4.8-86.0 - INFO    : xDoctor successfully updated to version 4.8-87.0
  1. If the environment is a multi-rack VDC, the new xDoctor package must be installed on the first node of each rack. To identify these rack primaries, run the below command. In this instance, there are four racks and four rack primaries highlighted
  1. Find the rack primary nodes (highlighted in bold below).
Command: 
# svc_exec -m "ip address show private.4 |grep -w inet"
Example: 
admin@ecsnode1~> svc_exec -m "ip address show private.4 |grep -w inet"
svc_exec v1.0.2 (svc_tools v2.1.0)                 Started 2021-12-20 14:03:33
 
Output from node: r1n1                                retval: 0
    inet 169.254.1.1/16 brd 169.254.255.255 scope global private.4
 
Output from node: r2n1                                retval: 0
    inet 169.254.2.1/16 brd 169.254.255.255 scope global private.4
 
Output from node: r3n1                                retval: 0
    inet 169.254.3.1/16 brd 169.254.255.255 scope global private.4
 
Output from node: r4n1                                retval: 0
    inet 169.254.4.1/16 brd 169.254.255.255 scope global private.4
  1. Copy the package from the first node of the system (R1N1) to the other rack primaries per below:
Example: 
admin@ecs-n1: scp xDoctor4ECS-4.8-87.0.noarch.rpm 169.254.2.1:/home/admin/
xDoctor4ECS-4.8-84.0.noarch.rpm                                                                                                                        100%   32MB  31.9MB/s   00:00
admin@ecsnode1~> scp xDoctor4ECS-4.8-87.0.noarch.rpm 169.254.3.1:/home/admin/
xDoctor4ECS-4.8-84.0.noarch.rpm                                                                                                                        100%   32MB  31.9MB/s   00:00
admin@ecsnode1~> scp xDoctor4ECS-4.8-87.0.noarch.rpm 169.254.4.1:/home/admin/
xDoctor4ECS-4.8-87.0.noarch.rpm                                                                                                                        100%   32MB  31.9MB/s   00:00
admin@ecsnode1~>
  1. Per step 1 above, run the same xDoctor install command on each of the above rack primaries identified previously. 
Command: 
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-87.0.noarch.rpm


2-A Add list of external IPs.

  • A Management user must perform this activity with a Security Administrator role. In the example below we use the root user as that also has a Security Administration capability.
  • It can be done on any node on ECS VDC.
  • In a federation with multiple VDCs, it must be performed after all VDCs in the federation have been upgraded. The procedure can then be applied to each VDC.
  • NOTE: The allowlist is replicated across federated VDCs, so adding allowlist entries on one cluster replicates those entries to the other cluster and overwrites anything already added on the other cluster
  • The tool throws a "401 - Unauthorized" Exception if the user is not Security Administrator, or the wrong credential is used.
  1. Prepare the list of server names (IP, Hostname, and FQDN) to be added into trust list. Consolidate the list into a file, each line with one server name. If the management network is separated, then FQDN of management nodes should also be added into the trust list. Many customers may have to add the names of the nodes; both short using svc_exec "hostname -s" and long using svc_exec hostname
To check if network separation for management, run the below and search for "mgmt" highlighted.

Command: 
# getrackinfo -n
Example: 
#getrackinfo -n
Named networks
==============
Node ID       Network          Ip Address        Netmask            Gateway            VLAN               Interface
1            mgmt              1xx.xxx.xx.xxx    255.25x.xxx.x      19x.xxx.xx.x       xxx                public
1            mgmt              1xx.xxx.xx.xxx    255.25x.xxx.x      19x.xxx.xx.x       xxx                public
1            mgmt              1xx.xxx.xx.xxx    255.25x.xxx.x      19x.xxx.xx.x       xxx                public
1            mgmt              1xx.xxx.xx.xxx    255.25x.xxx.x      19x.xxx.xx.x       xxx                public
1            mgmt              1xx.xxx.xx.xxx    255.25x.xxx.x      19x.xxx.xx.x       xxx                public
1            mgmt              1xx.xxx.xx.xxx    255.25x.xxx.x      19x.xxx.xx.x       xxx                public
Example below with server names list (replace with the real ones). 
# cat trust.server.list
ecs1-n1
ecs1-n1.dell.com
ecs1-n2
ecs1-n2.dell.com
ecs1-n3
ecs1-n3.dell.com
ecs1-n4
ecs1-n4.dell.com 
ecs1-n5
ecs1-n5.dell.com 
Load_balancer1_IP
Load_balancer1_Hostname
Load_balancer1_FQDN
Load_balancer2_IP
Load_balancer2_Hostname
Load_balancer2_FQDN
  1. Check the logs to verify the server name for any 403 errors:  

            Command: 

svc_log -f 'ERROR  HostHeaderAcceptListFilter.java' -sr objcontrolsvc -start '2 hours ago'

            Example:

# svc_log -f 'ERROR  HostHeaderAcceptListFilter.java' -sr objcontrolsvc -start '2 hours ago'
svc_log v1.0.31 (svc_tools v2.13.1)                 Started 2023-10-10 18:00:11

Running on nodes:              <All nodes>
Time range:                    2023-10-13 17:00:00 - 2023-10-13 18:11:47
Filter string(s):              'ERROR  HostHeaderAcceptListFilter.java'
Show filename(s):              True
Show nodename(s):              True
Search reclaim logs (if any):  False

169.254.1.1 objcontrolsvc.log 2023-10-10T17:00:07,188 [qtp47151359-4412] ERROR  HostHeaderAcceptListFilter.java (line 81) The host header server name didn't match the accepted list ecsdev1-m1
169.254.1.1 objcontrolsvc.log 2023-10-10T17:00:38,609 [qtp47151359-4386] ERROR  HostHeaderAcceptListFilter.java (line 81) The host header server name didn't match the accepted list ecsdev1-m1
169.254.1.1 objcontrolsvc.log 2023-10-10T17:01:07,499 [qtp47151359-4310] ERROR  HostHeaderAcceptListFilter.java (line 81) The host header server name didn't match the accepted list ecsdev1-m1
169.254.1.2 objcontrolsvc.log 2023-10-10T17:05:01,743 [qtp1754883736-3393] ERROR  HostHeaderAcceptListFilter.java (line 81) The host header server name didn't match the accepted list ecsdev1-m2
169.254.1.2 objcontrolsvc.log 2023-10-10T17:05:01,756 [qtp1754883736-3397] ERROR  HostHeaderAcceptListFilter.java (line 81) The host header server name didn't match the accepted list ecsdev1-m2
169.254.1.3 objcontrolsvc.log 2023-10-10T17:30:01,533 [qtp1289718177-2883] ERROR  HostHeaderAcceptListFilter.java (line 81) The host header server name didn't match the accepted list ecsdev1-m3
169.254.1.3 objcontrolsvc.log 2023-10-10T17:30:01,546 [qtp1289718177-2944] ERROR  HostHeaderAcceptListFilter.java (line 81) The host header server name didn't match the accepted list ecsdev1-m3
Example below with server names list (replace with the real ones). 
# cat trust.server.list
ecs1-n1
ecs1-n1.dell.com
ecs1-n2
ecs1-n2.dell.com
ecs1-n3
ecs1-n3.dell.com
ecs1-n4
ecs1-n4.dell.com 
ecs1-n5
ecs1-n5.dell.com
ecsdev1-m1
ecsdev1-m2
ecsdev1-m3
Load_balancer1_IP
Load_balancer1_Hostname
Load_balancer1_FQDN
Load_balancer2_IP
Load_balancer2_Hostname
Load_balancer2_FQDN

Add trusted servers with the provided list supplying the <mgmt_user>. User is prompted to input the password for the mgmt_user, enter the password and then press enter to continue.

Command:  
svc_param asn add -u <mgmt_user> -p -f <server.list>
Example: 
# svc_param asn add -u root -p -f trust.server.list
Password:
svc_param v1.1.3 (svc_tools v2.7.0)                 Started 2022-11-08 19:27:16
 
add trusted servers successfully!
  1. Retrieve the trusted server list. The list can either be a) printed to a screen for quick check if it is short, or b) saved to a file for backup or inspection.
  1. Print the result to the screen. You are prompted to input the password, press enter to continue.
Command: 
 svc_param asn list -u <mgmt_user> -p
Example: 
# svc_param asn list -u root -p
Password:
svc_param v1.1.3 (svc_tools v2.7.0)                 Started 2023-10-10 19:32:58
 
Found 19 trusted servers:
 
        Load_balancer1_FQDN
        Load_balancer1_Hostname
        Load_balancer1_IP
        Load_balancer2_FQDN
        Load_balancer2_Hostname
        Load_balancer2_IP
        ecs-n1
        ecs-n1.nas2008.com
        ecs-n2
        ecs-n2.nas2008.com
        ecs-n3
        ecs-n3.nas2008.com
        ecs-n4
        ecs-n4.nas2008.com
        ecs-n5
        ecs-n5.nas2008.com
        ecsdev1-m1
        ecsdev1-m2
        ecsdev1-m3
  1. Save the result to a file. You are prompted to input the password, press enter to continue.
Command:  
svc_param asn list -u <mgmt_user> -p -o <output file>
Example: 
# svc_param asn list -u root -p -o server.list.backup
Password:
svc_param v1.1.3 (svc_tools v2.7.0)                 Started 2022-10-10 19:38:05
 
Found 19 trusted servers:
 
Please find the result in /home/admin/server.list.backup
 
# cat /home/admin/server.list.backup
Load_balancer1_FQDN
Load_balancer1_Hostname
Load_balancer1_IP
Load_balancer2_FQDN
Load_balancer2_Hostname
Load_balancer2_IP
ecs-n1
ecs-n1.nas2008.com
ecs-n2
ecs-n2.nas2008.com
ecs-n3
ecs-n3.nas2008.com
ecs-n4
ecs-n4.nas2008.com
ecs-n5
ecs-n5.nas2008.com
ecsdev1-m1
ecsdev1-m2
ecsdev1-m3

Note: In the Case where a Load Balancer is NOT used and still getting this same nginx 403 Forbidden error in the ECS user interface, check if the customer is using VLAN Network. Separation using the getrackinfo -n command per step 1. If so then add the management DNS names and management IP addresses from the customer network into the Trust Server List for all their nodes.

(Such as below)

Example:

# cat trust.server.list
10.x.x.1
10.x.x.2
10.x.x.3
10.x.x.4
10.x.x.5
10.x.x.6
test-n1.domain.example
test-n2.domain.example
test-n3.domain.example
test-n4.domain.example
test-n5.domain.example
test-n5.domain.example



2-B Remove list of external IPs.

If you must remove entries from a trusted list, you are prompted to input the password, and press enter to continue. A server.to.remove.list should be created.

Example:  
# cat server.list.to.remove
Load_balancer1_Hostname
Load_balancer2_IP
Command to remove: 
 svc_param asn remove -u <mgmt_user> -p -f <server.to.remove.list>
(You are prompted to input the password, press enter to continue.)

Example: 
# svc_param asn remove -u root -p -f server.list.to.remove
Password:
svc_param v1.1.3 (svc_tools v2.7.0)                 Started 2022-11-08 19:42:00
 
remove trusted servers successfully!

List the entries again to confirm that entries are removed. 
Command:  
svc_param asn list -u root -p
(You are prompted to input the password, press enter to continue.)

Example: 
# svc_param asn list -u root -p
Password:
svc_param v1.1.3 (svc_tools v2.7.0)                 Started 2022-11-08 19:43:19
 
Found 4 trusted servers:
 
        Load_balancer1_FQDN
        Load_balancer1_IP
        Load_balancer2_FQDN
        Load_balancer2_Hostname

Additional Information

For more details and context regarding this change, see the following documents available on Dell Support:

  • ECS 3.8.0 or ECS 3.8.1 Release Notes
  • ECS 3.8.0 or ECS 3.8.1 Administration Guide
  • ECS 3.8.0 or ECS 3.8.1 Security Guide

Affected Products

ECS, ECS Appliance, Elastic Cloud Storage
Article Properties
Article Number: 000205031
Article Type: How To
Last Modified: 14 Nov 2024
Version:  30
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.