ECS: Oracle RMAN application encountering HTTP 501 error Not Implemented after upgrade to ECS 3.8.0.1 - 3.8.0.4

Summary: Due to an ECS regression, Oracle Recovery Manager (RMAN) backup application may encounterHTTP error 501 "Not Implemented" after upgrading to ECS 3.8.0.1 - 3.8.0.4.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

After upgrading to ECS 3.8.0.1 - 3.8.0.4, application requests may fail with HTTP error "501 Not Implemented" if all parameters used in the request are not on an ECS internal supported list.
  • Oracle Recovery Manager (RMAN) Backup App has been confirmed to be impacted by this issue. 
  • The Issue impacts Oracle RMAN if writing directly to ECS. If Oracle RMAN writes to ECS through another application, then it would not be impacted. 
  • Other applications may be impacted if they use a similar configuration, where all parameters in the request are not part of the supported list. 
 
Error seen on RMAN: 
RMAN-03009: failure of backup command on ORA_SBT_TAPE_1 channel at 11/20/2023 15:15:13 ORA-19506: failed to create sequential file,
name="DB_FIP_20231120_mj2bv7g0_503507_1_1", parms="" ORA-27028: skgfqcre: sbtbackup returned error ORA-19511: non RMAN, but media manager or vendor specific failure, 
error text: KBHS-01404: See trace file /soft/oracle19/diag/rdbms/fip/fip/trace/sbtio_25047_140117933847552.log for details KBHS-00700: HTTP response error 'Not Implemented'

Verify problem on ECS:

Command: 
svc_log -f "does not support operation with signature B" -start '<Timeline errors received in ECS> -sr dataheadsvc -n all -sf

Example: 
# svc_log -f "does not support operation with signature B" -start '1 hours ago' -sr dataheadsvc -n all -sf
svc_log v1.0.31 (svc_tools v2.14.0)                 Started 2023-11-22 15:34:17

Running on nodes:              <All nodes>
Time range:                    2023-11-21 03:34:17 - 2023-11-22 15:34:17
Filter string(s):              'does not support operation with signature B'
Show filename(s):              True
Show nodename(s):              True
Search reclaim logs (if any):  False

169.254.1.5 dataheadsvc.log.20231121-134402.gz 2023-11-21T12:01:51,006 [qtp379782668-6225-88a5e8cf:18bedd2e9f0:d5e:4-s3-136.165.238.100] 
ERROR  S3Service.java (line 638) does not support operation with signature B?not-implemented, request Id 88a5e8cf:18bedd2e9f0:d5e:4

Taking the requestID as example, further trace the error as below:

Command: 
# svc_log -f "request ID highlighted from previous step" -sr dataheadsvc dataheadsvc-access -start <Timeline of requestID> -sf
Example: 
#svc_log -f "88a5e8cf:18bedd2e9f0:d5e:4" -sr dataheadsvc dataheadsvc-access -start 1h -sf
svc_log v1.0.31 (svc_tools v2.14.0)                 Started 2023-11-22 15:45:40

Running on nodes:              <All nodes>
Time range:                    2023-11-21 03:45:40 - 2023-11-22 15:45:40
Filter string(s):              '88a5e8cf:18bedd2e9f0:d5e:4'
Show filename(s):              True
Show nodename(s):              True
Search reclaim logs (if any):  False

<output truncated>
.
.
169.254.1.5 dataheadsvc.log.20231121-134402.gz 2023-11-21T12:01:51,006 [qtp379782668-6225-88a5e8cf:18bedd2e9f0:d5e:4-s3-136.165.238.100] 
ERROR  S3Service.java (line 638) does not support operation with signature B?not-implemented, request Id 88a5e8cf:18bedd2e9f0:d5e:4

169.254.1.5 dataheadsvc-access.log 2023-11-21 12:01:51,007 88a5e8cf:18bedd2e9f0:d5e:4 136.165.232.207:9020 136.165.238.100:4249 dba-rman-prod - HEAD ul-rman-prod ul-rman-prod-01 
ul-rman-prod-01 format=xml&x-BlockSize=262144&x-CopyNumber=0&x-Dbid=808640723&x-Dbname=RCAT&x-FileName=DB_RCAT_20231121_1i2c0uuu_63538_1_1&x-Incarnation=fpC63dLILKPb&x-NetTest=FALSE&x-Node=node1.hostname&x-OpenTime=2023-11-21%2012%3A01%3A51
&x-ReqCnt=34&x-ReqTime=2023-11-21%2012%3A01%3A51&x-SbtApi=sbtbackup&x-SbtOp=BeginUpload&x-SbtVersion=19.0.0.1&x-SessionId=0AA9580D05342089E063F7EDA5880453&x-System=Linux%20x86%2064-bit&x-
SystemId=13&x-User=oracle HTTP/1.1 501 2 - - 1 - - - - 'x-amz-date: 20231121T120151Z' 'Connection: close' 'X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
.
.
<output truncated>

Cause

This issue is due to a regression introduced in ECS STORAGE-28517.
  1. If the application is not using any parameters, it does not encounter the HTTP 501 error. 
  2. If the application is using parameters:
    1. If you are using at least one parameter from the supported parameter list (below), then no HTTP 501 errors are reported. 
    2. If all the parameters used in the query are not on the supported parameter list, then the HTTP 501 error is reported. 
Below is the supported parameter list. If your application is not using any of these parameters as part of the query to ECS, then a 501 error is reported.
 
"acl",
"lifecycle",
"object-lock",
"enable-object-lock",
"location",
"logging",
"notification",
"policy",
"requestPayment",
"torrent",
"delete",
"uploadId",
"uploads",
"versioning",
"versions",
"website",
"tagging",
"cors",
"accessmode",
"fileaccess",
"ping",
"cors",
"endpoint",
"isstaleallowed",
"query",
"replication",
"replicationInfo",
"legal-hold",
"retention",
"retentionUpdate",
"searchmetadata",
"snapshot",
"snapshots",
"snapshotId",
"empty-bucket-status",
"select",
"detailedReplicationStatus"

Resolution

The issue is fixed starting ECS 3.8.0.5.
  • If impacted by this issue after upgrading to ECS 3.8.0.1 - 3.8.0.4, then open a case with Technical Support to get the fix.
  • If using Oracle RMAN and plan to upgrade to ECS 3.8.0.1 - 3.8.0.4, inform Services while planning for the upgrade so that they request a special patch to use for the upgrade. This special patch would have the fix for this issue. 
    NOTE: As the issue is fixed in newer code, it is recommended to upgrade to a version with the fix. 

Affected Products

ECS, ECS Appliance, ECS Appliance Gen 1, ECS Appliance Gen 2, ECS Appliance Gen 3, ECS Appliance Hardware Series, ECS Appliance Software with Encryption, ECS Appliance Software without Encryption, ECS Software, Elastic Cloud Storage
Article Properties
Article Number: 000219799
Article Type: Solution
Last Modified: 11 Mar 2024
Version:  8
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.