ECS: Objects under S3 Object Lock retention can be deleted or overwritten.
Summary: S3 has been configured with object lock retention. Even if retention is set, the object can be deleted or be overwritten.
Symptoms
Cause
ECS is working compatible to AWS S3 documentation.
When object lock is enabled, versioning is enabled as well. The retention on S3 PUT operation is now configured for this exact version ID.
The object is now considered the "current" version.
When a "DELETE" is now sent without pointing to the version ID, a "dmarker" is being created. The object version under retention is seen as "non-current" version.
When "overwriting" the object it is the same process - the current version becomes non-current and a new version is created. The new version is under the configured retention too!
Example:
The user in this example has every permission but "s3:BypassGovernanceRetention"
See the "ECS Data Access Guide" for details on "Compliance" and "Governance" Retention Methods.
Enabling object lock on Bucket "ObjectLockBucket"
mathias@client:/usr/share/s3curl$ ./s3curl.pl --debug --id=b_mathias_iam -- http://loadbalancer.fq.dn:9020/ObjectLockBucket?enable-object-lock -X PUT mathias@client:/usr/share/s3curl$ ./s3curl.pl --debug --id=b_mathias_iam -- http://loadbalancer.fq.dn:9020/ObjectLockBucket?object-lock |xmllint --format - <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ObjectLockConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <ObjectLockEnabled>Enabled</ObjectLockEnabled> </ObjectLockConfiguration>
From the ECS we see that along "Object Lock" the "Versioning" is enabled:
admin@ecs-n2:~> svc_bucket info ObjectLockBucket
svc_bucket v1.0.39 (svc_tools v2.15.0) Started 2024-02-23 10:33:37
Bucket ID b_mathias_ns.ObjectLockBucket
Name ObjectLockBucket
Namespace b_mathias_ns
Owner User urn:ecs:iam::b_mathias_ns:root
Owner VDC Name VDC1
Owner zone/VDC ID urn:storageos:VirtualDataCenterData:c65ec181-5c44-4491-94d1-d8c75beb2963
Keypool Hash ID 03faef6729cc80adb7fbeb228a0b50536cea6791973eabc99b44f1a30a28cad3
Keypool Policy None
Bucket ACL:
Type Affects Name Access Rights
user file/dir urn:ecs:iam::b_mathias_ns:root FULL_CONTROL
Replication Group (vpool) Name RG1
Replication Group (vpool) ID urn:storageos:ReplicationGroupInfo:b94655e8-e8ad-4276-80ca-3eb095147f99:global
Bucket Creation Date 2024-02-23 10:29:38 (1708684178025)
Temp Failed (TSO) False
API Type S3
FS Access Enabled False
Encryption Enabled False
Versioning State Enabled <<<<<<<<<<<<<<<<<<<<<<<<<<<<
ADO Enabled False
ADO Read-Only False
Object Lock Enabled True <<<<<<<<<<<<<<<<<<<<<<<<<<<<
Auto commit period (seconds) None
Enforce Retention False
Lifecycle Policy None
Default FS Access permissions (for owner group):
File Dir
Read Write Exec Read Write Exec
Unk Unk Unk Unk Unk Unk
Object count 0
Total Object Size 0 GB
Setting a default policy, type governance and 5 days retention:
mathias@client:/usr/share/s3curl$ ./s3curl.pl --debug --id=b_mathias_iam -- http://loadbalancer.fq.dn:9020/ObjectLockBucket?object-lock -X PUT \
> -d "<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled>
> <Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>5</Days></DefaultRetention></Rule></ObjectLockConfiguration>"
mathias@client:/usr/share/s3curl$ ./s3curl.pl --debug --id=b_mathias_iam -- http://loadbalancer.fq.dn:9020/ObjectLockBucket?object-lock |xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ObjectLockConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<ObjectLockEnabled>Enabled</ObjectLockEnabled>
<Rule>
<DefaultRetention>
<Mode>GOVERNANCE</Mode>
<Days>5</Days>
</DefaultRetention>
</Rule>
</ObjectLockConfiguration>
Creating the "Object-under-retention" object, using the debug option in s3curl to receive the version ID.
The version ID, which is the epoch timestamp, is seen in the response:
mathias@client:/usr/share/s3curl$ ./s3curl.pl --debug --id=b_mathias_iam --put=1k -- http://loadbalancer.fq.dn:9020/ObjectLockBucket/Object-under-retention > PUT /ObjectLockBucket/Object-under-retention HTTP/1.1 > Host: loadbalancer.fq.dn:9020 > User-Agent: curl/7.81.0 > Accept: */* > Date: Fri, 23 Feb 2024 11:00:16 +0000 > Content-Length: 28 > Expect: 100-continue > * Mark bundle as not supporting multiuse < HTTP/1.1 100 Continue * We are completely uploaded and fine * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Date: Fri, 23 Feb 2024 11:00:16 GMT < Server: ViPR/1.0 < x-amz-request-id: 0a3c0814:18dcc3986d8:178c:f42b9 < x-amz-id-2: c7eb1c173a8121b873ce4973eb16c326b7b02a920ca4d002fd3715b7b53adbf5 < ETag: "9f8e337286aa2bf5147061f1a9b8b934" < Last-Modified: Fri, 23 Feb 2024 11:00:16 GMT < x-emc-mtime: 1708686016872 < x-emc-previous-object-size: 0 < x-amz-version-id: 1708686016872 <<<<<<<<<<<<<<<<<<<<<<<<<<<< Version under retention. < Content-Length: 0 < * Connection #0 to host loadbalancer.fq.dn left intact
This version is our "current" version, a head to this object displays the retention.
It also displays our version ID 1708686016872, which we put under retention.
mathias@client:/usr/share/s3curl$ ./s3curl.pl --id=b_mathias_iam --head -- http://loadbalancer.fq.dn:9020/ObjectLockBucket/Object-under-retention HTTP/1.1 200 OK Date: Fri, 23 Feb 2024 11:00:54 GMT Server: ViPR/1.0 x-amz-request-id: 0a3c0814:18dcc3986d8:1d87:6180 x-amz-id-2: c7eb1c173a8121b873ce4973eb16c326b7b02a920ca4d002fd3715b7b53adbf5 x-amz-version-id: 1708686016872 <<<<<<<<<<<<<<<<<<<<<<<<<<<< x-amz-object-lock-mode: GOVERNANCE <<<<<<<<<<<<<<<<<<<<<<<<<<<< x-amz-object-lock-retain-until-date: 2024-02-28T11:00:16.872Z <<<<<<<<<<<<<<<<<<<<<<<<<<<< ETag: "9f8e337286aa2bf5147061f1a9b8b934" Last-Modified: Fri, 23 Feb 2024 11:00:16 GMT x-emc-mtime: 1708686016872 Content-Type: application/octet-stream Content-Length: 28
We can also address the versionID for the same result:
mathias@client:/usr/share/s3curl$ ./s3curl.pl --id=b_mathias_iam --head -- http://loadbalancer.fq.dn:9020/ObjectLockBucket/Object-under-retention?VersionId=1708686016872 HTTP/1.1 200 OK Date: Fri, 23 Feb 2024 11:01:41 GMT Server: ViPR/1.0 x-amz-request-id: 0a3c0814:18dcc3986d8:ff2:25dc4f x-amz-id-2: c7eb1c173a8121b873ce4973eb16c326b7b02a920ca4d002fd3715b7b53adbf5 x-amz-version-id: 1708686016872 x-amz-object-lock-mode: GOVERNANCE x-amz-object-lock-retain-until-date: 2024-02-28T11:00:16.872Z ETag: "9f8e337286aa2bf5147061f1a9b8b934" Last-Modified: Fri, 23 Feb 2024 11:00:16 GMT x-emc-mtime: 1708686016872 Content-Type: application/octet-stream Content-Length: 28
Deleting the current results in HTTP 1/1 200 - success.
This is due to versioning, this is not a delete, but a "dmarker" is created.
The version remains under retention. The "dmarker" has version ID 170868621362.
Deleting: mathias@client:/usr/share/s3curl$ ./s3curl.pl --id=b_mathias_iam --head -- http://loadbalancer.fq.dn:9020/ObjectLockBucket/Object-under-retention 404 is reported correctly! The "current" for "Object-under-retention" no longer exists. But the VersionId is different: HTTP/1.1 404 Not Found Date: Fri, 23 Feb 2024 11:03:44 GMT Server: ViPR/1.0 x-amz-request-id: 0a3c0814:18dcc3986d8:1d94:124b x-amz-id-2: c7eb1c173a8121b873ce4973eb16c326b7b02a920ca4d002fd3715b7b53adbf5 Content-Type: application/xml Date: Fri, 23 Feb 2024 11:03:44 GMT x-amz-request-id: 0a3c0814:18dcc3986d8:1d94:124b Server: ViPR/1.0 x-amz-delete-marker: true x-amz-version-id: 1708686213620 <<<<<<<<<<<<<<<<<<<<<<<<<<<
Object version 1708686016872 still exists and is under retention:
mathias@client:/usr/share/s3curl$ ./s3curl.pl --id=b_mathias_iam --head -- http://loadbalancer.fq.dn:9020/ObjectLockBucket/Object-under-retention?versionId=1708686016872 HTTP/1.1 200 OK Date: Fri, 23 Feb 2024 11:08:20 GMT Server: ViPR/1.0 x-amz-request-id: 0a3c0814:18dcc3986d8:1d94:36e8 x-amz-id-2: c7eb1c173a8121b873ce4973eb16c326b7b02a920ca4d002fd3715b7b53adbf5 x-amz-version-id: 1708686016872 x-amz-object-lock-mode: GOVERNANCE x-amz-object-lock-retain-until-date: 2024-02-28T11:00:16.872Z ETag: "9f8e337286aa2bf5147061f1a9b8b934" Last-Modified: Fri, 23 Feb 2024 11:00:16 GMT x-emc-mtime: 1708686016872 Content-Type: application/octet-stream Content-Length: 28
This version cannot be deleted:
mathias@client:/usr/share/s3curl$ ./s3curl.pl --id=b_mathias_iam --delete -- http://loadbalancer.fq.dn:9020/ObjectLockBucket/Object-under-retention?versionId=1708686016872 <Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>ObjectLockBucket/Object-under-retention</Resource><RequestId>0a3c0814:18dcc3986d8:1c48:26d90</RequestId></Error>
Resolution
Object lock in ECS requires the version being addressed to see retention.
New versions can be created with the same name, also "dmarker" can be created.
See also the AWS S3 documentation for this behavior:
https://aws.amazon.com/s3/features/object-lock/
https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html