ECS 3.6.2 Data Access Guide

Object lifecycle management

ECS supports S3 Lifecycle Configuration on both version-enabled buckets and non-version-enabled buckets.

Where you need to modify objects and delete objects, but need to ensure that the objects are still retained for a period, you can enable versioning on a bucket and use the lifecycle capability to determine when deleted versions of objects will be removed from ECS.

Versioning and lifecycle are standard S3 features. However, lifecycle expiration is closely related to retention, which is an ECS extension. If the lifecycle expires before the retention period expires, the object will not be deleted until the retention period is over.

Lifecycle cannot be enabled on FS enabled buckets.
Lifecycle is a bucket level concept.
Maximum of 1000 lifecycle rules per bucket is applicable.
There may be a delay between the expiration date and the date at which S3 removes an object.
Always round up the resulting time to the next day midnight UTC.
For expiration, the days are calculated since the last modified date (= Creation date for the objects not yet modified/deleted).
If you delete the data accidentally, raise a Service Request (SR) with the support team. For more information about recovering the data, see KB 539120.
For noncurrentexpiration, the days are calculated since the object became noncurrent.
The date-based rules trigger action on all objects created on or before this date.

Example lifecycle configurations for ECS

The following are some lifecycle configurations examples.

Aborting old MPU's (versioning and non-versioning enabled buckets)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>lifecycle-v2-expire-non-current-and-dmarkers-and-mpu</ID>
    <Filter/>
    <Status>Enabled</Status>
    <AbortIncompleteMultipartUpload>
      <DaysAfterInitiation>1</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
  </Rule>
</LifecycleConfiguration>

Expiring objects after a certain # of days (versioning and non-versioning enabled buckets)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>lifecycle-v2-expire-one-year</ID>
    <Filter/>
    <Status>Enabled</Status>
    <Expiration>
      <Days>365</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Expiring NoncurrentVersions of objects after a certain # of days (versioning enabled buckets only)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>lifecycle-v2-expire-non-current</ID>
    <Filter/>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>1</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
</LifecycleConfiguration>

Removing expired object delete markers (versioning enabled buckets only)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>lifecycle-v2-expire-dmarkers</ID>
    <Filter/>
    <Status>Enabled</Status>
    <Expiration>
      <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Expire all non-current versions, dmarkers and incomplete MPU's after 1 day

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>lifecycle-v2-expire-non-current-and-dmarkers-and-mpu</ID>
    <Filter/>
    <Status>Enabled</Status>
    <Expiration>
      <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
    </Expiration>
    <AbortIncompleteMultipartUpload>
      <DaysAfterInitiation>1</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>1</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
</LifecycleConfiguration>

PUT/GET lifecycle with s3curl examples

The following are PUT and GET lifecycle with s3curl examples. See Using s3curl with ECS for more information.

PUT lifecycle

admin@:/usr/share/s3curl> cat lifecycle.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>lifecycle-v2-non-current-expiration</ID>
    <Filter/>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>1</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
</LifecycleConfiguration>
admin@:/usr/share/s3curl>
admin@:/usr/share/s3curl> sudo perl ./s3curl.pl --debug --id=emc --put=lifecycle.xml --calculateContentMd5 -- "http://192.0.2.0:9020/emc_lifecycle?lifecycle" -v
s3curl: Found the url: host=10.32.169.121; port=9020; uri=/emc_lifecycle; query=lifecycle;
s3curl: replaced string: lifecycle
s3curl: ordinary endpoint signing case
s3curl: StringToSign='PUT\nFjZKcAgVegBUaGdqfEh/Ig==\n\nTue, 06 Nov 2018 17:28:58 +0000\n/tom_lifecycle?lifecycle'
s3curl: exec curl -v -H 'Date: Tue, 06 Nov 2018 17:28:58 +0000' -H 'Authorization: AWS emc:xDTXdXSF+qVIQ4EreEe+iqlHRns=' -L -H 'content-type: ' -H 'Content-MD5: FjZKcAgVegBUaGdqfEh/Ig==' -T lifecycle.xml http://192.0.2.0:9020/tom_lifecycle?lifecycle -v
* Hostname was NOT found in DNS cache
*   Trying 192.0.2.0...
* Connected to 192.0.2.0 (192.0.2.0) port 9020 (#0)
> PUT /emc_lifecycle?lifecycle HTTP/1.1
> User-Agent: curl/7.37.0
> Host: 192.0.2.0:9020
> Accept: */*
> Date: Tue, 06 Nov 2018 17:28:58 +0000
> Authorization: AWS emc:xDTXdXSF+qVIQ4EreEe+iqlHRns=
> Content-MD5: FjZKcAgVegBUaGdqfEh/Ig==
> Content-Length: 376
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Tue, 06 Nov 2018 17:28:58 GMT
* Server ViPR/1.0 is not blacklisted
< Server: ViPR/1.0
< x-amz-request-id: 0a20a979:166c6842ba5:82ba:5
< x-amz-id-2: 6687ce5967202724ed9a94d44c939438d39cabae9abc5a2c48a60c2c5355f95e
< Content-Length: 0
<
* Connection #0 to host 10.32.169.121 left intact
 

Troubleshooting LDS:
Enabling debug logging for LDS
LDS log is in resourcesvc-log4j2.xml
<Logger name="com.emc.storageos.data.object.impl.resource.LifeCycleDeleteScanner" level="DEBUG"/>
 
 
Other relevant classes for troubleshooting lifecycle issues from blobsvc-log4j2.xml   
<Logger name="com.emc.storageos.data.object.impl.gc.DeleteJobScanner" level="DEBUG"/>
<Logger name="com.emc.storageos.data.object.impl.file.directoryTable.ObjectDirectoryOperation" level="DEBUG"/>
<Logger name="com.emc.storageos.data.object.impl.file.directoryTable.BlobsvcOperationBase" level="DEBUG"/>
<Logger name="com.emc.storageos.data.object.impl.file.ObjectExpirationHelper" level="DEBUG"/>
         
dataheadsvc-log4j2.xml
<Logger name="com.emc.storageos.data.object.RESTAccess.ObjectListingHelper" level="DEBUG"/>

GET lifecycle

:/usr/share/s3curl # perl ./s3curl.pl --id=EMC -- "http://192.0.2.0:9020/test-bucket/?lifecycle" -s | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>lifecycle-v2-abortmpu-one-week</ID>
    <Filter/>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>1</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
</LifecycleConfiguration>
:/usr/share/s3curl #

Supported lifecycle configuration elements

Table 1. Supported lifecycle configuration elementsThis table describes the supported lifecycle configuration elements
Name	Description	Required
AbortIncompleteMultipartUpload	Container for specifying when an incomplete multipart upload becomes eligible for an abort operation. When you specify this lifecycle action, the rule cannot specify a tag-based filter. Child: DaysAfterInitiation Type: Container Ancestor: Rule	Yes, if no other action is specified for the rule.
And	Container for specify rule filters. These filters determine the subset of objects to which the rule applies. Type: String Ancestor: Rule	Yes, if you specify more than one filter condition (for example, one prefix and one or more tags).
Date	Date when you want S3 to take the action. The date value must conform to the ISO 8601 format. The time is always midnight UTC. Type: String Ancestor: Expiration or Transition	Yes, if Days and ExpiredObjectDeleteMarker are absent.
Days	Specifies the number of days after object creation when the specific rule action takes effect. Type: Nonnegative Integer when used with Transition, Positive Integer when used with Expiration. Ancestor: Expiration, Transition	Yes, if Date and ExpiredObjectDeleteMarker are absent.
DaysAfterInitiation	Specifies the number of days after initiating a multipart upload when the multipart upload must be completed. If it does not complete by the specified number of days, it becomes eligible for an abort operation and S3 aborts the incomplete multipart upload. Type: Positive Integer. Ancestor: AbortIncompleteMultipartUpload	Yes, if ancestor is specified.
Expiration	This action specifies a period in an object's lifetime when S3 should take the appropriate expiration action. The action S3 takes depends on whether the bucket is versioning-enabled. If versioning has never been enabled on the bucket, S3 deletes the only copy of the object permanently. Otherwise, if your bucket is versioning-enabled (or versioning is suspended), the action applies only to the current version of the object. A versioning-enabled bucket can have many versions of the same object, one current version, and zero or more noncurrent versions. Instead of deleting the current version, S3 makes it a noncurrent version by adding a delete marker as the new current version. NOTE: If your bucket state is versioning-suspended, S3 creates a delete marker with version ID null. If you have a version with version ID null, then S3 overwrites that version. To set expiration for noncurrent objects, you must use the NoncurrentVersionExpiration action. Type: Container Children: Days or Date Ancestor: Rule	Yes, if no other action is present in the Rule.
Filter	Container for elements that describe the filter identifying a subset of objects to which the lifecycle rule applies. If you specify an empty filter (<Filter></Filter>), the rule applies to all objects in the bucket. Type: String Children: Prefix, Tag Ancestor: Rule	Yes
ID	Unique identifier for the rule. The value cannot be longer than 255 characters. Type: String Ancestor: Rule	No
Key	Specifies the key of a tag. A tag key can be up to 128 Unicode characters in length. Tag keys that you specify in a lifecycle rule filter must be unique. Type: String Ancestor: Tag	Yes, if <Tag> parent is specified.
LifecycleConfiguration	Container for lifecycle rules. You can add as many as 1,000 rules. Type: Container Children: Rule Ancestor: None	Yes
ExpiredObjectDeleteMarker	On a versioned bucket (versioning-enabled or versioning-suspended bucket), you can add this element in the lifecycle configuration to direct S3 to delete expired object delete markers. On a nonversioned bucket, adding this element in a policy is meaningless because you cannot have delete markers and the element does not do anything. When you specify this lifecycle action, the rule cannot specify a tag-based filter. Type: String Valid values: true \| false (the value false is allowed, but it is no-op and S3 does not take action if the value is false) Ancestor: Expiration	Yes, if Date and Days are absent.
NoncurrentDays	Specifies the number of days an object is noncurrent before S3 can perform the associated action. Type: Nonnegative Integer when used with NoncurrentVersionTransition, Positive Integer when used with NoncurrentVersionExpiration. Ancestor: NoncurrentVersionExpiration or NoncurrentVersionTransition	Yes
NoncurrentVersionExpiration	Specifies when noncurrent object versions expire. Upon expiration, S3 permanently deletes the noncurrent object versions. You set this lifecycle configuration action on a bucket that has versioning enabled (or suspended) to request that S3 delete noncurrent object versions at a specific period in the object's lifetime. Type: Container Children: NoncurrentDays Ancestor: Rule	Yes, if no other action is present in the Rule.
Prefix	Object key prefix identifying one or more objects to which the rule applies. Empty prefix (<Prefix></Prefix>) indicates there is no filter based on key prefix. NOTE: ECS supports <Prefix> with and without <Filter>. PUT Bucket lifecycle with <Filter> <Filter> <Prefix>value</Prefix> </Filter> PUT Bucket lifecycle (Deprecated) without <Filter> <Prefix>value</Prefix> There can be at most one Prefix in a lifecycle rule Filter. Type: String Ancestor: Filter or And (if you specify multiple filters such as a prefix and one or more tags)	No
Rule	Container for a lifecycle rule. A lifecycle configuration can contain as many as 1,000 rules. Type: Container Ancestor: LifecycleConfiguration	Yes
Status	If Enabled, S3 executes the rule as scheduled. If Disabled, S3 ignores the rule. Type: String Ancestor: Rule Valid values: Enabled, Disabled.	Yes
Value	Specifies the value for a tag key. Each object tag is a key-value pair. Tag value can be up to 256 Unicode characters in length. Type: String Ancestor: Tag	Yes, if <Tag> parent is specified.

Enabling Lifecycle Delete Scanner (LDS)

The purpose of the LDS scanner is to initiate expiration of objects/versions created before the lifecycle is applied. So for instance, if there is a bucket created sometime ago and has been in use and now there is a requirement to apply lifecycle, in such cases LDS must be enabled for lifecycle policies to cover previous objects/versions.

NOTE: LDS is disabled by default. For enabling pre 3.2.1, contact .

For enabling 3.2.1 and higher versions, set the com.emc.ecs.resource.lifecycledeletescanner.enable parameter value as true.

svc_param set com.emc.ecs.resource.lifecycledeletescanner.enable -v "true" -r "Enable LDS"

Welcome

Welcome to Dell