ECS: Object Lifecycle policy is not expiring versions

Note: The below issue does not apply to Suspended or versioning disabled buckets.

1. In a version-enabled bucket, a lifecycle policy is set as in the example below to expire Objects. The Policy was configured to automatically expire and delete objects after 1 day.

<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.companyaws.com/doc/2006-03-01/">
<Rule>
<ID>Expire Objects</ID>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>1</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>

2. For objects that never had versions created, the object expiry policy above results in a delete marker inserted to to replace the current object. And thus the current object becoming a non-current version instead of getting deleted

Example:
Before the policy is met,  there one object and is latest

[VERSION_ID]: 1558539564138 [IS_LATEST]: true  [MTIME]: 2019-05-22T15:39:24.138Z [SIZE]: 54270        [OBJECT]: sample.object

After the policy matches (after 1 day) , the expiry happens but the object turns into a noncurrent version and a dmarker is inserted as the latest object. 
 

[VERSION_ID]: 1558539371262 [IS_LATEST]: false [MTIME]: 2019-05-22T15:36:11.262Z [SIZE]: 54270        [OBJECT]: sample.object        >>> original object becomes non current version
[VERSION_ID]: 1558539564138 [IS_LATEST]: true  [MTIME]: 2019-05-22T15:39:24.138Z [SIZE]: DMARKER      [OBJECT]: sample.object  >>> DMarker inserted as latest version.

Users expecting from this policy to see the object get fully deleted. 
 

The expectation from the policy example mentioned in the previous section is wrong for versioning buckets. It is however true for non-versioning buckets, as in non-versioning buckets the object expires right away and get deleted becoming unrecoverable.

In contrast, with Versioning buckets the S3 protocols handles objects expiry differently, it instead moves the object to a non-current version. And adds a delete marker as place holder. With versioning buckets the object version can be recovered, which allows a grace period recovery depending on the policy settings. This is an expected behavior. 

For further details on lifecycle policies , versioning and expiration in S3 please see reference in the "additional info" section at the bottom of this article.

In a version enabled bucket, it is better to use the below lifecycle policy example. Which instead expires noncurrent versions and also delete markers (which are a result of a direct delete to an object in a versioning bucket). 

Example:

In this example, S3 removes current versions (objects) 60 days after they are created by adding a delete marker for each of the current object versions. This process makes the current version noncurrent, and the delete marker becomes the current version.  

The NoncurrentVersionExpiration action in the same S3 Lifecycle configuration removes noncurrent objects 30 days after they become noncurrent. Thus, in this example, all object versions are permanently removed 90 days after object creation. 

IMPORTANT NOTE : Lifecycle policies should handled be with care, specify the number of days and actions that matches your application needs. The policy below is only an example, it expires both dmarkers and non-current versions which make deletions unrecoverable. 

<LifecycleConfiguration>
    <Rule>
        ...
        <Expiration>
           <Days>60</Days>
        </Expiration>
        <NoncurrentVersionExpiration>     
            <NoncurrentDays>30</NoncurrentDays>    
        </NoncurrentVersionExpiration>
    </Rule>
</LifecycleConfiguration>

Further details on policy expirations behavior in versioned and non-versioned buckets can be found here ; 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html#intro-lifecycle-rules-actions
https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html#lifecycle-config-conceptual-ex7

Further explanations can be also found in the ECS data-access guide with details of how to set lifecycle policy 

https://dl.dell.com/content/manual53094723-ecs-3-8-x-data-access-guide.pdf