ECS: How to enforce version limit for versioning-enabled buckets objects
Summary: This knowledge article is illustrating how to change and enforce the versions limit for versioning-enabled buckets objects.
Instructions
Important notes:
The following should be noted before making any changes:
- The feature is available on ECS 3.7 onwards.
- By default the versions limit for versioning-enabled buckets is 50,000.
- Versioning limit enforcement is enabled by default for new ECS installs on ECS 3.7 or higher.
- The versioning limit is not enforced (disabled by default) for ECS systems upgraded to 3.7 or higher. Unless customers choose to enable them manually.
- It is not recommended to increase the versions limit above 50,000, we only recommend reducing this limit to a recommended of 10K which should be typically within normal app requirements.
- Excessive versioning can lead to high blobsvc memory and may cause significant impacting events to production. As a result we do not recommend increasing the limit any higher than 50,000.
If in doubt and require any assistance or questions answered open a case with ECS Customer Services.
1. How to change the versions limit for versioning-enabled buckets:
-----------------------------------------------------------------------------------------
| WARNING |
Decreasing the limit when it is not yet enforced is harmless. Decreasing the limit when the limit is already enforced risks affect apps updating objects that were near the limit already. Hence, it is best to reduce the limit only when the limit is not enforced. The system alerts in the UI whenever any object was active and has reached the WARNING or ERROR levels, 50%, and 80% respectively. |
Change the below parameter to change the limit:
com.emc.ecs.blobsvc.object.version.limit.create_forbidden_threshold
Verify the parameter value which is set to a default of 50,000:
admin@ecsnode1:~> svc_param info -n com.emc.ecs.blobsvc.object.version.limit.create_forbidden_threshold svc_param v1.2.4 (svc_tools v2.17.0) Started 2024-05-26 09:22:03 Param: com.emc.ecs.blobsvc.object.version.limit.create_forbidden_threshold Description Maximum number of versions for an object Default value (active) 50000 Configured Value < Never modified > Modification reason None Last modified None
Change the parameter to a limit of 10,000 for example:
admin@ecsnode1:~> svc_param set -n com.emc.ecs.blobsvc.object.version.limit.create_forbidden_threshold -v 10000 -r "Changing the versions limit for versioning enabled buckets" svc_param v1.2.4 (svc_tools v2.17.0) Started 2024-05-26 09:35:13 Param value changed. Param: com.emc.ecs.blobsvc.object.version.limit.create_forbidden_threshold Description Maximum number of versions for an object Default value 50000 Configured Value (active) 10000 Modification reason Changing the versions limit for versioning enabled buckets Last modified 2024-05-26 09:35:13 UTC (1716716113626)
The above changes the notification limit only, if we must block the write access we must follow step 2.
2. How to block the write access once the limit is reached:
------------------------------------------------------------------------------
Change the parameter to true to enforce the versioning limit, client accessing ECS receives 403 errors when they try to create more versions for objects that already reached the threshold.
| WARNING |
Enforcing the limit prevents clients from creating more versions, only for objects that already breach the set limit. Those clients encounter 403 errors until their versions are deleted and returned below set limit. Clients should be informed before the change happens so they ensure that the change will not impact them. ECS UI should be consulted for alerts of any objects recently breaching the limit. |
| Recommendation |
If you have lowered the limit in step 1. It is best to wait for a few days and monitor the UI for any spike in alerts over objects exceeding the versioning limit. Contact the owners of those objects before enforcing the limit. Advise owners that they must reduce their version count for the affected objects to avoid 403 errors when enforcement is enabled. |
com.emc.ecs.blobsvc.object.version.limit.enabled
Verify parameter value:
admin@ecsnode1:~> svc_param info com.emc.ecs.blobsvc.object.version.limit.enabled svc_param v1.2.4 (svc_tools v2.17.0) Started 2024-05-26 09:26:07 Param: com.emc.ecs.blobsvc.object.version.limit.enabled Description Enable or disable max object version count enforcement Default value true Configured Value (active) false Modification reason CMF Changes for 3.7.0.4 Last modified 2023-07-13 07:53:12 UTC (1689234792375)
Change the parameter to block access to objects based on the current configured limit:
admin@ecsnode1:~> svc_param set -n com.emc.ecs.blobsvc.object.version.limit.enabled -v true -r "Enforce limit" svc_param v1.2.3 (svc_tools v2.16.0) Started 2024-05-26 09:04:43 Param value changed. Param: com.emc.ecs.blobsvc.object.version.limit.enabled Description Enable or disable max object version count enforcement Default value true Configured Value (active) true Modification reason Block access to objects Last modified 2024-05-26 09:04:44 UTC (1716714284396)