ECS: CAS error -10204 - Bucket Hard Quota exceeded

The Filenet application team has reported an issue when attempting to write blobs to the ECS.

Requested the Filenet Log file from the incident time, and the below error (-10204) associated with "the use of this operation is restricted" were seen in the Filenet logs:

[6/29/20 13:36:50:025 BST] 00000117 SystemOut     O 2020-06-29 13:36:50,025 ERROR [ContentQueueWorker_PERSONALLN_S_Worker background task processing282] 
- Failed to create new tag  FN.3029D828-A10B-4719-BEA1-6BA4357E4FD3.0 (Centera error code: -10204)
com.filepool.fplibrary.FPLibraryException: The use of this operation is restricted (transid='sl008509/644175/WRITE_BLOB')
	at com.filepool.fplibrary.FPTag.BlobWrite(Unknown Source)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.addTagAndContent(CenteraAdaptor.java:2188)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.saveDocument(CenteraAdaptor.java:333)
	at com.filenet.engine.content.fcprovider.centera.CenteraFCP.createMultipart2(CenteraFCP.java:367)
	at com.filenet.engine.content.FixedContentArea.rollfwdMigrateContent(FixedContentArea.java:1865)
	at com.filenet.engine.content.FixedContentArea.executeRollFwdCommand(FixedContentArea.java:1617)
	at com.filenet.engine.content.BaseContentArea.executeRollFwdCommand(BaseContentArea.java:1805)
	at com.filenet.engine.queueitem.ContentQueueWorker.processBatch(ContentQueueWorker.java:463)
	at com.filenet.engine.queueitem.ContentQueueWorker.execute(ContentQueueWorker.java:160)
	at com.filenet.engine.tasks.BackgroundTask.safeExecute(BackgroundTask.java:258)
	at com.filenet.engine.tasks.BackgroundTask$BackgroundTaskPriviledgedExceptionAction.run(BackgroundTask.java:1090)
	at com.filenet.engine.context.CallState.doAsSystem(CallState.java:549)
	at com.filenet.engine.tasks.BackgroundTask.run(BackgroundTask.java:210)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:949)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:973)
	at java.lang.Thread.run(Thread.java:790)
[6/29/20 13:36:50:026 BST] 00000117 SystemOut     O 18734167 [ContentQueueWorker_PERSONALLN_S_Worker background task processing282] 
ERROR filenet_error.fixed_content_provider.com.filenet.engine.content.fcprovider.centera.CenteraAdaptor  
- Failed to create new tag  FN.3029D828-A10B-4719-BEA1-6BA4357E4FD3.0 (Centera error code: -10204)
com.filepool.fplibrary.FPLibraryException: The use of this operation is restricted (transid='sl008509/644175/WRITE_BLOB')
	at com.filepool.fplibrary.FPTag.BlobWrite(Unknown Source)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.addTagAndContent(CenteraAdaptor.java:2188)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.saveDocument(CenteraAdaptor.java:333)
	at com.filenet.engine.content.fcprovider.centera.CenteraFCP.createMultipart2(CenteraFCP.java:367)
	at com.filenet.engine.content.FixedContentArea.rollfwdMigrateContent(FixedContentArea.java:1865)
	at com.filenet.engine.content.FixedContentArea.executeRollFwdCommand(FixedContentArea.java:1617)
	at com.filenet.engine.content.BaseContentArea.executeRollFwdCommand(BaseContentArea.java:1805)
	at com.filenet.engine.queueitem.ContentQueueWorker.processBatch(ContentQueueWorker.java:463)
	at com.filenet.engine.queueitem.ContentQueueWorker.execute(ContentQueueWorker.java:160)
	at com.filenet.engine.tasks.BackgroundTask.safeExecute(BackgroundTask.java:258)
	at com.filenet.engine.tasks.BackgroundTask$BackgroundTaskPriviledgedExceptionAction.run(BackgroundTask.java:1090)
	at com.filenet.engine.context.CallState.doAsSystem(CallState.java:549)
	at com.filenet.engine.tasks.BackgroundTask.run(BackgroundTask.java:210)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:949)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:973)
	at java.lang.Thread.run(Thread.java:790)

Ran the KPI command to check for the current CAS operations status, and found 73.53% of total Write Blob operations failing with error code -10204: 

# kpi.sh -cas -s -errorcode "-10204" -start "2 days ago"

kpi.sh Version 2.5.14                                                           

Report start: 2020-06-27 14:19:10                Report end:   2020-06-29 14:19:10
Extracting RequestLog data:        DONE
Preprocessing data:                DONE

                                   All Requests                                             -10204 Errors
                 READ      READ      WRITE     WRITE                         READ    READ    WRITE   WRITE
Node             CLIP      BLOB      CLIP      BLOB      DELETE   Total      CLIP    BLOB    CLIP    BLOB    DELETE  Total

10.xxx.xxx.xxx   325587    413917    124736    435750    16198    1316188    0       0       0       321319  301     321620
10.xxx.xxx.xxx   334412    405561    130012    440337    9245     1319567    0       0       0       322607  301     322908
10.xxx.xxx.xxx   323114    414638    133056    437784    9348     1317940    0       0       0       321415  308     321723
10.xxx.xxx.xxx   335851    407194    120876    435475    17422    1316818    0       0       0       320809  304     321113
10.xxx.xxx.xxx   327851    409858    128350    431039    16141    1313239    0       0       0       316541  310     316851
10.xxx.xxx.xxx   320987    410635    136813    441088    8477     1318000    0       0       0       323263  301     323564
10.xxx.xxx.xxx   319899    409053    138668    434242    8257     1310119    0       0       0       319452  307     319759
10.xxx.xxx.xxx   337687    409034    119097    435702    17143    1318663    0       0       0       321973  308     322281

Req Totals:      2625388   3279890   1031608   3491417   102231   10530534   0       0       0       2567379 2440    2569819
Req Error %:                                                                 0.00    0.00    0.00    73.53   2.39    24.40

The issue is caused by bucket hard quota exceed, therefore the write operations are not allowed.

Must extend the bucket Hard Quota to resume the write operations.

KB ECS: Setting Bucket Soft and Hard Quota and ECS: How to increase the quota set on an ECS Bucket should be followed.