ECS: CAS 오류 -10204 - 버킷 하드 할당량 초과

Filenet 애플리케이션 팀에서 ECS에 Blob을 쓰려고 할 때 발생하는 문제를 보고했습니다.

인시던트 시간 및 관련 아래 오류(-10204)에서 Filenet 로그 파일을 요청했습니다. "the use of this operation is restricted" Filenet 로그에서 볼 수 있습니다.

[6/29/20 13:36:50:025 BST] 00000117 SystemOut     O 2020-06-29 13:36:50,025 ERROR [ContentQueueWorker_PERSONALLN_S_Worker background task processing282] 
- Failed to create new tag  FN.3029D828-A10B-4719-BEA1-6BA4357E4FD3.0 (Centera error code: -10204)
com.filepool.fplibrary.FPLibraryException: The use of this operation is restricted (transid='sl008509/644175/WRITE_BLOB')
	at com.filepool.fplibrary.FPTag.BlobWrite(Unknown Source)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.addTagAndContent(CenteraAdaptor.java:2188)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.saveDocument(CenteraAdaptor.java:333)
	at com.filenet.engine.content.fcprovider.centera.CenteraFCP.createMultipart2(CenteraFCP.java:367)
	at com.filenet.engine.content.FixedContentArea.rollfwdMigrateContent(FixedContentArea.java:1865)
	at com.filenet.engine.content.FixedContentArea.executeRollFwdCommand(FixedContentArea.java:1617)
	at com.filenet.engine.content.BaseContentArea.executeRollFwdCommand(BaseContentArea.java:1805)
	at com.filenet.engine.queueitem.ContentQueueWorker.processBatch(ContentQueueWorker.java:463)
	at com.filenet.engine.queueitem.ContentQueueWorker.execute(ContentQueueWorker.java:160)
	at com.filenet.engine.tasks.BackgroundTask.safeExecute(BackgroundTask.java:258)
	at com.filenet.engine.tasks.BackgroundTask$BackgroundTaskPriviledgedExceptionAction.run(BackgroundTask.java:1090)
	at com.filenet.engine.context.CallState.doAsSystem(CallState.java:549)
	at com.filenet.engine.tasks.BackgroundTask.run(BackgroundTask.java:210)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:949)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:973)
	at java.lang.Thread.run(Thread.java:790)
[6/29/20 13:36:50:026 BST] 00000117 SystemOut     O 18734167 [ContentQueueWorker_PERSONALLN_S_Worker background task processing282] 
ERROR filenet_error.fixed_content_provider.com.filenet.engine.content.fcprovider.centera.CenteraAdaptor  
- Failed to create new tag  FN.3029D828-A10B-4719-BEA1-6BA4357E4FD3.0 (Centera error code: -10204)
com.filepool.fplibrary.FPLibraryException: The use of this operation is restricted (transid='sl008509/644175/WRITE_BLOB')
	at com.filepool.fplibrary.FPTag.BlobWrite(Unknown Source)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.addTagAndContent(CenteraAdaptor.java:2188)
	at com.filenet.engine.content.fcprovider.centera.CenteraAdaptor.saveDocument(CenteraAdaptor.java:333)
	at com.filenet.engine.content.fcprovider.centera.CenteraFCP.createMultipart2(CenteraFCP.java:367)
	at com.filenet.engine.content.FixedContentArea.rollfwdMigrateContent(FixedContentArea.java:1865)
	at com.filenet.engine.content.FixedContentArea.executeRollFwdCommand(FixedContentArea.java:1617)
	at com.filenet.engine.content.BaseContentArea.executeRollFwdCommand(BaseContentArea.java:1805)
	at com.filenet.engine.queueitem.ContentQueueWorker.processBatch(ContentQueueWorker.java:463)
	at com.filenet.engine.queueitem.ContentQueueWorker.execute(ContentQueueWorker.java:160)
	at com.filenet.engine.tasks.BackgroundTask.safeExecute(BackgroundTask.java:258)
	at com.filenet.engine.tasks.BackgroundTask$BackgroundTaskPriviledgedExceptionAction.run(BackgroundTask.java:1090)
	at com.filenet.engine.context.CallState.doAsSystem(CallState.java:549)
	at com.filenet.engine.tasks.BackgroundTask.run(BackgroundTask.java:210)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:949)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:973)
	at java.lang.Thread.run(Thread.java:790)

KPI 명령을 실행하여 현재 CAS 작업 상태를 확인한 결과, 오류 코드 -10204로 인해 총 쓰기 Blob 작업의 73.53%가 실패하는 것을 발견했습니다. 

# kpi.sh -cas -s -errorcode "-10204" -start "2 days ago"

kpi.sh Version 2.5.14                                                           

Report start: 2020-06-27 14:19:10                Report end:   2020-06-29 14:19:10
Extracting RequestLog data:        DONE
Preprocessing data:                DONE

                                   All Requests                                             -10204 Errors
                 READ      READ      WRITE     WRITE                         READ    READ    WRITE   WRITE
Node             CLIP      BLOB      CLIP      BLOB      DELETE   Total      CLIP    BLOB    CLIP    BLOB    DELETE  Total

10.xxx.xxx.xxx   325587    413917    124736    435750    16198    1316188    0       0       0       321319  301     321620
10.xxx.xxx.xxx   334412    405561    130012    440337    9245     1319567    0       0       0       322607  301     322908
10.xxx.xxx.xxx   323114    414638    133056    437784    9348     1317940    0       0       0       321415  308     321723
10.xxx.xxx.xxx   335851    407194    120876    435475    17422    1316818    0       0       0       320809  304     321113
10.xxx.xxx.xxx   327851    409858    128350    431039    16141    1313239    0       0       0       316541  310     316851
10.xxx.xxx.xxx   320987    410635    136813    441088    8477     1318000    0       0       0       323263  301     323564
10.xxx.xxx.xxx   319899    409053    138668    434242    8257     1310119    0       0       0       319452  307     319759
10.xxx.xxx.xxx   337687    409034    119097    435702    17143    1318663    0       0       0       321973  308     322281

Req Totals:      2625388   3279890   1031608   3491417   102231   10530534   0       0       0       2567379 2440    2569819
Req Error %:                                                                 0.00    0.00    0.00    73.53   2.39    24.40

이 문제는 버킷 하드 할당량 초과로 인해 발생하므로 쓰기 작업이 허용되지 않습니다.

쓰기 작업을 재개하려면 버킷 고정 할당량을 확장해야 합니다.

KB ECS: 버킷 소프트 및 하드 할당량과 ECS 설정: ECS 버킷에 설정된 할당량을 늘리는 방법을 따라야 합니다.