PowerScale:讀取 CloudPools 記錄訊息
Summary: 如何剖析 CloudPools 記錄中的消息。
Instructions
CloudPools 會根據正在執行的活動或工作,以及遇到的問題或訊息原因來記錄訊息。這些消息中的細節經常被遺漏,但為解釋提供了重要資訊。您可以在訊息檔案中找到這些訊息 isi_cpool_d.log or in isi_cpool_io_d.log。
以下是幾個範例:
2017-07-03T13:03:59-06:00 <3.3> isilon-1 isi_cpool_d[2547]: [0x800710610]: /b/mnt/src/isilon/lib/isi_cpool_d/task_processing_thread_pool.cpp:run_consumer_function:196: run_consumer_function complete for task ([cache-invalidation/1:054d:b5b7::HEAD]): Writing Cache Header: Read-only file system from write_header (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_cache.cpp:360) from clear_state (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_cache.cpp:907): clearing cacheheader state from isi_cbm_invalidate_cache_i (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_invalidate.cpp:262): Could not reset CACHED state flag for 1:054d:b5b7 from isi_cbm_invalidate_cached_file_opt (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_invalidate.cpp:518)
縮小消息來源和原因範圍的一個重要關鍵是任務資訊:
Task ([cache-invalidation/1:054d:b5b7::HEAD])
任務類型(緩存失效)指向生成消息的雲作業。在此情況下,工作 2 的快取失效工作:
tf80-1# isi cloud jobs list ID Description Effective State Type -------------------------------------------------------------------------------------- 1 Write updated data back to the cloud running cache-writeback 2 Expire CloudPools cache running cache-invalidation 3 Clean up cache and stub file metadata running local-garbage-collection 4 Clean up unreferenced data in the cloud running cloud-garbage-collection 491 error archive 492 error archive>
LIN 可用於追蹤該系統上的 LIN (如果其仍然存在)。
如果消息中 未 調用任務類型,您仍然可以確定操作類型。
以下範例:
如果您看到以下訊息:
2018-08-13T12:49:38+02:00 <3.6> Isilon-1 isi_cpool_d[4520]: [0x80f7d0610]: /b/mnt/src/isilon/lib/isi_cloud_api/cl_provider.cpp:do_with_retry_common:73: Operation: get_object, current try: 0, total retries: 140733193388032, total failures: 1, error: clapi error: CL_OBJ_NOT_FOUND; AEC_NOT_FOUND 2018-08-13T12:49:38+02:00 <3.3> Isilon-1 isi_cpool_d[4520]: [0x80f7d0610]: /b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_error_util.cpp:send_cloudpools_event:314: isi_cpool_cbm error in isi_cph_read_md_blob: CloudPools Object not found. provider RAN, devid 1, account X210_cloud, entitypath m0007430b5526ccc7be54980e747b9db7883ai11/007034b0250780fde1b5283187d98e49_00000000_1, filename , lin 0, offset -1, errormessage clapi error: CL_OBJ_NOT_FOUND; AEC_NOT_FOUND: [error code: CBM_CLAPI_OBJECT_NOT_FOUND]此資訊告訴您進程和線程。您可以在紀錄檔中進一步回顧(在本例中,
isi_cpool_d.log) 來初始化該線程。以下是該線程的初始化:
2018-05-23T07:51:43+02:00 <3.6> Isilon-1 isi_cpool_d[4520]: [0x80f7d0610]: /b/mnt/src/isilon/lib/isi_cpool_d/task_processing_thread_pool.cpp:start_thread:506: starting normal thread for type cloud-gc
一個雲端垃圾回收程序發生此錯誤。「找不到物件」訊息並不表示資料遺失。由於這會在例行垃圾回收和清理程序期間發生,因此已排程刪除物件。如果這是一次召回操作,那麼錯誤會更嚴重。
有時會提供此問題的原因:
Writing Cache Header: Read-only file system Could not reset CACHED state flag for 1:054d:b5b7
系統無法重置 LIN 的緩存狀態標誌,因為檔案系統處於唯讀狀態。在此特定情況下,這是 SyncIQ 目標路徑上的正常和預期訊息,但應調查路徑是否為唯讀。
再比如:
2017-06-19T09:13:15-04:00 <3.7> isilon-2 isi_cpool_d[4531]: [0x80ef97c10]: /b/mnt/src/isilon/lib/isi_cpool_d/task_processing_thread_pool.cpp:run_consumer_function:196: run_consumer_function complete for task ([archive/1:5d01:0232::HEAD]/CloudPool-5yrs): File modified after stubbing began; lin 1:5d01:0232: [error code: CBM_STALE_STUB_ERROR] from write_map_core (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_mapper.cpp:1615) from write_map (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_mapper.cpp:1653) from isi_cph_create_stub (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_archive.cpp:1073) from isi_cbm_archive_common (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_archive.cpp:934) from isi_cbm_archive (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_archive.cpp:312)
在這種情況下,正在處理存檔任務:
archive/1:5d01:0232::HEAD
確定檔案池原則:
CloudPool-5yrs
無法根據此錯誤判斷特定的 CloudPools 歸檔工作。
發生了什麼:
File modified after stubbing began; lin 1:5d01:0232.
檢查該檔並查看它是否可以存根(它應該在該路徑的下一個存檔期間)。
其他一些 CloudPools 錯誤訊息會提供更多隱晦的錯誤。在這些情況下,您通常可以將任務確定為線索的一部分,但可能需要進行一些研究才能找出錯誤的含義。
例如:
varlog.tar/log/messages: 2017-04-20T18:47:52-06:00 <3.3> isilon-6 isi_cpool_d[3856]: [0x80f24f710]: /b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_error_util.cpp:send_cloudpools_event:314: isi_cpool_cbm error in isi_cph_read_md_blob: CloudPools Object not found. provider AWS, devid 6, account isilon archive, entitypath m000743091d8878539854870242892ceaa45di36/007034b0dea80d459f85b4a16d82cc48_00000000_0, filename , lin 0, offset -1, errormessage clapi error: CL_OBJ_NOT_FOUND; The specified key does not exist.: [error code: CBM_CLAPI_OBJECT_NOT_FOUND]
這確實告訴你一點:
- 什麼是雲端提供商:
Provider AWS - 哪個節點遇到錯誤:
Devid 6 - 使用了什麼帳戶:
Account isilon archive - 在雲端中尋找的物件路徑是什麼:
entitypath m000743091d8878539854870242892ceaa45di36/007034b0dea80d459f85b4a16d82cc48_00000000_0. - 問題是什麼:
error message clapi error: CL_OBJ_NOT_FOUND; The specified key does not exist.: [error code: CBM_CLAPI_OBJECT_NOT_FOUND
在這種情況下,請查看是否可以在雲存儲中找到該物件。
其他一些情況:
An error occurred on the Cloud Storage - Check the cloud storage logs or errors: CL_SVR_INTERNAL_ERROR; We encountered an internal error. Please try again.: [error code: CBM_CLAPI_SERVER_ERROR] A network error occurred - make sure that Node can connect to the cloud: CL_COULDNT_CONNECT; Couldn't connect to server: error buf: couldn't connect to host: [error code: CBM_CLAPI_COULDNT_CONNECT] Authentication Failed - Make sure that the account identified has access on the Cloud target: CL_AUTHENTICATION_FAILED; 401: [error code: CBM_CLAPI_AUTHENTICATION_FAILED]