PowerScale:读取 CloudPools 日志消息
Summary: 如何解析 CloudPools 日志中的消息。
Instructions
CloudPools 根据正在执行的活动或任务以及遇到的问题或消息的原因来记录消息。这些消息中的详细信息经常会被遗漏,但为解释提供了重要信息。您可以在消息文件中找到这些消息 isi_cpool_d.log or in isi_cpool_io_d.log。
以下是几个示例:
2017-07-03T13:03:59-06:00 <3.3> isilon-1 isi_cpool_d[2547]: [0x800710610]: /b/mnt/src/isilon/lib/isi_cpool_d/task_processing_thread_pool.cpp:run_consumer_function:196: run_consumer_function complete for task ([cache-invalidation/1:054d:b5b7::HEAD]): Writing Cache Header: Read-only file system from write_header (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_cache.cpp:360) from clear_state (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_cache.cpp:907): clearing cacheheader state from isi_cbm_invalidate_cache_i (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_invalidate.cpp:262): Could not reset CACHED state flag for 1:054d:b5b7 from isi_cbm_invalidate_cached_file_opt (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_invalidate.cpp:518)
要缩小消息的来源和原因范围,一个重要的关键是任务信息:
Task ([cache-invalidation/1:054d:b5b7::HEAD])
任务类型 (cache-invalidation) 指向生成消息的云作业。在本例作业 2 中,高速缓存失效作业:
tf80-1# isi cloud jobs list ID Description Effective State Type -------------------------------------------------------------------------------------- 1 Write updated data back to the cloud running cache-writeback 2 Expire CloudPools cache running cache-invalidation 3 Clean up cache and stub file metadata running local-garbage-collection 4 Clean up unreferenced data in the cloud running cloud-garbage-collection 491 error archive 492 error archive>
LIN 可用于跟踪该系统上的 LIN(如果仍然存在)。
如果消息中 未 指出任务类型,您仍然可以确定操作的类型。
下面是一个示例:
如果您看到这些消息:
2018-08-13T12:49:38+02:00 <3.6> Isilon-1 isi_cpool_d[4520]: [0x80f7d0610]: /b/mnt/src/isilon/lib/isi_cloud_api/cl_provider.cpp:do_with_retry_common:73: Operation: get_object, current try: 0, total retries: 140733193388032, total failures: 1, error: clapi error: CL_OBJ_NOT_FOUND; AEC_NOT_FOUND 2018-08-13T12:49:38+02:00 <3.3> Isilon-1 isi_cpool_d[4520]: [0x80f7d0610]: /b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_error_util.cpp:send_cloudpools_event:314: isi_cpool_cbm error in isi_cph_read_md_blob: CloudPools Object not found. provider RAN, devid 1, account X210_cloud, entitypath m0007430b5526ccc7be54980e747b9db7883ai11/007034b0250780fde1b5283187d98e49_00000000_1, filename , lin 0, offset -1, errormessage clapi error: CL_OBJ_NOT_FOUND; AEC_NOT_FOUND: [error code: CBM_CLAPI_OBJECT_NOT_FOUND]此信息告诉您进程和线程。您可以在日志文件中进一步回顾(在本例中为
isi_cpool_d.log) 用于初始化该线程。下面是该线程的初始化:
2018-05-23T07:51:43+02:00 <3.6> Isilon-1 isi_cpool_d[4520]: [0x80f7d0610]: /b/mnt/src/isilon/lib/isi_cpool_d/task_processing_thread_pool.cpp:start_thread:506: starting normal thread for type cloud-gc
云垃圾数据收集进程遇到此错误。“找不到对象”消息不表示数据丢失。由于这会在例行垃圾收集和清理过程中发生,因此已计划删除对象。如果这是召回操作,那么错误会更严重。
有时会提供此问题的原因:
Writing Cache Header: Read-only file system Could not reset CACHED state flag for 1:054d:b5b7
系统无法重置 LIN 的高速缓存状态标记,因为文件系统处于只读状态。在此特定情况下,这是 SyncIQ 目标路径上的正常预期消息,但应调查该路径是只读还是应该为只读。
另一个例子:
2017-06-19T09:13:15-04:00 <3.7> isilon-2 isi_cpool_d[4531]: [0x80ef97c10]: /b/mnt/src/isilon/lib/isi_cpool_d/task_processing_thread_pool.cpp:run_consumer_function:196: run_consumer_function complete for task ([archive/1:5d01:0232::HEAD]/CloudPool-5yrs): File modified after stubbing began; lin 1:5d01:0232: [error code: CBM_STALE_STUB_ERROR] from write_map_core (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_mapper.cpp:1615) from write_map (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_mapper.cpp:1653) from isi_cph_create_stub (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_archive.cpp:1073) from isi_cbm_archive_common (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_archive.cpp:934) from isi_cbm_archive (/b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_archive.cpp:312)
在这种情况下,正在处理归档任务:
archive/1:5d01:0232::HEAD
确定文件池策略:
CloudPool-5yrs
无法从此错误确定特定的 CloudPools 归档作业。
发生了什么事:
File modified after stubbing began; lin 1:5d01:0232.
检查该文件并查看它是否可以存根(在该路径的下一次归档期间应该这样做)。
其他一些 CloudPools 错误消息提供了更多隐晦的错误。在这些情况下,您通常可以将任务确定为线索的一部分,但可能需要进行一些研究才能弄清楚错误的含义。
例如:
varlog.tar/log/messages: 2017-04-20T18:47:52-06:00 <3.3> isilon-6 isi_cpool_d[3856]: [0x80f24f710]: /b/mnt/src/isilon/lib/isi_cpool_cbm/isi_cbm_error_util.cpp:send_cloudpools_event:314: isi_cpool_cbm error in isi_cph_read_md_blob: CloudPools Object not found. provider AWS, devid 6, account isilon archive, entitypath m000743091d8878539854870242892ceaa45di36/007034b0dea80d459f85b4a16d82cc48_00000000_0, filename , lin 0, offset -1, errormessage clapi error: CL_OBJ_NOT_FOUND; The specified key does not exist.: [error code: CBM_CLAPI_OBJECT_NOT_FOUND]
这确实告诉你一点:
- 什么是云提供商:
Provider AWS - 哪个节点遇到错误:
Devid 6 - 使用了什么帐户:
Account isilon archive - 在云中查找的对象路径是什么:
entitypath m000743091d8878539854870242892ceaa45di36/007034b0dea80d459f85b4a16d82cc48_00000000_0. - 问题是什么:
error message clapi error: CL_OBJ_NOT_FOUND; The specified key does not exist.: [error code: CBM_CLAPI_OBJECT_NOT_FOUND
在这种情况下,请查看是否可以在云存储中找到该对象。
其他一些情况:
An error occurred on the Cloud Storage - Check the cloud storage logs or errors: CL_SVR_INTERNAL_ERROR; We encountered an internal error. Please try again.: [error code: CBM_CLAPI_SERVER_ERROR] A network error occurred - make sure that Node can connect to the cloud: CL_COULDNT_CONNECT; Couldn't connect to server: error buf: couldn't connect to host: [error code: CBM_CLAPI_COULDNT_CONNECT] Authentication Failed - Make sure that the account identified has access on the Cloud target: CL_AUTHENTICATION_FAILED; 401: [error code: CBM_CLAPI_AUTHENTICATION_FAILED]