ІДПА: Сервіс подій вузла погіршується через дублювання маршруту 169.254.0.1

Помилка попередньої перевірки RUCK firmware_readiness:

[ERROR] Firmware pre-upgrade checks failed. [ 10.100.XX.XXX ]: The cached response with Node Event Service is disabled.
Node Event Service is in a degraded state,iDRAC Service Module is not available/active at this time.
Check iDRAC Service Module/iDRAC status.

При виконанні нижче команди idraccache призведе до помилки або погіршиться.
 

[root@idpa-esx1:~] /opt/dell/DellPTAgent/tools/pta_call get agent/info
Request sent to DellPTAgent @ https://192.168.100.101:8086
{
    "libstorelibit.so": "07.05",
    "uptime": "177266 seconds ( 2 days 1 hour 14 minutes 26 seconds )",
    "host_pass_thru_ip": "169.254.0.2",
    "ism_version": "3.6.0",
    "model": "R640 IDPA",
    "os_version": "6.7.0 build-17700523",
    "process_id": "2101654",
    "libstorelibir-3.so": "15.03-0",
    "default_server_cert": "true",
    "TPM Present": "false",
    "MarvellLibraryVersion": "5.0.13.1109",
    "servicetag": "D3WCCD3",
    "mfr": "Dell Inc.",
    "system_uuid": "62dcf7ee-334e-8f96-f507-78ac4426a310",
    "status": {
        "iSM": "N/A",
        "agent": "Error/degraded",
        "idracConnection": "OK",
        "idraccache": "Error/degraded"

iSM працює, але не може обмінюватися даними з журналами налагодження iDRAC:

PTAgent /scratch/log/pta_debug.log

2023/10/23 14:29:33[UTC] [19394369:246258496] WARN  - WSManClient::isValidResponse: Http request to host: 169.254.0.1, failed with status code: -4
2023/10/23 14:29:34[UTC] [19394369:246786880] WARN  - WSManClient::isValidResponse: Http request to host: 169.254.0.1, failed with status code: -4
2023/10/23 14:29:34[UTC] [19394369:246258496] WARN  - ISMMonitor::isISMServiceRunning: Command to check iSM status failed with error code <2>. <ism is active (running limited functionality)

Цей додатковий маршрут створюється на рівні хост-ОС і базується на поточній конфігурації мережі. Така поведінка зазвичай спостерігається, коли в конфігурації ESXi визначено кілька статичних маршрутів.

Можлива причина включає використання внутрішнього SSH-тунелю від ESXi до iDRAC.

iSM не створює додаткові маршрути, а також не виявляє та не лікує жодних додатково визначених маршрутів.

1. Підтвердьте неправильний запис у таблиці маршрутизації ESXi, який не дозволяє iSM використовувати 169.254.0.2 для зв'язку з 169.254.0.1 iDRAC.

Приклад мережевого середовища, в якому виникає ця проблема, наведено нижче. Об'єкт vmk0 Адреса шлюзу інтерфейсу для додаткового маршруту для мережі 169.254.0.1/32.

[root@idpa-esxi:~] esxcli network ip route ipv4 list
Network         Netmask          Gateway        Interface  Source
--------------  ---------------  -------------  ---------  ------
default             0.0.0.0                10.100.10.254   vmk0       MANUAL
10.100.10.0     255.255.255.0    0.0.0.0               vmk0       MANUAL
169.254.0.0     255.255.255.0    0.0.0.0               vmk2       MANUAL
169.254.0.1     255.255.255.255  10.100.xx.xxx  vmk0       MANUAL
192.168.100.96  255.255.255.224  0.0.0.0          vmk1       MANUAL

2. Видаліть запис, щоб вирішити проблему з маршрутизацією між iSM та iDRAC.

esxcli network ip route ipv4 remove -g 10.100.xx.xxx -n 169.254.0.1/32

3. Запис видалено:

[root@idpa-esxi:~] esxcli network ip route ipv4 list
Network         Netmask          Gateway        Interface  Source
--------------  ---------------  -------------  ---------  ------
default             0.0.0.0                10.100.10.254   vmk0       MANUAL
10.100.10.0     255.255.255.0    0.0.0.0               vmk0       MANUAL
169.254.0.0     255.255.255.0    0.0.0.0               vmk2       MANUAL
192.168.100.96  255.255.255.224  0.0.0.0          vmk1       MANUAL

4. Як тільки зайвий маршрут буде видалено, перезапустіть PTAgent і службу iSM.

/etc/init.d/DellPTAgent restart

/etc/init.d/dcism-netmon-watchdog restart

5. Запустіть готовність прошивки, щоб підтвердити УСПІХ

ДП4400:

curl -k -i -H "Content-Type:application/json" -X POST  https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": false, "hostList": [{"hostIP": "192.168.100.101", "esxiUser": "root", "esxiPassword": "IDPAPASSWORD"}]}'

IDPAPASSWORD = IDPA common password

Приклад:

curl -k -i -H "Content-Type:application/json" -X POST  https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": false, "hostList": [{"hostIP": "192.168.100.101", "esxiUser": "root", "esxiPassword": "Idpa_12345"}]}'

Вихід:

HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:38:41 GMT

[{"id":814402494717874004,"hostIP":"192.168.100.101","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]

ДП5х00/ДП8х00:

for i in {1..3}; do curl -k -i -H "Content-Type:application/json" -X POST https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": true, "hostList": [{"hostIP": "192.168.100.10'${i}'", "esxiUser": "root", "esxiPassword": "IDPAPASSWORD"}]}'; done

IDPAPASSWORD = IDPA common password

Приклад:

for i in {1..3}; do curl -k -i -H "Content-Type:application/json" -X POST https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": true, "hostList": [{"hostIP": "192.168.100.10'${i}'", "esxiUser": "root", "esxiPassword": "Idpa_12345"}]}'; done

Вихід:

HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:39:05 GMT

[{"id":2337625064092707360,"hostIP":"192.168.100.101","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":false,"twoHopIsRequired":false,"invalidFirmwareList":[{"name":"Integrated Remote Access Controller","currentVersion":"3.36.103.36","minimumRequiredVersion":"4.40.10.00"}],"messages":"The IDPA system is ready for firmware update.","links":[]}]HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:39:44 GMT

[{"id":1389413431580575643,"hostIP":"192.168.100.102","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:40:19 GMT

[{"id":1184184063038749065,"hostIP":"192.168.100.103","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]

Перевірте стан кешу, виконавши наведену нижче команду:

/opt/dell/DellPTAgent/tools/pta_call get agent/info

Вихід:

Request sent to DellPTAgent @ https://192.168.100.101:8086
{
    "mfr": "Dell Inc.",
    "servicetag": "D3WCCD3",
    "host_epoch_time": "1699148602.56383 (secs.usecs)",
    "uptime": "204355 seconds ( 2 days 8 hours 45 minutes 55 seconds )",
    "system_uuid": "62dcf7ee-334e-8f96-f507-78ac4426a310",
    "model": "R640 IDPA",
    "process_id": "2101654",
    "libstorelibir-3.so": "15.03-0",
    "domain": "esx1-5800-crk.dp.ce.gslabs.lab.emc.com",
    "name": "esx1-5800-crk",
    "ptagentversion": "2.4.1-3",
    "idrac_ethernet_ip": "192.168.100.110",
    "os_version": "6.7.0 build-17700523",
    "ism_version": "3.6.0",
    "MarvellLibraryVersion": "5.0.13.1109",
    "libstorelib.so": "07.07",
    "host_pass_thru_ip": "169.254.0.2",
    "default_server_cert": "true",
    "status": {
        "iSM": "N/A",
        "agent": "OK",
        "idraccache": "OK",
        "idracConnection": "OK"
    },
    "idrac_pass_thru_ip": "169.254.0.1",
    "api_blocking_enabled": "false",
    "os": "VMWare ESXi",
    "rest_endpoints": "https://192.168.100.101:8086",
    "TPM Present": "false",
    "libstorelibit.so": "07.05"
}
Response: status: 200 [OK], size: 1063 bytes, latency: 0.144 seconds.