IDPA: Nodhändelsetjänsten försämras på grund av en dubblettväg 169.254.0.1

Fel vid RUCK-förkontroll firmware_readiness:

[ERROR] Firmware pre-upgrade checks failed. [ 10.100.XX.XXX ]: The cached response with Node Event Service is disabled.
Node Event Service is in a degraded state,iDRAC Service Module is not available/active at this time.
Check iDRAC Service Module/iDRAC status.

När du kör kommandot nedan kommer idraccache att felas eller försämras.
 

[root@idpa-esx1:~] /opt/dell/DellPTAgent/tools/pta_call get agent/info
Request sent to DellPTAgent @ https://192.168.100.101:8086
{
    "libstorelibit.so": "07.05",
    "uptime": "177266 seconds ( 2 days 1 hour 14 minutes 26 seconds )",
    "host_pass_thru_ip": "169.254.0.2",
    "ism_version": "3.6.0",
    "model": "R640 IDPA",
    "os_version": "6.7.0 build-17700523",
    "process_id": "2101654",
    "libstorelibir-3.so": "15.03-0",
    "default_server_cert": "true",
    "TPM Present": "false",
    "MarvellLibraryVersion": "5.0.13.1109",
    "servicetag": "D3WCCD3",
    "mfr": "Dell Inc.",
    "system_uuid": "62dcf7ee-334e-8f96-f507-78ac4426a310",
    "status": {
        "iSM": "N/A",
        "agent": "Error/degraded",
        "idracConnection": "OK",
        "idraccache": "Error/degraded"

iSM körs men kan inte kommunicera med iDRAC:

PTAgent felsökningsloggar /scratch/log/pta_debug.log

2023/10/23 14:29:33[UTC] [19394369:246258496] WARN  - WSManClient::isValidResponse: Http request to host: 169.254.0.1, failed with status code: -4
2023/10/23 14:29:34[UTC] [19394369:246786880] WARN  - WSManClient::isValidResponse: Http request to host: 169.254.0.1, failed with status code: -4
2023/10/23 14:29:34[UTC] [19394369:246258496] WARN  - ISMMonitor::isISMServiceRunning: Command to check iSM status failed with error code <2>. <ism is active (running limited functionality)

Den här extra vägen skapas på värdoperativsystemets lager och baseras på befintlig nätverkskonfiguration. Det här beteendet uppstår ofta när flera statiska vägar har definierats i ESXi-konfigurationen.

Möjlig orsak är användning av en intern SSH-tunnel från ESXi till iDRAC.

iSM skapar inte de ytterligare rutterna och identifierar och läker inte heller några extra definierade rutter.

1. Bekräfta en ogiltig post i ESXi-routningstabellen som förhindrar att iSM använder 169.254.0.2 för att kommunicera med iDRAC:s 169.254.0.1.

Ett exempel på en nätverksmiljö som har det här problemet visas nedan. Informationen vmk0 Gränssnittets gatewayadress för en extra väg för nätverk 169.254.0.1/32.

[root@idpa-esxi:~] esxcli network ip route ipv4 list
Network         Netmask          Gateway        Interface  Source
--------------  ---------------  -------------  ---------  ------
default             0.0.0.0                10.100.10.254   vmk0       MANUAL
10.100.10.0     255.255.255.0    0.0.0.0               vmk0       MANUAL
169.254.0.0     255.255.255.0    0.0.0.0               vmk2       MANUAL
169.254.0.1     255.255.255.255  10.100.xx.xxx  vmk0       MANUAL
192.168.100.96  255.255.255.224  0.0.0.0          vmk1       MANUAL

2. Ta bort posten för att åtgärda routningsproblemet mellan iSM och iDRAC.

esxcli network ip route ipv4 remove -g 10.100.xx.xxx -n 169.254.0.1/32

3. Posten har tagits bort:

[root@idpa-esxi:~] esxcli network ip route ipv4 list
Network         Netmask          Gateway        Interface  Source
--------------  ---------------  -------------  ---------  ------
default             0.0.0.0                10.100.10.254   vmk0       MANUAL
10.100.10.0     255.255.255.0    0.0.0.0               vmk0       MANUAL
169.254.0.0     255.255.255.0    0.0.0.0               vmk2       MANUAL
192.168.100.96  255.255.255.224  0.0.0.0          vmk1       MANUAL

4. När den extra vägen har tagits bort startar du om PTAgent och iSM-tjänsten.

/etc/init.d/DellPTAgent restart

/etc/init.d/dcism-netmon-watchdog restart

5. Kör förberedd fast programvara för att bekräfta SUCCESS

DP4400:

curl -k -i -H "Content-Type:application/json" -X POST  https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": false, "hostList": [{"hostIP": "192.168.100.101", "esxiUser": "root", "esxiPassword": "IDPAPASSWORD"}]}'

IDPAPASSWORD = IDPA common password

Exempel:

curl -k -i -H "Content-Type:application/json" -X POST  https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": false, "hostList": [{"hostIP": "192.168.100.101", "esxiUser": "root", "esxiPassword": "Idpa_12345"}]}'

Utdata:

HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:38:41 GMT

[{"id":814402494717874004,"hostIP":"192.168.100.101","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]

DP5x00/DP8x00:

for i in {1..3}; do curl -k -i -H "Content-Type:application/json" -X POST https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": true, "hostList": [{"hostIP": "192.168.100.10'${i}'", "esxiUser": "root", "esxiPassword": "IDPAPASSWORD"}]}'; done

IDPAPASSWORD = IDPA common password

Exempel:

for i in {1..3}; do curl -k -i -H "Content-Type:application/json" -X POST https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": true, "hostList": [{"hostIP": "192.168.100.10'${i}'", "esxiUser": "root", "esxiPassword": "Idpa_12345"}]}'; done

Utdata:

HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:39:05 GMT

[{"id":2337625064092707360,"hostIP":"192.168.100.101","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":false,"twoHopIsRequired":false,"invalidFirmwareList":[{"name":"Integrated Remote Access Controller","currentVersion":"3.36.103.36","minimumRequiredVersion":"4.40.10.00"}],"messages":"The IDPA system is ready for firmware update.","links":[]}]HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:39:44 GMT

[{"id":1389413431580575643,"hostIP":"192.168.100.102","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:40:19 GMT

[{"id":1184184063038749065,"hostIP":"192.168.100.103","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]

Kontrollera cachestatusen genom att köra kommandot nedan:

/opt/dell/DellPTAgent/tools/pta_call get agent/info

Utdata:

Request sent to DellPTAgent @ https://192.168.100.101:8086
{
    "mfr": "Dell Inc.",
    "servicetag": "D3WCCD3",
    "host_epoch_time": "1699148602.56383 (secs.usecs)",
    "uptime": "204355 seconds ( 2 days 8 hours 45 minutes 55 seconds )",
    "system_uuid": "62dcf7ee-334e-8f96-f507-78ac4426a310",
    "model": "R640 IDPA",
    "process_id": "2101654",
    "libstorelibir-3.so": "15.03-0",
    "domain": "esx1-5800-crk.dp.ce.gslabs.lab.emc.com",
    "name": "esx1-5800-crk",
    "ptagentversion": "2.4.1-3",
    "idrac_ethernet_ip": "192.168.100.110",
    "os_version": "6.7.0 build-17700523",
    "ism_version": "3.6.0",
    "MarvellLibraryVersion": "5.0.13.1109",
    "libstorelib.so": "07.07",
    "host_pass_thru_ip": "169.254.0.2",
    "default_server_cert": "true",
    "status": {
        "iSM": "N/A",
        "agent": "OK",
        "idraccache": "OK",
        "idracConnection": "OK"
    },
    "idrac_pass_thru_ip": "169.254.0.1",
    "api_blocking_enabled": "false",
    "os": "VMWare ESXi",
    "rest_endpoints": "https://192.168.100.101:8086",
    "TPM Present": "false",
    "libstorelibit.so": "07.05"
}
Response: status: 200 [OK], size: 1063 bytes, latency: 0.144 seconds.