20 Posts
0
4099
R630 with Intel 10G X710 rNDC - Lifecycle Controller Error SUP0528
Lifecycle Controller: v2.81.81.81
Bios: v2.13.0
Intel x710: v20.0.17
Hello
Having installed an Intel 10G X710 rNDC (#68m95), the Lifecycle controller is failing with a critical error "SUP0528" (Unable to generate a firmware comparison table). According to the resources, this is an internal error and restarting the process does not clear the fault. Reflashing the iDrac firmware does not resolve the fault. Updates are taking place via the F10 lifecycle controller and connected via HTTPS to download the catalog. Catalog downloads but the proces fails on the subsequent step.
The fault is reproduceable. A second R630 with an identical configuration, additional x710 and all updated firmware levels gives the same result. Removing the x710 card and attempting the update process however succeeds.
It appears the Lifecycle controller fails with this NDC installed. Can Dell confirm please?
Rgds
Pops
Popolou
20 Posts
0
September 24th, 2021 06:00
@DELL-Joey C, @DELL-Marco B, hi
Problem identified and solved. However, for the life of me i cannot understand why.
Both the 4P 10G x710 cards (68M95) were new. They were installed at the same time in both R630 units. Both Lifecycle controllers we then run independently for updates and both gave the same SUP0528 critical error. Installing the same rNDC NIC into a third R530 to serve as a testbed gave the same results. Initial suspicions were conflicts with other hardware but this did not turn out to be the case in subsequent tests. Attention was turned back to the rNDC card.
Up until now, neither of the two rNDC NICs were used operationally. They were connected to the mainboard and powered up but no actual data was passed through the ports. Having then connected one of the four SFP ports to our switches and enabled it, inexplicably, the Lifecycle controller now does not flag the critical error. Turning to the second server and enabling a port on the second rNDC NIC and the precisely the same behaviour occurred.
It seems that once the lifecycle controller downloaded the online catalog and proceeded to interrogate the hardware, it may have been reading a value from the x710 daughtercards that it did not expect - perhaps a register or a count (for data throughput, say). Whatever it is was cleared the moment traffic was flowing through the cards. Cold booting both units and the Lifecycle controller is able to now continue through to the firmware listing table.
A very odd behaviour and one that was quite unexpected. It may be helpful to Teir 2 support or if this is passed on to the engineers. I am unsure how common this is with other x710 variants but there is evidently something with the Lifecycle firmware that it did not like until the rNDC cards were 'used'.
Cheers
Pops
DELL-Joey C
Moderator
Moderator
•
3.4K Posts
0
September 22nd, 2021 22:00
Hi @Popolou,
I might misunderstood your issue. Are you trying to update the rNDC card via Lifecycle Controller and encountering error, or Lifecycle Controller is prompting error whenever the rNDC card is installed?
Can I check with you how was the rNDC updated to v20.0.17? It seems that R630's rNDC x710 firmware is at the latest of v19.5. If the firmware updates from catalog, it won't update to v20.0.17.
Popolou
20 Posts
0
September 23rd, 2021 01:00
@DELL-Joey C, hi
Both actually. Using the Lifecyle Controller to update the rNDC led to the critical error. "SUP0528" occurs at the processing stage just after it downloads the catalog. I cannot get to the firmware table/listing whilst the 4P 10G x710 is installed. Removing it and powercycling restores the Lifecycle controller to normal operation.
Updating the firmware was only possible via the iDRAC using the payload from latest available (Network_Firmware_DK4G2_WN64_20.0.17_A00). This completed without error & the iDrac correctly reports the latest version up and running.
The x710 was at firmware level v17.5.11. The SUP0528 error occurred under both v17.5.11 and v20.0.17 (apparently rrespective of the firmware revision) and it is currently preventing normal operations of the Lifecycle controller. Both cards in the two R630 units are exhibiting identical issues.
Rgds
Pops
DELL-Joey C
Moderator
Moderator
•
3.4K Posts
0
September 23rd, 2021 02:00
Hi @Popolou,
Seems that Network_Firmware_DK4G2_WN64_20.0.17_A00 is not listed as compatible with R630. The version that I found Network_Firmware_YP4R0_WN64_19.5.12_A00 would be the latest for R630. There might be a corrupted firmware that is causing this issue.
Popolou
20 Posts
0
September 23rd, 2021 02:00
@DELL-Joey C, hi
Thanks for pointing this out. No effect i'm afraid. Downgraded one of the two R630 via iDrac and rebooted. Running an online update on the Lifecycle controller still returns critical error SUP0528 - Unable to generate a firmware comparison table.
Rgds
Pops
Popolou
20 Posts
0
September 23rd, 2021 03:00
@DELL-Joey C, hi
Just tested on third R630 which has a different config (PERC H730 and different RAM, nothing consequential to the issue) and the third lifecycle controller flags the same SUP0528 critical error.
Initial indications are there is a mismatch with the iDrac/Lifecycle controller firmware. I can further investigate with downgrading it but will await to hear back from Dell.
Rgds
Pops
DELL-Marco B
Moderator
Moderator
•
3.5K Posts
0
September 23rd, 2021 06:00
Hello,
are you using a repository to update firmware? It's sound like an issue connecting to the repository
Following this guide, are you selecting Network Share (Https) and not FTP?
https://dell.to/3lRe7Yp
It could be an issue also with DNS or firewall.
Do you have an enterprise license?
Can you try to downgrade iDRAC version and see if problem is the same?
Thanks
Marco
Popolou
20 Posts
0
September 23rd, 2021 07:00
@DELL-Marco B, hi
Yes, i will need to speak sourcing but they should still be covered. I'm going to experiment with the firmware and components to see i can make any sense of this in the meantime.
Cheers
pops
DELL-Marco B
Moderator
Moderator
•
3.5K Posts
0
September 23rd, 2021 07:00
Yes, we can involve our L2 techs or escalate internally.
The servers are still under warranty?
Let us know if after downgrade problem is the same.
Thanks
Marco
Popolou
20 Posts
0
September 23rd, 2021 07:00
@DELL-Marco B, hi
Yes, all carried out in the usual way via Https via downloads.dell.com. No problems with connectivity in this case and yes, full enterprise licensing.
I could attempt a downgrade for testing purposes. Are Dell able to put this to the engineers in parallel to see if they can replicate the issue?
Rgds
Pops
Popolou
20 Posts
0
September 23rd, 2021 15:00
@DELL-Marco B @DELL-Joey C, hi
Firmware rollback testing each successive revsion through to v2.63.60.61 presented no change in the result. I limited my investigations to the previous five firmware revisions which is early 2019 and was uncomfortable taking it back any further.
Rgds
Pops
DELL-Joey C
Moderator
Moderator
•
3.4K Posts
0
September 23rd, 2021 19:00
Hi @Popolou,
I was wondering, when you were doing the testing, was all of the 3 servers containing PN# 68M95, it's just 1 that you have moved among all 3 server? Would you like to try a soft reset of iDRAC? https://dell.to/3APhBkg
If the soft reset fails, I would suggest to contact support to check on the issue with an engagement of higher level support (L2) to try replicate the issue that you are having, since it is reproduceable on 3 units of yours.
DELL-Marco B
Moderator
Moderator
•
3.5K Posts
0
September 24th, 2021 07:00
Hello,
thanks for your analysis and glad that problem is solved now.
As Joey suggested, can you contact support and check if they can escalate the issue, replicating the problem? That could be interesting.
Thanks
Marco
Gavino
1 Rookie
1 Rookie
•
4 Posts
1
September 28th, 2021 04:00
My environment is very similar with Dell PowerEdge R730 with Intel 10G X710 rNDC
Lifecycle Controller: v2.81.81.81
Bios: v2.13.0
Intel x710: v20.0.17
I too had Lifecycle Controller Error SUP0528 and solved the issue by switching from UEFI to legacy BIOS mode, and then booting back into F10 - Lifecycle controller and trying again. Then it worked for me (HTTPS firmware update checks)
This might be helpful to someone who can't patch any of the ports on the rNDC.
Cheers,
Gav
Popolou
20 Posts
0
September 30th, 2021 09:00
Excellent workaround. I would never have thought flipping the Bios boot mode like this would have an effect on the matter. It further adds to rather the odd nature of this issue.
Cheers
Pops