There ended up being no sane fix for this problem. Despite what other people had reported about BIOS & BMC being out-of-date, that wasn't the case here as I mentioned.
As we had to leave the data centre and travel inter-city, we were forced to work out a fix. That ended up being to re-flash the BMC via FreeDOS which led to several changes:
IP addressing was reset (yay!)
VLAN ID was removed (yay!)
NIC went back to Shared
All other config reset
(Presumably)
We then power cycled and configured the IP addressing via BIOS > IPMI, which accepted the changes unlike before the re-flash. The NIC was re-configured back to the dedicated card, though it's worth mentioning that the reported MAC address doesn't change in BIOS despite changing the setting UNTIL YOU REBOOT AGAIN.
From there, as the MAC had changed after a confirmation check in BIOS, we then had to update our MAC-to-IP binding configuration to match the dedicated NIC's MAC address. We disconnected the NIC, left it for 1-2 minutes, then reconnected and were able to access the IPMI interface through a web browser. Splendid.
It's a shame that there's no known fix for this, however we're grateful that the BMC flashing process wipes the configuration. That's not something that I've ever said in my ICT history, or at least not that I can recall!
Hopefully this helps someone in the future, as we were stuck on these bugged-out nodes out of a plethora of hardware that was reconfigured without incident, and were starting to question our sanity.
Thanks for sharing your resolution to the issue that you are having, apologies I couldn't reply sooner to try help out or suggest some methods to check if it does help you.
Just curious, did you run ipmitool lan set 1 vlan id off, and got the error "LAN parameter Data does not match! Write may have failed"?
Definitely there are fixes on iDRAC and BIOS to the error that you face, but I'm unsure if they are fixed on the server that you had on. In case if in future you face same issue, could you try running and outsource IPMITool ver. 1.8.18 to check if the issue persist. Alternative, you can se the raw command to disable vlanid: "ipmitool raw 0x0c 0x1 1 20 0x2 0x0".
You're right - that's the error we got when we ran that command as you've written it; same occurred when we tried to specify the IP address, subnet mask and default gateway. However, changing the mode from DHCP to/from static worked OK. Also fine was enabling or disabling channel access. Others have reported such behaviour.
Attempting a warm reset was unsuccessful, and while cold resets claimed to have sent the command to the BMC, there was no evidence of that completing properly. Reminded me of UDP traffic - "send and hope it's fine".
I've just confirmed that the version of ipmitool in-use on all servers was 1.8.18.
As for the raw command, while it's good to have on-hand for the future, could you please provide a reference for all Dell ipmitool raw commands? It seems to be a mixed bag online, though perhaps we've missed a manual?
Could you please check with others within Dell to see if the PEC6100 has been patched?
Unfortunately I'm not too familiar with the old C6000 model. I share some documents that I could find after researching and that I think might be helpful;
Appreciate the links, we'll keep those to-hand for the future.
With the PEC6100, could you please explain how we'd go about updating other components aside from the BIOS and BMC? For example, the RAID card (which doesn't seem to be listed on the website), fan module, etc.
This system is pre-DSU so we can't use our typical Dell update process. It could well be that there are no updates required for the chassis aside from what's been done, though it'd be great to have the resources on-hand.
Well, C6100 is a relatively an old server, the new update utilities like SUU or DRM is out of support for C6100. So there is not ways to update the components, other than what you have done, using BMC and FreeDOS (which you have successfully done and I'm not knowledgeable in that) or to do the update in OS. Last I checked on the C6100 updates on RAID and Fan module are dated 2012/2013.
LST-AU
6 Posts
0
January 6th, 2021 00:00
G'day,
There ended up being no sane fix for this problem. Despite what other people had reported about BIOS & BMC being out-of-date, that wasn't the case here as I mentioned.
As we had to leave the data centre and travel inter-city, we were forced to work out a fix. That ended up being to re-flash the BMC via FreeDOS which led to several changes:
We then power cycled and configured the IP addressing via BIOS > IPMI, which accepted the changes unlike before the re-flash. The NIC was re-configured back to the dedicated card, though it's worth mentioning that the reported MAC address doesn't change in BIOS despite changing the setting UNTIL YOU REBOOT AGAIN.
From there, as the MAC had changed after a confirmation check in BIOS, we then had to update our MAC-to-IP binding configuration to match the dedicated NIC's MAC address. We disconnected the NIC, left it for 1-2 minutes, then reconnected and were able to access the IPMI interface through a web browser. Splendid.
It's a shame that there's no known fix for this, however we're grateful that the BMC flashing process wipes the configuration. That's not something that I've ever said in my ICT history, or at least not that I can recall!
Hopefully this helps someone in the future, as we were stuck on these bugged-out nodes out of a plethora of hardware that was reconfigured without incident, and were starting to question our sanity.
Cheers,
LST-AU
DELL-Joey C
Moderator
•
4.1K Posts
0
January 6th, 2021 01:00
Hi,
Thanks for sharing your resolution to the issue that you are having, apologies I couldn't reply sooner to try help out or suggest some methods to check if it does help you.
Just curious, did you run ipmitool lan set 1 vlan id off, and got the error "LAN parameter Data does not match! Write may have failed"?
Definitely there are fixes on iDRAC and BIOS to the error that you face, but I'm unsure if they are fixed on the server that you had on. In case if in future you face same issue, could you try running and outsource IPMITool ver. 1.8.18 to check if the issue persist. Alternative, you can se the raw command to disable vlanid: "ipmitool raw 0x0c 0x1 1 20 0x2 0x0".
LST-AU
6 Posts
0
January 6th, 2021 02:00
Many thanks for your reply, Joey.
You're right - that's the error we got when we ran that command as you've written it; same occurred when we tried to specify the IP address, subnet mask and default gateway. However, changing the mode from DHCP to/from static worked OK. Also fine was enabling or disabling channel access. Others have reported such behaviour.
Attempting a warm reset was unsuccessful, and while cold resets claimed to have sent the command to the BMC, there was no evidence of that completing properly. Reminded me of UDP traffic - "send and hope it's fine".
I've just confirmed that the version of ipmitool in-use on all servers was 1.8.18.
As for the raw command, while it's good to have on-hand for the future, could you please provide a reference for all Dell ipmitool raw commands? It seems to be a mixed bag online, though perhaps we've missed a manual?
Could you please check with others within Dell to see if the PEC6100 has been patched?
Thank you,
LST-AU
DELL-Erman O
Moderator
•
3K Posts
0
January 6th, 2021 05:00
Hello,
Unfortunately I'm not too familiar with the old C6000 model. I share some documents that I could find after researching and that I think might be helpful;
Dell PowerEdge C System Management https://dell.to/395ISlV;
Using IPMItool raw commands for remote management of dell PowerEdge Servers https://dell.to/3rULGLz
Using the Baseboard Management Controller pg 18 https://dell.to/2KXmG5J
https://dell.to/39lOCbF;
from github https://dell.to/3pP0mcY;
LST-AU
6 Posts
0
January 7th, 2021 17:00
Thank you Erman,
Appreciate the links, we'll keep those to-hand for the future.
With the PEC6100, could you please explain how we'd go about updating other components aside from the BIOS and BMC? For example, the RAID card (which doesn't seem to be listed on the website), fan module, etc.
This system is pre-DSU so we can't use our typical Dell update process. It could well be that there are no updates required for the chassis aside from what's been done, though it'd be great to have the resources on-hand.
Many thanks,
LST-AU
DELL-Joey C
Moderator
•
4.1K Posts
0
January 7th, 2021 18:00
Hi,
Well, C6100 is a relatively an old server, the new update utilities like SUU or DRM is out of support for C6100. So there is not ways to update the components, other than what you have done, using BMC and FreeDOS (which you have successfully done and I'm not knowledgeable in that) or to do the update in OS. Last I checked on the C6100 updates on RAID and Fan module are dated 2012/2013.