PE R620 Orders Nics in wrong order

Hi,

 

Firstly, FreeNas is not a verified OS for the server, so I couldn't provide a proper support on configuration and anything OS related, as there is no documentation for me to refer. 

 

Apologies but I'm not able to understand the issue, is it because the Firmware Inventory is listing the Ports in a different order than the Network Devices listed? 

 

Could you try running the Lifecycle Controller Hardware inventory: https://dell.to/3nKPuvm and check if the Ports and MAC address matches? The Firmware Inventory list might be listing the port is a different order. 

 

FreeNas is not really involved, but it led me to discover the issue.

The iDrac card reports the nic's in two different orders, which leads to ANY OS to seeing port1 as port 3, port 2 as port4, port 3 as port 1 and port4 as port 2

If needed I can supply screen shots of the results of what is happening, I just don't understand WHY it's happening, as I have never seen this on any server with the integrated 4 port Broadcom.

My hope is that possibly using sysmgmt the incorrect order could be changed to match the other part. The part I am hoping to be addressed is the differences in the two card inventories I carefully laid out in the first post.

I used the example of freenas, to show the results the wrong order can cause, with almost any OS.

Hello @universal4,

 

I am afraid there is no configuration possible on the iDRAC to modify how the OS interprets and orders the NICs. That is a translation done by the OS through the driver.

And it's basically the same behavior in Windows OS. However, in Windows you can use PowerShell to Identify Physical Ports in Multiple Ports Network Adapter:

Identify Physical Port in Multiple Ports Network Adapter on Windows Servers with PowerShell

And eventually renaming Network Adapters by Using PowerShell:
Renaming Network Adapters by Using PowerShell

 

As my colleague @DELL-Joey C mentioned, we have no documentation for FreeNAS to help you with this configuration, so all I can suggest you is try to contact the support forum of this OS and ask if there is such a similar procedure.

 

Regards.

Hello again @universal4,

 

On a second read of your message I noticed your issue is with the boot and the NIC order... so maybe there is something you can try.. Are you using PXE boot? if so, you can configure Boot Order Via Dell RACADM Command.

 

First, you need to enable PXE on the NIC card. So, once you have identified the names of the network interfaces then you can enable PXE boot on the NIC port you want it to be.

 

Example:

# Get Nic config for integrated port 1
  racadm -r $ip -u root -p xxxxxxx get nic.nicconfig.1 | grep Legacy
# Set to PXE
  racadm -r $ip -u root -p xxxxxxx set nic.nicconfig.1.legacybootproto PXE
# Verify it's set to PXE (pending)
  racadm -r $ip -u root -p xxxxxxx get nic.nicconfig.1 | grep Legacy
# Create a job to enable the changes following the reboot
  racadm -r $ip -u root -p xxxxxxx jobqueue create NIC.Integrated.1-1-1
# reboot so that the configur job will execute
  ipmitool -I lanplus -H $ip -U root -P xxxxxxx chassis power reset

 

Finally, you would have to configure the BIOS boot order.

 

Example:

# Get Bios Boot sequence
  racadm -r $ip -u root -p xxxxxxx get BIOS.biosbootsettings.BootSeq | grep BootSeq
# Set Bios boot sequence
  racadm -r $ip -u root -p xxxxxxx set BIOS.biosbootsettings.BootSeq NIC.Integrated.1-1-1,NIC.Integrated.1-3-1,NIC.Slot.3-1-1,Optical.SATAEmbedded.J-1,HardDisk.List.1-1
# Create a BIOS reboot job so that the boot order changes are applied
  racadm -r $ip -u root -p xxxxxxx jobqueue create BIOS.Setup.1-1 -r pwrcycle -s TIME_NOW -e TIME_NA

Hope this helps or at least, give you any idea to fix your issue.

Regards.

 

 

No I am not using PXE boot and really have no interest to do so.

The fact that the iDrac shows 2 different orders of the Nic's has zero to do with the OS.

The differences in the Nic order presents itself with or without an OS enabled.

I have proven this in testing, as my OS is installed and loads from a USB drive, and during a trip to the Data Center last night I unplugged the OS drive, booted the server and looked at the drac and nothing had changed. 2 locations, 2 nic orders, the SERVER is reporting to the operating system the wrong nic orders.

In my opinion, this is a FLAW in the firmware, and PXE boot should not be needed to fix it. If there is no permanent fix without additional scripting (such as pxe booting) that tells the firmware to boot in the correct hardware order, I may have to face living with this flaw.This thing already has a slow enough boot time at 6.5 minutes before the nic pings. (this is also without a perc bios loading, as that would add another 45 seconds or more) Even the 710's boot in one third the time with raid and 5 times the amount of memory. (I think the boot speed is hindered by the drac software, not anything else hardware related)

However, I am personally hesitant to ever look at a R620 again, too bad since overall I used to think it was a great piece of equipment.

In this particular case, my work around is to always try and remember that bge0 and bge1 is actually port #3 and #4

Conclusion, the firmware detects and reports the correct order of the Nic's then the firmware decides all by itself that it will ignore it's own recommendation and boot the cards in a different order.

Hi universal4
I got the same issue. I installing 16 new PowerEdge R750. of the 16 servers, 4 have that same issue.

The order on the ports is reversed. Very frustrating, spend the last few day resolve this

Integrated NIC 1 Port 2-1  << 
Integrated NIC 1 Port 1-1

Integrated NIC 1 Port 1-1
Integrated NIC 1 Port 2-1

 

Embedded NIC 1: Broadcom Gigabit Ethernet BCM5720
Not connect or used (disabled)

Integrated NIC 1: Intel(R) Ethernet 10G 2P X710 OCP - 68:05:CA:EE:E4:E0
Port 1 connected

NIC Slot 6: Intel(R) Ethernet Converged Network Adapter X710 - 6C:FE:54:3E:C4:D0
Port 1 connected (disabled)

This order is correct of 12 of the 16 servers

 

The order should be the other way around.

This is to disable the other networks during the build:

RACADM set BIOS.IntegratedDevices.EmbNic1Nic2 DisabledOs
RACADM set BIOS.SlotDisablement.Slot6 Disabled
RACADM get BIOS.IntegratedDevices.IntegratedNetwork1
RACADM get BIOS.IntegratedDevices.EmbNic1Nic2
RACADM get BIOS.SlotDisablement.Slot6
racadm set Bios.BiosBootSettings.HddPlaceholder Enabled
racadm get Bios.BiosBootSettings.HddPlaceholder
racadm jobqueue create BIOS.Setup.1-1
racadm jobqueue view

If you found a way around this issue let me know

the interesting thing is that I tried the other NIC in slot 6.
The order is also incorrect. The lowest PCI/MAC is the wrong way again.
Seem like this is indeed a BOIS bug

 

Embedded NIC 1: Broadcom Gigabit Ethernet BCM5720
Not connect or used (disabled)

Integrated NIC 1: Intel(R) Ethernet 10G 2P X710 OCP - 68:05:CA:EE:E4:E0
Port 1 connected (disabled)

NIC Slot 6: Intel(R) Ethernet Converged Network Adapter X710 - 6C:FE:54:3E:C4:D0
Port 1 connected (enable)

Hey folks,

Did anyone get a resolution to this issue? If so, then what was the solution to reordering the PCIe NICs?