Broadcom NIC Gigabit connection

Back again.

I forgot to mention previously that we have been having problems with one of our PE6650 servers in that it's Broadcom NIC connection just turns off for no particular reason. I have read other people have had similar issues with PE2650 servers with the connection just going to sleep.

We logged a call with Dell about this particular problem and they asked use to uninstall and then reinstall both the drivers and management software again, which we have done but the problem still occurs. One thing I have noticed with this particular machine is that the Output Queue Length counter in Perfmon is extremely high (~45000000) and data transfer is very slow, it's lucky to pump-out anymore then 200Mb-300Mb /min even though it is configured and connected to a Gigabit switch.

Other strange things I've seen in Perfmon is a very large amount of Page Faults/sec and Page Reads/sec, even though the server has not memory issues as it has 2Gb of memory and about 900Mb is always free, the page file is hardly touched and is set at 3500Mb. The server is SAN attached also.

Anyone else seen similar problems on their servers where they have had Broadcom NIC issues?

Interesting results......have you checked with your network guys on the interface stats of the Cisco Cat 6500 ?

show mac x/y  ................where x is the module and y is the specifc port on that module for your servers, look for errors, duplx or discards, this will indicate a potential mismatch of server nic to swithc config

show port x/y    to check its configured as you think

Are you using "Jumbo frames" on your Gig Nic - this allows frames to be  9216 bytes in length rather than 1480, and saves mutilple TCP fragments for file transfer apps

show port jumbo ..............will tell you if this enable or not

set port jumbo x/y...........will configure it on port x/y

 

Hi Sentinel,

Thanks for the reply. I got one of our network guys to look at those stats for the port as suggested and got the following results:

---

XXXXXXXXS01> (enable) sh mac 8/4
 
Port     Rcv-Unicast          Rcv-Multicast        Rcv-Broadcast
-------- -------------------- -------------------- --------------------
 8/4              56197811659                    0               772094
 
Port     Xmit-Unicast         Xmit-Multicast       Xmit-Broadcast
-------- -------------------- -------------------- --------------------
 8/4              12884873856            137809440             22145841
 
Port     Rcv-Octet            Xmit-Octet
-------- -------------------- --------------------
 8/4           82884286506563         953843487723
 
MAC      Dely-Exced MTU-Exced  In-Discard Out-Discard
-------- ---------- ---------- ---------- -----------
 8/4              0          0          0           2
 
Last-Time-Cleared
--------------------------
Sun Sep 22 2002, 10:39:29
XXXXXXXXS01> (enable) sh po 8/4
Port  Name               Status     Vlan       Duplex Speed Type
----- ------------------ ---------- ---------- ------ ----- ------------
 8/4  SDBO5211           connected  112          full  1000 1000BaseT
 
 
Port  Security Violation Shutdown-Time Age-Time Max-Addr Trap     IfIndex
----- -------- --------- ------------- -------- -------- -------- -------
 8/4  disabled  shutdown             0        0        1 disabled     218
 
Port  Num-Addr Secure-Src-Addr   Age-Left Last-Src-Addr     Shutdown/Time-Left
----- -------- ----------------- -------- ----------------- ------------------
 8/4         0                 -        -                 -        -         -
 
Port     Broadcast-Limit Broadcast-Drop
-------- --------------- --------------------
 8/4                   -                    0
 
Port   Send FlowControl    Receive FlowControl   RxPause TxPause Unsupported
       admin    oper       admin    oper                         opcodes
-----  -------- --------   -------- --------     ------- ------- -----------
 8/4   desired  off        off      off          0       0       0
 
Port  Status     Channel              Admin Ch
                 Mode                 Group Id
----- ---------- -------------------- ----- -----
 8/4  connected  off                    209     0
 
Port  Align-Err  FCS-Err    Xmit-Err   Rcv-Err    UnderSize
----- ---------- ---------- ---------- ---------- ---------
 8/4           0          0          0          0         0
 
Port  Single-Col Multi-Coll Late-Coll  Excess-Col Carri-Sen Runts     Giants
----- ---------- ---------- ---------- ---------- --------- --------- ---------
 8/4           0          0          0          0         0         0         0
 
Last-Time-Cleared
--------------------------
Sun Sep 22 2002, 10:39:29
XXXXXXXXS01> (enable) sh port jumbo
Jumbo frames MTU size is 9216 bytes.
No ports configured to allow jumbo frames.
XXXXXXXXS01> (enable) exit

---

Nothing seems too out of the ordinary to me, there doesn't seem to be a issue with lots of errors or discards happening or anything else of that nature, unless there is something there you see differently?.

We do have jumbo frames disabled though and have not yet enabled it. I read that it is best to turn on jumbo frames for all ports and not select ports or else problems might occur, but I might talk further with on of our Network gurus and ask him about it.

Still at my wits end though about troubleshooting the performance issue but it hasn't dropped the connection again since we moved the network connection to the 2nd Broadcom NIC port on the server.

I am having about the same problem, slow performance between various servers using Gigabit NIC cards into a cisco switch.

My network guy has enabled and disabled jumbo frames onthe Cisco switch with no change in symptoms. Drivers have been updated on all the various Dell machines to the latest available from Dell.

We are at a loss, one of the guys even has a sniffer plugged into the wire to see if he can see anything wierd going on.

 

What does DELL Support say about all this?

can you try to set your Cisco switch ports to Auto/Auto (speed and dux) and test again. BTW, in IEEE standard, gigabit is possible only with autonegatiation on.

Because you use a simple file copy between 2 servers, you probably want to set every partner to Auto (i.e. 2 NICs and 2 switch ports).

If... for some reason.... it wont help with performance, then, set all to 100 Half Duplex and test again.

Don't waste your time with half-duplex - you have the most powerful multilayer switch that exists a Cat6500  - it can deal with wire-rate speeds and switches were built for full-duplex environments, since a switch port has dedicated port buffering per ASIC.

Have you tried using other reference test software - in case its a software issue (i.e can't use TCP windowing)

Try using TFTP file transfers between the two servers to check, in case its your filecopy s/w ( reminds me of the days of Novell and IPX bust mode.........ahh but thats another era and in the past )

Also check with you network guys if the file copy is within the same VLAN or being routed (VLAN x to VLAN Y), and also check whether any ingress or egress rate-limiting is turned on the ports with access-control lists. (The Cat6509 can do access-list control at wire rate too)

Well, I went and did it. I went and fired up PERFMON on the server in question and found an interesting bit of information.

The OUTPUT QUEUE LENGTH counter shows I have 240+ packets waiting to depart the system via the Broadcom NIC.

This flys in the face of the 2 or less at all times recommendations from Microsoft and others.   The question still remains:  "Why".

 

JoMat.

I have several PE2650's that will drop there connections for no reason and can not reestablish there connection until I disable\enable the NIC or restart the server.  To my knowledge I have the most recent driver.  This has been happening on my SQL and Print servers on a regular basis and am now thinking about switching to Intel Pro/1000 MT adapters as downtime is becoming problematic. 

Any suggestions to help save me $$$$ or would switching to the Intel's be my best bet.

John,

Thanks.  There's no slot for new pci card.  Emulex cards filled up all available slots.  I guessed I will install the latest Dell Broadcom driver and see what'll happen. 

The short answer for this is NO.  The better answer is; contact Dell support and see if they will send you Intel cards. Speak nicely to them and they may send them to you at no charge.

 

John M.

Anyone find the solution for this.  This is the problem we currently having with 4 PE 2650 servers.

On the issue of the NIC losing connection, you have disabled the power saver feature of this??? I have seen the Broadcom lose connection but disabling this stopped it...

We had major problems when network load would reach a certain level.  Dell, Microsoft and our network engineers could not resolve - we went with the base 100 Intel cards and that is the workaround that we have in place today.  We spent many many hours going nowhere.

Where do you disable the power save feature for the Broadcom NICs.  I have two Broadcom NetXtreme Gigabit NICs in a PE6650 and one keeps losing connectivity.  Thanks for your help.