Skip to main content
Start a Conversation

Unsolved

This post is more than 5 years old

T

TJaszo

5 Posts

37430

July 12th, 2007 12:00

DELL PowerEdge servers with 4GB RAM, but OS can't use all of this memory.

Description of problem: We have few DELL PowerEdge servers with 4GB RAM, but OS can't use all of this memory. PowerEdge 6650 -------------------- 6650_1:~> free total Mem: 3375272 6650_1:~> lspci 00:00.0 Host bridge: Broadcom CMIC-HE (rev 22) 00:00.1 Host bridge: Broadcom CMIC-HE 00:00.2 Host bridge: Broadcom CMIC-HE 00:00.3 Host bridge: Broadcom CMIC-HE 00:04.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 00:05.0 Class ff00: Dell Remote Access Card III 00:05.1 Class ff00: Dell Remote Access Card III 00:05.2 Class ff00: Dell Remote Access Card III: BMC/SMIC device not present 00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93) 00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93) 00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05) 00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge 00:10.0 Host bridge: Broadcom CIOB30 (rev 03) 00:10.2 Host bridge: Broadcom CIOB30 (rev 03) 00:11.0 Host bridge: Broadcom CIOB30 (rev 03) 00:11.2 Host bridge: Broadcom CIOB30 (rev 03) 00:12.0 Host bridge: Broadcom CIOB30 (rev 03) 00:12.2 Host bridge: Broadcom CIOB30 (rev 03) 03:01.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge 04:00.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge 04:01.0 SCSI storage controller: QLogic Corp. ISP12160 Dual Channel Ultra3 SCSI Processor (rev 06) 05:00.0 RAID bus controller: American Megatrends Inc. MegaRAID (rev 20) 0a:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14) 0a:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14) 15:01.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge 16:00.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge 16:01.0 SCSI storage controller: QLogic Corp. ISP12160 Dual Channel Ultra3 SCSI Processor (rev 06) 17:00.0 RAID bus controller: American Megatrends Inc. MegaRAID (rev 20) -------------------- 6650_2 :~> free total Mem: 3635360 6650_2:~> lspci 00:00.0 Host bridge: Broadcom CMIC-HE (rev 22) 00:00.1 Host bridge: Broadcom CMIC-HE 00:00.2 Host bridge: Broadcom CMIC-HE 00:00.3 Host bridge: Broadcom CMIC-HE 00:04.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 00:05.0 Class ff00: Dell Remote Access Card III 00:05.1 Class ff00: Dell Remote Access Card III 00:05.2 Class ff00: Dell Remote Access Card III: BMC/SMIC device not present 00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93) 00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93) 00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge 00:10.0 Host bridge: Broadcom CIOB30 (rev 03) 00:10.2 Host bridge: Broadcom CIOB30 (rev 03) 00:11.0 Host bridge: Broadcom CIOB30 (rev 03) 00:11.2 Host bridge: Broadcom CIOB30 (rev 03) 00:12.0 Host bridge: Broadcom CIOB30 (rev 03) 00:12.2 Host bridge: Broadcom CIOB30 (rev 03) 03:01.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge 04:00.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge 04:01.0 SCSI storage controller: QLogic Corp. ISP12160 Dual Channel Ultra3 SCSI Processor (rev 06) 05:00.0 RAID bus controller: American Megatrends Inc. MegaRAID (rev 20) 0a:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14) 0a:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14) PowerEdge 2560 -------------------- :~> free total Mem: 3895444 :~> lspci 00:00.0 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset) (rev 13) 00:00.1 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset) 00:00.2 Host bridge: Broadcom CMIC-LE 00:04.0 Class ff00: Dell Embedded Remote Access or ERA/O 00:04.1 Class ff00: Dell Remote Access Card III 00:04.2 Class ff00: Dell Embedded Remote Access: BMC/SMIC device 00:0e.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93) 00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93) 00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05) 00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge 00:10.0 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03) 00:10.2 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03) 00:11.0 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03) 00:11.2 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03) 03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) 03:08.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) 04:08.0 PCI bridge: Intel Corporation 80303 I/O Processor PCI-to-PCI Bridge (rev 01) 04:08.1 RAID bus controller: Dell PowerEdge Expandable RAID Controller 3/Di (rev 01) Version-Release number of selected component: all servers listed above uses RH4U4 with 2.6.9-42.ELsmp #1 SMP Sat Aug 12 09:39:11 CDT 2006 i686 i686 i386 GNU/Linux but we've checked amount of memory using memtest86 and shows the same amount of memory as stated above Actual results: non all of memory can be used by system Expected results: all memory should be accessible by system Additional info: When i removed some unused PCIX cards I freed 256MB of memory (see difference between 6650_1 and 6650_2) I've found some hardware related info: Server Products Not All Memory is Available after Installing 4GB or More of System Memory (http://support.intel.com/support/motherboards/server/sb/CS-010458.htm) Not All Physical Memory May Be Reported By The Operating System On Certain HP ProLiant Servers http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=PSD_EL041214_CW01 but only info at DELL sites that i've found suggests to use /3GB and PAE switch with Windows.... and we are using RHEL 4U4, Centos4 and other linux like :) btw. DELL PE1850 doesn't have this problem 1850:~> free total Mem: 4147184

TJaszo

5 Posts

July 12th, 2007 12:00

(sorry for above, just didn't thought that i have to use html coding in here...)

Description of problem:
We have few DELL PowerEdge servers with 4GB RAM, but OS can't use all of this
memory.

PowerEdge 6650
--------------------
6650_1:~> free
total
Mem: 3375272

6650_1:~> lspci
00:00.0 Host bridge: Broadcom CMIC-HE (rev 22)
00:00.1 Host bridge: Broadcom CMIC-HE
00:00.2 Host bridge: Broadcom CMIC-HE
00:00.3 Host bridge: Broadcom CMIC-HE
00:04.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:05.0 Class ff00: Dell Remote Access Card III
00:05.1 Class ff00: Dell Remote Access Card III
00:05.2 Class ff00: Dell Remote Access Card III: BMC/SMIC device not present
00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93)
00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93)
00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05)
00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge
00:10.0 Host bridge: Broadcom CIOB30 (rev 03)
00:10.2 Host bridge: Broadcom CIOB30 (rev 03)
00:11.0 Host bridge: Broadcom CIOB30 (rev 03)
00:11.2 Host bridge: Broadcom CIOB30 (rev 03)
00:12.0 Host bridge: Broadcom CIOB30 (rev 03)
00:12.2 Host bridge: Broadcom CIOB30 (rev 03)
03:01.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge
04:00.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge
04:01.0 SCSI storage controller: QLogic Corp. ISP12160 Dual Channel Ultra3 SCSI
Processor (rev 06)
05:00.0 RAID bus controller: American Megatrends Inc. MegaRAID (rev 20)
0a:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit
Ethernet (rev 14)
0a:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit
Ethernet (rev 14)
15:01.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge
16:00.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge
16:01.0 SCSI storage controller: QLogic Corp. ISP12160 Dual Channel Ultra3 SCSI
Processor (rev 06)
17:00.0 RAID bus controller: American Megatrends Inc. MegaRAID (rev 20)


--------------------
6650_2 :~> free
total
Mem: 3635360

6650_2:~> lspci
00:00.0 Host bridge: Broadcom CMIC-HE (rev 22)
00:00.1 Host bridge: Broadcom CMIC-HE
00:00.2 Host bridge: Broadcom CMIC-HE
00:00.3 Host bridge: Broadcom CMIC-HE
00:04.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:05.0 Class ff00: Dell Remote Access Card III
00:05.1 Class ff00: Dell Remote Access Card III
00:05.2 Class ff00: Dell Remote Access Card III: BMC/SMIC device not present
00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93)
00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93)
00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge
00:10.0 Host bridge: Broadcom CIOB30 (rev 03)
00:10.2 Host bridge: Broadcom CIOB30 (rev 03)
00:11.0 Host bridge: Broadcom CIOB30 (rev 03)
00:11.2 Host bridge: Broadcom CIOB30 (rev 03)
00:12.0 Host bridge: Broadcom CIOB30 (rev 03)
00:12.2 Host bridge: Broadcom CIOB30 (rev 03)
03:01.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge
04:00.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge
04:01.0 SCSI storage controller: QLogic Corp. ISP12160 Dual Channel Ultra3 SCSI
Processor (rev 06)
05:00.0 RAID bus controller: American Megatrends Inc. MegaRAID (rev 20)
0a:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit
Ethernet (rev 14)
0a:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit
Ethernet (rev 14)


PowerEdge 2560
--------------------
:~> free
total
Mem: 3895444

:~> lspci
00:00.0 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset) (rev 13)
00:00.1 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset)
00:00.2 Host bridge: Broadcom CMIC-LE
00:04.0 Class ff00: Dell Embedded Remote Access or ERA/O
00:04.1 Class ff00: Dell Remote Access Card III
00:04.2 Class ff00: Dell Embedded Remote Access: BMC/SMIC device
00:0e.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93)
00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93)
00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05)
00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge
00:10.0 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
00:10.2 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
00:11.0 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
00:11.2 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit
Ethernet (rev 15)
03:08.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit
Ethernet (rev 15)
04:08.0 PCI bridge: Intel Corporation 80303 I/O Processor PCI-to-PCI Bridge (rev 01)
04:08.1 RAID bus controller: Dell PowerEdge Expandable RAID Controller 3/Di (rev 01)



Version-Release number of selected component:
all servers listed above uses RH4U4 with
2.6.9-42.ELsmp #1 SMP Sat Aug 12 09:39:11 CDT 2006 i686 i686 i386 GNU/Linux
but we've checked amount of memory using memtest86 and shows the same amount of memory as stated above


Actual results:
non all of memory can be used by system

Expected results:
all memory should be accessible by system

Additional info:
When i removed some unused PCIX cards I freed 256MB of memory (see
difference between 6650_1 and 6650_2)

I've found some hardware related info:
Server Products
Not All Memory is Available after Installing 4GB or More of System Memory
(http://support.intel.com/support/motherboards/server/sb/CS-010458.htm)

Not All Physical Memory May Be Reported By The Operating System On Certain HP
ProLiant Servers
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=PSD_EL041214_CW01

but only info at DELL sites that i've found suggests to use /3GB and PAE switch
with Windows.... and we are using RHEL 4U4, Centos4 and other linux like :)


btw. DELL PE1850 doesn't have this problem

1850:~> free
total
Mem: 4147184

mmainprize

260 Posts

July 18th, 2007 04:00

You did not state what OS and service pack you are running.
 
But you can only access 3gig of memory with a 32bit OS or a 32bit CPU address bus.
 
You need to be running Win 2003 Enterprize SP1 or datacenter at least because 2003 standard dose not support over 4gig. Win 2000 server SP4 to support the chipset for the larger memory addresses.

TJaszo

5 Posts

July 18th, 2007 05:00

hmm...

"
Version-Release number of selected component:
all servers listed above uses RH4U4 with
2.6.9-42.ELsmp #1 SMP Sat Aug 12 09:39:11 CDT 2006 i686 i686 i386 GNU/Linux
"

And with exactly the same version of linux installed on PowerEdge 1850 shows all 4GB of RAM... so it's rather not OS issue.

"
DELL PE1850 doesn't have this problem

1850:~> free
total
Mem: 4147184
"

mmainprize

260 Posts

July 19th, 2007 04:00

Sorry, i am not up to date with Linux versions
 
I know for a fact that the PE18xx series servers support > 4 gig memory and addressing,
 
the 6650 specs are

Memory

Architecture

72-bit ECC PC-1600 compliant, DDR SDRAM registered DIMMs, with 4-way interleaving

Memory module sockets

sixteen 72-bit wide, 168-pin DIMM sockets on two memory riser cards

Memory module capacities

128-, 256-, 512-MB, or 1-GB registered SDRAM DIMMs

Minimum RAM

512 MB

Maximum RAM

16 GB

 
So it shoud support 4 or more gig, but it did not list that version of linux so maybe it is a chipset driver that you need for that linux version.


Message Edited by mmainprize on 07-19-2007 12:35 AM

Message Edited by mmainprize on 07-19-2007 12:36 AM

kghammond

20 Posts

July 27th, 2007 05:00

As I understand it, in the Windows world, without PAE, you can only access some amount of RAM under 4 GB with 32 bit windows.  It has to do with the memory mapped to PCIx, PCIe and other hardware devices.  That is why you noticed the 256MB change when you removed a PCIx device.
 
Depending on the bios and motherboard support, the OS support, you can access all the 4 GB and more with PAE and if the motherboard supports memory re-mapping (or something similar to this).
 
If you are using just standard Windows without PAE though, you will never be able to access all 4GB of RAM.  This is the same for Windows XP 32 bit with 4 GB of RAM.
 
64 bit versions of Windows differ depending on the motherboard and bios.
 
I hope some of this helps.
 
Kevin

TJaszo

5 Posts

July 27th, 2007 11:00

arhhhhhhhhhhhh.... I'm not using Windows....


sorry, I know that all of You want to help, but please read question carefully...


I ask again:

does any one of You know how to handle memory problem in PE2650 and PE6650 in configuration stated above under linux system (RHEL4U4)


I know that i can buy Windows ... and use some magic tricks, but I just can't... We are using RHEL in very stable version with other software and we just added 2GB of RAM to improve performance... and it appeared that instead of 4GB we can use less...

we tried some software tricks, but i didn't helped. I guess that maybe there is some hardware (bios?) tricks that we can try... So thats why I'm asking about it on _this_ forum...


thanks
--
Tomasz


I wish all of You happy SysAdmin Day ;)

TJaszo

5 Posts

July 27th, 2007 12:00

Ok, we are using kernel with PAE, identical OS installed on PE1850 shows 4GB of memory...

Intel and HP in similar case advices to upgrade bios - but in Dell BIOS change log I do not see any update regarding memory remapping...

so maybe there is some switch in BIOS config that we are missing... :/

kghammond

20 Posts

July 27th, 2007 12:00

Well the answer is still the same.  Without using some technique such as PAE to remap memory address space, a portion of the 4 GB of RAM is reserved for your devices PCIx, PCIe, etc.
 
Windows gets around this with PAE.  I am not sure what Linux does to get around this hardware issu.  Also from what I have read before, your hardware (The DELL PowerEdge) needs to support memory remapping as well to allow the different OS techniques to reclaim that memory.
 
64 bit solutions are different.  I did not pay as much attention to the details as I have been reading.

View More

View All

No Events found!

Top