jbvernejoux

1 Message

2681

February 24th, 2017 02:00

MD3200i Dual Controler: 1rst RAID controler displayed in MDSM as REMOVED, after power shutdown

hello,

I have a powervault PowerVault MD3200i with dual RAID controler. It was bought 5 or 6 years ago, so no more under warranty.
Several weeks ago, for an electric powersupply maintenance in my datacenter, I had to shutdown properly all my systems (servers, racks, network, storage,...).
Once maintenance done, all systems were restarted.

Since this time, the storage device MD3200i is running with an alarm state:
RAID Controller Module in slot 0 is REMOVED

- Symptoms:
In MDSM:
The tool Modular Disk Storage Management client shows an empty slot for this controler in the backside view of the MD3200i and sees it as REMOVED even if the controler is in.

In my rack:
Physically, on the back of the device, this controler is in the slot, well powered (some LEDs are on) and connected to the network (leds blinking for management and iSCSI RJ45)
I can ping the management interface with its IP own address.
Currently, all the I/O are processed by the another RAID controler (in slot 1). Production workloads keep running but with lower performances.

- Things tried to fix:
* Stop / restart all the MD3200I ==> same pb / no change
* Eject / reinsert the faulty controler ==> same pb / no change

- Diags tried:
I plugged the serial cable on the faulty controler and watch its boot sequence (after remove/reinsert):

It doesn't seem that the controler is in the lockdown mode (as I can seen before in others forums), because it doesn't even boot: it hangs before starting the lockdown check sequence.
Here is a dump of a boot:
------

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number f10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number f10000
Label:" " ...

RTC Error: Real-time clock device is not working
OK.

Adding 14588 symbols for standalone.

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
7000 SE iSCSI BE2 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
02/22/17-15:51:05 (tSystem): ERROR: FPGA FW is out of date
"Rhone03 rev17" currently in use
"Rhone03 rev20" available for update
02/22/17-15:51:09 (tNetCfgInit): NOTE: eth0: LinkUp event
02/22/17-15:51:10 (tNetCfgInit): NOTE: Network Ready
Bad File CRC after decompression
FAILED - ERROR READING FILE (errno = 0x610001)

Kernel initialization complete
Current date: 02/22/17 time: 15:51:21

Send for Service Interface or baud rate change

-------

In this state, nothing appens, even after wait 30/60 minutes while watching the
I can press Ctrl BREAK key to see the Service Interface :
'Press within 5 seconds: for Service Interface, for baud rate'
But even if "S" key pressed, nothing appens; no menu displayed as expected.

I would try to reflash the controler again but I don't know if I can do it manually via serial cable, without using the MSDM tool.

How do I can do it, if it is possible ?

Do you have any other ideas to try to fix this faulty controler if this issue is really only software ?

Thank for any help.

PS: attached a partial SupportBundle (without trace-buffers.7z, because oversized) to this post.

1 Attachment

APXsupportbundle22-02-2017.zip

Responses(11)

S

spaxton

30 Posts

0

May 24th, 2021 11:00

Hello Friend,

I know it's been a long time since you wrote this but, have You ever resolved the issue? I have something similar. However, I know what You should do to enter some menu after pressing ctrl+break. You should try this:

Press Ctrl-Break
Instead of hitting S press Esc
You should get a vxWorks login:
Username: shellUsr
passsword: DF4m/2>

L

lukehy

61 Posts

0

June 2nd, 2021 08:00

was this resolved ?

i have the same issue :

Bad File CRC after decompression
FAILED - ERROR READING FILE (errno = 0x610001)

DELL-Josh Cr

Moderator

•

8.5K Posts

0

June 2nd, 2021 09:00

Hi lukehy,

It sounds like there is some corruption. Are you able to do anything during the boot process? Does it work off the other controller? Let us know if you have any additional questions.

L

lukehy

61 Posts

0

June 3rd, 2021 05:00

thank you for the reply.

Both controllers wont boot.
we tried to boot only with 1 same issue

L

lukehy

61 Posts

0

June 7th, 2021 06:00

Hello Josh.

this is from Controller 0:

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId , serial number
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13891 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.

Controller has been locked down due to PCI errors:

================= EXCEPTION LOG =================
Serial number: 48900BR
Entry count: 19
Wrap-arounds: 0
First entry time:
Current date/time: JUN-07-2021 09:47:04 AM

---- Log Entry #0 SEP-23-2014 09:09:03 PM ----

WARNING: Reset by alternate controller

---- Log Entry #1 SEP-23-2014 09:11:24 PM ----

WARNING: Reset by alternate controller

---- Log Entry #2 SEP-23-2014 09:15:28 PM ----

WARNING: Reset by alternate controller

---- Log Entry #3 SEP-24-2014 01:04:43 AM ----

WARNING: Reset by alternate controller

---- Log Entry #4 MAR-03-2015 11:39:28 PM ----

WARNING: Reset by alternate controller

---- Log Entry #5 MAR-04-2015 12:05:00 AM ----

WARNING: Reset by alternate controller

---- Log Entry #6 MAR-13-2015 07:32:50 PM ----

WARNING: Reset by alternate controller

---- Log Entry #7 MAR-13-2015 09:11:46 PM ----

WARNING: Reset by alternate controller

---- Log Entry #8 MAR-13-2015 10:56:10 PM ----

WARNING: Reset by alternate controller

---- Log Entry #9 MAR-26-2019 11:32:10 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00000001
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001081

---- Log Entry #10 MAR-26-2019 11:32:42 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

Outbound Completion Error Stat 0x00204800, Addr 0x00000000F0200000

PCI SERR Exception
Meteor PCI-E (Unit 0)
VID 0x1000 DID 0x0064 B255:D255:F255
PCI Status = 0x4010
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00040000
Header Log 0 = 0x30008000
Header Log 1 = 0x00000033
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x000011C1
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001001

---- Log Entry #11 JAN-02-2020 03:12:28 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001181
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001101

---- Log Entry #12 JAN-02-2020 03:12:56 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x000011C1

---- Log Entry #13 JAN-02-2020 03:13:24 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x000011C1
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00000001

---- Log Entry #14 MAY-25-2021 07:54:35 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001181
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001101

---- Log Entry #15 MAY-25-2021 08:24:23 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x000010C1
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001001

---- Log Entry #16 MAY-25-2021 08:37:38 PM ----

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 Pare ntBridge=PLX PCI-E Bridge to Host Card
REBOOTING.

---- Log Entry #17 MAY-25-2021 09:29:05 PM ----

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 Pare ntBridge=PLX PCI-E Bridge to Host Card
TRIGGERING LOCKDOWN.

PCI DEVICE NOT ACCESSIBLE: Host-side SAS Device=0 Function=0 ClassCode=0x10700 P arentBridge=PLX PCI-E Bridge to Host Card

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 Pare ntBridge=PLX PCI-E Bridge to Host Card
TRIGGERING LOCKDOWN.

PCI DEVICE NOT ACCESSIBLE: Host-side SAS Device=0 Function=0 ClassCode=0x10700 P arentBridge=PLX PCI-E Bridge to Host Card

---- Log Entry #18 MAY-25-2021 09:45:28 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
Meteor PCI-E (Unit 0)
VID 0x1000 DID 0x0064 B255:D255:F255
PCI Status = 0x4010
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00040000
Header Log 0 = 0x30008000
Header Log 1 = 0x00000033
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00000081
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001101

Current date: 06/07/21 time: 16:42:20

Send for Service Interface or baud rate change
06/07/21-09:47:08 (tNetCfgInit): NOTE: eth0: LinkUp event
06/07/21-09:47:08 (tNetCfgInit): NOTE: Acquiring network parameters for interface
06/07/21-09:47:12 (ipdhcpc): NOTE: netCfgDhcpReplyCallback :: received OFFER on
06/07/21-09:47:13 (ipdhcpc): WARN: **WARNING** The DHCP Server did not assign
06/07/21-09:47:13 (ipdhcpc): WARN: Network access to this controll
06/07/21-09:47:13 (ipdhcpc): NOTE: DHCP server:
06/07/21-09:47:13 (ipdhcpc): NOTE: Client IP routers:
06/07/21-09:47:13 (ipdhcpc): NOTE: Client DNS name servers:
06/07/21-09:47:13 (ipdhcpc): NOTE: DNS domain name:
06/07/21-09:47:13 (ipdhcpc): NOTE: Assigned IP address:
06/07/21-09:47:13 (ipdhcpc): NOTE: Assigned subnet mask:
06/07/21-09:47:13 (tNetReset): NOTE: Network Ready

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId , serial number
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13891 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
Bad File CRC after decompression
FAILED - ERROR READING FILE (errno = 0x610001)

Kernel initialization complete

Current date: 06/07/21 time: 01:35:54

Send for Service Interface or baud rate change
06/07/21-09:59:54 (ipdhcpc): NOTE: netCfgDhcpReplyCallback :: received OFFER on interface
06/07/21-09:59:55 (ipdhcpc): WARN: **WARNING** The DHCP Server did not assign a permanent IP
06/07/21-09:59:55 (ipdhcpc): WARN: Network access to this controller may eventually fail.
06/07/21-09:59:55 (ipdhcpc): NOTE: DHCP server:
06/07/21-09:59:55 (ipdhcpc): NOTE: Client IP routers:
06/07/21-09:59:55 (ipdhcpc): NOTE: Client DNS name servers:
06/07/21-09:59:55 (ipdhcpc): NOTE: DNS domain name:
06/07/21-09:59:55 (ipdhcpc): NOTE: Assigned IP address:
06/07/21-09:59:55 (ipdhcpc): NOTE: Assigned subnet mask:
06/07/21-09:59:55 (tNetReset): NOTE: Network Ready

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId , serial number
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13891 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick
-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId , serial number
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13891 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
Bad File CRC after decompression
FAILED - ERROR READING FILE (errno = 0x610001)

Kernel initialization complete

Current date: 06/07/21 time: 02:22:12

Send for Service Interface or baud rate change
^Z

<------------------->

D)isplay R)etry C)ontinue M)onitor Shell A)bort ?

D)isplay R)etry C)ontinue M)onitor Shell A)bort ?
6D50 LSISAS2008 IOC 1
02 Register Address Lines Test Passed
03 Register Data Lines Test 06/07/21-10:55:25 (tDiag): ERROR: bcmPciCfgWrite error for bus 2, dev 0, func 0, reg 0x11, size 1. Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.

Controller has been locked down due to PCI errors:

================= EXCEPTION LOG =================
Serial number:
Entry count: 20
Wrap-arounds: 0
First entry time:
Current date/time: JUN-07-2021 10:55:25 AM

---- Log Entry #0 SEP-23-2014 09:09:03 PM ----

WARNING: Reset by alternate controller

---- Log Entry #1 SEP-23-2014 09:11:24 PM ----

WARNING: Reset by alternate controller

---- Log Entry #2 SEP-23-2014 09:15:28 PM ----

WARNING: Reset by alternate controller

---- Log Entry #3 SEP-24-2014 01:04:43 AM ----

WARNING: Reset by alternate controller

---- Log Entry #4 MAR-03-2015 11:39:28 PM ----

WARNING: Reset by alternate controller

---- Log Entry #5 MAR-04-2015 12:05:00 AM ----

WARNING: Reset by alternate controller

---- Log Entry #6 MAR-13-2015 07:32:50 PM ----

WARNING: Reset by alternate controller

---- Log Entry #7 MAR-13-2015 09:11:46 PM ----

WARNING: Reset by alternate controller

---- Log Entry #8 MAR-13-2015 10:56:10 PM ----

WARNING: Reset by alternate controller

---- Log Entry #9 MAR-26-2019 11:32:10 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00000001
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001081

---- Log Entry #10 MAR-26-2019 11:32:42 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

Outbound Completion Error Stat 0x00204800, Addr 0x00000000F0200000

PCI SERR Exception
Meteor PCI-E (Unit 0)
VID 0x1000 DID 0x0064 B255:D255:F255
PCI Status = 0x4010
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00040000
Header Log 0 = 0x30008000
Header Log 1 = 0x00000033
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x000011C1
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001001

---- Log Entry #11 JAN-02-2020 03:12:28 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001181
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001101

---- Log Entry #12 JAN-02-2020 03:12:56 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x000011C1

---- Log Entry #13 JAN-02-2020 03:13:24 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x000011C1
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00000001

---- Log Entry #14 MAY-25-2021 07:54:35 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001181
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001101

---- Log Entry #15 MAY-25-2021 08:24:23 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
PCIe Link Width is 0
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x000010C1
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001001

---- Log Entry #16 MAY-25-2021 08:37:38 PM ----

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card
REBOOTING.

---- Log Entry #17 MAY-25-2021 09:29:05 PM ----

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card
TRIGGERING LOCKDOWN.

PCI DEVICE NOT ACCESSIBLE: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card
TRIGGERING LOCKDOWN.

PCI DEVICE NOT ACCESSIBLE: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card

---- Log Entry #18 MAY-25-2021 09:45:28 PM ----

Root Complex TLP header[0] 30008000
Root Complex TLP header[1] 01200033
Root Complex TLP header[2] 00000000
Root Complex TLP header[3] 00000000

PCI SERR Exception
Meteor PCI-E (Unit 0)
VID 0x1000 DID 0x0064 B255:D255:F255
PCI Status = 0x4010
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00004000
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Switch (Unit 0)
VID 0x10b5 DID 0x8632 B0:D0:F0
PCI Status = 0x4010
Bridge Secondary PCI Status = 0x4000
PCI-E Device Status = 0x0004
PCI-E AER Uncorrectable Status = 0x00040000
Header Log 0 = 0x30008000
Header Log 1 = 0x00000033
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PLX PCI-E Bridge to Host Card (Unit 1)
VID 0x10b5 DID 0x8632 B1:D4:F0
PCI Status = 0x4010
PCI-E Device Status = 0x0005
PCI-E AER Uncorrectable Status = 0x00000020
Header Log 0 = 0x00000000
Header Log 1 = 0x00000000
Header Log 2 = 0x00000000
Header Log 3 = 0x00000000
PCI-E AER Correctable Status = 0x00001100
Host-side SAS (Unit 0)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00000081
Host-side SAS (Unit 1)
VID 0x1000 DID 0x0072 B2:D0:F0
PCI-E Device Status = 0x0001
PCI-E AER Correctable Status = 0x00001101

---- Log Entry #19 JUN-07-2021 05:42:40 PM ----

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card
TRIGGERING LOCKDOWN.

PCI DEVICE NOT ACCESSIBLE: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card
ERROR: PLX NTB Port 4 Bad DLLP Count 4779 exceeds threshold 16

Current date: 06/07/21 time: 17:50:41

Send for Service Interface or baud rate change
06/07/21-10:55:30 (tNetCfgInit): NOTE: eth0: LinkUp event
06/07/21-10:55:30 (tNetCfgInit): NOTE: Acquiring network parameters for interface
06/07/21-10:55:40 (ipdhcpc): NOTE: netCfgDhcpReplyCallback :: received OFFER on interface , unit 0
06/07/21-10:55:41 (ipdhcpc): WARN: **WARNING** The DHCP Server did not assign a permanent IP
06/07/21-10:55:41 (ipdhcpc): WARN: Network access to this controller may eventually fail.
06/07/21-10:55:41 (ipdhcpc): NOTE: DHCP server:
06/07/21-10:55:41 (ipdhcpc): NOTE: Client IP routers:
06/07/21-10:55:41 (ipdhcpc): NOTE: Client DNS name servers:
06/07/21-10:55:41 (ipdhcpc): NOTE: DNS domain name:
06/07/21-10:55:41 (ipdhcpc): NOTE: Assigned IP address:
06/07/21-10:55:41 (ipdhcpc): NOTE: Assigned subnet mask:
06/07/21-10:55:41 (tNetReset): NOTE: Network Ready

Controller 1:

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId , serial number
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13891 symbols for standalone.

PCI DEVICE 0 NOT FOUND: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card
TRIGGERING LOCKDOWN.

PCI DEVICE NOT ACCESSIBLE: Host-side SAS Device=0 Function=0 ClassCode=0x10700 ParentBridge=PLX PCI-E Bridge to Host Card
Length: 0x13c Bytes
Version ver03.0A

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines ERROR: PLX NTB Port 4 Bad DLLP Count 4779 exceeds threshold 16 Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test ** Failed **

*** Error (0001): Register data miscompare

Address Expected Observed Info
0004 46 40 Host-side SAS

*** Current Loop Count: 1 Date: 06/07/2021 Time: 10:47:27

*** Master Error Code: 2660-6D50-01-0001

D)isplay R)etry C)ontinue M)onitor Shell A)bort ?

DELL-Chris H

Moderator

•

8.4K Posts

0

June 7th, 2021 10:00

Lukehy,

The controller is in a lockdown state per the serial capture that you posted, so to start you will need to clear the lockdown with vxshellusr.

Let me know how it goes.

L

lukehy

61 Posts

0

June 7th, 2021 12:00

Hello Chris,

are these the steps to take it out of lockdown?

Do Ctrl +A, Ctrl + B, hit escape

VXlogin: shellUsr

password: DF4m/2>

Run: lemClearLockdown

DELL-Josh Cr

Moderator

•

8.5K Posts

0

June 7th, 2021 12:00

That’s correct. Those are the right commands.

L

lukehy

61 Posts

0

June 15th, 2021 01:00

Hello josh.

tried that command but got :

C interp: unknown symbol name 'lemClearLockdown'

DiegoLopez

4 Operator

•

2.7K Posts

1

June 15th, 2021 04:00

Hello @lukehy,

Could you please, share a screenshot of the serial client? Do this error appear after login or before?

Regards.

L

lukehy

61 Posts

0

June 21st, 2021 13:00

Hello @DiegoLopez .

i have replaced the controller and it booted correctly.

same will be done to the other controller.

View All

No Events found!