N3024ET-ON ecc errors at startup

Hi Ian,

How many switches are having the issue? Were they all ordered at the same time? It shouldn’t be a problem since the ECC is doing its job.

Yes, every one to a varying degree.  They are off the same order.  I was wondering if there was an issue with vendor supply chain as the memory on these is on board and not replaceable dimms.  Thank you for taking time to look at this.   

I am not seeing any known issue, but if you do have any further problems let us know.

Please re-install OS6 via ONIE on the affected units.

Thank you for the help, installed OS6 (Ver 6.7.0.34) via ONIE, appeared ok on initial boot after installation, no errors showing.  However powered off switch and let sit for a few minutes and when I plugged back in and it powered up they were back.  see below. 

boot in 3 s
Creating 1 MTD partitions on "nand0":
0x000004e00000-0x000040000000 : "mtd=6"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   262144 bytes (256 KiB)
UBI: logical eraseblock size:    253952 bytes
UBI: smallest flash I/O unit:    4096
UBI: VID header offset:          4096 (aligned 4096)
UBI: data offset:                8192
8 ecc_errors after reading 15781000:0
8 ecc_errors after reading 19501000:0
8 ecc_errors after reading 19541000:0
8 ecc_errors after reading 19581000:0
8 ecc_errors after reading 19601000:0
8 ecc_errors after reading 19641000:0
8 ecc_errors after reading 19681000:0
8 ecc_errors after reading 196c1000:0
8 ecc_errors after reading 19701000:0
8 ecc_errors after reading 19741000:0
8 ecc_errors after reading 19781000:0
8 ecc_errors after reading 197c1000:0
8 ecc_errors after reading 19801000:0
8 ecc_errors after reading 19841000:0
8 ecc_errors after reading 198c1000:0
8 ecc_errors after reading 19901000:0
8 ecc_errors after reading 19941000:0
8 ecc_errors after reading 19981000:0
8 ecc_errors after reading 199c1000:0
8 ecc_errors after reading 3ff81000:0
8 ecc_errors after reading 3ffc1000:0
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=6"
UBI: MTD device size:            946 MiB
UBI: number of good PEBs:        3784
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     1
UBI: available PEBs:             43
UBI: total number of reserved PEBs: 3741
UBI: number of PEBs reserved for bad PEB handling: 37
UBI: max/mean erase counter: 331/3
160 ecc_errors after reading 31562000:0
160 ecc_errors after reading 315a2000:0
UBIFS: recovery needed
16 ecc_errors after reading 19a03000:0
UBIFS: recovery deferred
UBIFS: mounted UBI device 0, volume 0, name "open"
UBIFS: mounted read-only
UBIFS: file system size:   936067072 bytes (914128 KiB, 892 MiB, 3686 LEBs)
UBIFS: journal size:       33521664 bytes (32736 KiB, 31 MiB, 132 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: LZO
UBIFS: reserved for root:  5182151 bytes (5060 KiB)
32 ecc_errors after reading 31603000:0
8 ecc_errors after reading 39b3f000:0
Loading file '/image1' to addr 0x70000000 with size 36583024 (0x022e3670)... 

Hello @IanFLA,

I would like to recommend you to contact phone support for a level 2 escalation. As per your message, this is happening in several models at the same time. I think support will request you a log to check this and they probably will update to system engineer team in case they don't have this issue reported previously.

Regards.

Ok All, 

Apparently the issue is firmware driven that leads to an ECC Strength Mismatch.  When reviewing release notes under "issues resolved" on the current release there is a section that addressed this issue.   

Release 6.6.3.16 / 6.6.3.46

Summary:

Upgrade to 6.6.2.x, 6.6.3.x fails with ECC errors in N30xx-EP box with FLASH. [FIELD-6319, FIELD-6727]

User Impact:

N3000E-ON with certain flash type and/or HW Rev.6 exhibits ECC errors on bootup due to ECC strength mismatch

Resolution:

N3000E-ON with certain flash type and/or HW Rev.6 exhibits ECC errors on bootup due to ECC strength mismatch

• After downloading image version 6.6.3.46 to backup partition and entering "boot system backup” command from CLI, the recovery logic will look for the flash types MT29F32G08AB AAAWP, MT29F8G08ABA CA and/or HW Rev. 6, then perform filesystem recovery. For other HW variants, it will just set the FLAGS as needed in recovery logic.
• This recovery logic triggers on each unit (both N3000EON & N3132PXON) in the stack and it’s applicable to standalone unit as well
• The ECC recovery logic will set a flag “recovery_done” at U-boot and create a file “recovery_done” under /mnt/fastpath directory as well. This flag at uboot and file under /mnt/fastpath directory will be created as part of recovery process on N3000E-ON and N3132PX-ON switches
• The ECC recovery logic will set a flag “recovery_done” at U-boot and create a file “recovery_done” under /mnt/fastpath directory as well. This flag at uboot and file under /mnt/fastpath directory will be created as part of recovery process on N3000E-ON and N3132PX-ON switches
• The ECC recovery logic will set a flag “recovery_done” at U-boot and create a file “recovery_done” under /mnt/fastpath directory as well. This flag at uboot and file under /mnt/fastpath directory will be created as part of recovery process on N3000E-ON and N3132PX-ON switches
• Also it is recommended not to downgrade after upgrade to 6.6.3.16 as it may cause file system corruption with N3000E-ON revision 6 and above. Please look into downgrade Note section for details

Affected Platform:

N3000 E-ON & N3132 PXON

Hope this helps anyone else with this Issue. 

Ian