Connectrix MDS: EXT3-fs Error (device sdc1): ext3_readdir: directory #2 contains a hole at offset 0 - kernel
Summary: Connectrix MDS switch streaming the following error: %KERN-2-SYSTEM_MSG: [63661120.723386] EXT3-fs error (device sdc1): ext3_readdir: directory #2 contains a hole at offset 0 - kernel.
Symptoms
Connectrix MDS switch streaming the following error:
%KERN-2-SYSTEM_MSG: [63661120.723386] EXT3-fs error (device sdc1): ext3_readdir: directory #2 contains a hole at offset 0 - kernel
The below is seen in the show logging output:
2016 Jul 26 19:54:22 Switch Jul 26 19:54:21 %KERN-2-SYSTEM_MSG: [63661120.680361] EXT3-fs error (device sdc1): ext3_readdir: directory #2 contains a hole at offset 0 - kernel 2016 Jul 26 19:54:22 Switch Jul 26 19:54:21 %KERN-2-SYSTEM_MSG: [63661120.702907] EXT3-fs error (device sdc1): ext3_readdir: directory #2 contains a hole at offset 0 - kernel 2016 Jul 26 19:54:22 Switch Jul 26 19:54:21 %KERN-2-SYSTEM_MSG: [63661120.723386] EXT3-fs error (device sdc1): ext3_readdir: directory #2 contains a hole at offset 0 - kernel
The below is seen in the show system internal flash:
`show system internal flash` Mount-on 1K-blocks Used Available Use% Filesystem / 409600 65564 344036 17 /dev/root /proc 0 0 0 0 proc /sys 0 0 0 0 none /isan 1572864 539352 1033512 35 none /var 51200 460 50740 1 none /etc 5120 1624 3496 32 none /nxos/tmp 51200 336 50864 1 none /var/log 51200 8992 42208 18 none /var/home 5120 0 5120 0 none /var/tmp 307200 11628 295572 4 none /var/sysmgr 2097152 124 2097028 1 none /var/sysmgr/ftp 512000 51968 460032 11 none /var/sysmgr/srv_logs 102400 0 102400 0 none /var/sysmgr/ftp/debug_logs 10240 0 10240 0 none /dev/shm 1572864 234276 1338588 15 none /volatile 512000 12 511988 1 none /debug 2048 28 2020 2 none /dev/mqueue 0 0 0 0 none /debugfs 0 0 0 0 nodev /mnt/plog 242430 4458 225455 2 /dev/sdc1 /mnt/fwimg 121219 4127 110833 4 /dev/sdc3 /mnt/cfg/0 75797 4729 67154 7 /dev/sda5 /mnt/cfg/1 75797 4729 67154 7 /dev/sda6 /logflash 15392672 33064 14577684 1 /dev/sdb1 /bootflash 3653168 244900 3222696 8 /dev/sda3 /cgroup 0 0 0 0 vdccontrol /var/sysmgr/startup-cfg 409600 10272 399328 3 none /dev/pts 0 0 0 0 devpts /mnt/pss 38156 5482 30704 16 /dev/sda4 /bootflash_sup-remote 3653168 244896 3222704 8 127.1.1.6:/bootflash/ `show module` Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 2 48 2/4/8/10/16 Gbps Advanced FC Module DS-X9448-768K9 ok 3 48 2/4/8/10/16 Gbps Advanced FC Module DS-X9448-768K9 ok 4 48 2/4/8/10/16 Gbps Advanced FC Module DS-X9448-768K9 ok 5 0 Supervisor module-3 DS-X97-SF1-K9 active * 6 0 Supervisor module-3 DS-X97-SF1-K9 ha-standby `show system redundancy status` Redundancy mode --------------- administrative: HA operational: HA This supervisor (sup-1) ----------------------- Redundancy state: Active Supervisor state: Active Internal state: Active with HA standby Other supervisor (sup-2) ------------------------ Redundancy state: Standby Supervisor state: HA standby Internal state: HA standby
Cause
The flash storage device is not mounted as per Cisco issue "CSCuv67115."
This issue initially occurs only on supervisors in standby state. The issue is not corrected by an ISSU/D, system switchover, or reload, and is observed on a supervisor in an active state.
Logflash is used only to store the output of debug commands. It is not required for stable system operation.
Resolution
Workaround:
Power cycle the affected supervisor.
In dual-supervisor systems, this is done by powering down the supervisor with the out-of-service module command and powering it back up with the no poweroff module command. Ensure that the supervisor is in standby mode so that the recovery does not affect the stability of the system.
It is not sufficient to reboot the module with the reload or system switchover commands.
Example:
Commands:
-
Confirm that the standby supervisor is in HA Standby:
show system redundancy status
-
Power down the standby supervisor module:
out-of-service module <standby sup>
-
Power up the standby supervisor:
no poweroff <standby sup>
-
Wait for HA Standby:
show system redundancy status
Monitor the switch show logging output to see if the EXT3-fs error message is still logging on the switch. If the error is still logging, switchover to the standby supervisor and repeat steps 1-4 above.