Highlighted
mackendw
1 Nickel

VNX 5300 in a clustered setup

In an clustered setup (RHEL 6), using a VNX 5300 as the storage for a bunch of EXT4 filesystems.  The cluster is setup ACTIVE/PASSIVE where only one node is using the filesystems at a time.

The WRITE CACHE is enabled on the VNX.

Having issues when unmounting the filesystems on the ACTIVE node and then going to the other node and mounting the filesystems.  Getting EXT4-fs errors in the /var/log/messages about deleted inodes, etc.  The active system typically is up for 1 week using the filesystem data which contains mostly small files (many) and a bunch of ISAM files.

Could the CACHE on the VNX be holding pending updates to the storage that the formerly passive node is not seeing when it attempts the mount operations?

Tags (2)
0 Kudos
14 Replies
Rainer_EMC
5 Rhenium

Re: VNX 5300 in a clustered setup

Not likely on the VNX cache

Check if you are properly flushing the Linux buffer cache

0 Kudos
mackendw
1 Nickel

Re: VNX 5300 in a clustered setup

The buffers held by the OS in the filesystem *should* be flushed with the unmount of the filesystem...one would think.

Does the WRITE CACHE get flushed to physical disk with the unmounting of the filesystems on the first node?

What we are seeing is the 2nd node is getting an older / incomplete view of the actual data on the SAN filesystems.

I could do a sync to flush the filesystems before they are unmounted.....but I think that would be un-needed.  Is the /proc/sys/vm/drop_caches also required to be set to 2 after syncing to purge directory and file inodes?

0 Kudos
Rainer_EMC
5 Rhenium

Re: VNX 5300 in a clustered setup

The VNX write cache doesnt have to get flushed since any other host accessing a block would just automatically the the data from the write cache

Did you have a look at the EMC Linux host configuration guide and the knowledgebase ?

If you are using a volume manager you should also check if its cluster capable

0 Kudos
mackendw
1 Nickel

Re: VNX 5300 in a clustered setup

"Did you have a look at the EMC Linux host configuration guide and the knowledgebase ?"

I'm just the consultant trying to pick up the pieces and get this thing working again...;)  Can you send me a link to both pls.

No volume manager is being used.  All filesystems on the SAN (24 of them) are EXT4 filesystems.

0 Kudos
mackendw
1 Nickel

Re: VNX 5300 in a clustered setup

Note:  I contacted RedHat support and they suggested upgrading the SAN firmware. 

The VNX5300 is running version 5.31 firmware if that helps.  Comments are welcome.

see:

ext4 file system corrupted with ext4_mb_generate_buddy messages seen in the logs - Red Hat Customer ...

0 Kudos
Rainer_EMC
5 Rhenium

Re: VNX 5300 in a clustered setup

An upgrade to a recent VNX OE is always a good idea

5.31 isnt the full version number – which one are you running exactly ?

0 Kudos
Rainer_EMC
5 Rhenium

Re: VNX 5300 in a clustered setup

Just see support.emc.com – there is a link for kb search and a search box to find the host config guide for linux

0 Kudos
brettesinclair
3 Silver

Re: VNX 5300 in a clustered setup

I would definitely look to upgrade that OE, 5.31 is circa 2011. The 5.32 code has numerous improvements.

Do you have current support on the array ?

0 Kudos
mackendw
1 Nickel

Re: VNX 5300 in a clustered setup

05.31.000.5.720 (correction)

0 Kudos