Unsolved
This post is more than 5 years old
16 Posts
0
3121
VNX 5300 in a clustered setup
In an clustered setup (RHEL 6), using a VNX 5300 as the storage for a bunch of EXT4 filesystems. The cluster is setup ACTIVE/PASSIVE where only one node is using the filesystems at a time.
The WRITE CACHE is enabled on the VNX.
Having issues when unmounting the filesystems on the ACTIVE node and then going to the other node and mounting the filesystems. Getting EXT4-fs errors in the /var/log/messages about deleted inodes, etc. The active system typically is up for 1 week using the filesystem data which contains mostly small files (many) and a bunch of ISAM files.
Could the CACHE on the VNX be holding pending updates to the storage that the formerly passive node is not seeing when it attempts the mount operations?
Rainer_EMC
8.6K Posts
0
December 21st, 2015 15:00
Not likely on the VNX cache
Check if you are properly flushing the Linux buffer cache
mackendw
16 Posts
0
December 21st, 2015 15:00
The buffers held by the OS in the filesystem *should* be flushed with the unmount of the filesystem...one would think.
Does the WRITE CACHE get flushed to physical disk with the unmounting of the filesystems on the first node?
What we are seeing is the 2nd node is getting an older / incomplete view of the actual data on the SAN filesystems.
I could do a sync to flush the filesystems before they are unmounted.....but I think that would be un-needed. Is the /proc/sys/vm/drop_caches also required to be set to 2 after syncing to purge directory and file inodes?
Rainer_EMC
8.6K Posts
0
December 22nd, 2015 01:00
The VNX write cache doesnt have to get flushed since any other host accessing a block would just automatically the the data from the write cache
Did you have a look at the EMC Linux host configuration guide and the knowledgebase ?
If you are using a volume manager you should also check if its cluster capable
mackendw
16 Posts
0
December 22nd, 2015 06:00
"Did you have a look at the EMC Linux host configuration guide and the knowledgebase ?"
I'm just the consultant trying to pick up the pieces and get this thing working again...;) Can you send me a link to both pls.
No volume manager is being used. All filesystems on the SAN (24 of them) are EXT4 filesystems.
mackendw
16 Posts
0
December 22nd, 2015 12:00
Note: I contacted RedHat support and they suggested upgrading the SAN firmware.
The VNX5300 is running version 5.31 firmware if that helps. Comments are welcome.
see:
ext4 file system corrupted with ext4_mb_generate_buddy messages seen in the logs - Red Hat Customer Portal
Rainer_EMC
8.6K Posts
0
December 22nd, 2015 13:00
Just see support.emc.com – there is a link for kb search and a search box to find the host config guide for linux
brettesinclair
715 Posts
0
December 22nd, 2015 13:00
I would definitely look to upgrade that OE, 5.31 is circa 2011. The 5.32 code has numerous improvements.
Do you have current support on the array ?
Rainer_EMC
8.6K Posts
0
December 22nd, 2015 13:00
An upgrade to a recent VNX OE is always a good idea
5.31 isnt the full version number – which one are you running exactly ?
mackendw
16 Posts
0
December 22nd, 2015 15:00
05.31.000.5.720 (correction)
Rainer_EMC
8.6K Posts
0
December 23rd, 2015 03:00
I don’t see 717 as an official release - .716 was released April 2012 and .727 in July 2013
Both are long EOSL by now
Is your Linux version also 3 years old ?
I would strongly suggest to upgrade
mackendw
16 Posts
0
December 23rd, 2015 05:00
Its running in a secure environment off the grid so to speak. Upgrading can be done but usually requires an "act of god". Will push for this but I doubt the OS part will happen in the current contract. Unless of course the OS is the problem here.
Which Windows version does the SAN run? Is it supported by Microsoft?
We are under support for the SAN from EMC via Unisys.
Rainer_EMC
8.6K Posts
0
December 24th, 2015 03:00
R31.5.720 7.0.53.2 (Posted July 2012)
mackendw
16 Posts
0
December 24th, 2015 04:00
Is this the current version for the VNX5300? (R31.5.720 7.0.53.2 (Posted
July 2012))
On Thu, Dec 24, 2015 at 7:40 AM, Rainer_EMC
Rainer_EMC
8.6K Posts
0
December 24th, 2015 13:00
no - all R31 versions are EOSL
current is the latest R32 code
Sent from my mobile