Unsolved
This post is more than 5 years old
20 Posts
0
5256
VNX - Tree Quota Out of Sync
We are migration data from a Celerra to a VNX using Replicator.
After switchover users can no longer save files on filesystems with tree quotas enabled.
We have concluded that the problem is that the "Used" space reported on the VNX is corrupted.
Before replication:
Report for tree quotas on filesystem fs_name mounted on /root_vdm_5/fs_name
+------------+---------------------------------+-------------------------------+| Tree | Bytes Used (1K) | Files |+------------+--------+-------+-------+--------+--------+------+------+--------+| | Used | Soft | Hard |Timeleft| Used | Soft | Hard |Timeleft|+------------+--------+-------+-------+--------+--------+------+------+--------+|#1 |86663995|94371840|94371840| | 100385| 0| 0| |
After replication:
[nasadmin@vnx ~]$ nas_quotas -report -tree -fs fs_name
Report for tree quotas on filesystem fs_name mounted on /root_vdm_20/fs_name
+------------+-----------------------------------------------+-----------------------------------------------+| Tree | Bytes Used (1K) | Files |+------------+------------+------------+------------+--------+------------+------------+------------+--------+| | Used | Soft | Hard |Timeleft| Used | Soft | Hard |Timeleft|+------------+------------+------------+------------+--------+------------+------------+------------+--------+|#1 | 693311960| 94371840| 94371840| EXPIRED| 100385| 0| 0| |
We have tried to run:
nas_quotas -check -start -mode online -tree -fs fs_name -path /projects/Courses
This process started and
-check -status returned
100% complete. The result is that used can write to Courses! 
Nontheless starting another check fails! 
Attempts to run -check -stop, and -check -status fails with error 5005: I/O Error.
Another attempt to start a check cause nas_quotas command to hang for hours.
Any ideas, on how we can get the VNX to respond to nas_quotas commands again? And what to do to get the tree quotas synced with the actual filesystem usage?
Rainer_EMC
8.6K Posts
0
August 11th, 2011 12:00
A data mover reboot would most likely help.
In your case I really suggest to open a service request to that this gets analyzed properly
Rainer
Rainer_EMC
8.6K Posts
0
August 11th, 2011 15:00
What's the DART version of the source system?
Harald Jensas
20 Posts
0
August 11th, 2011 17:00
NAS version: 7.0.13-1
Support is currently reebooting the DM after an hours troubleshooting...
Harald Jensas
20 Posts
0
August 11th, 2011 18:00
And we are stuck again...
We wrote a script that run the nas_quota -check -start -mode online -tree -fs -path command looping on all filesystems and verifying that the job completes with nas_quota -check -status before moving on. When we got to a large filesystem, the status reports 100% complete. But it never changes to the "not running" state that we want before moving on to the next FS. 
Harald Jensas
20 Posts
0
August 11th, 2011 18:00
Ok, the reboot got us out of the hang situation...
Running the nas_quota -check -start -mode online -tree -fs -path to fix/recalculate the quota.
Thanks!
Rainer_EMC
8.6K Posts
0
August 12th, 2011 00:00
No - I meant the DART code on the non VNX
Do you have a SR # ?
Harald Jensas
20 Posts
0
August 15th, 2011 02:00
Hi,
SR # 42617546
New SR # 42617658
DART on the Celerra was upgraded to support EMC Replicator between Celerra and VNX, the code running is 5.6.51-3
Btw:
Do you know what the -quotadb -upgrade does? And when would it be appropriate to use this command?
---
The problem escalated, we have now confirmed that we have the same problem on filesystems with User & Group Quotas...
---
Any way I can list the processes running on the Datamover, identify the quota check and send a "clean" kill signal? (This way we can get away from rebooting everytime a check gets stuck at 100%) I have also verified now that once it runs to 100% and we reboot the DM the QuotaDB and a du command reports the same data...
Rainer_EMC
8.6K Posts
0
August 15th, 2011 08:00
Hi Harald,
my guess would be because the source is at 5.6
In 6.0.41.3 we added the ability for greater than 4TB treequota’s and the Online Quota update tool.
In order to that some changes were done to the file system structures – if you upgrade a system to 6.0.41.3 or later it would automatically convert them during the upgrade.
If you still have a file system to migrate I would suggest to run a manual upgrade of the quota db after Replicator is finished and stopped and before you use it in production.
The procedure is in the 6.0.41/42 release notes:
Upgrade quota database limits for file systems
You can upgrade your quota databases to the new 256 TB limit from
the existing 4 TB limit. The nas_quotas command has a new option
called -quotadb that, when run, performs the upgrade. The file
system will be unavailable during the upgrade.
The Command Reference Manual contains more detailed information
about the -quotadb command.
Rainer
Harald Jensas
20 Posts
0
August 15th, 2011 11:00
Thank you for the clarification on the -quotadb -update option.
Right now status is that we have been waiting 9, NINE, hours for EMC Support to call us back. Aprox 1.5 hours ago I spoke to the VNX Manager On-Call, but it did'nt seem to help.
It does not help to have good engineers when they are this unavailable...
Rainer_EMC
8.6K Posts
0
August 15th, 2011 14:00
I would suggest again and ask to escalate because of DU
There also seems to be problem with matching the serial number to the site ID for your SR
Harald Jensas
20 Posts
0
August 16th, 2011 05:00
We finally got through and spent 2 shifts in WebEx...
The issue has been escalated and 2 instances of panic dumps have been uploaded for Engineering to look at.
zerothehero
64 Posts
0
September 21st, 2013 04:00
Sorry to pull out this old thread, but yesterday i ran into the same situation:
Migrated Filesystems with Replicator from a Celerra NS40 to a VNX 5500 with a lot of tree quotas.
Celerra is running version 6.0.65, VNX is running 7.1.x
After Switchover the users started complaining about not being able to put any more files in their home drives.
In Unisphere we could clearly see that Users had exceeded their hard quotas by up to several hundred per cents. In reality they maybe used only a few per cent of their quota.
We then ran nas_quotas -check -start -mode online -tree -fs -path and things got normal again.
My questions are now:
Rainer_EMC
8.6K Posts
0
September 23rd, 2013 03:00
Could be because of some slight changes to the quota feature between 6.0 and 7.1