This post is more than 5 years old
20 Posts
0
1633
LUN in storagepool not using all spindles in pool
Hi everybody,
We have a performance problem with one of our VNX5400 systems. We have a 70-disk storage pool in RAID5 4+1, so it contains 14 RAID sets. If we run a heavy dskspd test on 8 LUN's in that storagepool we see only activity on 4 of the 14 RAID groups. If would expect activity on all RAID groups. Looking at the numbering of the RAID groups it suggests the the storagepool started with only 20 disks and the LUN's were created in this config. If the remaining 50 drives were added later, i would think when adding the disks in the pool should have rebalanced the LUN's over all the new drives. Does anybody know if this should have been the case ?
kelleg
4.5K Posts
0
February 25th, 2016 15:00
The new data is distributed on the internal raid groups starting with the first raid group and moving to the next one as the first gets full. If you put one 10GB LUN in a 1TB Pool, the data will most like get put on the first raid group. It will not spread out unless you have FAST VP installed and Auto-tiering running. If Auto-tiering detects that the disks in the internal raid groups are getting overloaded, it will relocate those "hot" slices to other raid groups that have a lower temp. If the temp of the slices does not exceed the upper temp for the raid group (based on the disk type, raid type of relative temp) then the slices will not be relocated. This was adding in Flare 32 for the VNX1.
When adding more disks to a pool when you do not have FAST VP installed, the re-balancing will only re-balance based on capacity, not performance temp.
glen
rewalmilo
62 Posts
0
February 25th, 2016 00:00
First do you have the Fast-VP enabler installed? Without the Array is only redistruting slices when the pool is expanded capacity vice but no performance values are taken into account..
All luns are distributed in 256 MB slices across the private rg's. So depending on the size of luns and the locality of your test pattern. it can happen that private raidgroups are unevenly utilized.
StefanWilmes
20 Posts
0
February 25th, 2016 01:00
No FastVP enabler installed, I still dont understand. Can i conclude that existing LUN's are not redistributed among the new RAID groups when expanding a storage pool ? Is that what you are saying. My assumption is this is th way it worked with older flasre releases but this was changed in the newer ones. I understand FastVP will help here and it could do no harm but we don't have the FAST suite. So only a LUN migration will do the trick here i guess.
StefanWilmes
20 Posts
0
February 25th, 2016 02:00
Exactly, a once-off rebalance, thats what should have happened. But is doesn't look that has happened for some kind of reason.
brettesinclair
2 Intern
2 Intern
•
715 Posts
0
February 25th, 2016 02:00
It was my understanding that without FAST VP, a pool that is expanded would get a once-off rebalance at the time it is expanded, but no rebalancing / intra-tier relocations thereafter.
I think this was introduced in Flare 32 (Inyo) for series 1 VNX.
johngmr
6 Posts
0
February 25th, 2016 03:00
Maybe wrong, but in some official papers I guess I saw a warning about that - LUNs do not rebalance automagically after expanding a pool. I guess you can try a workaround - create new LUN of similar size and make a migration. It should be distributed across all drives. If enabling tiering won't help of course.
Rainer_EMC
8.6K Posts
0
February 25th, 2016 03:00
If you have Spcollects from the time when you added the disks it should show
StefanWilmes
20 Posts
0
February 25th, 2016 05:00
Thx Rainer, will check that. I've done some checking. I can clearly see the following behaviour during our dskspd test. The test runs on 8 LUN's in parallel. All 8 LUN's show IOPS like this LUN:
Looking at the 14 RAID groups in the pool, on 4 of them i can see a similair pattern:
All the remaining 10 RAID groups are doing far less, they average round 300 IOPS, there are also other LUN's in the pool. Another picture showing the IOPS:
Clearly on unbalance of the slices of the LUN's over the PvtLUNs/RAID groups in my opinion.
Rainer_EMC
8.6K Posts
0
February 25th, 2016 05:00
You shold be able to check like this when the last rebalancing happened:
/nas/sbin/navicli -h autotiering -info -opstatus -loadbalance
Storage Pool Name: Pool 0
Storage Pool ID: 0
Relocation Start Time: 07/06/12 15:42
Relocation Stop Time: 07/06/12 15:42
Relocation Status: Completed
Relocation Type: Rebalance
Relocation Rate: Low
Data to Move Up (GBs): 0.00
Data to Move Down (GBs): 0.00
Data to Move Within Tiers (GBs): 0.00
Data Movement Completed (GBs): 0.00
Estimated Time to Complete: N/A
Schedule Duration Remaining: N/A
StefanWilmes
20 Posts
0
February 25th, 2016 22:00
Glen, so i conclude, always buy/install the FAST VP license, even when there is only 1 tier in the pool. Correct ?
Yesterday we did a LUN migration to newly created LUN's. First conclusions are that the iops are better spread over all the RAID groups and lower response times. Still a couple of RAID groups are doing; 5 RAID groups are doing 2000 IOPS and the other 9 are doing 1000 IOPS. Better than before.
kelleg
4.5K Posts
1
February 26th, 2016 08:00
If your Pool is over about 70% full, the chances that doing a LUN migration will spread the LUN over more of the private raid groups is probably higher than if the Pool is mostly empty.
FAST VP is recommended whenever you use Pools.
glen