Issue with Storage Pools for File (Capacity Shown Is Incorrect - vnx5300)

can you post output from "nas_disk -l"

- went into storage configuration -> Storage Pools for file and initially saw 8TB of available space.  I started messing around with the LUNs and what storage groups they were assigned to.  Did some rescans of the storage inbetween and seemingly, my available storage has now dropped to 4TB available (I actually saw this drop to 6TB and thought it was because I assigned one of the LUNs to another storage group but undid that change).

If you simply removed the LUNs from the underlying ~filestorage storage group there is likely now an inconsistency from what the data movers expects is presented versus what actually exists and possibly needs to be cleared up.  If this is the case this may be why it is seemingly preventing anything new from the underlying block storage pool from properly being discovered/added as potential capacity to the mapped storage pool for file.  Let's first verify as follows:

Can you perform the following:

1) nas_storage -check -all

- If it simply returns "done" then that means the scan didn't report any errors.

As Dynamox had suggested...

2) nas_disk -l

- Will be used to compare to the following dump of the storage group

3) /nas/sbin/navicli -h SPA storagegroup -list -gname ~filestorage

- SPA is an actual entry in /etc/hosts so use it as-is

- Only interested in the content under: "HLU/ALU Pairs:"

- This also let's us verify that none of the LUN's you added were assigned a HOST ID less than 16.  The system makes an effort to enforce this and will renumber, but an easy check that someone it wasn't accidentally "forced"

For next time, the proper way to gracefully unpresent LUNs (via the mapped dvol) from the data movers is:

nas_disk -delete -perm

Optionally you can also specify (-unbind) which also deletes the LUN.

the unbalanced LUNs between SPA and SPB is also from when the lun wizard was used ...

ok, i see the HOST LUN ID for the other 4 LUNs somehow all were set lower than 16 during my testing and only HLU 16/17 show up in the mapped pool space.

I removed them from the ~filestorage group but I see when I add them back to ~filestorage group -- they're added back in with a lower than 16 ID ... do I have to use naviseccli with the -hlu or -addhlu command?

If that's the case, this is befuddling to have to change between cli and gui to do this. I see no unification at all ...

Here's the output:

./nas_storage -check -all

Discovering storage on cs0 (may take several minutes)

done

./nas_disk -l

id   inuse  sizeMB    storageID-devID   type  name          servers

1     y      11260  APMxx-2007 CLSTD root_disk     1,2

2     y      11260  APMxx-2008 CLSTD root_ldisk    1,2

3     y       2038  APMxx-2009 CLSTD d3            1,2

4     y       2038  APMxx-200A CLSTD d4            1,2

5     y       2044  APMxx-200B CLSTD d5            1,2

6     y      65526  APMxx-200C CLSTD d6            1,2

7     n    2047999  APMxx-012E CAPAC d7            1,2

8     n    2047999  APMxx-012D CAPAC d8            1,2

9     n    2047999  APMxx-0001 CAPAC d9            1,2

10    n    2047999  APMxx-0000 CAPAC d10           1,2

11    n    2047999  APMxx-0192 CAPAC d11           1,2

12    n    2047999  APMxx-0191 CAPAC d12           1,2

/nas/sbin/navicli -h SPA storagegroup -list -gname ~filestorage

HLU/ALU Pairs:

  HLU Number     ALU Number

  ----------     ----------

    5               4090

    4               4091

    3               4092

    2               4093

    1               4094

    0               4095

   6               301

    7               302

    8               0

    17              401

    16              402

    9               1

Special:               YES

Shareable:             YES

So, looking at HLUs listed -- the ones with ALU 301, 302, 401, 402 are all part of ~filestorage.  But, I have two luns named CIFS_303, CIFS_304 in the group that are not showing up in this HLU/ALU listing.  They show 'ID' 0, 1 on my side.

But, CIFS_303, CIFS_304 were added using the 'LUN WIZARD' at a later point when I was messing around.  So, why did it do that with the ID numbers?  I thought I read in another forum post that the requirement to be above ID 16 was no longer needed since the VNX took steps to rename or something when you added LUNs.

Outside of the CLI command listed -- can I simply unmap them all from the unisphere GUI and re-add them to ~filestorage?   Ideally, I'd like to use to the GUI for these operations as CLI isn't too always the best for guys who may be working on the system (though, I guess they should never be making these operations in general).

It doesn't seem too obvious to me ... why would it show 4TB if 301, 302, 401, 402 are present in the above HLU/ALU?  I thought the RAID6 was the underlying block storage pool and not based on my luns presented?

When using the Unisphere GUI, go to the ~filestorage storage group and select connect LUNS.  Add the 4 LUNS and as the LUNS are added in the bottom box, you can select the HLU to be something greater than 16.  Once you click in the HLU column for the LUN added, a dropdown will appear to select an HLU.

sdcy00 wrote: Here's the output: ./nas_storage -check -all Discovering storage on cs0 (may take several minutes) done ./nas_disk -l id  inuse  sizeMB    storageID-devID  type  name          servers 1    y      11260  APMxx-2007 CLSTD root_disk    1,2 2    y      11260  APMxx-2008 CLSTD root_ldisk    1,2 3    y      2038  APMxx-2009 CLSTD d3            1,2 4    y      2038  APMxx-200A CLSTD d4            1,2 5    y      2044  APMxx-200B CLSTD d5            1,2 6    y      65526  APMxx-200C CLSTD d6            1,2 7    n    2047999  APMxx-012E CAPAC d7            1,2 8    n    2047999  APMxx-012D CAPAC d8            1,2 9    n    2047999  APMxx-0001 CAPAC d9            1,2 10    n    2047999  APMxx-0000 CAPAC d10          1,2 11    n    2047999  APMxx-0192 CAPAC d11          1,2 12    n    2047999  APMxx-0191 CAPAC d12          1,2 /nas/sbin/navicli -h SPA storagegroup -list -gname ~filestorage HLU/ALU Pairs:   HLU Number    ALU Number   ----------    ----------     5              4090     4              4091     3              4092     2              4093     1              4094     0              4095   6              301     7              302     8              0     17              401     16              402     9              1 Special:              YES Shareable:            YES

Thanks for the output.  At least the data movers are recognizing each of the LUNs; just not adding them as potential storage to the mapped storage pool for file.  If you didn't already pick up on it, while there are other ways of mapping the dvol to the LUN ID (ALU), from the output above the "storageID-devID" and the "name" column (from the nas_disk -l output) lets us derive that.  You simply need to know that the devID is the ALU in hex.

Using an example:

APMxx-012E CAPAC d7

==================

a) 012E in base 10 (decimal) is = 302

b) ALU 302 <=> (dvol) d7

Let's first get the data LUNs assigned to an HLU >15 and take it from there.  While you may be testing, this would also be the time to perform the mentioned calculations and work towards final placement.

Why do we take all these precautions to prevent being able to allocate user data to any filestorage LUNs with an HLU of 6-15 (Control LUNs already assigned: 0-5)?  As part of the SCSI protocol, ID's 0 - 15 are required to boot from.  Currently the data movers are comfortably booting from the 6 control LUNs; however let's assume later the code increases, and in consideration of an upgrade we require a 7th (would not increase an existing volume).  If the next available ID (6) is taken, then we would need to free that up; however, if that LUN were allocated to a filesystem, then you'd have to migrate data off of that first so that the underlying dvol no longer is "inuse".

This was an issue in the past prior to the VNX.  Precautions such as this have been taken to minimize it automatically.  While it didn't renumber it automatically upon storage group assignment (again, if you were to instead right-click on the LUN and select "Add to storage group" it would have), the next defense is to not automatically add it as available capacity to what should be the mapped storage pool for file.

When adding data LUNs to the ~filestorage group, depending on how it is accomplished you are correct the system will make an effort to renumber the HLU to something >15.  For instance, if you were to instead right-click on the LUN(s) and select "Add to storage group", it would have done so.  On the other hand, while I would have to review exactly how you assigned them, generally speaking since if you are adding them yourself from the storage group properties you are more-or-less "forcing" it, and they would take the next available HLU which will be 6+ (control LUNs assigned 0 - 5) if you didn't manually assign it.

As already mentioned by joetarin, from the GUI, you are able to manually modify the Host ID column.  While not obvious, if you click into the Host ID column as you add the LUN to the storage group, this will expose a drop-down menu.  Keep in mind, this is *only* available when you add the LUN; after that you would have to remove it and add the LUN back to reassign it.  Also, as a reminder, ideally throughout this you should be unpresenting gracefully first from the data movers' as noted above with the specified nas_disk command before you take them out of the storage group.  That is unless you use the "-unbind" switch which removes it from the storage group and also deletes the LUN.

Earlier you had noted: "ignore the number of luns as i'm just testing at this point" so I did ; however, you should probably, if you haven't already, review some of the best practices related to using a (block) storage pool versus using a traditional RG.  Please refer to the following paper:

"EMC VNX Unified Best Practices for Performance Applied Best Practices Guide"

https://support.emc.com/docu42660_Applied_Best_Practices_Guide:_EMC_VNX_Unified_Best_Practices_for_Performance.pdf

Page 19

[...]

When creating LUNs:

• Create approximately 1 LUN for every 4 drives in the storage pool.
• Create LUNs in even multiples of 10
• Number of LUNs = (number of drives in pool divided by 4), rounded up to nearest multiple of 10
• Make all LUNs the same size
• Balance LUN ownership across SPA and SPB

[...]

Also, keep the following in mind should you choose to use (block) storage pools for filestorage over traditional RAID Groups (which has its own logic):

• Use thick LUNs only (over-subscription/allocate on demand is available at the filesystem level)
• Considering you chose (block) storage pools over traditional RG's, I am assuming you own a FAST (VP) license; even with a single tier  (with more than one private RG of course) the system with an active policy will tier horizontally.
• Dedicate the pool for filestorage (do not mix with block and file) if possible
• Leave 5% free capacity in the pool to optimize tiering (with the metadata and pool LUN overhead it will actually come closer to 4% but to keep the math simple, you do not need to calculate the overhead ahead of time)
• Not well known and not mentioned in the document, when using FAST VP, we also recommend using thin enabled filesystem (not thin LUNs)

Why?  Well to quote an engineer:

[...]

By using Thin-enabled file systems (via the starting size, maximum size, and high watermark settings) we ensure denser file placements, keeping files and their meta-data closer together. After an auto-extension, new allocations will also be written into a smaller location in the file system and then that data will typically be hotter and the isolated hot areas can more effectively leverage Flash drives. This concept is illustrated below and, as evidenced by workload testing, this solution will maintain a localized area of hot data, better suited for tiered pools using FAST VP

[...]

Therefore, ideally you should be calculating as follows:

1) Calculate the total available capacity of the pool and subtract 5%

2) Divide that amount into 10 equal sized LUNs (based on calculation above and input the 16 drives you have assigned to the pool) balancing LUN ownership: 5x SP A and 5x SP B

This has to do with the logic of how AVM groups the pool members (5 LUNs, same size, balanced between SP A and SP B, etc).  This is described in the "Managing Volumes and File Systems with VNX AVM" document available on support.emc.com.

I like 5 x 1TB approach, my personal reasoning behind it is that while backend pool consists of the same 8 drives but since File OE is just another operating system it would benefit from having more i/o queues by having more LUNs. There is a limit to how many LUNs you can present to file datamovers ( i think 256 ?) so if you plan for your box to get much bigger that's something to consider.

Question about creating LUNs from storage pool for file.

It appears that the recommendations for the number of LUNs has changed from 1 per 4 disks (round up to nearest 10) to simply 5 LUNs.

When creating the LUNs, should you fully, almost, or partially allocated the existing capacity in the storage pool?  We have 8 3TB NL-SAS disks in a RAID6 6+2 disk group that makes up the storage in the pool.  The pool capacity is about 16.5TB.  From earlier answers in this forum thread, it sounds like allocating 100% of the space is not desired.  Should we create 5 3TB thick LUNs and hand them off to the data mover, leaving somewhere between 5-10% of space unallocated in the pool or is it ok to create 5 smaller LUNs, say 1TB each, giving 5TB over to file system use now and then at future intervals, based on usage needs, create additional 1TB LUNs and give 5TB additional over to the data mover?