Ask the Expert: VNXe front-end Networks with VMware

Well, it doesn't appear to be a time thing, I just tied again, and it is still only showing 764GB available to create shareed folder storage.

Any thoughts or does this sound like it's time to open a support case?

Dave

Sean,

Storage DRS is interesting, but I'm only running Essentials Plus, so I don't get to play with the fancy toys!

Thanks,

Dave

Henri,

I also see the full amount of free space when creating a new vmfs data store.

Please try creating a shared folder.

My original volume was 1.5TB.

I tried closing the browser, also checked from home this morning. Still can only see 7XX GB for shared folders

Dave

Dave,

I tested similar situation (had 5 disk RG with 2TB datastore on it, added 5 more disks to the pool, created second 2TB datastore and deleted the first datastore) and I can see 2TB available when trying to create VMFS datastore or NFS share. Try closing the browser (yes this might help sometimes) and then try again. If you are still having this issue I would definitely contact support. How big was your datastore on the 5 disk RG?

etinj wrote: Recently purchased a VNXe3100 and built a 3 node VM cluster (ESXi 5).  I'm only likely to have about 8-10 VMs all lightly loaded.

One SP should easily handle 8-10 VMs lightly loaded VMs.

Ugh, so I probably shuffled everything around for no reason? Any good way to remedy this without deleting everything and starting from scratch? Any way to verify what drives / raid groups the data stores reside on?

Might be able to tuck a few of my important vms into direct attached storage and just blow out my test vms, delete the entire contents of VNXe, and start over, but I'd really rather not....

Thanks for trying the shared folder thing. Mine is still not showing he correct free space. I guess I'll have to open a case if I get back to the office at a decent hour today...

I can see all the space that is availabe

Here is one think that came to my mind about your configuration: You had 2TB 5 disk RG with 1.5TB datastore on it leaving you about 500GB free space on the RG. You then added five more disks to the pool having total of 2.5TB free space on the pool (500GB on the first RG and 2TB on the added RG). You then created new 1.5TB datastore. What I've understood datastore will not be striped across the RGs if there is not equal amount of free space on the RGs. So your new datastore might fully reside on the new 5 disk RG. Matt correct me if I'm wrong.

@henriwithani

Lots of streams of thought here so let's break it down:

Henri - great point on free space, though the striping algorithm is a good bit more advanced than just matching free space on RGs. The logic is based on a combination of free space, drive utilization and optimization of spindles.

Dave - You should see freespace available within a trivial amount of time of the deletion. Q: did you happen to exit Unisphere during the datastore deletion? The act of deletion or allocation of storage via Unisphere requires it to remain open. If it isn't you can be left in a state that requires EMC support.

Let's also answer the question directly of "where is my storage?" If you're as savvy as Henri and spend time on the CLI, you may be familiar with the Service Commands which all begin with "svc_" and are usable by the Service account. If you want to know more about these commands, download the documentation here.

Now I would use one command in particular - svc_storagecheck- that could come in handy here. After you deleted the datastore, your storage should return to looking like this:

service@egreen-867-spb spb:~> svc_storagecheck -l | more

======================= [Tue Jan 24 20:50:18 UTC 2012] Beginning Run  ======

=================

======================= Now running ./nas_disk -list ... ===================

====

id   inuse  sizeMB    storageID-devID   type  name          servers

1     y       2044  FNM00113000402-0000 CLSAS root_disk     1,2

2     y       2038  FNM00113000402-0001 CLSAS root_ldisk    1,2

11    y      20479  FNM00113000402-0010 CLSAS d11           1

12    y      20479  FNM00113000402-0011 CLSAS d12           2

::::: blah blah blah ::::::

If the nas_disk -list portion of this output is more than those 4 lines then you still have storage provisioned. Also note the use of the Linux piping command "|" and more since this output is rather long.

Let me know how that goes.

Hello Experts,

I am trying to ascertain the best way to setup the vnxe 3100 storage array with NFS.  I have gone through most of the white papers presented, but still not sure of configuration.  So this is what I came up with....

Create 2 shared folder servers.  One server for SPA and one for SPB.  Aggregate ports 2/3 together and assign to one of the shared servers and aggregate 10/11 together for the other shared server.  Hopefully this is sound thus far.

I have and will be using cisco 3750 switches cross stacked.

I am putting both of the shared folders on the same subnet ( Not clear if SPA should be one subnet and SPB be on another).

My ESXi hosts have 8 physical nics and I will be using 2 pNICS for IP storage on a single vswitch with 2 vmkernels assigned to it (will be using IP hash). 

Does this sound like a sound setup?

Thanks

Felix

Matt,

Thanks for the info.  I don't believe it I exited Unisphere during the deletion or creation of any datastores.  However, that revelation kind of worries me....  If accidentally closing a browser window would leave the unit in an inconsistent state then the VNXe seems kind of fragile for something that I'm basically planning to trust with most of our companies infrastructure and data.

As for the "trivial amount of time" for the free space to show up, it took quite a while last night.  I'd say probably about 10-15 minutes after deletion before it appeared in the free space graph.  I noticed it missing, grabbed some screenshots (one attached) and wrote a post  that I never submited.  Just before hitting send, I checked again and the space showed as available.

I then tried to created the shared folder storage and again ran into the missing space there, which I did post about.

Henri -- If you still have your unit setup in a condition to test, can you try creating the shared folder associated with a CIFS server rather than NFS?  I don't think it should make a diference, but it might be worth a shot if it's not too much trouble.

Matt -- As for the CLI, I can't seem to log in.  SSH is enabled and I can get to the unit.  I tried to log in as both admin and root, with the admin and service passwords, but no go, access denied.

Thanks,

Dave

Hey Dave, I'm not a fan of deeming Unisphere or the VNXe as "fragile," but I would fully admit business logic inside Unisphere itself is not ideal. The approach allows for the most responsive interactivity for the user by allowing triggers to be passed and returned to Unisphere. 99% of business logic doesn't run into this case.

To be fair, my use of "trivial" is also subjective. I'm thinking about all the layers of the software stack it's traversing and consistency checks it's ensuring to make sure your storage is seamlessly and 100% available to you as the user and I consider 10 min for a multi-TB datastore as reasonable. That's a better word. With great simplicity of use comes great complexity of development.

The CLI is only accessible by the special "service" user account. SSH as service and you'll be able to login.

Best,

Matt

@mjbrender

Matt,

Sorry if I offended your or any of your colleagues with the "fragile" comment, I know you guys worked / are working very hard on this thing.  I'm new to the storage array concept, and it still feels a little like I'm putting all my eggs in one basket with an array.  So I just get a little uneasy when I think about making a "little" change down the road and having something as silly as a web browser crashing possibly take out my entire cluster of VM's (yes, backups, etc, but downtime is never fun).

I also understand your point about the time it takes to do anything to something that ends with a TB.  I just thought all the internal checks / etc would be finished up during the "Please Wait" or whatever message appears when deleting / creating a datastore.

Back to your point about closing Unisphere, I didn't close it, but I probably switched around to other tabs while it was deleting the datastore, would this cause any problems?

Thanks for the tip about the service account, that did the trick.  Looking at the output of svc_storagecheck now.

Dave

Matt,

======================= Beginning Run

=======================

======================= Now running ./nas_disk -list ...

=======================

id inuse sizeMB storageID-devID type name servers

1 y 2044 APM00XXXXXXXXX-0000 CLSAS root_disk 1,2

2 y 2038 APM00XXXXXXXXX-0001 CLSAS root_ldisk 1,2

15 y 1751123 APM00XXXXXXXXX-0011 CLSAS d15 1

16 y 244219 APM00XXXXXXXXX-0010 CLSAS d16 1

17 y 244219 APM00XXXXXXXXX-0012 CLSAS d17 1

18 y 1142160 APM00XXXXXXXXX-0013 CLSAS d18 1

It's certainly more than 4 lines, but I do still have the one 1.5TB

datastore in use.

Oddly I can see the space if I try to create another vmware datastore

but not when I try to create the shared folder storage.

Going back to where you and Henri were discussing the striping

algorithm, how can I tell if my new datastore is striped across all 10

disks? I'm sure it's probably somewhere in the output of the

svc_storagecheck, but I'm getting lost trying to follow the chain of

stripes and slices. Any hints on where to look would certainly be

appreciated.

Thanks again for all your help.

Dave

Dave,

I'm not able to test the CIFS because of the network configuration that I have.

I did some quick testing with the 5 disk RG, creating datastore, expanding it etc. After every step I compared the svc_storagecheck -l output and after I did same steps that you have done my output looks like this:

======================= Now running ./nas_disk -list ... =======================

id   inuse  sizeMB    storageID-devID   type  name          servers

1     y       2044  APMxxxxxxxxxxx-0000 CLSAS root_disk     1,2

2     y       2038  APMxxxxxxxxxxx-0001 CLSAS root_ldisk    1,2

28    y    1751123  APMxxxxxxxxxxx-0010 CLSAS d28           1

29    y     441603  APMxxxxxxxxxxx-0011 CLSAS d29           1

30    y     441603  APMxxxxxxxxxxx-0012 CLSAS d30           1

31    y     736845  APMxxxxxxxxxxx-0013 CLSAS d31           1

This is what I figuret out from the output. Disk id 28 is the one that was created on the 5 disk RG. Now this is a bit larger than the 1.5TB datastore that was created on it, so there is some free space on that disk left. After adding 5 disks to the RG and creating new 1.5TB datastore on the pool disks 29, 30 and 31 were created. If you look further on the output you can see something like this:

id                         = 59

name                    = StoragePool00_dart0

description            = StoragePool00

acl                        = 0

in_use                  = True

clients                  = fs1327439116,fs1327440267

members              = d28,stripe1327440217,d31

fs132nnn are the datastores. If you look further on the output you start seeing something like this

id          = 145

name        = v145

acl         = 0

in_use      = True

type        = meta

volume_set  = s69

disks       = d28

clnt_filesys= fs1327439116

id = 144

id          = 144

name        = s69

acl         = 0

in_use      = True

type        = slice

slice_name  = s69

slice_of    = d28

offset(MB) = 0

size  (MB) = 1620049

disks       = d28

clnt_volume = v145

So this is the first 1.5TB datastore that I created on the 5 disk RG and it is only using the d28.

If you start digging more you can see how the other 1.5TB datastore is layed on the storage after the pool is extended:

id          = 153

name        = v153

acl         = 0

in_use      = True

type        = meta

volume_set  = s70,s71,s72

disks       = d28,d29,d30,d31

clnt_filesys= fs1327440267

So the second datastore uses all four disks and three volume sets. Now if we look at the dependensies of the volume_sets and disks you can see how the data is placed on the system.

id          = 150

name        = s70

acl         = 0

in_use      = True

type        = slice

slice_name  = s70

slice_of    = d28

offset(MB) = 1620049

size  (MB) = 131074

disks       = d28

clnt_volume = v153

So s70 is a slice of the disk 28 (that was created on the first RG) and it's size is 131074 MB

id          = 151

name        = s71

acl         = 0

in_use      = True

type        = slice

slice_name  = s71

slice_of    = stripe1327440217

offset(MB) = 0

size  (MB) = 883207

disks       = d29,d30

clnt_volume = v153

id          = 148

name        = stripe1327440217

acl         = 0

in_use      = True

pool        = StoragePool00_dart0

type        = stripe

stripe_size = 32768

volume_set  = d29,d30

disks       = d29,d30

clnt_slices = s71

s71 seems to be stripe across disks 29 and 30 and it's size is 883207 MB

id          = 149

name        = d31

acl         = 0

in_use      = True

pool        = StoragePool00_dart0

type        = disk

disk_name   = d31

disks       = d31

clnt_slices = s72

id          = 152

name        = s72

acl         = 0

in_use      = True

type        = slice

slice_name  = s72

slice_of    = d31

offset(MB) = 0

size  (MB) = 605768

disks       = d31

clnt_volume = v153

And s72 is 605768MB size slice from disk 31. Now adding the sizes to gether we get 1620049MB. So It seems that VNXe created two equal size disks on both RGs and then created stripe acros those. Then one part was was placed on the first created disk (d28) and one part on the disk 31 that seems to be on the new RG. Using these components 1.5TB metalun was created for the datastore.

After this I deleted the first 1.5TB datastore and all the links to clnt_filesys= fs1327439116 on the output file were gone except the d28 because the s70 slice is still residing on it.

Now all this is something that I figured out just looking in to the output of svc_storagecheck -l and what Matt had said. Matt correct me again if I'm wrong. At least for me this sounds right.

I also created two 1.5TB datastores on empty pool and VNXe created nice equal size disks and stripes:

32    y     875562  APMxxxxxxxxxxx-0010 CLSAS d32           1

33    y     875562  APMxxxxxxxxxxx-0011 CLSAS d33           1

34    y     810024  APMxxxxxxxxxxx-0012 CLSAS d34           1

35    y     810024  APMxxxxxxxxxxx-0013 CLSAS d35           1

id          = 161

name        = v161

acl         = 0

in_use      = True

type        = meta

volume_set  = s74

disks       = d32,d33

clnt_filesys= fs1327441776

id          = 156

name        = d32

acl         = 0

in_use      = True

type        = disk

disk_name   = d32

disks       = d32

clnt_volume = stripe1327441770

id          = 157

name        = d33

acl         = 0

in_use      = True

type        = disk

disk_name   = d33

disks       = d33

clnt_volume = stripe1327441770

id          = 158

name        = stripe1327441770

acl         = 0

in_use      = True

pool        = StoragePool00_dart0

type        = stripe

stripe_size = 32768

volume_set  = d32,d33

disks       = d32,d33

clnt_slices = s74,s75

id          = 160

name        = s74

acl         = 0

in_use      = True

type        = slice

slice_name  = s74

slice_of    = stripe1327441770

offset(MB) = 0

size  (MB) = 1620049

disks       = d32,d33

clnt_volume = v161

@henriwithani

Looks like when I responded via email, it ate half my post... let me try to paste this via the web form:

Henri,

Thank you again for taking the time to get into this in such detail.

I looked at my output again, and it does make much more sense after your explanation.

So I ended up with a similar situation as what you outlined with the resulting volume on 4 disks and 3 volume sets.  What I didn't follow was how you made the leap to what is living on each raid group (other than based on observing what disks existed before and after adding the second RG).

What I got out of this is that VMStoragPool1 is spread across: d15, stripe1326833437 (which lives on d16 and 17), and d18.  But what physical disks / RG's do these "disks" live on?

I tried to put this in some sort of order that made sense:

id   inuse  sizeMB    storageID-devID   type  name          servers

1     y       2044  APM00XXXXXXXXX-0000 CLSAS root_disk     1,2

2     y       2038  APM00XXXXXXXXX-0001 CLSAS root_ldisk    1,2

15    y    1751123  APM00XXXXXXXXX-0011 CLSAS d15           1

16    y     244219  APM00XXXXXXXXX-0010 CLSAS d16           1

17    y     244219  APM00XXXXXXXXX-0012 CLSAS d17           1

18    y    1142160  APM00XXXXXXXXX-0013 CLSAS d18           1

------------------------------

id                   = 59

name                 = VMStoragePool1_dart0

description          = VMStoragePool1

acl                  = 0

in_use               = True

clients              = fs1323200211,fs1326833485

members              = d15,stripe1326833437,d18

default_slice_flag   = True

is_user_defined      = True

virtually_provisioned= False

disk_type            = CLSAS

server_visibility    = server_2

template_pool        = N/A

num_stripe_members   = N/A

stripe_size          = N/A

------------------------------

id          = 121

name        = v121

acl         = 0

in_use      = True

type        = meta

volume_set  = s63

disks       = d15

clnt_filesys= fs1323200211

id          = 120

name        = s63

acl         = 0

in_use      = True

type        = slice

slice_name  = s63

slice_of    = d15

offset(MB) = 1620049

size  (MB) = 10547

disks       = d15

clnt_volume = v121

id          = 116

name        = d15

acl         = 0

in_use      = True

pool        = VMStoragePool1_dart0

type        = disk

disk_name   = d15

disks       = d15

clnt_slices = s63,s64

------------------------------

id          = 129

name        = v129

acl         = 0

in_use      = True

type        = meta

volume_set  = s64,s65,s66

disks       = d15,d16,d17,d18

clnt_filesys= fs1326833485

id          = 126

name        = s64

acl         = 0

in_use      = True

type        = slice

slice_name  = s64

slice_of    = d15

offset(MB) = 1630596

size  (MB) = 120527

disks       = d15

clnt_volume = v129

id          = 116

name        = d15

acl         = 0

in_use      = True

pool        = VMStoragePool1_dart0

type        = disk

disk_name   = d15

disks       = d15

clnt_slices = s63,s64

------------------------------

id          = 127

name        = s65

acl         = 0

in_use      = True

type        = slice

slice_name  = s65

slice_of    = stripe1326833437

offset(MB) = 0

size  (MB) = 488439

disks       = d16,d17

clnt_volume = v129

id          = 124

name        = stripe1326833437

acl         = 0

in_use      = True

pool        = VMStoragePool1_dart0

type        = stripe

stripe_size = 32768

volume_set  = d16,d17

disks       = d16,d17clnt_slices = s65

id          = 122

name        = d16

acl         = 0

in_use      = True

type        = disk

disk_name   = d16

disks       = d16

clnt_volume = stripe1326833437

id          = 123

name        = d17

acl         = 0

in_use      = True

type        = disk

disk_name   = d17

disks       = d17

clnt_volume = stripe1326833437

------------------------------

id          = 128

name        = s66

acl         = 0

in_use      = True

type        = slice

slice_name  = s66

slice_of    = d18

offset(MB) = 0

size  (MB) = 1011083

disks       = d18

clnt_volume = v129

id          = 125

name        = d18

acl         = 0

in_use      = True

pool        = VMStoragePool1_dart0

type        = disk

disk_name   = d18

disks       = d18

clnt_slices = s66

Henri,

Thank you again for taking the time to get into this in such detail.

I looked at my output again, and it does make much more sense after your

explanation.

So I ended up with a similar situation as what you outlined with the

resulting volume on 4 disks and 3 volume sets. What I didn't follow was

how you made the leap to what is living on each raid group (other than

based on observing what disks existed before and after adding the second

RG).

What I got out of this is that VMStoragPool1 is spread across: d15,

stripe1326833437 (which lives on d16 and 17), and d18. But what

physical disks / RG's do these "disks" live on?

I tried to put this in some sort of order that made sense:

id inuse sizeMB storageID-devID type name servers

1 y 2044 APM00XXXXXXXXX-0000 CLSAS root_disk 1,2

2 y 2038 APM00XXXXXXXXX-0001 CLSAS root_ldisk 1,2

15 y 1751123 APM00XXXXXXXXX-0011 CLSAS d15 1

16 y 244219 APM00XXXXXXXXX-0010 CLSAS d16 1

17 y 244219 APM00XXXXXXXXX-0012 CLSAS d17 1

18 y 1142160 APM00XXXXXXXXX-0013 CLSAS d18 1