This post is more than 5 years old
14.3K Posts
0
6880
DD890 extension
Today at two locations we had DD890 extended with additional ES30 disk shelves. Originally we had 3, then 4 and now we went to 8 by adding additional 4. However, EMC support partner who carried out this got into trouble claiming we miss license and this is why only 6 disk shelves are visible and fs is extended with only 2.
To my knowledge, DD890 requires DS license which comes with disk shelf so that's ok (we have 8 of those for each DD). I'm told we might miss capacity extension license, but to my knowledge you need that one only if going beyond default capacity (and I think with ES30 using 2TB disks you can go up to 12 disk shelves and I'm unaware that beyond 6 you need any extra license nor we were ever told that).
While I might be barking at the wrong tree, I do not like disks in shelves added neither. disk show hardware is showing then being different manufacture or type than those before. So, previous 4 shelves were all HUA and now I got HUS, HUA and ST. All with different firmware. I thought best practice was to use same type if possible or this does not matter (note that this is type per enclosure; in single enclosure all disks are same type).
disk show state shows all disks in problematic enclosures as U (unknown), enclosure show summary shows them all as online, but enclosure show topology doesn't show it connected on one DD system affected (on other apparently it is). In short:
DD1 (broken)
hcrvelin@DD1# disk show state
Enclosure Disk
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
--------- ---------------------------------------------
1 s . . .
2 . . . . . . . . . . . . . . s
3 . . . . . . . . . . . . . . s
4 . . . . . . . . . . . . . . s
5 . . . . . . . . . . . . . . s
6 . . . . . . . . . . . . . . s
7 . . . . . . . . . . . . . . s
8 U U U U U U U U U U U U U U U
9 U U U U U U U U U U U U U U U
--------- ---------------------------------------------
Legend State Count
------ ------------- -----
. In Use Disks 87
s Spare Disks 7
U Unknown Disks 30
------ ------------- -----
Total 124 disks
hcrvelin@DD1# enclosure show topology
Port enc.ctrl.port enc.ctrl.port
---- - ------------- - -------------
2a > 2.A.H: 2.A.E > 6.A.H: 6.A.E
2b > 7.B.H: 7.B.E > 3.B.H: 3.B.E
2c > 5.A.H: 5.A.E > 9.A.H: 9.A.E
2d > 8.B.H: 8.B.E > 4.B.H: 4.B.E
3a > 3.A.H: 3.A.E > 7.A.H: 7.A.E
3b > 6.B.H: 6.B.E > 2.B.H: 2.B.E
3c > 4.A.H: 4.A.E > 8.A.H: 8.A.E
3d > 9.B.H: 9.B.E > 5.B.H: 5.B.E
---- - ------------- - -------------
Encl WWN Serial #
---- ---------------- --------------
2 N/A CKM00134600159
3 N/A CKM00131000739
4 N/A CKM00143300554
5 N/A CKM00143301817
6 N/A CKM00131000747
7 N/A CKM00131000740
8 N/A CKM00143301816
9 N/A CKM00132002227
---- ---------------- --------------
Error Message:
-----------------
No error detected
-----------------
ES20 rear view:
CONTROLLER-B CONTROLLER-A OP-PANEL
+--------------------------------------------------------+
| | E | H | |
| | | | |
| | H | E | |
+--------------------------------------------------------+
ES30 rear view:
+--------------------------------------------------------+
CONTROLLER-B | E H |
+--------------------------------------------------------+
+--------------------------------------------------------+
CONTROLLER-A | H E |
+--------------------------------------------------------+
DD2 (broken)
hcrvelin@DD2# disk show state
Enclosure Disk
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
--------- ---------------------------------------------
1 s . . .
2 . . . . . . . . . . . . . . s
3 . . . . . . . . . . . . . . s
4 . . . . . . . . . . . . . . s
5 U U U U U U U U U U U U U U U
6 . . . . . . . . . . . . . . s
7 . . . . . . . . . . . . . . s
8 . . . . . . . . . . . . . . s
9 U U U U U U U U U U U U U U U
--------- ---------------------------------------------
Legend State Count
------ ------------- -----
. In Use Disks 87
s Spare Disks 7
U Unknown Disks 30
------ ------------- -----
Total 124 disks
hcrvelin@DD2# enclosure show topology
Port enc.ctrl.port enc.ctrl.port enc.ctrl.port
---- - ------------- - ------------- - -------------
2a > 2.A.H: 2.A.E > 6.A.H: 6.A.E > 8.A.H: 8.A.E
2b > 9.B.H: 9.B.E > 7.B.H: 7.B.E > 3.B.H: 3.B.E
2c > 4.A.H: 4.A.E > 5.A.H: 5.A.E
2d
3a > 3.A.H: 3.A.E > 7.A.H: 7.A.E > 9.A.H: 9.A.E
3b > 8.B.H: 8.B.E > 6.B.H: 6.B.E > 2.B.H: 2.B.E
3c > 5.B.H: 5.B.E > 4.B.H: 4.B.E
3d
---- - ------------- - ------------- - -------------
Encl WWN Serial #
---- ---------------- --------------
2 N/A CKM00143301810
3 N/A CKM00131000360
4 N/A CKM00132101820
5 N/A CKM00132002275
6 N/A CKM00134801588
7 N/A CKM00131001009
8 N/A CKM00131001011
9 N/A CKM00143301809
---- ---------------- --------------
Error Message:
-----------------
No error detected
-----------------
ES20 rear view:
CONTROLLER-B CONTROLLER-A OP-PANEL
+--------------------------------------------------------+
| | E | H | |
| | | | |
| | H | E | |
+--------------------------------------------------------+
ES30 rear view:
+--------------------------------------------------------+
CONTROLLER-B | E H |
+--------------------------------------------------------+
+--------------------------------------------------------+
CONTROLLER-A | H E |
+--------------------------------------------------------+
Except onsite engineer reporting issue with two disk shelves, I see following:
- DD2 doesn't show 2 DS connected at all - DD1 at least does
- DD1 and DD2 show different connectivity diagram (for example 2 versus 3 enc.ctrl.port columns)
I'm not sure if above suggest cabling is wrong (as this one done by EMC partner) nor why different outputs, but I also checked quickly document titled "ES30 Expansion Shelf Hardware Guide (DD OS 5.2)" (5.2.5.0 here) and page 27 seems to be clear how this should be connected (not being physically onsite I can't say how this exactly was done). I suspect DD2 is using 3-3-2 topology (8xDS3) while DD1 doesn't used it. Support guy who was onsite doing this suspects license which I'm not aware of. I'm a bit bothered why cabling is not the same and why I have now mix of different disk types within same unit.
Anyone with experience or hint what might be going on here? Right now I'm left waiting for support guys talking to their support guys who eventually will reach DD guys most likely tomorrow
jbrooksuk
208 Posts
0
September 12th, 2014 09:00
DD890 ships with 96GB memory in all cases and supports up to 180TB of raw storage before the expanded storage license is required.
The DD890 is full when it has 384TB of raw storage.
Why 384TB? It’s a legacy because that’s 12x 32TB ES20 shelves (16 disks of 2TB), which we obviously no longer sell and you never had.
EMC recommends filling a set (loop) in full before moving to the next set – 3 shelves across the 4 sets (8 SAS ports across 2x HBA's).
If you already have a couple of ES20’s and these obviously can’t share with ES30’s, you will essentially lose shelf count and won’t be able to reach 384TB at all. For this EMC make an allowance to let you add up to 16 shelves but this is intended to account for loss of a loop to those 2x ES20’s that essentially stole 2 spaces for shelves, in essence allowing them to other loops and reaching 4 per set on 2 of them. Or if you have some 15TB/16TB shelves in the system.
In this case you have 6x 30TB ES30’s (2TB x 15 slots) and that is 180TB raw, so the expanded storage license is required to add any more storage.
You will in reality be full at 360TB raw.
Bare in mind though, that the ES20 has 2 spares - not 1, so you are no worse off with the useable space, so don't get drawn into a concern of loss of storage space because you are using ES30's and can't reach the stated 384TB.
i.e: 2TB x 12 shelves = 24TB, which are just spare disks.
As long as there are 2 paths to each shelf from different SAS controllers the DD will be fine (though standard SAS port topology should be followed), also that the loop is reversed through the shelves on the second path but I agree it’s possibly annoying to see 2 identical systems cabled differently.
In both cases of your 2 DDR's shown, both achieve redundancy by loop reversal and using each SAS cards for each set path.
I guess you possibly just had 2 different engineers or they did it online and were forced to add the shelved in a specific way based on how it was already cabled but not sure.
but as suggested, you can shutdown, pull the SAS cables and have it re-cabled how you want it, it's 30 minute job with the correct length SAS cables to achieve it.
Hope that helps,
Jonathan Brooks
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
September 11th, 2014 10:00
admin@dd890# system show version
Data Domain OS 5.4.0.8-404909
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
1
September 11th, 2014 10:00
i can confirm you need the expansion license key once you get to x number of shelves. We had to buy one for our DD890 as well.
ble1
14.3K Posts
0
September 11th, 2014 10:00
Damn, that's not what I was hoping to hear. Well, not sure why that was not delivered then (since it was not cheap despite inclusion of HUS disks which I read are cheaper than HUA I had before).
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
September 11th, 2014 10:00
here is mine
bobt5
49 Posts
1
September 11th, 2014 13:00
Hrzoje,
The cabling is fine on both systems. 3-3-2 or 2-2-2-2 are perfectly acceptable ways to cable the system. It's easy enough to change if you want both systems to look the same. If you had a need to install ES20 enclosures, then the 3-3-2 configuration is better as it leaves two HBA ports available for the ES20 enclosures. If you have no intention of ever installing ES20's, you are fine with the 2-2-2-2 configuration. You can mix different types of disk sizes and manufacturers together as long as they are in the same enclosure type.. You cannot mix ES20 and ES30 enclosures on the same SAS chain.
As long as you received all SATA disks, what you were shipped are all qualified disks for a DD system. The DD860 does not support SAS disks.
You do need the expanded storage license to add the 2 remaining enclosures. The documentation says two things, both of which apply to your situation. On a DD860, expanding your active tier storage beyond your initial purchase (entry capacity) or above a 64TB raw capacity requires an expanded storage license.
You also would have added another 36GB of memory to add more than 64TB of raw storage.
ble1
14.3K Posts
0
September 11th, 2014 16:00
Hi,
Thank you for input. I was not aware of additional license and memory requirement. License part I guess is easy - memory not so sure. This is DD890 (not DD860) and pure ES30 since day one (when we bought it ES20 was already going to retirement). System comes with 96GB memory - are you sure we also need to add memory? (I can see this to be the case for DD860 if you have model with lower memory, but DD890 should be fine up to 384TB with 96GB of memory).
bobt5
49 Posts
0
September 11th, 2014 17:00
Sorry, I must have misread something.
You are correct, a DD890 with 96GB of memory will support the maximum disk configuration.
As far as the license goes, you are still going to need one.
Starting with DD OS 5.1, any expansion beyond shipped capacity is to require an Expanded-Storage license.
ble1
14.3K Posts
0
September 12th, 2014 17:00
Thanks Jonathan!