UNSOLVED
M
MaximUnited
111 Posts
0
3040
September 2nd, 2009 09:00
Advice on best practice
Hi guys,
I'm facing a large project of upgrading our Data Warehouse environment.
Currently DMX3, oracle 10g, HPUX11v2, raw devices on vxvm.
We bought a DMX-4 4500 with about 900 drives [300GB 15k rpm] + 8 flash drives [200GB]. 132GB of cache.
We also have a new HP Superdome that will hold the DB [oracle 11g, no ASM].
More details:
Facts:
24 HBA connections to 24 dedicated FA's.
Superdome with new sx2000 chipset (64 processors)
hyper size 29.76GB + some hypers 0.5GB.
Production DB is RAID-1
Clone of production is RAID-5 (3+1)
HPUX 11.31, VxVM foundation suite 5.0 (used for both FS and Volume manager).
Service Guard cluster.
Oracle 11g
Assumptions:
Symmtrix data:
Main DB devices are 244 Metas of 238GB [8 hypers each]. 58TB total
For the archives: 21 Metas of 238GB [8 hypers each]. 5TB total
for redo logs:12 metas of 8GB [consists of 16 hypers of 0.5GB]. 98GB total. Currently will reside on flash drives (I probably know that this is a bad idea).
After a future meeting with oracle experts we'll probably select a different are for flash like temporary oracle data. Indexes are also an option, but they are probably around 5TB (flash is only 1.2TB).
Host config:
Aggrigate each 16 metas into a vxvm device group.
Create a logical volume that will spread on 4 metas stripped, sized 238GB
On each logical volume will create a VxFS
I can't tell why these assumptions are used. If you can make a good case against we sure consider to change.
What am I looking for?
Missed parameters, like stripe size and block size. Both vxvm and vxfs.
General advice.
I'm sorry for asking a lot, but it's a really large project for me and my company. So I'd really appreciate any help and insights.
Thanks a lot in advance!
Maxim
I'm facing a large project of upgrading our Data Warehouse environment.
Currently DMX3, oracle 10g, HPUX11v2, raw devices on vxvm.
We bought a DMX-4 4500 with about 900 drives [300GB 15k rpm] + 8 flash drives [200GB]. 132GB of cache.
We also have a new HP Superdome that will hold the DB [oracle 11g, no ASM].
More details:
Facts:
24 HBA connections to 24 dedicated FA's.
Superdome with new sx2000 chipset (64 processors)
hyper size 29.76GB + some hypers 0.5GB.
Production DB is RAID-1
Clone of production is RAID-5 (3+1)
HPUX 11.31, VxVM foundation suite 5.0 (used for both FS and Volume manager).
Service Guard cluster.
Oracle 11g
Assumptions:
Symmtrix data:
Main DB devices are 244 Metas of 238GB [8 hypers each]. 58TB total
For the archives: 21 Metas of 238GB [8 hypers each]. 5TB total
for redo logs:12 metas of 8GB [consists of 16 hypers of 0.5GB]. 98GB total. Currently will reside on flash drives (I probably know that this is a bad idea).
After a future meeting with oracle experts we'll probably select a different are for flash like temporary oracle data. Indexes are also an option, but they are probably around 5TB (flash is only 1.2TB).
Host config:
Aggrigate each 16 metas into a vxvm device group.
Create a logical volume that will spread on 4 metas stripped, sized 238GB
On each logical volume will create a VxFS
I can't tell why these assumptions are used. If you can make a good case against we sure consider to change.
What am I looking for?
Missed parameters, like stripe size and block size. Both vxvm and vxfs.
General advice.
I'm sorry for asking a lot, but it's a really large project for me and my company. So I'd really appreciate any help and insights.
Thanks a lot in advance!
Maxim
967
0
Posted September 14th, 2009 15:00
Although I would stay by Raid1, I think there are other reasons why you...
Starting by the idea to use flash drives for redo logs...I would do it. But how?
Actually you need some more (flash) drives in order to be able to built 4 Raid5 Groups for you redologs. That means (and I jump over all abstraction layers) you place the
- Redo-LogM10 to the first Raid5 Group and the Red-LogM11 to the second Raid5 Group
- Redo-LogM20 to the third Raid5 Group and the Red-LogM21 to the fourth Raid5 Group
- Redo-LogM30 to the first Raid5 Group and the Red-LogM11 to the second Raid5 Group
- ....
While your database is writing in Redo-LogM20[M21] after a log switch, the read-performance of the first and second Raid5 Group is exclusive for archiving the data of Redo-LogM10[M11]... although I believe that a good portion of the writes (performed in Redo-LogM10[M11]) which have to be archived will be read-hits.
With 8 flash drives you can form only two Raid5 Groups (2x[3+1]=8). That means you have to skip redolog mirroring in order to be able to distinguish the load. But no redolog mirroring is as far as I know not welcomed.
So far according to write performance. For the rest try to spread you production tablespaces over all disks. With 900 disks you should be able to manage at least a load of 600-800 MB/sec (Write:Read=40%:60) and more.
By the way, the best balanced perfromance profiles in reference of Oracle Databases that I have seen were in combination with ASM.967
0
Posted September 14th, 2009 19:00
Flash drives are good for all workloads.
However log files placed on well configured spinning disks will have the same performance as log files placed on flash drives. In both cases writes should be acknowledged by Symmetrix cache and should not be waiting on the drives.967
0
Posted September 15th, 2009 01:00
Thanks for your replies.
When I said "I probably know that this is a bad idea", I was talking about write hits. Since redo logs do only writes, and my cache is not utilized, there is no need to use flash drives. Correct?
Our current DWH env is on DMX3 and on 1440 drives (but only 10k rpm), and gives close to 3GB/s throughput. I'm hoping to top that.
How about other configurations? Meta size. Host striping. etc... What considerations for changing any of those parameters.
Maxim967
0
Posted September 15th, 2009 03:00
With 1440 drives in your DMX3, you have more than enough drives to saturate the system throughput. In seqential workloads, with low thread counts on the drive, a single disk can do about 50 MB/sec. Your overall limit isn't going to be the drives.967
0
Posted September 15th, 2009 03:00
That depends on how I configure the data to spread on those disks. How can I get the best performance possible from those 900 15k rpm drives?967
0
Posted September 15th, 2009 04:00
Make sure they are all used evenly, striping is good, but try to avoid many levels of striping. If you are using RAID5 and meta volumes, that is already 2 levels of striping, so host based striping may not be needed. For instance, if you used RAID 7+1 with 16 way metas, each logical would be spread across 128 drives.
You should discuss this with a Symmetrix performance guru. They could model your workload. The drives should not be your concern.561
0
Posted September 15th, 2009 05:00
A lot of the performance details of you system are better discussed with a performance guru, not on a public forum.
For filesystem stripe size, 64,128 or 256k are probably a good choices. If you or someone you know attended EMC World, there was a presentation given around general Symmetrix performance best practices that could be helpful.967
0
Posted September 15th, 2009 05:00
Sorry, I missed that. Too many details
There is a decent white paper on PowerLink about using flash drives with Oracle. You should read that for ideas on what to do with your EFDs.561
0
Posted September 15th, 2009 05:00
It looks like you already have a full working production env. Have you profiled that to see where the current bottlenecks are? You mentioned 3GB/s bandwidth on current DMX-3. What other numbers did you see at the FA, DA and Disk levels?
I have gotten excellent results with RAID1 hardware and striping at the host level. It has been a while, but I used default Veritas volume manager striping of 128K. Using large block sizes at the vxfs level is a good idea. Match it to oracle db blocksizes or a multiple of that.
I would do some host level application profiling to see where the current bottlenecks are (if you think there are).
Also, how you lay out your devices on FA's and the interaction with multipathting software will impact your performance (powerpath, dmp etc.)
MaximUnited
111 Posts
967
0
Posted September 14th, 2009 08:00
I'd really appreciate any advise.
Maxim