Tier 1 Full

Doug,

Yes, you will definitely see a performance degradation when your tier one is full.  In essence, when it gets full your lower tiered drives (the slower ones) will try to accommodate the work load.  

I'd strongly recommend re-evaluating your data progression profiles.  Try to keep your tier 1 available for your writes, and use lower tiers for your read data.  

If you need additional help with this, let me know.

We were having the same issue and modified our storage profiles which cleaned out all but about 5% in Tier 1 Raid 10 (SSD) according to Enterprise Manager under Storage Types.  However, we noticed that no data is migrating to Raid 5 in Tier 1 additionally under Disks in Enterprise Manager under SLSSD it shows that they are 99% allocated.  Any idea what's happening?  Seems like conflicting info.

Hey, thanks for the reply.  Let me take this question a little further.

If our Tier1 write is full, for both standard and fast track, but the Tier1 Reply/Read still has some room left, which leaves Tier1 with some free space over all, will the write start to cannibalize the remaining space for the reads to use for writes?

Copilot helped us figure this out.  When the array was installed it was installed with a Raid 5-9 setup in tier 1, however there wasn't enough SSDs to support that.  Therefore when data progression would run it would just kick the Raid 10 data in tier 1 down to Raid 5 in tier 2.  They changed tier 1 to be Raid 5-5 and that resolved the issue. Thanks.

I had a similar thing happen to me on several occasions - except it was in Tier 3. We have only 5 active SATA disks and the default was RAID5-9. Tier1 filled up because no data could get migrated down. Annoyingly this appears to be some sort of default, we had the same thing corrected twice on a single storage center.

When performance degradation occurrs because Teir1 gets full, and the lower tiered drives (the slower ones) will try to accommodate the work load for the writes (and the reads...)

... just how much degredation are we talking about?   Are we talking about a fractional percentage degradation in the 1% to 100% range?   Or are we talking about a performance degradation of several orders of magnitude, or hundreds of orders of magnitude?

It really depends on your disk configuration. In our environment we'd see a significant hit to I/O if Tier 1 filled up. Our Tier 1 has 11 active 15K FC disks. If that filled up then writes would be done to 5 active 7200 RPM SATA disks. Reads would still come from wherever the data lives.

One of our arrays, for example, is all 10K drives, so the differences in the tiers is essentiall just raid10 vs raid5.

During data progression, we go from writes completing in 2 milliseconds per write, to hundreds and even thousands of milliseconds per writes...  For most types of writes, the array essentially becomes un-writable.

What ways can data-progression be configured, so that data progression can occur, but not monopolize the back-end IO while it is occuring?  It is just while data progression is running that our performance degredation is unbearable.  Can data progression be limited to a single thread, instead of 5 or 8 threads, and be allowed to run whenever it needs to, instead of starting a specific time and running as long as it takes?

I believe data progression can be configured in a few ways, but the settings are hidden. You'll need to get Copilot to make those changes.

I'm told by our data storage team the Copilot has told us they wont support us until we reduce our IOPS load from 12000 to 8300.   I presume Copilot means the BE IO/Sec, because our FE IO/sec averages around 3000 and peaks at 7500 occasionally.

The BE IO/sec graph shows an overal average at about 7500, but we do have daily periods that run 10,000 and spikes to 12,500.     But, the times when we experience extreme performance degradation are not every time when IP spikes are sustained over 10,000... but rather sometimes when IO is high (over 8000) ~and~ data progression is running.  When we get copilot to force data progression to stop, performance returns to excellent.

I'm not directly on our storage team, and I dont work directly with copilot, though.  So it sort of seems like I'm sitting at a table where a customer has offended the service staff, and so now we're gettting cold food, etc...

We've got a lot invested in the compellent arrays that we have... But to reduce our IO so that our sustained peaks of usage stay below 8300 IOPs, we pretty much are only going to be able to use about 30% of the storage capacity.

Is anyone else succesful at getting adequate performance at a high IOPs level, and succesful at consming more like 60% or 80% of the storage capacity?

No feeback from anyone about sustainable or peak IOPS expectations the compellents should or do live up to?

Or about "usable" capacity?   I cant find any good information about published IOPS metrics for compellents by googling, so maybe I'll post a new topic to this forum asking for some benchmarks of IOPS levels from actual real-world usage...

For us, things start to slow down around 80% capacity, but we've run our SAN over 90% in the past.   As you start filling up the SAN, you will notice your IOPS and latency spike.  Depending on what you are running, it's possible to make the SAN crawl at 30% capacity, it all depends on the average IOPS per disk.  For example, you don't want to run a crazy busy SQL DB on a handful of SATA disks.

The max IOPS per disk that I like to see are more conservative than Compellent.  For *me* I like to see a max of 130 per 15K disk, 110 per 10K, and 80 per 7K.   I believe Compellent is cool with seeing 150, 120, 90 respectively.

We don't start data progression until 11:59pm, otherwise it impacts our backup window too much.  In the past, Backups  + Data Progression made the SAN unusable (we're talking 15 minute reboot times for servers).  This was also back when we ran 15k Tier 1 (FC) and 7K Tier 3 (SATA).  We are now running SSD, 10k (SAS), 7K (SAS).  We haven't tried moving data progression back to 7pm, because the 11:59pm time continues to work so well for us.

Something else that burned us in the past was having mixed drive sizes in the same tier.  In our case it was 300gig and 450gig 15K's in the same tier.  That was just a terrible situation.  We'd have some drives doing 100 IOPS while other did 20.  Striping didn't work properly as space utilization went up.  So, stay away from that if possible.

In short..  study the IO Usage graphs in Enterprise Manager.  Without EM, you are flying blind.  I typically set my time frame to show me about 15 or 30 minutes of usage so that I can get a good idea as to what is going on.

I hope this helps...

I Need Bill of Materials for Dell server PoweEdge R730 xd

Who is help me ? pls contact via email: van.chau@giaiphapit.vn or van.chau.van@gmail.com

Hi van.chau,

You can call Dell Vietnam support line 1800 545455.

Rgds,