Live Optics | Optical Prime | Queue Depth: A deeper look

Disk Queue is often thought of the first indicator of poor application performance, but it is frequently blamed too early. The following explanation is a quick-and-dirty guide to understanding some basic approaches to demystifying Disk Que.

Let us break this up into two parts. The basic hit and run survival guide for those of you that do not have the time to read on and a more in-depth understanding of why Disk Queue got to be a focal point in the first place. 

The Survival Guide:

AN Optical Prime project shows the number of outstanding IOs from the OS's perspective for each sample throughout the recording period. If Disk Queue is the problem, it should be tightly associated with latency in the same period. So, from good to worse:

- Low Disk Queue and Low Latency = A likely happy application and user experience

- High Disk Queue and Low Latency = If latency stays desirable this should be OK.

- Low Disk Queue and High Latency = Needs attention, but it is unlikely to be your storage.

- High Disk and High Latency = Should look at your storage as a potential bottleneck.

This last one should be investigated and here is where the value of Optical Prime representing performance over time is important. If Disk Queue is causing the Latency, one should see tightly correlated patterns between the two values.  

Here is an entry that shows an example of a good correlation between latency and Disk Queue.

To understand the basics of Disk Queue Length think of a check-out line at your local "Food-Mart." Everyone knows the drill… you select your items, you get in line to be checked out, when it is your turn you pay, and finally you own the item.

Everyone has also been there on holiday or late at night when the line is long and the poor check-out clerk has a queue of upset people saying, "Why does not management just open up more check-out lanes!"

At a basic level of definition, Disk Queue is the number of outstanding disk operations that are "waiting in line" and thus the reason that it is often looked at to indicate storage trouble.

We all know that adding more check-out clerks at Food-Mart would make the line fan-out and go faster and it is doing so because we increased our degree of parallel work. The same basic principles can be applied to IO requests. If I had only one disk in my server trying to do all that work or even, let us say, a small RAID 5 trying to do that work. Then we could conceive that the application would generate a workload demand where the check-out line for I/Os would get backed up. This high Disk Queue phenomenon is called being "Spindle Bound." Plain and simply put the disks cannot keep up with the demand so a line is formed and that manifests itself as Latency to the Operating System.

The basic guideline is that a Disk Queue of more than 2-4 is bad.

Easy, right? Well it gets more complicated unfortunately.

The rule is that a Disk Queue of more than 2-4 per disk is bad… the reason it gets hard is that Optical Prime does not tell you how many disks make up that "F: Drive."

Seems trivial enough, why do not we grab the number of disks and call it a day? Well, we cannot always do that. Some drives are really partitions and an E: and F: drive might be on the same disk. A better mask of the truth comes from storage arrays themselves.

Any external disk array that might be representing a Volume or a LUN to an Operating system could be masking any number of drives from the OS. For example, an array could have a RAID group of 4 or 9 disks making up the LUN that is represented to the Windows OS as the "F: Drive"… so if we have a Disk Queue of 15 is it bad… or is it OK?

Application Interference
Some applications can manage the Disk Queue or be responsive to it. A management technique application such as SQL server can throttle back the I/O as to not create too many outstanding IOs. If they see the Disk Queue climbing, it can mask the problem by not allowing it to get out of hand.

Data Patterns
Back at the check-out counter at Food-Mart… When the manager does finally wake up and opens three new check-out lanes the people can fan-out and go through the lines because their purchases are unrelated. They are like Random I/O. Each person goes through the line independent of anyone else. Random IO is the same. Each operation wants to finish as quickly as possible and does not really care about anyone else.

Sequential data is the opposite and can be thought of more like a movie. A movie is a series of still frame photos that are played "in sequence" to give you the effect of a motion picture. For the movie to make sense those frames must be played in order and are dependent on that order for the movie to make sense. (minus any Quentin Tarantino films of course)

Often sequential I/O cannot be broke into parallel activity. Depending on the nature of the program running the sequential workload, you may or may not see an increased amount of Disk Queue and Latency, but might see a similar correlation extending to the I/O Transfer size. To learn more about this, read the post on how I/O Transfer Size can effect Latency.

Summary
Today with SSDs and virtualized storage the chances of having disks being the bottleneck are not like they were when 15 K RPM drives were the highest tier. Nonetheless, it is something worth looking into every time you are hunting down a latency issue.  

It is almost easier to rule out that disk is causing the latency than find the cause of the latency. But a least you can have one less place to look :).

If you have any questions, please reach out to Live Optics Support at liveoptics.support@dell.com.