Skip to main content
Start a Conversation

Solved!

Go to Solution

stfconsulting

1 Rookie

 • 

28 Posts

67

March 13th, 2025 15:42

Dell ME4 Series Read Latency

Hi Everyone, We manage a bunch of ME4 and ME5 Arrays. One of them has some read latency that we are looking into. I would appreciate anyone's experience.

It is a ME4 Running the latest firmware 280-14. All of the hard disks have had the firmware updated. 

(23 x 2.4TB 10k SAS SED) and 1 1.92 TB SED Read Cache Disk. 

When you look at the response times and queue depth there are 4 disks that have abnormally high queue depth and response times (see attached). 

Curious if anyone else has hit this and how it was resolved. 

I was thinking about replacing one of the disks that is experiencing high response times to see if it goes away. I just find it odd that 4 disks are doing this?

Location   URL Vendor Rev Speed (kr/min) Size Total I/Os Data Transferred I/O Resp Time
0.2 SEAGATE SSED 10 2400.4GB 261202 133.7MB 19595
0.12 SEAGATE SSED 10 2400.4GB 267763 137.0MB 17826
0.18 SEAGATE SSED 10 2400.4GB 246646 126.2MB 17764
0.15 SEAGATE SSED 10 2400.4GB 242740 124.2MB 16682
0.1 SEAGATE SSED 10 2400.4GB 105176 53.8MB 6107
0.4 SEAGATE SSED 10 2400.4GB 104442 53.4MB 6042
0.6 SEAGATE SSED 10 2400.4GB 122691 62.8MB 5815
0.8 SEAGATE SSED 10 2400.4GB 119969 61.4MB 5636
0.11 SEAGATE SSED 10 2400.4GB 112459 57.5MB 5583
0.2 SEAGATE SSED 10 2400.4GB 150802 77.2MB 5529
0.14 SEAGATE SSED 10 2400.4GB 118933 60.8MB 5448
0.19 SEAGATE SSED 10 2400.4GB 125482 64.2MB 5426
0.1 SEAGATE SSED 10 2400.4GB 98341 50.3MB 5248
0.5 SEAGATE SSED 10 2400.4GB 101104 51.7MB 5226
0.13 SEAGATE SSED 10 2400.4GB 131425 67.2MB 5205
0.17 SEAGATE SSED 10 2400.4GB 89642 45.8MB 5171
0.7 SEAGATE SSED 10 2400.4GB 111858 57.2MB 5051
0.3 SEAGATE SSED 10 2400.4GB 108587 55.5MB 4940
0.9 SEAGATE SSED 10 2400.4GB 118494 60.6MB 4875
0.21 SEAGATE SSED 10 2400.4GB 129662 66.3MB 4778
0 SEAGATE SSED 10 2400.4GB 110085 56.3MB 4679
0.16 SEAGATE SSED 10 2400.4GB 101942 52.1MB 4513
0.22 SEAGATE SSED 10 2400.4GB 108502 55.5MB 4511
0.23 TOSHIBA B322 0 1920.3GB 463916 237.5MB
357

DELL-Charles R

Moderator

 • 

4.2K Posts

March 24th, 2025 17:20

Hello,

 

I got a read of the logs. You can review the information below. I hope this formatting comes across good.

 

 

  • Speeds are better but still higher than others
    • 0.2        Up        0        A        Standard        14529        10 ms        32 C        2400.4GB        SAS 10K        SEAGATE ST2400MM0149        SSEE        WBM5AM4L
    • 0.12        Up        0        A        Standard        14601        8 ms        33 C        2400.4GB        SAS 10K        SEAGATE ST2400MM0149        SSEE        WBM6RMPB
    • 0.15        Up        0        A        Standard        18617        10 ms        30 C        2400.4GB        SAS 10K        SEAGATE ST2400MM0149        SSEE        WBM3QXBD
    • 0.18        Up        0        A        Standard        10861        10 ms        32 C        2400.4GB        SAS 10K        SEAGATE ST2400MM0149        SSEE        WBM89HP3
  • Disks are all in ADAPT
  • Disk stats since 02/07/2025
    • 0.2        WBM5AM4L              14529   50.1KB   1    385555704   327190875  20.4TB    23.7TB       0B            0B               2025-02-07 17:55:13
    • 0.12       WBM6RMPB              14601   30.2KB   1    394203220   333654710  20.5TB    23.3TB       0B            0B               2025-02-07 17:55:13
    • 0.15       WBM3QXBD              18617   34.8KB   1    382064489   323281254  20.3TB    23.4TB       0B            0B               2025-02-07 17:55:13
    • 0.18       WBM89HP3              10861   0B       0    396852076   335831895  20.9TB    24.2TB       0B            0B               2025-02-07 17:55:13
  • These drives are in the top 4 of 5 most read drives since 2/7
    •  

Location

Bps

IOPS

Reads

Writes

Data Read

Data Written

0.23

2162.6KB

9

1337043371

632052252

85.5

59.1

0.18

0B

0

396852076

335831895

20.9

24.2

0.12

30.2KB

1

394203220

333654710

20.5

23.3

0.2

50.1KB

1

385555704

327190875

20.4

23.7

0.15

34.8KB

1

382064489

323281254

20.3

23.4
  • They are also the top 4 of 5 most written drives since 2/7
    •  

Location

Bps

IOPS

Reads

Writes

Data Read

Data Written

0.23

2162.6KB

9

1337043371

632052252

85.5

59.1

0.18

0B

0

396852076

335831895

20.9

24.2

0.2

50.1KB

1

385555704

327190875

20.4

23.7

0.15

34.8KB

1

382064489

323281254

20.3

23.4

0.12

30.2KB

1

394203220

333654710

20.5

23.3
  • The only drive that does more work is the Read Cache drive which is #1
  • It appears that the disks may be higher latency as they are accessed the most of any of the drives
    • This implies that the data on those disks are accessed more frequently
    • There is not much that can be done to change that other than operations that force data movement
    • Normally we would suggest splitting up the data into different pools to handle the different loads in the data
    • Another normal suggest would be to have a SSD tier so the hot data could move to the SSDs, but we only have 1 SSD

 

POA

  • As the disks are within expected performance for spinning disks, there is not much to do

Suggestions

  1. Purchase 1-3 SSDs
    1. Offload the data
    2. Delete the pool
    3. Swap 1-3 spinning drives for SSD
      1. Keep the spinning disks for spares
    4. Create a new pool with
      1. SSD tier with the 2-4 SSDs
      2. ADAPT on the spinning disks
    5. Recreate volumes and reload data
    6. Now with SSD tier, frequently accessed data will be on the SSDs for faster response times
  2. Recreate pools
    1. Offload the data
    2. Delete the pool
    3. Split up the spinning disks into 2 different pools
    4. Designate 1 pool to be the faster one and add the RC SSD
    5. Only load frequently read data into this pool/DG/volume
    6. Load the other pool with the other data
    7. Load balance as needed with file level copies
  3. Add new enclosure with disks
    1. Add SSDs to new enclosure
    2. Remove the RC SSD as read cache
    3. Add SSD tier instead

 

 

stfconsulting

1 Rookie

 • 

28 Posts

March 13th, 2025 15:44

Here is the table again. I can't get anything to format properly.

Location   URL Vendor Rev Speed (kr/min) Size I/O Resp Time Health
0.2 SEAGATE SSED 10 2400.4GB 19595 OK
0.12 SEAGATE SSED 10 2400.4GB 17826 OK
0.18 SEAGATE SSED 10 2400.4GB 17764 OK
0.15 SEAGATE SSED 10 2400.4GB 16682 OK
0.1 SEAGATE SSED 10 2400.4GB 6107 OK
0.4 SEAGATE SSED 10 2400.4GB 6042 OK
0.6 SEAGATE SSED 10 2400.4GB 5815 OK
0.8 SEAGATE SSED 10 2400.4GB 5636 OK
0.11 SEAGATE SSED 10 2400.4GB 5583 OK
0.2 SEAGATE SSED 10 2400.4GB 5529 OK
0.14 SEAGATE SSED 10 2400.4GB 5448 OK
0.19 SEAGATE SSED 10 2400.4GB 5426 OK
0.1 SEAGATE SSED 10 2400.4GB 5248 OK
0.5 SEAGATE SSED 10 2400.4GB 5226 OK
0.13 SEAGATE SSED 10 2400.4GB 5205 OK
0.17 SEAGATE SSED 10 2400.4GB 5171 OK
0.7 SEAGATE SSED 10 2400.4GB 5051 OK
0.3 SEAGATE SSED 10 2400.4GB 4940 OK
0.9 SEAGATE SSED 10 2400.4GB 4875 OK
0.21 SEAGATE SSED 10 2400.4GB 4778 OK
0 SEAGATE SSED 10 2400.4GB 4679 OK
0.16 SEAGATE SSED 10 2400.4GB 4513 OK
0.22 SEAGATE SSED 10 2400.4GB 4511 OK
0.23 TOSHIBA B322 0 1920.3GB 357 OK

DELL-Charles R

Moderator

 • 

4.2K Posts

March 13th, 2025 20:35

Hello,

 

Did you try swapping out a drive to see if it helped?

 

 

Let's get a log for review.

 

How to Collect support logs on ME4 models

https://www.dell.com/support/kbdoc/000120519

 

 

How to Share Log Files, Screenshots, and Error Messages with Dell

https://www.dell.com/support/kbdoc/en-us/000212038/how-to-share-log-files-screenshots-and-error-messages-with-dell?lang=en

 

 

I will send you a Direct Message with the case number you can upload the log to.

 

Direct Messages can be accessed in the top right of the forum page.

 

stfconsulting

1 Rookie

 • 

28 Posts

March 13th, 2025 20:38

Hey Charles, not yet. I would be happy to share some logs with you. 

We had another ME4 with the latest firmware on everything do something similar to this recently. In that case it was just one drive however it was different model than the 22 other drives. That one we are still waiting for a maintenance window to swap it.

stfconsulting

1 Rookie

 • 

28 Posts

March 13th, 2025 21:57

Logs are uploaded Charles. Thanks for your time.

DELL-Joey C

Moderator

 • 

3.7K Posts

March 14th, 2025 02:09

Hi,

 

Was the drives recently replaced? After a brief look at the logs, the drives show lower POH (PowerOnHours) than other drives. 

 

In the logs, the drive firmware are on SSED version, there is an update for the drives, SSEE. Could you check if they can be updated to the latest: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=3nf59&oscode=wst14&productcode=powervault-me4024

 

Do feedback with Charles about the swapping slot, just to make sure that it's not the slots which are causing the issue. It is advisable to power down the storage and swap drive, probably just swap 2 drives will do, say 0.2 with 0.3. The reason of powering down the storage is because of metadata storage clearance. 

stfconsulting

1 Rookie

 • 

28 Posts

March 14th, 2025 12:08

No not recently. Thanks for pointing that. I am going to read the release notes for SSEE. Appreciate your pointing that out. The read latency is not horrible (Average on most of the ME4/ME5 that we manage is 5-7ms) This one is 10-11ms. If it gets busy it can spike more than it should.

stfconsulting

1 Rookie

 • 

28 Posts

March 14th, 2025 12:26

Do you guys agree that they latency on these 4 drives is abnormal?

DELL-Charles R

Moderator

 • 

4.2K Posts

March 14th, 2025 12:38

Hello,

 

Thank you for the logs. I will get a review as soon as I can.

 

Have you had a chance to update the firmware Joey provided?

 

I can follow up with you as soon as I have more information.

DELL-Charles R

Moderator

 • 

4.2K Posts

March 14th, 2025 17:11

Hello,

 

I have worked with an Engineer and outlined a POA:

  • We do see higher response times on the disks but not outside expectations of spinning disks at 12-14ms (over 20 is concerning)
    • Get the drives updated as there are several fixes in the list that could affect things
  • Get maintenance window
  • Stop IO to the storage
  • Update HDDs to the latest
    • ST2400MM0149 to SSEE
  • Recheck after changes

 

stfconsulting

1 Rookie

 • 

28 Posts

March 14th, 2025 17:14

Thank you Charles. I will work on this and report back.

stfconsulting

1 Rookie

 • 

28 Posts

March 24th, 2025 11:31

Charles, I was able to upgrade the firmware on all the drives. At first glance is seems better however w few of the drives still have larger than normal response times. I tried to post the logs to the case but it will not let me. If you give me a new place to upload the logs I will post them for review.

DELL-Charles R

Moderator

 • 

4.2K Posts

March 24th, 2025 12:24

Hello Sean,

 

Can you try to upload again to the same case number?

 

stfconsulting

1 Rookie

 • 

28 Posts

March 24th, 2025 14:38

Logs uploaded. Thank you!

DELL-Charles R

Moderator

 • 

4.2K Posts

March 24th, 2025 14:45

Thank you for letting me know.
I'll get the log for review and follow up with you.

View More

View All

No Events found!

Top