NetWorker: How to Analyze IO Performance Using ddboost_precert.log

Precert logging is enabled as a troubleshooting step where NetWorker Client Direct backup issues are thought to be related to the client's Data Domain DD Boost library. This utility can identify performance issues from the precert logs collected.

Preparing ddboost_precert.log

As a simple usage example, create ddboost_precert.ini on the UNIX root, or root of C:\ for Windows. The file should contain: 

precert_enabled=true
num_log_files=4
log_file_size=512
log_file_name=ddboost_precert.log
log_file_dir=/DDLog
ddcl_logs_include=true
ddcl_logs_sev=4

See article Data Domain: Enable DD Boost API logging with the precert tool (precert logs) for complete procedural details for collecting precert logs. (login to Dell Support is required to view this article)

Preparing to use the script

Prerequisite:  Python 3.x 
Download the script from: Solutions NetWorker Tools (dell.com)

1. Save the script to a working directory containing the precert logs as precert-io-perf.py.
2. On UNIX systems, ensure that the script is executable: chmod +x precert-io-perf.py
3. Run the script: ./precert-io-perf.py

Input Prompts

• Enter the full path to the DD Boost precert log file when prompted.
• Enter Yes or No when prompted whether to include MB/s for each I/O operation

Output Summary

The script displays a summary of the write and read operations, including:

• Total number of write and read operations
• Average write and read duration
• Maximum and minimum write and read duration along with their respective throughput (MB/s)
• Total write and read time
• Percentage of total time spent on write and read
• Average write and read throughput (MB/s)
• Detailed I/O Operations

The script logs detailed I/O operations to two CSV files:

• io-write.log - contains detailed information about each write operation
• io-read.log - contains detailed information about each read operation

Example Output

# ./precert-io-perf.py
Please enter the full path to the log file: ./ddboost_precert.log
Do you want to include MB/s for each I/O operation? (yes/no): yes

    Write Durations Summary:

    Total number of write operations: 2646
    Average write duration: 0.005389 seconds
    Maximum write duration: 0.041000 seconds (24.39 MB/s)
    Minimum write duration: 0.000001 seconds (1000000.00 MB/s)
    Total write time: 14.259001 seconds
    Percentage of total time spent on writes: 22.99%
    Average write throughput: 185.566997 MB/s
    Note: High MB/s values for very short durations indicate efficient operations.


    Read Durations Summary:

    Total number of read operations: 2651
    Average read duration: 0.018018 seconds
    Maximum read duration: 31.838000 seconds (0.03 MB/s)
    Minimum read duration: 0.001000 seconds (1000.00 MB/s)
    Total read time: 47.766000 seconds
    Percentage of total time spent on reads: 77.01%
    Average read throughput: 55.499728 MB/s
    Note: High MB/s values for very short durations indicate efficient operations.


Top 20 Slowest Write I/O Operations:
1. I/O Number: 608, Duration: 0.041000 seconds, MB/s: 24.39
2. I/O Number: 2276, Duration: 0.038000 seconds, MB/s: 26.32
3. I/O Number: 1548, Duration: 0.025000 seconds, MB/s: 40.00
4. I/O Number: 2076, Duration: 0.025000 seconds, MB/s: 40.00
5. I/O Number: 636, Duration: 0.024000 seconds, MB/s: 41.67
6. I/O Number: 195, Duration: 0.023000 seconds, MB/s: 43.48
7. I/O Number: 892, Duration: 0.023000 seconds, MB/s: 43.48
8. I/O Number: 1451, Duration: 0.023000 seconds, MB/s: 43.48
9. I/O Number: 1539, Duration: 0.023000 seconds, MB/s: 43.48
10. I/O Number: 1987, Duration: 0.023000 seconds, MB/s: 43.48
11. I/O Number: 604, Duration: 0.022000 seconds, MB/s: 45.45
12. I/O Number: 896, Duration: 0.022000 seconds, MB/s: 45.45
13. I/O Number: 1283, Duration: 0.022000 seconds, MB/s: 45.45
14. I/O Number: 1532, Duration: 0.022000 seconds, MB/s: 45.45
15. I/O Number: 1544, Duration: 0.022000 seconds, MB/s: 45.45
16. I/O Number: 595, Duration: 0.021000 seconds, MB/s: 47.62
17. I/O Number: 616, Duration: 0.021000 seconds, MB/s: 47.62
18. I/O Number: 652, Duration: 0.021000 seconds, MB/s: 47.62
19. I/O Number: 1512, Duration: 0.021000 seconds, MB/s: 47.62
20. I/O Number: 1552, Duration: 0.021000 seconds, MB/s: 47.62

Top 20 Slowest Read I/O Operations:
1. I/O Number: 5900, Duration: 31.838000 seconds, MB/s: 0.03
2. I/O Number: 1614, Duration: 0.041000 seconds, MB/s: 24.39
3. I/O Number: 4950, Duration: 0.039000 seconds, MB/s: 25.64
4. I/O Number: 4550, Duration: 0.026000 seconds, MB/s: 38.46
5. I/O Number: 1670, Duration: 0.025000 seconds, MB/s: 40.00
6. I/O Number: 3494, Duration: 0.025000 seconds, MB/s: 40.00
7. I/O Number: 788, Duration: 0.023000 seconds, MB/s: 43.48
8. I/O Number: 1606, Duration: 0.023000 seconds, MB/s: 43.48
9. I/O Number: 2182, Duration: 0.023000 seconds, MB/s: 43.48
10. I/O Number: 3300, Duration: 0.023000 seconds, MB/s: 43.48
11. I/O Number: 3476, Duration: 0.023000 seconds, MB/s: 43.48
12. I/O Number: 3486, Duration: 0.023000 seconds, MB/s: 43.48
13. I/O Number: 4372, Duration: 0.023000 seconds, MB/s: 43.48
14. I/O Number: 2190, Duration: 0.022000 seconds, MB/s: 45.45
15. I/O Number: 2964, Duration: 0.022000 seconds, MB/s: 45.45
16. I/O Number: 3422, Duration: 0.022000 seconds, MB/s: 45.45
17. I/O Number: 3462, Duration: 0.022000 seconds, MB/s: 45.45
18. I/O Number: 4628, Duration: 0.022000 seconds, MB/s: 45.45
19. I/O Number: 5548, Duration: 0.022000 seconds, MB/s: 45.45
20. I/O Number: 1588, Duration: 0.021000 seconds, MB/s: 47.62
Detailed I/O operations have been logged to io-write.log and io-read.log


example of above analysis:

Percentage of total time spent on writes: 22.99%
Percentage of total time spent on reads: 77.01%

This example demonstrates how the performance issue appears with the read operation of the data as opposed to the write operations.