Recurring problems with long-running backup jobs TL2000

Question

We have several TL2000 tape loaders at remote sites used for backup. Periodically the long-running jobs will somewhat consistently fail with an E00084ED error in backup exec, indicating a communication error with the tape drive. These are accompanied in the event log by event ID 129 'Reset to device, \Device\RaidPort1, was issued', and event ID 11 'The driver detected a controller error on \Device\RaidPort1. ITDT diagnostics show no problems with the drive or communication. Small jobs to tape work and verify, long jobs to disk work and verify, long jobs to take nearly always fail. In the past we have replaced cleaned drives with new cleaning cartridges, replaced entire tape drives, switched to brand new tapes, none of which had any effect. Only splitting the jobs up into smaller ones (< 12 hr run time including verify) stopped the failures, and this is becoming impractical.  The event log errors seem to point to something with the storage controller, a Dell 6Gbps SAS HBA. LSI cards and autoloaders seem to have these issues frequently, Microsoft has a trouobleshooting document :https://support.microsoft.com/en-us/help/154690/how-to-troubleshoot-event-id-9,-event-id-11,-and-event-id-15-error-messages that includes the following advice:  Slow the transfer rate settings if the timeouts are associated with tape drives - using 5MBS transfer rate usually cures the timeouts. Where would I 'slow the transfer rate'? Are there any other troubleshooting steps I can take? PowerEdge T610 server running Windows Server 2008r2, BE 2010, Dell 6Gbps SAS HBA, TL2000 autoloader with single LTO5 drive.

tapelibraryfixer · Answer

event codes and error codes can be misleading a lot of the time, some people have come back with software issues some say cable issues, if you say your tapes and tape drive are ok, next step is either try updating firmware, drivers and even backup exec. i would think if its happening on multiple units its more software related

also the transfer rate change, i would think that's more to do with older scsi devices than an LTO, your not gonna want an LTO5 running at 5mb second.

SGUS_1 · Answer

I've updated drivers and firmware at the HBA card, the library, and the drive, no apparent impact on the problem. I did find something potentially interesting though- if I sort my jobs by 'job rate', which appears to be BE's data throughput metric of the job, I see a nearly 100% failure rate for the fastest jobs down to about  1,700 MB/min. Below that, probably a 90%+ success rate. I checked the other server that was having the same problem and found the same pattern and even the same cutoff point of about 1,700 MB/min. The correlation between failure and job rate is even stronger than failure and job elapsed time. Since I don't know how it's calculated I could be looking at a symptom rather than a cause. However, it seems logical that the pattern indicates that some of the components of the process can operate at higher speeds but others cannot, and job rates over the 1,700 MB/min threshold are always failing. Is there a way I can throttle the card, drive, or something to less than 1,700 MB/min as a test?

DELL-Sam L · Answer

Hello SGUS_1

There is not a way that I am aware of to slow the transfer speed on your TL2000. Your TL2000 will spin the drive up once it receives the command from your backup software that data is coming. I am not sure if you can make that adjustment on the HBA as well.

Please let us know if you have any other questions.

PowerVault

Was this post helpful?