Error: I/O bus was reset on PowerEdge 2600

Question

First of all, if anyone can be of assistance on this issue, I would greatly appreciate it.

I am currently using Veritas Backup Exec ver. 8.6 on our DELL PowerEdge 2600 server. The server has a DELL (tm) PowerVault (tm) 110T1/VStape Drive.

Over the past few weeks, our backups have started giving this error:

Storage device "DELL 1" reported an error on a request to write data to media.

Error reported:

The I/O bus was reset.

^^^^^

Once that error is given, the backups will fail. Typically, the backup would run for about 6 hours and fail. Thinking it was a timing issue, I moved most of the data being backed up to another server so that this server was only backing up data locally. However, after 3 hours last night, the backups again failed with this error. Last weekend, I did apply all updates to the BIOS, DELL Perc4DI, etc. This was done after calling DELL Tech Support and explaining my plight to them. The Saturday after the updates were done, the backups actually went through successfully. However, on Monday night's backups, they again failed with the error. I have already had the tape drive replaced, but am wondering if the issue may lie elsewhere. I attempted to update the SCSI controller drivers/firmware, but do not even see a SCSI card listed on the server under Device Manager. As far as I know, the tape drive runs off of a daughterboard attached to the motherboard and isn't a separate device. Basically, I need to get back to the point where this error does not occur.

If anyone can give any insight on this issue, it is greatly appreciated.

Thanks in advance.

Jeffrey G. Kowalke

DELTEC Information Solutions

JeremyMordkoff · Answer

I suspect we have the same problem and it must be pretty low level software or hardware.

I have a redat 7.3 system that fails to fsf quite often during restores with an i/o error. in the log I find

Jul 28 09:38:37 lux1 kernel: st0: Error with sense data: Current st09:00: sense key Unit Attention
Jul 28 09:38:37 lux1 kernel: Additional sense indicates Power on,reset,or bus device reset occurred

The funny thing is, if I restore every single file from the tape, it seems to be ok. It only has trouble fsf'ing past certain (large) files.

JLM

WilsonTorres · Answer

Jeffrey,

I am the Systems Support Analyst for my company and you are not going to believe this but I have the exact same problem. Every line you described on your problem applies to us, I am yet to update any BIOS or configurations.
We purchase the same server model/type from DELL (PowerEdge 2600) about six months ago with the Veritas Backup Execute Ver. 8.6 The backup started failing few weeks after it was installed, about once per week. We bough new tapes and started clearing the drive everyday instead of every so often, this did not work very well, the backup kept failing, so I went and moved about 9.0GB of data from this backup to see if it was a time out issue but it keep failing. Currently we are at the point where we are ready to purchase/replace the backup drive.
Question? Did DELL replaced your? was it under guarantee?
How are things running for you now?
Since you and I are running the same server model/type and have ran into the same problems, maybe we can keep in touch for future problems/comments/concerns.

Thank you for your attention and prompt response

Wilson Torres
Universal Packaging Systems, INC

jgkowalke · Answer

Wilson,

I saw that I had some responses to the post I had made a while back and came back to revisit it. The interesting part about all of this is that I am now rarely having the issues that I had had before. My conclusion on this is that it could have been the media that was causing the issues. I have obtained several successful backups since the last time I have received the message. Also, when the job had failed due to the error, I marked the tape as such, and made sure it wasn't used again.

Now, with the DLT drives, I have found that the drive cannot be cleaned regularly. The reason we did have one tape drive replaced was that I was doing my proactives on the drive or cleaning it regularly. However, after talking with DELL, I found that the only time I should be cleaning the drive was when it asked me too. DELL informed me that I was actually ruining the tape mechanism by cleaning the drive too often. From my work in the past, I didn't know this could be true, but I hadn't worked with DLT drives before, only AIT, DAT, etc. From this, I thought that some of the media might have gotten damaged when I was using the old tape backup drive and having issues with it.

Since August 2nd, the backup job has failed with the error only once, on August 6th. Every backup during that time has completed successfully. When the rotation comes back around to use the tape, I plan on swapping out the tape for another and seeing where the job goes from there.

I hope this information helps you. I will keep my eye open for messages from the board if you respond.

Jeffrey G. Kowalke

DELTEC Information Solutions

WilsonTorres · Answer

FYI: Problem has been resolve

The following steps were taken to resolve our backup drive jobs failing due to the following error message:

Storage device “DELL 1” reported an error on a request to write to write to media
Error reported:
The I/O bus was reset
^^^^^

After contacting DELL Support the first step was to find out if the backup drive was connected directly to the motherboard or to the 39160 SCSI Control Card.

We were unable to see the drive configuration under the Device Manager so the server was shutdown in order to remove the side panel and physically check the connections.
We find out that the backup drive was connected to the motherboard but it was not configured properly in the original setup and the 38160 SCSI Control Card was just
not being utilized in any shape or form.
We then located the backup drive SCSI cable and unplugged it from the motherboard SCSI id B (the motherboard uses 2 build in SCSI controllers, SCSI id A in use with the hard drives) and moved it into the 38160 SCSI Control Card ids A.
The server then was started up halted at the BIOS setting for the new SCSI Controller with the ALT/A key combinations; the card was check at this point for connectivity.
Once the server rebooted all the way up we downloaded the following set of drivers, patches and updates for the SCSI Control card and the Backup drive.

WIN2KA13.exe for 39160 SCSI Control Card Drivers
160FW310.exe for 39160 SCSI Control Card Firmware
R62854.EXE for Power Vault 110T DLT VS80 Tape Drive upgrade/utility

Once the download completed the drivers were installed, server rebooted
Then the firmware was applied to the new configuration, server rebooted
At this time while the server was rebooting a set of warning messages were coming up indicating that the SCSI card configuration did not match and to reboot again, this process took 3 reboots to totally removed all warning messages.
Finally, the backup drive update/utility was installed and sets of 4 different backup tests were performed to a successful completion.

At this time everything had been configured and tested, the nightly backup was reset to its original setting, which ran to completion the same night.

The origin of the problem came when the backup drive was installed directly to the motherboard instead of the SCSI control card, the motherboard is not capable of keeping up with the I/O requests of the backup job it only process data transfers request at 80 seconds interval while the SCSI control card does this at only 20 eliminating data overruns.

jgkowalke · Answer

Wilson,

This is great information! I am going to check out the server this weekend to see if I have the same configuration. As far as I remember, the configuration is exactly how you described with the tape drive being connected direction to the motherboard. Thank you very much for the update. I may be on the wrong track completely with the media idea, and will check out the hardware configuration regardless.

Thanks again!

Jeffrey G. Kowalke

DELTEC Information Solutions.

WilsonTorres · Answer

Jeffrey,

Thank you! you gave me an starting point to resolve this problem.
I check this forum from time to time, maybe we can help each other next time.

Wilson Torres
Systems Support Analyst
Universal Packaging Systems, INC.

jgkowalke · Answer

Wilson,

Hopefully are you still checking this post. I was able to take down the server this evening while working on another issue. What I found when I opened up the server was that the server didn't contain a separate SCSI controller card; none that I could see anyway. Is the SCSI controller card you are referring you a separate device in the onboard expansion slots? As of right now, there is nothing in those slots whatsoever. If there is supposed to be, I may have to go back to the original specs on ordering the server and see if something is missing. Let me know when you have a moment. Thanks again.

Jeffrey G. Kowalke

DELTEC Information Solutions

jgkowalke · Answer

I did want to add also that no SCSI controller card is listed in the Device Manager also.  Thanks in advance. Jeffrey G. Kowalke DELTEC Information Solutions

WilsonTorres · Answer

Jeffrey,

YES, You do need to install an SCSI Card Controller for the Backup drive to work properly, we are using a 39160 SCSI Card (recommended by DELL. The drivers, firmware and backup drive software utilities were listed on my previous e-mail.
Eh, if you like you can e-mail directly at work at W.Torres@upsilab.com if you have any further questions.
PS: All my backup have completed successfully since I made these changes.

Hope you make this changes soon to avoid future headaches.

Sincerely,

Wilson Torres
Systems Support Analyst
UPSI

shatf · Answer

It took me awhile to find the solution but adding the 39160 solved this same problem. I wish Dell would clearly point this out when purchasing these for use with newer dell servers.

wyckoffj · Answer

I was having the same problems with my PE2600 server and tape backup using Backup Exec 8.6, giving me I/O errors and saying the bus had been reset. After reading this forum, I called Dell Tech Support and they had some ideas other than just buying another SCSI card.

My configuration is the on-board SCSI controller controlling the RAID configuration on one channel and the Tape drive on the other channel.

Dell Tech Support had me update three things, and now I am getting backups every night error-free. They told me to update the SCSI driver, the SCSI firmware, and then to update Backup Exec 8.6. All three updates were available from the Dell FTP site.

For the RAID Driver (do NOT use this if you're not using a RAID configuration):
ftp://ftp.dell.com/scsi-raid/RAID-DRVR-R63411.exe

For the RAID Firmware (do NOT use this if you're not using a RAID configuration):
ftp://ftp.dell.com/scsi-raid/RAID-FRMW-R63812.exe

For the Veritas Backup Exec Update:
ftp://ftp.dell.com/tape-sw/bnt86idrv36_253800.exe

Please only use these updates if you have the same configuration that I have. If your configuration is different, it might be a good idea to contact Dell Tech Support to get the right updates for your machine.

Good luck!

Oliver Trowman · Answer

Wow!!!  At last there appears to be a fix for this problem.  I had this more or less straight away when we bought our PE2600.  We actually had the tape drive swapped out because it physically broke, and only recently has this cropped up again - best check with Tech support that this fix will work on our machine first!!

pmolaughlin · Answer

We have been having problems with our backups on our 2600's for a while now.  As soon as we applied these patches, our problems seemed to disappear.  Thanks for the fix, wyckoffj!

DBakerMan · Answer

Just to be complete, you should also know that sometimes this poblem can be only be fixed by replacing the tape drive. We have had the issue on a number of machines and while yes, the patches do help, so does setting the SCSI bus speed on the tape channel to the slower speed.

In one case though, the problem only went away after we had Dell replace the drive all together.

PE2600 · Answer

Hi,

we got around this problem without recourse to buying another SCSI card by demanding that DELL send us a new card for free on the basis that they sold goods unfit for the purpose for which it was sold (ability to reliably restore files would come under this I think !)..

The new cards that DELL sent us work and we now have 100% successful daily backups.

The thought of having to flash a raid bios in a live environment without being able to guarantee a restore in the event of losing all data, if unsuccessful, doesn't bear thinking about.. that's my humble opinion anyway..

good luck and regards

PowerVault

Error: I/O bus was reset on PowerEdge 2600

Was this post helpful?