hi all ,
we are facing a strange issue, briefly our backup structure is as follow:
we have two NetApp filers : 1st 7-mode and the 2nd cluster mode connected to a SAN switch and connected to the backup server which have EMC networker v 8.1 and then connected to IBM tape library which have 10 drives.(we use NDMP backup)
backup was forking fine till suddenly the cluster mode filer was unable to take backups; suddenly the networker gives error read open error the device is in use for the two drives dedicated for this filer so i had to re-configure the library on the networker and assign another two tape drives but apparently the previous two tapes drives are not available to be assigned again for this particular filer while being available to the others.
this error message kept on coming again and again, now am out of any tape drive for this filer they seem to be undiscoverable or they just disappeared.
running inquire -N -on 1st filer am able to see my 10 drives
running inquire -N -2nd filer (cluster mode) am able to see only a library and other tape drives are seen busy as below
email@example.com:IBM ULT3580-TD6 C7S4|Tape, nrst3a
ATNN=IBM ULT3580-TD6 00078B00A9
0 Tape, 1 Autochanger (Jukebox), Total: 1
Device 'nrst6a' is busy. It cannot be opened at this time.
Device 'nrst7a' is busy. It cannot be opened at this time.
Device 'nrst1a' is busy. It cannot be opened at this time.
Device 'nrst4a' is busy. It cannot be opened at this time.
Device 'nrst0a' is busy. It cannot be opened at this time.
Device 'nrst2a' is busy. It cannot be opened at this time.
Device 'nrst5a' is busy. It cannot be opened at this time.
Device 'nrst8a' is busy. It cannot be opened at this time.
any ideas why could this be happening here ?
cluster mode can be bitch. When you storage tape show-tape-drive and system node hardware tape drive show - how many and which do you see assigned? Which OnTap version do you use? How do you access drives - over node or cluster alias?
all are showed perfectly from the netapp filer as below:
OnTap version 8.2.4.p6
we use node scope level.
BPcDOT::> tape drive show
(system node hardware tape drive show)
Node Device Id Drive Description NDMP Path
------ --------- -------------------- --------------------------------------
Main-SAN-Switch:1-1.126 nrst0l nrst0m nrst0h nrst0a
IBM LTO-6 ULT3580 rst0l rst0m rst0h rst0a
urst0l urst0m urst0h urst0a
Main-SAN-Switch:1-14.126 nrst8l nrst8m nrst8h nrst8a
IBM LTO-6 ULT3580 rst8l rst8m rst8h rst8a
urst8l urst8m urst8h urst8a
Main-SAN-Switch:1-2.126 nrst2l nrst2m nrst2h nrst2a
IBM LTO-6 ULT3580 rst2l rst2m rst2h rst2a
urst2l urst2m urst2h urst2a
Main-SAN-Switch:1-3.126 nrst3l nrst3m nrst3h nrst3a
IBM LTO-6 ULT3580 rst3l rst3m rst3h rst3a
urst3l urst3m urst3h urst3a
Main-SAN-Switch:1-4.126 nrst5l nrst5m nrst5h nrst5a
IBM LTO-6 ULT3580 rst5l rst5m rst5h rst5a
urst5l urst5m urst5h urst5a
Main-SAN-Switch:1-5.126 nrst1l nrst1m nrst1h nrst1a
IBM LTO-6 ULT3580 rst1l rst1m rst1h rst1a
urst1l urst1m urst1h urst1a
Main-SAN-Switch:1-6.126 nrst7l nrst7m nrst7h nrst7a
IBM LTO-6 ULT3580 rst7l rst7m rst7h rst7a
urst7l urst7m urst7h urst7a
Main-SAN-Switch:1-7.126 nrst6l nrst6m nrst6h nrst6a
IBM LTO-6 ULT3580 rst6l rst6m rst6h rst6a
urst6l urst6m urst6h urst6a
Main-SAN-Switch:1-8.126 nrst4l nrst4m nrst4h nrst4a
IBM LTO-6 ULT3580 rst4l rst4m rst4h rst4a
urst4l urst4m urst4h urst4a
Main-SAN-Switch:1-9.126 nrst9l nrst9m nrst9h nrst9a
IBM ULT3580-TD2 rst9l rst9m rst9h rst9a
urst9l urst9m urst9h urst9a
10 entries were displayed.
And volumes writing to that tape as on the same volume or do they send data over interconnect? Since you started this thread I was wondering if tape goes off/busy due some failover mechanism which keeps hanging it in some strange state.I assume you already checked for active NDMP sessions on node to see there is nothing running? What happens if you restart ndmp service? Are those tape drives back as not busy?
they are still listed busy though we restarted the NDMP services, we tried also restarting the backup server and nothing actually happened, i agree with you that there might be a service that make the tape drives go busy but actually the drives are working fine with the other netapp and they are not seen busy.
Yes, but others are mode7, right? NDMP was not top of the head when cluster mode came out and you can see with each new version OnTap is closing this gap, but there are still certain things left to be desired. I have cluster mode, but we don't use DSA instead (as we use DD so I just push things over network).