Safely Remove Hardware

Try to use the Safely Remove Hardware feature to remove my USB External Hard Drive and keep getting message that " The Generic volume' cannot be stopped right now. Try stopping the device later."  This has been going on for some time now. Tried to find out if it was using any resources but can't tell. The only thing on it that could possibly affect it is Norton Ghost.

Any suggestions appreciated in that I don't care to loose any data on the drive.