First of all , your environment is not supported as your NetWorker server OS is Windows 2008 R2 with NetWorker 7.6 SP1 , where the supported version for that OS is 7.6 SP2 and higher , so before any troubleshooting you have to upgrade your NetWorker version on the backup server and the remote storage nodes as well. The latest now is NetWorker 7.6.3.3 which can be downloaded from the following FTP link:
After the upgrade you have to check that you have the latest updates and firmwares for all your hardwares ( Tape drives, HBA, etc.. ) , If you have any available updates you have to patch it firstly so that you environment be in upto-date state. Also Check for corrupt NIC (network card) driver and those cards are configured properly.
I recommend you to run the bigasm test , in order to measure tape device writiing speed, you can find how to do that test from the following Powerlink KB article:
I would normally agree with you, but this is a very isolated problem. backups are fine across various LAN and WAN links, issues are only encountered on that specific server (which was rebuilt 3 weeks ago, and i have done all the checks for driver upgrade etc)
Oh and i made a mistake, backup server is 2008 SP1, sorry about the confusion.
1 - 1 hop, being routed through cisco nexus from new network to old. all on 1Gbps switches. Cisco nexuses are configured as your run-of-the-mill switches.
2 - yes and yes
3 - no encryption, tried with compression on and off, same result...
4 - no, while it is slow, it hits 30-40mbps easy, which would be enough to satisfy the business and all people hanging over my head
5 - I backup a good 90 servers, various back office applications, and only this one is causing me greif. Spoke to the DBA who is in charge of the system, i am backing up backups of multple SQL DBs and logs. One of the LUNs the files are easy 10-50GB in size, other lun is hdfs. performance is the same.
On a side note, i am backing up good 40 servers of the new network (one on the cisco nexus backbone) and i am not having any issues with any of the jobs.
Can you define the problem a little more clearly? Is this one new client affected or more than one client affected? Were the CommVault backups of this server/these servers successful and fast? Is it correct that this backup does sometimes go at reasonable speeds but average speed over time is slow?
It's 2 clients actually now, one is MSCS primary node other is just a hot standby for a smaller san (EVE 3000). Issues is the same with both backups. Further investigation - backing up just the local recourses for both servers gives me the same issue as well. You are correct, i get bursts that can go as high as 65MBps, but it drops almost instantly to avg 5MBps. CommVault backups were done direct to tape and they were averaging 40 MBps from what i was told.
Thank you and let me know if there are any other questions
The fact that we see bursts of 65Mbps suggests that the problem is not on the side of the network but something to do with the client itself, you may want to do some monitoring there and see if anything unusual or unexpected is happening.
My thinking is the same, but my theory is based on nothing more than gut feeling. What do bursts like that tell you? I am not writing direct to tape so while it looks like shoe-shining effect where it offloads from cache, it definately isnt the case in my situation. I would like to know your thinking process if you do not mind.
You have confirmed this is backing up to a VTL and other backups are working fine so we can make a reasonable assumption that an issue does not occur on the VTL (obviously shoe-shining does not affect virtual tape drives anyway!).
We see bursts of the kind of speed which we would expect to see which suggests the network does not have a configuration issue that is causing it to run slow.
Our problem therefore shifts back to the client; if performance is tempermental my thought is that perhaps there is some kind of overload that causes the system to lockup.
But then the argument against the problem being there is that when you were backing up with the Commvault agent you had no problems!
Thierry101
2 Intern
•
326 Posts
0
April 17th, 2012 17:00
Hi Mikka
A few things you can check:-
1) How many hops to reach the storage node?
2) Between the hops, are all servers with same eth Mb/s and Full duplex?
3) Any encryption enabled?
4) Without going to VTL, have you tried just copy and paste to SN. Is the MB/s same?
5) Are all apps that going to this VTL have perf issues or a few apps?
Bebo2k
544 Posts
0
April 17th, 2012 20:00
Hi Mikka,
First of all , your environment is not supported as your NetWorker server OS is Windows 2008 R2 with NetWorker 7.6 SP1 , where the supported version for that OS is 7.6 SP2 and higher , so before any troubleshooting you have to upgrade your NetWorker version on the backup server and the remote storage nodes as well. The latest now is NetWorker 7.6.3.3 which can be downloaded from the following FTP link:
ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/7.6/
After the upgrade you have to check that you have the latest updates and firmwares for all your hardwares ( Tape drives, HBA, etc.. ) , If you have any available updates you have to patch it firstly so that you environment be in upto-date state. Also Check for corrupt NIC (network card) driver and those cards are configured properly.
I recommend you to run the bigasm test , in order to measure tape device writiing speed, you can find how to do that test from the following Powerlink KB article:
https://solutions.emc.com/emcsolutionview.asp?id=esg53018
Hope this helps you to figure out the root cause of that performance.
Thanks,
Ahmed Bahaa
mikka1984
1 Rookie
•
5 Posts
0
April 18th, 2012 01:00
Hi Ahmed
I would normally agree with you, but this is a very isolated problem. backups are fine across various LAN and WAN links, issues are only encountered on that specific server (which was rebuilt 3 weeks ago, and i have done all the checks for driver upgrade etc)
Oh and i made a mistake, backup server is 2008 SP1, sorry about the confusion.
Thank you
mikka1984
1 Rookie
•
5 Posts
0
April 18th, 2012 01:00
Hi Thierry
Thank you for your responce
1 - 1 hop, being routed through cisco nexus from new network to old. all on 1Gbps switches. Cisco nexuses are configured as your run-of-the-mill switches.
2 - yes and yes
3 - no encryption, tried with compression on and off, same result...
4 - no, while it is slow, it hits 30-40mbps easy, which would be enough to satisfy the business and all people hanging over my head
5 - I backup a good 90 servers, various back office applications, and only this one is causing me greif. Spoke to the DBA who is in charge of the system, i am backing up backups of multple SQL DBs and logs. One of the LUNs the files are easy 10-50GB in size, other lun is hdfs. performance is the same.
On a side note, i am backing up good 40 servers of the new network (one on the cisco nexus backbone) and i am not having any issues with any of the jobs.
DavidHampson-rY
294 Posts
0
April 19th, 2012 02:00
Hi Mikka
Can you define the problem a little more clearly? Is this one new client affected or more than one client affected? Were the CommVault backups of this server/these servers successful and fast? Is it correct that this backup does sometimes go at reasonable speeds but average speed over time is slow?
Regards
David
mikka1984
1 Rookie
•
5 Posts
0
April 19th, 2012 03:00
Hi David
It's 2 clients actually now, one is MSCS primary node other is just a hot standby for a smaller san (EVE 3000). Issues is the same with both backups. Further investigation - backing up just the local recourses for both servers gives me the same issue as well. You are correct, i get bursts that can go as high as 65MBps, but it drops almost instantly to avg 5MBps. CommVault backups were done direct to tape and they were averaging 40 MBps from what i was told.
Thank you and let me know if there are any other questions
DavidHampson-rY
294 Posts
0
April 19th, 2012 04:00
The fact that we see bursts of 65Mbps suggests that the problem is not on the side of the network but something to do with the client itself, you may want to do some monitoring there and see if anything unusual or unexpected is happening.
mikka1984
1 Rookie
•
5 Posts
0
April 19th, 2012 04:00
Hi David
My thinking is the same, but my theory is based on nothing more than gut feeling. What do bursts like that tell you? I am not writing direct to tape so while it looks like shoe-shining effect where it offloads from cache, it definately isnt the case in my situation. I would like to know your thinking process if you do not mind.
Thank you
DavidHampson-rY
294 Posts
0
April 19th, 2012 05:00
Mikka
My thinking is this:
You have confirmed this is backing up to a VTL and other backups are working fine so we can make a reasonable assumption that an issue does not occur on the VTL (obviously shoe-shining does not affect virtual tape drives anyway!).
We see bursts of the kind of speed which we would expect to see which suggests the network does not have a configuration issue that is causing it to run slow.
Our problem therefore shifts back to the client; if performance is tempermental my thought is that perhaps there is some kind of overload that causes the system to lockup.
But then the argument against the problem being there is that when you were backing up with the Commvault agent you had no problems!