Actually this is not a fix at least for the second problem. As I stated I have this problem also on a 2008 R2 server. If there would be a problem with VSS itself, shouldn't it complain in the same way, when you take a manual quiesced snapshot through vSphere?
I have other 2008 R2 servers, that works ok, but this one is still causing problems with the same error messages.
(Shadow)
About the dns suggestion. Why would this one be different from all the others, which mainly are named as the dns without FQDN?
The name in networker is the same as dns name. I have tried with and without the VADP_VM_NAME=VMWARENAME line in the Application information.
Haven't anybody else had this problem?? I still have with this same client. I just deleted the client completely and recreated it through the wizard. The problem still remains. If I take a quiesced snapshot manually through vCenter it works ok, but when my VADP backup tries to make a snapshot, it fails with the stupid "Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine." error message.
How is the VADP snapshot different from a manual snapshot, with "Snapshot the virtual machine's memory" and "Quiesce guest file system" checked?
I have several working VADP client backups and they seem to be rather stable. If they start to work, then they usually work all the time.
But then I have a couple of VADP backup clients, that just doesn't seem to work. There seems to be two different scenarios, these are both Windows 2003 SP2 servers, but I think the first problem also was on a 2008 R2 :
1. The snapshot is created, but when the VADP proxy is trying to "Download the VM configuration for [client]" it fails with following errors:
* :guest/client:*FULL* 2011-06-07T10:52:33Z SnapShot created Successfully for VM 'guest/client'. * :guest/client.*FULL* Downloading VM configuration for guest/client. * :guest/client.*FULL* 2011-06-07T10:54:36Z 83065:nsrvadp_save: The following internal error occurred: libcurl error: Timeout was reached * :guest/client.*FULL* 83687:nsrvadp_save: Could not download VM configuration for VM 'guest/client'. * :guest/client.*FULL* 2011-06-07T10:54:36Z Temporary directory 'D:\\VADP\guest/client' for VADP deleted.
Sometimes it leaves the snapshot, sometimes not. There are no errors in the vCenter to be found. Nothing in the guest itself either.
2. The creating of the snapshot is failed because of "Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine." This is the whole output from the log:
* :guest/client.*FULL* 2011-06-07T10:48:56Z Creating snapshot for 'vm-36481' ... * :guest/client.*FULL* Task is -1% complete * :guest/client.*FULL* 2011-06-07T10:49:04Z Task is 95% complete * :guest/client.*FULL* 2011-06-07T10:49:11Z 74854:nsrvadp_save: The following VMware related error occured: Unable to create snapshot '_VADP-BACKUP_' for VM: vm-36481. Details: 'An error occurred for task 'CreateSnapshot_Task'. Details: Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.'. * :guest/client.*FULL* 80405:nsrvadp_save: Could not create Snapshot for VM 'guest/client. * :guest/client.*FULL* 2011-06-07T10:49:11Z Temporary directory 'D:\\VADP\guest/client' for VADP deleted.
The same error message can be found in the vCenter, but if I take a quiesced snapshot manually from this client it works ok. I have googled around and this has possibly something to do with VSS, because in the event log on the guest i found these errors
But I haven't found anything wrong with VSS, vssadmin list writers outputs what is expected.
A workaround for this would be to take a nonquiesced snapshot, but even though there was a way to do it with the old VCB backup, I haven't found a way to do it with VADP. Another workaround is to install vmware tools without VSS support, but I don't want to do that. because I sometimes want a quiesced snapshot taken manually.
Form the VADP Integration Guide: Vmware tools must be installed on the VM to ensure that backups are taken in a
consistent state.
After installing VMware tools, if only 2008 machines are affected:
The vmware tools installer contains a program, comreg.exe , which has a Windows 2008 R2 bug that prevents it from registering the VMware Snapshot Provider with VSS.
VADP backups of a Windows Server 2008 R2 or Windows 7 VM may fail for certain
versions of ESX 4.0.0 due to this issue.
Instructions for fixing this issue are provided at the following location:
No. First of all, as I already several times stated. I can take quiesced snapshots manually. So the problem is not the above. And secondly, we already use ESX 4.1.0.
No. First of all, as I already several times stated. I can take quiesced snapshots manually. So the problem is not the above. And secondly, we already use ESX 4.1.0.
1. VMware tools is needed, ensure its installed
2. Follow this article to find the pre-requisites/services/vss-providers, and other related configuration areas that need to be checked on affected systems.
3. ' Unexpected error calling routine RegSaveKeyExW' : Seems to be a problem around registry, which exact OS is this for 2003SP2 / 2008 R2?
1. The vmware tools are indeed installed and up to date, as always. With the VSS providers. The manual quiesced snapshot would fail if it wasn't.
2. Checked those things plenty of times.
3. This is a 2008 R2 server. But the EventID in the original post may be coincidental. Have tried taking VADP backups several times after that without anything in the EventLog.
There must be a difference between VADP snapshots and Manual snapshots. If somebody could tell me what the difference is, then maybe one could work out why VADP snapshots are failing.
Although we have no structured documenet for this - the only difference between a manual_quiesced_snapshot from VCenter vs a Networker VADP_snapshot is that the later is is triggered programmatically.
I have heard of this same issue occuring once before and I created esg121746. In this case, manual snapshots were also successful. The root cause of the issue was a "malfunctioning VSS on the host managing the VM being backed up". Unfortunately I did not get any further details at the time but I'm trying again to see if I can get more info on exactly what the VSS issue was, symptoms, tests etc but in the meantime, have a look at
Interesting reading and it has been updated since I last read it. And if I the manual quiesced snapshots would fail, then this would be an exact match.
Anyway, this server is the only one (for now) with this problem and I will probably try the workaround by removing the vmware-tools VSS-component.
I just wonder why you can't choose to take a non-quiesced snapshot with VADP. With VCB this was an option! I imagine it shouldn't be too hard to implement?
But that is a dirty workaround. This will affect all snapshots, not only those made by VADP.
I just found out that there are new Cumulative_hotfixes on the ftp-site...7.6.2.5 (never even tested 7.6.2.4?). Installed it on a test vadp-proxy just to find out, that there still is no option to make a non-quiesced snapshot.
Will this option ever be possible? When? If not, why not? Please EMC:ers, tell me.
I did some following up and there is an RFE filed (NW131294) to provide an option to turn off the queisce flag for a VADP backup but it has not been committed to a release at this time. I'm also following up to have the details of the workaround doc'ed in the NetWorker VMWare Integration Guide on Powerlink:
The steps below require a reboot of the virtual machine. VMware recommends scheduling downtime before performing this action.
Uninstall VMware Tools from the VM.
Allow the system to reboot.
Reinstall VMware Tools. Ensure to clickCustom Install.
DeselectVSS.
If this differs from what you did, can you please post back?
No, that is exactly what i did. But you should probably add, that this affects all snapshots. Sometimes you would like to have a quiesced manual snapshot and after this change it is not possible anymore.
AllanW1
334 Posts
0
June 7th, 2011 07:00
Hi Tony,
This post looks similar: https://community.emc.com/message/361293
In short- Hvoje posted:This is a MS issue, check their support page. Perhaps rebooting the box will bounce VSS services back to operational state.
and jog posted:This knowledgebase article resolved the issue in a similiar case of mine
http://support.microsoft.com/kb/940032/en-us
Hope this helps,
Allan
Martin_Schrijve
1 Message
0
June 9th, 2011 03:00
Hi Tony,
I have encountered this problem also (together with some others).
One of the things I had to changes was the naming of the VM in VC manager, it had to be the same as the dns name on the network.
regardless of the available mapping filed (VM name in VC versus the real name on the network).
I changed some more thing at that time so I am not sure if this solves you problem with the product.
Yust wanted th shar this information.
With regards,
Martin
tonykl
27 Posts
0
August 12th, 2011 05:00
(AllanW)
Actually this is not a fix at least for the second problem. As I stated I have this problem also on a 2008 R2 server. If there would be a problem with VSS itself, shouldn't it complain in the same way, when you take a manual quiesced snapshot through vSphere?
I have other 2008 R2 servers, that works ok, but this one is still causing problems with the same error messages.
(Shadow)
About the dns suggestion. Why would this one be different from all the others, which mainly are named as the dns without FQDN?
The name in networker is the same as dns name. I have tried with and without the VADP_VM_NAME=VMWARENAME line in the Application information.
So I am still waiting for more suggestions...
tonykl
27 Posts
0
September 27th, 2011 05:00
Haven't anybody else had this problem?? I still have with this same client. I just deleted the client completely and recreated it through the wizard. The problem still remains. If I take a quiesced snapshot manually through vCenter it works ok, but when my VADP backup tries to make a snapshot, it fails with the stupid "Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine." error message.
How is the VADP snapshot different from a manual snapshot, with "Snapshot the virtual machine's memory" and "Quiesce guest file system" checked?
TheStorageProto
27 Posts
0
September 27th, 2011 06:00
@ tonyk
Looking at Issue 2:
Form the VADP Integration Guide: Vmware tools must be installed on the VM to ensure that backups are taken in a
consistent state.
After installing VMware tools, if only 2008 machines are affected:
The vmware tools installer contains a program, comreg.exe , which has a Windows 2008 R2 bug that prevents it from registering the VMware Snapshot Provider with VSS.
VADP backups of a Windows Server 2008 R2 or Windows 7 VM may fail for certain
versions of ESX 4.0.0 due to this issue.
Instructions for fixing this issue are provided at the following location:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displa
yKC&externalId=1022720
To resolve this issue, upgrade to ESX 4.0 update 2 or ESX 4.1, or to upgrade your ESX 4.0.0
server with a VMware patch, go to the following link:
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1
013127
Cheers!
tonykl
27 Posts
0
September 27th, 2011 06:00
No. First of all, as I already several times stated. I can take quiesced snapshots manually. So the problem is not the above. And secondly, we already use ESX 4.1.0.
TheStorageProto
27 Posts
0
September 27th, 2011 07:00
1. VMware tools is needed, ensure its installed
2. Follow this article to find the pre-requisites/services/vss-providers, and other related configuration areas that need to be checked on affected systems.
3. ' Unexpected error calling routine RegSaveKeyExW' : Seems to be a problem around registry, which exact OS is this for 2003SP2 / 2008 R2?
tonykl
27 Posts
0
September 27th, 2011 07:00
1. The vmware tools are indeed installed and up to date, as always. With the VSS providers. The manual quiesced snapshot would fail if it wasn't.
2. Checked those things plenty of times.
3. This is a 2008 R2 server. But the EventID in the original post may be coincidental. Have tried taking VADP backups several times after that without anything in the EventLog.
There must be a difference between VADP snapshots and Manual snapshots. If somebody could tell me what the difference is, then maybe one could work out why VADP snapshots are failing.
TheStorageProto
27 Posts
0
September 28th, 2011 08:00
Although we have no structured documenet for this - the only difference between a manual_quiesced_snapshot from VCenter vs a Networker VADP_snapshot is that the later is is triggered programmatically.
Here's an interesting case: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1037754
Cheers!
ddunn1
42 Posts
0
September 28th, 2011 09:00
Hi Tony,
I have heard of this same issue occuring once before and I created esg121746. In this case, manual snapshots were also successful. The root cause of the issue was a "malfunctioning VSS on the host managing the VM being backed up". Unfortunately I did not get any further details at the time but I'm trying again to see if I can get more info on exactly what the VSS issue was, symptoms, tests etc but in the meantime, have a look at
http://kb.vmware.com/selfservice/documentLink.do?externalID=1018194µsiteID=null if you haven't already. This VMware article is what I was advised to reference when I created the kb article.
Debbie
tonykl
27 Posts
0
September 28th, 2011 23:00
Yes, well this particular server has only two disks, so this doesn't apply here.
tonykl
27 Posts
0
September 28th, 2011 23:00
Interesting reading and it has been updated since I last read it. And if I the manual quiesced snapshots would fail, then this would be an exact match.
Anyway, this server is the only one (for now) with this problem and I will probably try the workaround by removing the vmware-tools VSS-component.
I just wonder why you can't choose to take a non-quiesced snapshot with VADP. With VCB this was an option! I imagine it shouldn't be too hard to implement?
tonykl
27 Posts
1
October 5th, 2011 03:00
Removing the vmware-tools VSS component did help!
But that is a dirty workaround. This will affect all snapshots, not only those made by VADP.
I just found out that there are new Cumulative_hotfixes on the ftp-site...7.6.2.5 (never even tested 7.6.2.4?). Installed it on a test vadp-proxy just to find out, that there still is no option to make a non-quiesced snapshot.
Will this option ever be possible? When? If not, why not? Please EMC:ers, tell me.
--
tonyk
ddunn1
42 Posts
0
October 5th, 2011 06:00
Hi Tony,
I did some following up and there is an RFE filed (NW131294) to provide an option to turn off the queisce flag for a VADP backup but it has not been committed to a release at this time. I'm also following up to have the details of the workaround doc'ed in the NetWorker VMWare Integration Guide on Powerlink:
The steps below require a reboot of the virtual machine. VMware recommends scheduling downtime before performing this action.
If this differs from what you did, can you please post back?
Debbie
tonykl
27 Posts
0
October 5th, 2011 07:00
No, that is exactly what i did. But you should probably add, that this affects all snapshots. Sometimes you would like to have a quiesced manual snapshot and after this change it is not possible anymore.