Unsolved
This post is more than 5 years old
9 Posts
0
2690
VMware image backups slow after migration to a new vCenter
We recently deployed a new vCenter running version 6 at our main data center and the previous vCenter was running version 5.5. From reading the documentation the best practice was to retire the clients in Avamar in the old vcenter and re-add them to the new vcenter. I've been noticing a trend in the backups of the virtual servers since the move. The backup times of the virtual servers have dramatically increased. From what took less than a half an hour to an hour for the biggest servers is now taking closer to 6 hours. Me and the VMware admin has been a little stumped.
old vcenter - ***-***-vcent01 running version 5.5
new vcenter - ***-***-vcentb running version 6
Virtual servers are in Dynamic folders
server ***-***-fs02 went from around 1 hour on average to 5:30-6:15 to complete. 2.5 TB virtual server with 4 vhdks.
server ***-***-lfiche went from 20-30 minutes to around 2 hours 600 GB server
server ***-***-ise went from 10-20 minutes to around 2 hours 600gb server not much data changed since it's a Cisco ISE server.
Proxies have auto mapping to all the available data stores. None of the backup groups from either vcenter have auto proxy mapping since 1 proxy server is in a DMZ and my production servers were trying to use in when auto mapping.
After we added the new vCenter in Avamar and before the move I copied the dataset, schedule, retention, over for the new vcenter and created the backup groups for the production VMs and a separate backup group for the big file server.
Retired virtual servers from vcent01 (v 5.5)
Re-added virtual servers to vcentb (v 6). Which was around 75-80 servers.
Originally 1 proxy server for the new vcenter but I deleted the proxies from the old vcenter and added them into the new one. 3 total now and The old one I was running 2.
Increased the available ram on vcentb to match what vcent01 had.
Updated UTC of the proxies (doubt that had anything to do with it)
Noticing high memory utilization of the proxies in the vcenter.
CBT is enabled in the dataset.
We're a little stumped on what's going on and wanted to see if anyone had some ideas to look at before I submit a request to EMC.
DPESury
17 Posts
1
December 8th, 2016 05:00
Basically after Vcenter migration, first backup usually takes some amount of time because it is as good as first new backup for that VM. We recently did similar upgrade of vcenter, and noticed change in backup window but subsequent backups were again back in minutes. Are you facing this issues for all the subsequent backups as well ? If yes then please share session logs.
DarthHerm1
9 Posts
0
December 9th, 2016 13:00
The vCenter migration occurred around November 28th. The servers Backup times of the VMs haven't gone back down to the levels they were before on the other vCenter. The initial did take some time and I anticipated it. One thing I had noticed is that I am unable to use Instant Access on the VMs. Might be related to the two. Won't have logs until Monday.
DarthHerm1
9 Posts
0
December 13th, 2016 11:00
I submitted a request to EMC support yesterday and submitted the logs of one of the virtual servers. CBT was enabled for use in the dataset, but it didn't appear that Enable change block tracking was enabled when the dynamic folders were added. The VM backups were running level 0 backups.
I ran some commands that the support tech recommended to enable block tracking. I am going through if it applied to the dynamic folder as well or just the vms. Me and another colleague were discussing about retiring the folders and vms and re-add them with Change block tracking enabled.
Current Disk '2000': file(base):'[VNX1016_AT_DS09] XXX-XXX-DC02MIG/XXX-XXX-DC02MIG.vmdk', backItUp=1
snapshot(base) file:'[VNX1016_AT_DS09] XXX-XXX-DC02MIG/XXX-XXX-DC02MIG-000002.vmdk'
snapshot(curr) file:'[VNX1016_AT_DS09] XXX-XXX-DC02MIG/XXX-XXX-DC02MIG-000001.vmdk'
prior size(KB):0, current size(KB):104857600, match=0
prior change block ID: ''
current change block ID:''
CBTenabled=0, thin=0, independent=0, SCSI=1, diskMode=persistent, backing=DiskFlatVer2
DPESury
17 Posts
0
December 14th, 2016 08:00
By dynamic folder, did you mean VM container configuration?
To ensure CBT stuff refer below info.
Try running below command and see if CBT is enabled.
java -jar proxycp.jar --cbtstatus –vm vm_name
if this shows as false and try to enable it with below command or navigate to GUI Policy>Client>edit option > under vmware tab > check/select checkbox for Enable CBT.
java -jar proxycp.jar --cbtstatus –vm vm_name –enablecbt
Sometimes, VM needs restart to have CBT take in effect or else try re-registering VM with enable CBT option.
Also, verify with you Vcenter team or VMware guys to verify if CBT is correctly configured from VM configuration. Refer below link for additional info on this.
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031873
Hope this information helps. If still issue exists, then please share session log once, will go through it.
DarthHerm1
9 Posts
0
December 14th, 2016 12:00
Sorry I meant VM container. I updated the containers to enable changed block tracking for auto-imported Virtual Machines with the instructions you provided me. After running the commands to enable CBT on the Virtual server clients the other day I noticed that only two servers still show as CBT disabled. Those two servers are 20gb and 11gb in size and they’re averaging 15 minutes anyways so I’m not worried about them. It was my bigger virtual servers I was more worried about and from looking at the backups the last night or two they have gone back down to similar backup times as it was last month. I think we can go ahead and close out the service request.
DPESury
17 Posts
0
December 15th, 2016 08:00
Yes work with EMC, meantime can you check if Proxies are deployed within the same cluster as VM's/containers.
Will it be possible to attach session logs here.
DPESury
17 Posts
0
January 10th, 2017 12:00
Hello Darthem,
Were you able to get this fixed/resolved. Please share current status.