Good afternoon everyone. Hoping to get some help on this issue. I desperately need some insight to a unique situation.
We are on 7.1.2-21 Avamar running all jobs to a 4200 that's on 188.8.131.52. Select VMs will run as full after VCenter reboot. It's the same VMs each time. Additionally, it is across different VCenters (both 6.0, 5.5) connected to our single Avamar node.
2016-06-08 23:01:29 avvcbimage Info <11986>: Changed block tracking is engaged for this VM
2016-06-08 23:01:29 avvcbimage Info <11988>: A reference to a valid prior backup is not available so this will be a full level zero backup.
I've noticed that if I cancel the job, toggle CBT on/off, then run the job again I get
2016-06-07 23:00:47 avvcbimage Info <11986>: Changed block tracking is engaged for this VM
2016-06-07 23:00:47 avvcbimage Info <11987>: Reference backup is #99 captured on 2016-06-05 23:00:48
2016-06-07 23:00:47 avvcbimage Info <14680>: A reference to a valid prior backup is available so this will be an incremental.
So I have ensured we aren't impacted by any of the CBT bugs, even the most latest KB from VMware.
I have tried just about everything. I've deleted the backup, shutdown the VM, reset the CBT on said VM, ran a full and still noticed after VCenter reboot. I have tried every combination of Syncing with VCenter, restarting the VCenter service, etc. I've rebooted my Avamar node and all of the proxies. Finally I tried rebooting a 5.5 environment, and it too had two VMs that ran as full. This is looking more are like a cache issue. If I restart the MCS, a majority of the VMs will run normally, but there are still some, oddly enough, during the 12AM Backup windows that still run as full, when the others did not (after having restarting MCS).
If I reboot VCenter, the said VMs attempt to run as full. If I cancel, toggle CBT on/off, then run the job again, it runs normally. I've also noticed if I reboot and have said VMs run, cancel, then Sync with VCenter and run the job again (without toggling CBT) The job runs normally. This is very odd behavior. If anyone can suggest anything, it would be great. I have tickets open with EMC, VMware, etc. No one can quite pinpoint the issue, since it's move behavioral than anything.
I know best practice after VCenter crash is to restart MCS, restart VCenter connection and re-sync, but I should be able to reboot vcenter and not have this happen. And why is it just certain VMs? Across different Host, different datastores, different proxies, with some VM tools and hardware updated, some not. etc. There is no common denominator.
This ended up being disk order in VSphere. It apparently changes in 6 after reboot, therefore Avamar doesn't know what to do besides call a full backup.