Experience / Best Practices with long time VM snaps for huge VMFS stores?

From Josh Atwell:

I would support this concept as well.  The key to this is having them dictate the culture of the service they provide.  I have yet to see a reasonable use case for long term snapshots of any kind.

Even in that scenario it was less than 5 weeks. 

I'd suggest simply have the team push back and have all requests for longer snapshots defend a business case for allowing that feature. (risk mitigation, long term roll back, etc.)  If a business case is justified and approved by upper management (based on legitimate analysis of impact vs gain) then this could be implemented in an isolated environment as part of their service.  Dedicating specific datastores/luns where long term snapshots are allowed.  These would be thick provisioned to minimize performance impact during snapshot maintenance tasks, etc.

Just because you can, doesn't mean you should.  I ran into this a lot while at Cisco working behind their portal (CITEIS).  Here's how I approached feature requests.

1. Identify the business objectives and benefits of the portal/feature (need business justification for dangerous activities in portal)
2. Identify design that is operationally sustainable that meets those objectives.  What impact to non-standard capabilities have on ability to recover from failure, impact of environment, impact to neighbors, etc.
3. Implement automation to maintain design and prevent people from "going rogue" or trying to go around the portal process.

In the end if the behavior might impact SLAs of other tenants those customers were not given the full experience.  They would get isolated to a non-standard offering which extended deployment cycles and costs.  Limited portal capabilities.  That would typically force the app owners to think more critically about whether they really need an offering such as long term snapshots or not.  98/100 they decide they can live within the stricter constraints and you never hear from them about it again.

And from Rich Barlow:

Use RecoverPoint for long term snaps.  At the hypervisor level I'm in total agreement that snaps beyond a couple of days is asking for disaster because the technology is so different. I think we should make sure that we don't conflate these two technologies.

From Richard Anderson:

Why not suggest Avamar for incremental-forever deduplicated backups of the VMs. Even can do an incremental restore using block change tracking for fast recovery to a previous backup.

Just run a new incremental backup before a change and you are all set, no snapshots to worry about, no performance problems when it's time to cleanup snaps, or disk space consumed on the datastore, plus it's an actual backup and can be replicated offsite for DR recovery.

Josh Atwell's response:

Fair enough.  I speak solely from a portal/service standpoint.  The task is then to identify what the use case is really truly asking for, and if we anticipate that array snapshots and/or recover point can meet that objective.  Then it becomes a separate infrastructure related option that they can offer as separate "service", or ideally on an isolated tier w/ that capability.  Standard service offering would be virtual layer option that can be controlled in the portal that puts a limit on virtual layer snapshots at no additional cost since it uses native architecture and toolsets.  This will then limit the impact of needing/wanting to alter LUN sizes and isolate impact.

I also dealt with some long term snaps but in the end we found that we reclaimed data from them so infrequently that it wasn't worth making a service around.  Naturally I found this out after spending a week or two working on figuring out how to specifically enable users to do independently. :-)  We still kept the snap policy but requests for mounting and using the snapshots came with a best effort SLA.

My .02 cent

>offering more flexibility to app owners when doing patches and alike.

First thing to do is to define clearly the offering here. And then use the best technology to address the offering.

Here is an idea of the offering. Thinking out loud here so I'm positively missing things

and don't forget that the journey is more important than the destination

Functionals:

1- the offering should allow a point in time (PIT) recovery state

2- the offering should allow rollback to a previous point in time (PIT) recovery state

Non-Functionals:

1- PIT recovery should be easy and quick to enable (manageability)

2- PIT recovery should should have the minimum impact on VM performance

3- In case of consolidation process, it should have the minimum impact on VM performance as well

4- PIT recovery, rollback and eventually consolidate functions should take a minimum of time to process

Let create some abbreviations:

1- TTE = Time To Enable

2- TTR = Time To Rollback

3- TTC = Time To Consolidate

Here is a table to summarise my point

	VMW Snapshot	VMW aware Backup
TTE	Snapshot is fast	Backup takes longer than a snapshot
TTR		Restore takes longer than roll back of a snapshot
TTC	Variable. It depends on the size of the delta which usually is high since it captured the many changes.	There is no consolidation per se here and thus TTC=0. Note: snapshots are being used and consolidated during backup process. Though the goal is not the same.
PIT recovery should be easy and quick to enable
PIT recovery should should have the minimum impact on VM performance	Snapshot hinders VM performance as soon it is turned on	Once the backup is done, there is no impact on the VM performance
In case of consolidation process, it should have the minimum impact on VM performance	Consolidation of snapshot puts a great stress on the storage and thus hinders VM performance	There is no consolidation here

You can of course add non-functional qualities, such disk space requirement, security and cost to further add granularity to this table.

Feedback are welcome of course

Rgds,

Didier