AVE, protecting it's PostgreSQL DB

As far as the mechanics of how the system's PostgreSQL databases are protected, configuration files and a database dump for the MCS, the Avamar Installer, etc. are periodically backed up ("flushed") to the backup storage. These backups are then protected by replication or checkpoint backup to DD. I hope that clarifies things.

There are two options for protecting an AVE (or any single node, really) from a disaster. The first is replication. The second is checkpoint backup to Data Domain. Backing up just the MCS database is not sufficient to protect the AVE from a disaster.

Sounds like Replication is the way to go from what I'm hearing.  I was reading in another post about replicating the /MC_BACKUPS folder but I don't have a folder in my directory listing, does that mean something's not happening that should?

Also, I get confused as to what the difference is between a checkpoint and the hourly "flushes" and which is used in a restore procedure?  Sorry, I've got Avamar experience from a normal backup/restore operator but this is first time I'm having to manage the back-end and want to make darn sure I at least understand what it takes to restore an Avamar AVE instance if needed.  Thanks for the answers.

Replication is the only option for pure Avamar. For Avamar / DD you could use either and they each have advantages and disadvantages. You would have to get in touch with Dell EMC to actually perform a DR regardless of which option you choose.

The MC_BACKUPS account is a system account so it's hidden in the GUI. This is the account where the hourly flushes are stored.

The hourly flushes write the configuration and data for the MCS, AVI and other system services to the regular backup storage. On the other hand, checkpoints are used to protect the backup data that have been written to the system, including the flushes. These checkpoints are designed to protect the system in case of something like a software crash but would not protect you from, say, physical damage to the storage devices. This is the reason that use of either replication or checkpoint backup is mandatory for single node Avamar systems including AVE.

The basic process for recovering from a replication target in a disaster is:

• Repair or replace the damaged Avamar system
• Run a replicate restore to repopulate the Avamar user accounting system and restore the flushes for the system accounts
• Restore the MCS from flush
• Place the source system back into production (re-activate clients if necessary, etc.)

For checkpoint restore, things are similar but not exact:

• Repair or replace the damaged Avamar system
• Attach it to the Data Domain
• Run a checkpoint restore
• Roll back the Avamar server to the restored checkpoint (the MCS will be restored from the latest flush automatically)
• Run a new checkpoint and hfscheck
• Place the system back into production

Keep in mind that this is just a rough outline of each of the procedures and that the full procedure is something that only Dell EMC is authorized to perform.

If you have any further questions, please don't hesitate to ask.

Ok, think I'm getting it, the Checkpoints that are taken during the maintenance windows are more for minor incidents, like my esx server holding AVE crashes and comes back up in an inconsistent state and I rollback to the last good state. 

Whereas with replication, that's for a full disaster, like the esx server gets flooded and you have to rebuild from scratch.  And with replication, is that a policy you create by hand or once Avamar is aware of a replication partner it automatically backs up that /MC_BACKUPS, etc... directories, or basically the data needed to recover from a full disaster.  And stupid side question, does the replication partner replicate it's config back or because it's just a target it doesn't need to?

Either way you're saying you need to engage support to take care of any restore situations?  It gets confusing because I see in older posts(few years old) where it seems like people are doing their own rollbacks and whatnot and in the Admin guide there's a short section on restoring MCS data and it mentions using the GUI to browse backups out of the /MC_BACKUPS directory, that was getting me confused. 

Thanks again, I may have a follow up question or two but I'm starting to get it.  I always hear deploying a full grid is painful and then sales kind of makes AVE sound like it's as easy as deploying an ova file and taking off!

Ok, think I'm getting it, the Checkpoints that are taken during the maintenance windows are more for minor incidents, like my esx server holding AVE crashes and comes back up in an inconsistent state and I rollback to the last good state. Whereas with replication, that's for a full disaster, like the esx server gets flooded and you have to rebuild from scratch.

Exactly.

And with replication, is that a policy you create by hand or once Avamar is aware of a replication partner it automatically backs up that /MC_BACKUPS, etc... directories, or basically the data needed to recover from a full disaster.

The system accounts like MC_BACKUPS are automatically included in every replication job for plugin-based replication so if you have any replication job configured edit: and enabled, that will be enough.

And stupid side question, does the replication partner replicate it's config back or because it's just a target it doesn't need to?

If the replication target isn't ingesting backups directly, it doesn't need to replicate (if you lose it, you can just replace it or re-initialize it and let replication catch up). If it is ingesting backups, it should be configured to cross-replicate with its partner. To set up cross-replication, you just have to create a replication job on the "target" that replicates back to the other system. The /REPLICATE domain where replicated data lives will be excluded automatically so you won't end up with a replication loop.

Either way you're saying you need to engage support to take care of any restore situations?  It gets confusing because I see in older posts(few years old) where it seems like people are doing their own rollbacks and whatnot and in the Admin guide there's a short section on restoring MCS data and it mentions using the GUI to browse backups out of the /MC_BACKUPS directory, that was getting me confused.

It's only for a full system loss and restore from DR that you'd need to engage Dell EMC. Usually that's the Professional Services team.

Checkpoint rollbacks are fine to run yourself. If the system goes down hard for some reason, single nodes will generally roll back automatically on boot. If for some reason that doesn't happen, you'll be prompted to roll back when you try to start up the Avamar software. Don't pick the "restart without rollback" option if the system goes down hard because it will probably not come back up. Even if it does, the chances of it passing the next checkpoint validation are near-zero.

MCS flush restore I'd recommend against running on your own unless you know exactly what you're doing -- you don't want to put GSAN and MCS out of sync because that causes all kinds of issues.

Happy to help! Multi-node systems require a PS engagement to install. AVE has been customer installable for a couple of versions now.

Thanks again Ian; I've got more questions but on different topics so I'll start some more threads.  I'm moving on now to understanding more about replication in general.