This system is almost certainly failing its daily data integrity check (hfscheck). If a system is failing hfscheck, it will retain the last checkpoint that passed validation to ensure there is a known good point in time available for a rollback. Checkpoints are similar to snapshots, so as the checkpoint ages, checkpoint overhead will build up as differences between the validated checkpoint and the current data build up, leading to an increase in disk utilization.
Please contact support as soon as possible for assistance. Once the hfscheck issue is resolved and the system is able to validate a checkpoint, the old checkpoint will be rolled off and the disk space being consumed by checkpoint overhead will be freed. Hopefully this goes without saying but it is vitally important that the validated checkpoint not be removed. Deleting the validated checkpoint may lead to catastrophic system failure.
I rolled avamar to previous valid 'snapshot' but now MCS status is down, and when I running "mcserver.sh --start" then I have:
WARNING: Insufficient privilege to create accounts and perform backups. ERROR: gsan rollbacktime: 1591180617 does not match stored rollbacktime: 1588347908
The error message shown appears when the MCS has not been rolled back following a rollback of the GSAN. Normally, the dpnctl utility is used to roll a system back to ensure that GSAN and MCS are rolled back at the same time. The MCS and GSAN need to be in sync at all times in order to avoid issues like client ID mismatches that can cause subtle problems that are difficult and time consuming to troubleshoot.
The warning message you posted also makes me concerned that there is still some underlying issue with the GSAN that has not been addressed.
I strongly recommend that you work with support on this.
ionthegeek
2 Intern
•
2049 Posts
902
0
Posted June 3rd, 2020 05:00
This system is almost certainly failing its daily data integrity check (hfscheck). If a system is failing hfscheck, it will retain the last checkpoint that passed validation to ensure there is a known good point in time available for a rollback. Checkpoints are similar to snapshots, so as the checkpoint ages, checkpoint overhead will build up as differences between the validated checkpoint and the current data build up, leading to an increase in disk utilization.
Please contact support as soon as possible for assistance. Once the hfscheck issue is resolved and the system is able to validate a checkpoint, the old checkpoint will be rolled off and the disk space being consumed by checkpoint overhead will be freed. Hopefully this goes without saying but it is vitally important that the validated checkpoint not be removed. Deleting the validated checkpoint may lead to catastrophic system failure.