Avamar: The database encountered an error - more than one row returned by a subquery

A rollback was performed in the Avamar server and after that the Management Console Server (MCS) shows the error:

Event Code 22602 - The database encountered an error: more than one row returned by a subquery used as an expression.

The error repeats several times:

mcdb=# select date,time,summary from v_events where code='22602' ;
    date    |     time     |              summary
------------+--------------+------------------------------------
 2024-08-07 | 21:36:38.847 | The database encountered an error.
 2024-08-07 | 21:43:02.476 | The database encountered an error.
 2024-08-07 | 21:43:02.483 | The database encountered an error.
...
 2024-09-06 | 13:30:24.053 | The database encountered an error.
 2024-09-06 | 13:53:27.032 | The database encountered an error.
 2024-09-06 | 14:00:00.01  | The database encountered an error.
(427 rows) 

The exception is seen in /usr/local/avamar/var/mc/server_log/mcserver.log.0:

09/06-11:23:33.00868 [RMI TCP Connection(1304)-10.137.1.65#25952] com.avamar.mc.util.MCException.logException
WARNING: org.postgresql.util.PSQLException: ERROR: more than one row returned by a subquery used as an expression
  at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713)
  at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401)
  at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368)
  at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498)
  at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415)
  at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:335)
  at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:321)
  at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:297)
 at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:246)
  at com.avamar.mc.datatap.MCDataAccessObject.getBytesProtected(MCDataAccessObject.java:11484)
  at com.avamar.mc.datatap.MCDataAccessObject.getBytesProtectedLast24Hours(MCDataAccessObject.java:11531)
  at com.avamar.mc.mcsm.DataProtectedCache.getBytesProtectedLast24Hours(DataProtectedCache.java:117)
  at com.avamar.mc.mcsm.MCSManagerService.getBytesProtectedLast24Hours(MCSManagerService.java:796)
  at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)

Although the error message, the Avamar server is performing maintenance activities and backups correctly.

Further investigation, it was found duplicate entries in the management console database (mcdb):

admin@av-server:~/>: psql -p 5555 mcdb -c "select * from property_value;"
      property       |                 value
---------------------+----------------------------------------
 morning_cron_start  | -1
 evening_cron_start  | -1
 mcsnmp_cron_start   | 1
 clean_db_cron_start | 3
 morning_cron_start  | -1
 evening_cron_start  | -1
 mcsnmp_cron_start   | 1
 clean_db_cron_start | 3
 systemid            | 1234567890@00:50:56:AB:0E:C5
 systemid            | 1234567890@00:50:56:AB:0E:C5
 hfscreatetime       | 1234567890
 hfscreatetime       | 1234567890
 systemname          | av-server.lab.example.com
 systemname          | av-server.lab.example.com
 restoredFlushTime   | 2024-08-07 08:00:09 CST
 license_period_day  | 14
 license_period_day  | 14
 license_buffer_pct  | 10
 license_buffer_pct  | 10
 rollbacktime        | 1723065800
 rollbacktime        | 1723065800
(21 rows)
admin@av-server:~/>:

The Avamar group, dataset, schedule, and retention show a ".2" in their names:  

admin@av-server:~/>: mccli schedule show --recursive
0,23000,CLI command completed successfully.
Schedule                                 Domain
---------------------------------------- -------------------------------------------------------
Daily Schedule.2.2.2.2.2.2               /
Default Replication Schedule.2.2.2.2.2.2 /
Default Schedule.2.2.2.2.2.2             /
Default Tiering Schedule.2.2.2.2.2.2     /
Evaluation Schedule.2.2.2.2.2.2          /
Notification Schedule.2.2.2.2.2.2        /
Override Daily Schedule.2.2.2.2.2.2      /
Statistics Schedule.2.2.2.2.2.2          /
Usage Intelligence Schedule.2.2.2.2.2.2  /
...
admin@av-server:~/>: 

admin@av-server:~/>: mccli dataset show --recursive
0,23000,CLI command completed successfully.
Dataset                                  Domain                                Type
---------------------------------------- ------------------------------------- ------
Base Dataset.2                           /                                     backup
Default Dataset.2                        /                                     backup
OpenStack Image Dataset.2                /                                     backup
Oracle Cumulative Backup Dataset.2       /                                     backup
Oracle Differential Backup Dataset.2     /                                     backup
Oracle Full Backup Dataset.2             /                                     backup
Oracle Incremental Backup Dataset.2      /                                     backup
SQL Server Differential Backup Dataset.2 /                                     backup
SQL Server Full Backup Dataset.2         /                                     backup
SQL Server Incremental Backup Dataset.2  /                                     backup
Unix Dataset.2                           /                                     backup
VMware Image Dataset.2                   /                                     backup
Windows Dataset.2                        /                                     backup
...
admin@av-server:~/>: 

admin@av-server:~/>: mccli retention show --recursive
0,23000,CLI command completed successfully.
Name                                                Domain
--------------------------------------------------- -------------------------------------
Default Retention.2                                 /
End User On Demand Retention.2                      /
Minimal Retention.2                                 /
Monthly Retention.2                                 /
Weekly Retention.2                                  /
...
admin@av-server:~/>: 

admin@av-server:~/>: mccli group show --recursive
0,23000,CLI command completed successfully.
Group                          Domain                                                  Type   Auto Proxy Mapping Rule       Rule Enabled
------------------------------ ------------------------------------------------------- ------ ------------------ ---------- ------------
...
Default Group.2                /                                                       Normal false              Empty Rule false
Default Proxy Group.2          /                                                       Normal false              Empty Rule false
...
admin@av-server:~/>: 

The root cause is unknown. Upon further investigation, it was determined that the error began after a server rollback. By reviewing the history commands and other outputs, it was discovered that an incorrect MCS restore occurred, likely causing this issue.

The Avamar server was re-deployed preventing further analysis to identify the root cause and potential solution.

Contact Dell technical support immediately and use article 000022859 to collect the getlogs.