Avamar: Unable to Disable Invalid Flags after a Later Version Plugin Catalog is Installed.

In this instance, the issue related to avtier

The avtier job fails with messages relating to invalid flags.

The tiering log reports something similar to the following:

  <catalog>
...
    <flag type="checkbox" pidnum="1051" value="false" name="reversed-operation" />
...
    <flag type="checkbox" pidnum="1051" value="false" name="unmark-only" />
    <flag type="pulldown" pidnum="1051" value="0" name="verbose" />
  </catalog>
...  
2019-10-22 15:04:14 avtier Error <5062>: Invalid flag: "--reversed-operation"
2019-10-22 15:04:14 avtier Error <5062>: Invalid flag: "--unmark-only"

If a plugin catalog from a later Avamar version than the Avamar server version is installed in error, flags for the later version report errors in the UI that cannot be deleted.

• The dataset flags refer to the latest (max) plugin version.  
• If the plugin catalog is updated, all the datasets that were created after are affected as the flags are set to the latest plugin catalog version. 
• All backup policies using these datasets are affected.

In this specific instance, a grid running Avamar 7.5.1-101 had the plugin catalog from Avamar Version 18.1.100-33 installed in error.

1. Log in to the Avamar Utility Node as admin.
2. Elevate to root, and load the SSH keys. See article Avamar: How to Log in to an Avamar Server and Load Various Keys. 
3. Verify the Avamar server version:

avmgr --version |egrep "version|build" |grep -v OS

4. From the /usr/local/avamar/lib/plugins/ directory, verify that there are folders with two different versions (In this example, the versions are 7.5.101-101 and 18.1.100-33):

cd /usr/local/avamar/lib/plugins

find . -type d -ls |egrep -v "avamarbackup"

131073    4 drwxr-xr-x   4 root     root         4096 Dec 13 08:56 .
196725    4 drwxr-xr-x   3 root     root         4096 Nov  8 07:24 ./18.1.100-33
147533    4 drwxr-xr-x   3 root     root         4096 Nov  8 07:23 ./7.5.101-101

5. Verify that the symbolic links in that folder are pointing to the newer plugins folder 18.1.100-33:

ls -ltr

total 232
lrwxrwxrwx 1 root root    59 Apr  3  2019 ZFSndmp-15003.xml -> /usr/local/avamar/lib/plugins/18.1.100-33/ZFSndmp-15003.xml
lrwxrwxrwx 1 root root    66 Apr  3  2019 WINwinclusterfs-3036.xml -> /usr/local/avamar/lib/plugins/18.1.100-33/WINwinclusterfs-3036.xml
lrwxrwxrwx 1 root root    57 Apr  3  2019 WINvss-3015.xml -> /usr/local/avamar/lib/plugins/18.1.100-33/WINvss-3015.xml
lrwxrwxrwx 1 root root    61 Apr  3  2019 WINvmwfile-3019.xml -> /usr/local/avamar/lib/plugins/18.1.100-33/WINvmwfile-3019.xml
...
lrwxrwxrwx 1 root root    57 Apr  3  2019 AIXdb2-5009.xml -> /usr/local/avamar/lib/plugins/18.1.100-33/AIXdb2-5009.xml
lrwxrwxrwx 1 root root    59 Apr  3  2019 AIXavtar-5001.xml -> /usr/local/avamar/lib/plugins/18.1.100-33/AIXavtar-5001.xml
lrwxrwxrwx 1 root root    61 Apr  3  2019 AIXavagent-5000.xml -> /usr/local/avamar/lib/plugins/18.1.100-33/AIXavagent-5000.xml
drwxr-xr-x 3 root root  4096 Nov  8 07:23 7.5.101-101
drwxr-xr-x 3 root root  4096 Nov  8 07:24 18.1.100-33

6. If the links are pointing to the newer version, all symbolic links must be deleted and re-created:
   1. Generate a text file containing the name of the symbolic links:

ls -al | grep '^[-l]' |awk '{print $9}' |grep -v "filenames.txt" > filenames.txt

2. Run the following command to remove all existing symbolic links:

cat filenames.txt | while read line; do rm $line; done

3. Create new symbolic links using the correct (older) plugin version:

cat filenames.txt | while read line; do ln -s /usr/local/avamar/lib/plugins/7.5.101-101/"$line" "$line";done

4. Repeat step 5 above and verify that the symbolic links indicate the plugins folder that matches the Avamar version in use.

ls -ltr

total 232
lrwxrwxrwx 1 root root    59 Apr  3  2019 ZFSndmp-15003.xml -> /usr/local/avamar/lib/plugins/7.5.101-101/ZFSndmp-15003.xml
lrwxrwxrwx 1 root root    66 Apr  3  2019 WINwinclusterfs-3036.xml -> /usr/local/avamar/lib/plugins/7.5.101-101/WINwinclusterfs-3036.xml
lrwxrwxrwx 1 root root    57 Apr  3  2019 WINvss-3015.xml -> /usr/local/avamar/lib/plugins/7.5.101-101/WINvss-3015.xml
lrwxrwxrwx 1 root root    61 Apr  3  2019 WINvmwfile-3019.xml -> /usr/local/avamar/lib/plugins/7.5.101-101/WINvmwfile-3019.xml
...
lrwxrwxrwx 1 root root    57 Apr  3  2019 AIXdb2-5009.xml -> /usr/local/avamar/lib/plugins/7.5.101-101/AIXdb2-5009.xml
lrwxrwxrwx 1 root root    59 Apr  3  2019 AIXavtar-5001.xml -> /usr/local/avamar/lib/plugins/7.5.101-101/AIXavtar-5001.xml
lrwxrwxrwx 1 root root    61 Apr  3  2019 AIXavagent-5000.xml -> /usr/local/avamar/lib/plugins/7.5.101-101/AIXavagent-5000.xml
drwxr-xr-x 3 root root  4096 Nov  8 07:23 7.5.101-101
drwxr-xr-x 3 root root  4096 Nov  8 07:24 18.11.100-33

7. Update the plugin catalog by force:

mccli plugin update --force

0,23000,CLI command completed successfully.

8. Update the plugin catalog in the Avamar Administration UI to ensure that all policy flags are refreshed:
   1. Open the Avamar Administration UI
   2. Click Data Movement Policy
   3. Click Groups
   4. Select the group corresponding to your Cloud Tier policy
   5. Click Actions
   6. Click Edit Group
   7. Click Next until finish (DO NOT change anything)