ViPR Controller: An order fails due to "An error occurred while encoding/decoding of tokens"

A ViPR Controller order fails with the error message reporting:

• An error occurred while encoding/decoding of tokens

The ViPR Controller sasvc log reports the error as a result of a failed REST GET request:

vipr3 sasvc 2019-04-18 14:33:41,448 [pool-112-thread-9]  INFO  LoggingFilter.java (line 238) 138143 > GET https://<ViPR IP>:4443/vdc/tasks/<Task URN>
 

Note: Although the order's status is shown as failed, with the above error message, it is possible that the generated workflows and tasks associated to the order continue to run and complete successfully.
For example, if the order run was "Change Virtual Pool" then the order reports a failure but as the workflows and tasks continue to execute the volumes are successfully migrated.

ViPR Controller nodes have a key rotation interval of 24 hours, set at the time of the latest vApp/node reboot.

The ViPR Controller authsvc logs report the key rotation:

vipr3 authsvc 2019-04-18 14:33:38,690 [pool-24-thread-1]  INFO  TokenKeyGenerator.java (line 334) Current key in cache is old.  Reloading cache.
vipr3 authsvc 2019-04-18 14:33:38,693 [pool-24-thread-1]  INFO  TokenKeyGenerator.java (line 342) Current key in cache is still old after reload.  Rotating keys now.
vipr3 authsvc 2019-04-18 14:33:38,693 [pool-24-thread-1]  INFO  TokenKeyGenerator.java (line 496) Rotating keys...
vipr3 authsvc 2019-04-18 14:33:38,697 [pool-24-thread-1]  INFO  TokenKeyGenerator.java (line 499) Done rotating keys...
 

In rare cases, this key rotation can cause a momentary inconsistency where the active thread uses the old encoding strings to decode the key.

In the example above, a key rotation takes place at 14:33:38 on the vipr3 node.
Then at 14:33:41 the active thread runs a GET request on a Task and this uses the old encoding strings that have just been rotated.
The GET request fails which results in the order's status being reported as failed.
 

There are extremely rare cases when the daily ViPR Controller encoding key rotation sequence occurs during operations requiring password/token exchange and decoding.

1. If seen it should be confirmed whether there are any "in-progress" tasks associated to the order.
2. If there are tasks in progress, allow these tasks to run to completion.
3. If the tasks associated to the order have failed, or no tasks found, then the order can be re-submitted.
4. If subsequent orders/tasks fail with the same error contact the DELL EMC Customer Support Center or your service representative for technical assistance and quote this article ID.