PowerFlex Scli commands fail with "Invalid session"

Scenario

The customer has one of the following:

• Third-party monitoring software or scripts that issue periodic scli logins against the MDM. 

And possibly: 

• MDM variables "user_session_hard_timeout_secs" or "user_session_timeout_secs" are set above default.

Symptoms

Two symptoms can be observed here:

1. Scli commands fail with the following error:

Error: MDM failed command. Status: Invalid session. Please login and try again.

2- Login attempts fail with the following error:

cli --login --username dave@local.net --ldap_authentication
Enter password:
Error: MDM failed command. Status: A timeout occurred

Impact

Cannot log in properly and no configuration can be done to ScaleIO.

When "user_session_hard_timeout_secs" or "user_session_timeout_secs" are set above default values and a monitoring script/program logs in repeatedly, the MDM login session table can become full before the hard timeout occurs for the first session. When the session table is full, login succeeds but no session-id is available and commands will fail.

Workaround

There are a couple of fixes for this issue:

1. Find the user/service that is logging in repeatedly and then stop the logins and wait 10 minutes (default idle timeout)
2. If the user/service cannot be stopped for other reasons, increase its login interval time
3. If this is not possible, you can reduce the user session timeout. The default is 10 minutes. See the documentation here, Pg 452.

OR

1. From Master MDM, run scli --query_cluster and ensure the MDM cluster state is Normal.

2. On a slave MDM: Remove (or comment) the following two variables, if present, in mdm/cfg/conf.txt on both MDMs: 

user_session_hard_timeout_secs= (default 8 hrs)
user_session_timeout_secs= (default 10 mins)

3- Restart the MDM service:

  kill <mdm pid>

4- Repeat the above steps for each Slave MDM.

5- Switch MDM ownership to one of the slave MDMs

6- Ensure the MDM cluster state is Normal, and then run steps 2 ~ 3 on the previous Master MDM.

7- Wait for 10 minutes, and then try again to issue scli commands after logging in.

Impacted versions

1.32.x 2.x 

Fixed in version

 3.x