PowerFlex 2.x GUI not showing any information about frontend or backend

Issue Description

After a few days to a few weeks, the backend and frontend views in the UI show only a loading icon instead of the expected information. 

Scenario

The UI begins to experience a hang in loading the SDS, SDC, and Volume views after some time passes.

Symptoms

The following screenshots are what could be seen in either the frontend or backend views when this issue happens:

 
 

 

In the UI logs from "%appdata%\EMC\scaleio\logs\log.txt" thousands of the following can occur:

java.util.concurrent.TimeoutException: Could not achieve full QueryPoll response in 10,000 requests

Impact

Unable to manage or monitor the ScaleIO system from the UI.  

The root cause of this issue is when there are a thousand+ creation and deletions of Volumes (vtrees) over a period of time where the number of volumes still in existence in the ScaleIO system is above 1000+. This impacts the UI FilterController used by the BaseTreeTablePanel which is in use by both Backend and Frontend views, and also determines when the Backend and Frontend views are finally shown. Since the UI never gets a complete model, the Loading image keeps spinning until the model is fully loaded.

This type of issue can be seen in OpenStack deployments since there are a lot of create/map/unmap/delete events on volumes due to the nature of snapshots in OpenStack. 

The current workaround is to perform a switch MDM ownership with the sci: 

scli --switch_mdm_ownership --new_master_mdm_id <ID of Slave MDM>

Impacted versions

2.0.1.3, 2.0.1.4, 2.5, 2.6

Fixed in version

2.6.1, 3.0