VxRail: Invalid Refresh Token Error During Internet Download

Internet Download Failure

VxRail Internet download 8.0.330 fails with error message:

"Error occurred while downloading the upgrade bundle from the website. Retry downloading in a few minutes."

lcm-web.log

2025-07-23T07:39:06.256Z <3456540b4fc0dd8bd64899703970a084> lcm [ERROR] <1689> LcmStateManager.java logProgressError() (665): FAILED GETTING PROGRESS
2025-07-23T07:39:06.257Z <3456540b4fc0dd8bd64899703970a084> lcm [ERROR] <1689> BundleAcgDownloadProcessService.java waitDownloadComplete() (208): failed querying download progress
com.vce.lcm.exception.LCMException: Failed to complete the poll ACG download progress operation. Reeturn Code is: 261. Detail message is: EMC SERVER OTHER ERROR
        at com.emc.mystic.manager.upgrade.process.BundleAcgDownloadProcessService.queryDownloadProgress(BundleAcgDownloadProcessService.java:297)
        at com.emc.mystic.manager.upgrade.process.BundleAcgDownloadProcessService.waitDownloadComplete(BundleAcgDownloadProcessService.java:202)
        at com.emc.mystic.manager.upgrade.process.BundleFallbackDownloadProcessService.waitDownloadComplete(BundleFallbackDownloadProcessService.java:160)
        at com.emc.mystic.manager.upgrade.process.LcmBundleDownloadAndDeployProcess.waitDownloadThenContinueDeploy(LcmBundleDownloadAndDeployProcess.java:145)
        at com.emc.mystic.manager.upgrade.process.LcmBundleDownloadAndDeployProcess.access$000(LcmBundleDownloadAndDeployProcess.java:47)
        at com.emc.mystic.manager.upgrade.process.LcmBundleDownloadAndDeployProcess$1.run(LcmBundleDownloadAndDeployProcess.java:99)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2025-07-23T07:39:07.257Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> BundleAcgDownloadProcessService.java queryDownloadProgress() (276): get upgrade package download progress for Upgrade B
undle: 80331174
2025-07-23T07:39:41.079Z <3456540b4fc0dd8bd64899703970a084> lcm [ERROR] <1689> LcmStateManager.java logProgressError() (665): FAILED GETTING PROGRESS
2025-07-23T07:39:41.080Z <3456540b4fc0dd8bd64899703970a084> lcm [ERROR] <1689> BundleAcgDownloadProcessService.java waitDownloadComplete() (208): failed querying download progress
com.vce.lcm.exception.LCMException: Failed to complete the poll ACG download progress operation. Reeturn Code is: 261. Detail message is: EMC SERVER OTHER ERROR
        at com.emc.mystic.manager.upgrade.process.BundleAcgDownloadProcessService.queryDownloadProgress(BundleAcgDownloadProcessService.java:297)
        at com.emc.mystic.manager.upgrade.process.BundleAcgDownloadProcessService.waitDownloadComplete(BundleAcgDownloadProcessService.java:202)
        at com.emc.mystic.manager.upgrade.process.BundleFallbackDownloadProcessService.waitDownloadComplete(BundleFallbackDownloadProcessService.java:160)
        at com.emc.mystic.manager.upgrade.process.LcmBundleDownloadAndDeployProcess.waitDownloadThenContinueDeploy(LcmBundleDownloadAndDeployProcess.java:145)
        at com.emc.mystic.manager.upgrade.process.LcmBundleDownloadAndDeployProcess.access$000(LcmBundleDownloadAndDeployProcess.java:47)
        at com.emc.mystic.manager.upgrade.process.LcmBundleDownloadAndDeployProcess$1.run(LcmBundleDownloadAndDeployProcess.java:99)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2025-07-23T07:39:41.108Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> MysticClusterReaderDO.java isClusterTypeAS() (189): Return value from cluster DO: ['STANDARD', 'UNAVAILABLE']

2025-07-23T07:39:41.204Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> EventMicroWriter.java saveEvent() (424): Successfully saved new event.event_id:178
2025-07-23T07:39:41.204Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> EventMicroWriter.java saveFailedLCMUpgradeEvent() (162): Successfully saved event:VXR104114
2025-07-23T07:39:41.204Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateManagerService.java markError() (566): ERROR [LcmBundleDownload]
2025-07-23T07:39:41.204Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateManagerService.java markError() (569): currentState=[DOWNLOADING]
2025-07-23T07:39:41.204Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateManagerService.java updateOperationStatusInfo() (814): update operation status info for LcmBundleDownloadAndDeploy-cc16ad1a-f8bf-49e1-8525-a8f8c04834c3
2025-07-23T07:39:41.211Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> VxrailSystemDO.java getOperationWithDoModuleByRequestId() (171): operation status cache put LcmBundleDownloadAndDeploy-cc16ad1a-f8bf-49e1-8525-a8f8c04834c3
2025-07-23T07:39:41.220Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> VxrailSystemDO.java updateOperationById() (206): operation status cache update 328 LcmBundleDownloadAndDeploy-cc16ad1a-f8bf-49e1-8525-a8f8c04834c3
2025-07-23T07:39:41.220Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> OperationStatusInfoMicroWriter.java failRequest() (183): update OPERATION_STATUS,state=FAILED, endtime=1753256381211 in DO
2025-07-23T07:39:41.220Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateManagerService.java updateBundleRepository() (735): currentState=[DOWNLOADING]
2025-07-23T07:39:41.220Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateManagerService.java updateBundleRepository() (740): SET: targetState=[DOWNLOAD_ERROR]
2025-07-23T07:39:41.221Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateUpdater.java setState() (66): invalidate caches due to change lcm state from DOWNLOADING to DOWNLOAD_ERROR.
2025-07-23T07:39:41.221Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> CacheManager.java invalidate() (44): invalidate cache HostDO: CacheStats{hitCount=16, missCount=16, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=0}
2025-07-23T07:39:41.221Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> CacheManager.java invalidate() (44): invalidate cache VxrailSystemDO: CacheStats{hitCount=1365, missCount=14, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=12}
2025-07-23T07:39:41.223Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateUpdater.java reload() (176): invalidate caches due to change lcm state from DOWNLOADING to DOWNLOAD_ERROR.
2025-07-23T07:39:41.223Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> CacheManager.java invalidate() (44): invalidate cache HostDO: CacheStats{hitCount=16, missCount=16, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=0}
2025-07-23T07:39:41.223Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> CacheManager.java invalidate() (44): invalidate cache VxrailSystemDO: CacheStats{hitCount=1365, missCount=14, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=12}
2025-07-23T07:39:41.223Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> LcmStateManagerService.java markError() (671): >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025-07-23T07:39:41.231Z <3456540b4fc0dd8bd64899703970a084> lcm [INFO] <1689> VxrailSystemDO.java getOperationWithDoModuleByRequestId() (171): operation status cache put LcmBundleDownloadAndDeploy-cc16ad1a-f8bf-49e1-8525-a8f8c04834c3

short.term.log

2025-07-23T07:13:45.443Z <3456540b4fc0dd8bd64899703970a084> do-vxrail-system [INFO] <140006429554368> request.py log_request() (61): Request body: b'{""id"":328,""request_id"":""LcmBundleDownloadAndDeploy-cc16ad1a-f8bf-49e1-8525-a8f8c04834c3"",""owner"":""LcmBundleDownloadAndDeploy"",""start_time"":""2025-07-23 06:53:32.054000"",""end_time"":null,""state"":""IN_PROGRESS"",""error"":null,""progress"":0,""target"":null,""step"":null,""detail"":null,""extension"":""{\\""vcplugin_updated\\"":false,\\""retry\\"":0,\\""retry_maximum\\"":0,\\""details\\"":[{\\""action\\"":\\""Downloading upgrade bundle\\"",\\""message\\"":\\""Downloading upgrade bundle 8.0.331-29160392 from website\\""}]}""}'"
2025-07-23T07:13:45.445Z <3456540b4fc0dd8bd64899703970a084> do-vxrail-system [INFO] <140006429554368> request.py log_response() (118): [REQUEST END] The request finished successfully. Response status: 200 OK"
2025-07-23T07:13:45.446Z [INFO] <6501> middleware.go 1() (25): POST /v1/device-auth/request-token"
2025-07-23T07:13:45.446Z [INFO] <6501> device_access_token_controller.go GetAccessToken() (48): Enter Get Access Token"
2025-07-23T07:13:45.452Z [INFO] <6501> proxy_gateway_impl.go GetProxy() (37): start retrieving system proxy "
2025-07-23T07:13:45.518Z [INFO] <6501> http_client.go setTlsConfig() (63): Proxy does not exist, create http client without proxy"
2025-07-23T07:13:46.202Z [ERROR] <6501> di_gateway_impl.go RefreshToken() (208): Failed to call DI refresh access token API due to response status is :400"
2025-07-23T07:13:46.202Z [INFO] <6501> di_gateway_impl.go decodeErrorResponse() (320): Error response received from DI: {""error"":""invalid_request"",""error_description"":""P2:Refresh token is invalid or has already been revoked"",""error_uri"":""""}"
2025-07-23T07:13:46.203Z [INFO] <6501> di_gateway_impl.go decodeErrorResponse() (328): Decoded DI error response: {ErrorCode:invalid_request ErrorDescription:P2:Refresh token is invalid or has already been revoked ErrorUri:}"
"
2025-07-23T07:13:46.203Z [ERROR] <6501> token_access_usercase.go GetAccessToken() (41): Failed to get a new token because invalid_request"
2025-07-23T07:13:46.203Z [DEBUG] <6501> i18n.go TransWithReq() (76): Detected locale: en_US"
2025-07-23T07:13:46.203Z [INFO] <6501> device_access_token_controller.go func1() (50): End of Get Access Token."
2025-07-23T07:13:46.203Z [INFO] <6501> middleware.go 1() (29): POST /v1/device-auth/request-token 500"
2025-07-23T07:13:46.203Z <3456540b4fc0dd8bd64899703970a084> do-eservices [WARNING] <140246729619136> utils.py call() (212): Retryed 0 times, spending 0.7598 seconds, raise TokenServiceError('MCP-Token-Service error. {""ErrorCode"":""RELOGIN_REQUIRED"",""Message"":""Invalid refresh token. Try to login again.""}\n') error"
2025-07-23T07:13:46.944Z <00000000000000000000000000000000> hsm [INFO] <139837202677280> sync_task.py get() (129): ping_result[{'celery@do-network-5c456c8cfb-ctq67': {'ok': 'pong'}}, {'celery@ms-day2-cdf7d744d-x5k75': {'ok': 'pong'}}, {'celery@do-storage-56d8568cdd-xb524': {'ok': 'pong'}}, {'celery@do-host-786665f466-2vbkx': {'ok': 'pong'}}, {'celery@hsm-6b446fd4c6-fbx75': {'ok': 'pong'}}, {'celery@do-cluster-688794855f-s45zv': {'ok': 'pong'}}, {'celery@do-vm-9dd674f98-cv4xm': {'ok': 'pong'}}]."

mcp-token-service.log

"2025-07-23 07:54:23,232" microservice.mcp-token-service "2025-07-23T07:54:22.905722194Z stderr F 2025-07-23T07:54:22.905Z [ERROR] <10048> di_gateway_impl.go RefreshToken() (208): Failed to call DI refresh access token API due to response status is :400"
"2025-07-23 07:54:23,232" microservice.mcp-token-service "2025-07-23T07:54:22.905778189Z stderr F 2025-07-23T07:54:22.905Z [INFO] <10048> di_gateway_impl.go decodeErrorResponse() (320): Error response received from DI: {""error"":""invalid_request"",""error_description"":""S2:Refresh token is invalid or has already been revoked"",""error_uri"":""""}"
"2025-07-23 07:54:23,232" microservice.mcp-token-service "2025-07-23T07:54:22.9058447Z stderr F 2025-07-23T07:54:22.905Z [INFO] <10048> di_gateway_impl.go decodeErrorResponse() (328): Decoded DI error response: {ErrorCode:invalid_request ErrorDescription:S2:Refresh token is invalid or has already been revoked ErrorUri:}"
"2025-07-23 07:54:23,232" microservice.mcp-token-service "2025-07-23T07:54:22.905847339Z stderr F "
"2025-07-23 07:54:23,232" microservice.mcp-token-service "2025-07-23T07:54:22.905896616Z stderr F 2025-07-23T07:54:22.905Z [ERROR] <10048> token_access_usercase.go GetAccessToken() (41): Failed to get a new token because invalid_request"

Token Authentication Failure

The error messages in the logs indicate that the token authentication failed.
The refresh token is invalid or has already been revoked, causing the authentication failure.

"2025-07-23 07:54:23,232" microservice.mcp-token-service "2025-07-23T07:54:22.9058447Z stderr F 2025-07-23T07:54:22.905Z [INFO] <10048> di_gateway_impl.go decodeErrorResponse() (328): Decoded DI error response: {ErrorCode:invalid_request ErrorDescription:S2:Refresh token is invalid or has already been revoked ErrorUri:}"

From "Cluster" - "Configure" - "VxRail" - "Support" the "Dell Support Portal Access" shows configured and a future expiration date. This can be misleading as the log errors indicate an authentication failure.

Resolution Steps

To resolve the issue, follow these steps:

• Investigate the system logs to identify the cause of the authentication failure.
• Verify that authentication is properly set up for the Customer Support account.
• Run the reconfigure authentication option to update the authentication token even though it shows the correct status.
• After reconfiguring authentication, attempt to download the upgrade bundle again.
• If the issue persists, check the system logs for any errors or warnings related to authentication or token validation.