VxRail:升級錯誤無法設定自訂的 depot
Summary: vlcm 啟用時,VxRail 升級錯誤「無法上傳套裝:觸發設定自訂維修站遇到例外狀況」,線上維修站 URL 無效或現在無法連線。
Symptoms
VxRail 已啟用 vLCM,再升級至 VxRail 7.0.x。
接著在執行升級至 VxRail 7.0.x 時失敗,並顯示以下錯誤:
Failed to upload bundle: VxRail_COMPOSITE-XXXX.zip Trigger set customized depot meets exception, detail: Meet error in vlcm service request exchange.....
lcm-web.log:
2022-05-17 02:49:13,848 ERROR [SURROGATE] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#1-1] c.v.l.a.LCMServiceImpl [LCMServiceImpl.java:913] Trigger set customized depot meets exception, detail:
com.dellemc.vxrail.lcm.data.provider.out.VlcmServiceWriteException: Meet error in vlcm service request exchange, please check log for detail.{"message": "Internal Server Error"}
at com.dellemc.vxrail.lcm.data.provider.utils.HttpUtilsForVlcm.postVlcmHttpResponse(HttpUtilsForVlcm.java:94)
at com.dellemc.vxrail.lcm.data.provider.repositories.service.VlcmAPIService.setOnlineDepots(VlcmAPIService.java:164)
at com.dellemc.vxrail.lcm.data.provider.out.VlcmServiceMicroWriter.setCustomizedOnlineDepots(VlcmServiceMicroWriter.java:95)
at com.vce.lcm.api.LCMServiceImpl.vlcmCustomizedDepotProcess(LCMServiceImpl.java:890)
at com.vce.lcm.api.LCMServiceImpl.uploadLocalCompositeBundle(LCMServiceImpl.java:828)
at com.vce.lcm.api.LCMServiceImpl.resumeUploadCompositeBundle(LCMServiceImpl.java:1146)
at com.vce.lcm.api.LCMServiceImpl.uploadAcgCompositeBundle(LCMServiceImpl.java:1134)
at com.vce.lcm.api.LCMServiceImpl.uploadACGUpgradeBundle(LCMServiceImpl.java:362)
at com.emc.mystic.manager.upgrade.service.VirtualApplianceServiceImpl.startUploadOfUpgradeBundle(VirtualApplianceServiceImpl.java:775)
at com.emc.mystic.manager.upgrade.service.VirtualApplianceServiceImpl.triggerUploadOfUpgradeBundleFromDeployMessage(VirtualApplianceServiceImpl.java:557)
at com.emc.mystic.manager.upgrade.service.VirtualApplianceServiceImpl.processUploadOfUpgradeBundle(VirtualApplianceServiceImpl.java:495)
short.term.log:
2022-05-16-07:12:08 microservice.do-cluster "2022-05-16 07:12:08,275 [ERROR] <Dummy-1149:140446016676424> request.py log_response() (131): [REQUEST END] The request could not be understood by the server due to malformed syntax. Response status: 400 BAD REQUEST, Response body: ['{""error_type"": ""INVALID_ARGUMENT"", ""messages"": [{""args"": [""https://<vxm_ip_or_fqdn>/vlcm/depot/vlcm/components/ESXi-7.0.3_19482537-8f99f6c1/index.xml""], ""default_message"": ""Online Depot URL \'https://<vxm_ip_or_fqdn>/vlcm/depot/vlcm/components/ESXi-7.0.3_19482537-8f99f6c1/index.xml\' is not valid or cannot be reached now."", ""localized"": ""Online Depot URL \'https://<vxm_ip_or_fqdn>/vlcm/depot/vlcm/components/ESXi-7.0.3_19482537-8f99f6c1/index.xml\' is not valid or cannot be reached now."", ""id"": ""com.vmware.vcIntegrity.lifecycle.depots.online.Invalid""}]}']"
2022-05-16-07:12:08 microservice.vlcm "2022-05-16 07:12:08,276 - DEBUG - connectionpool(461) - http://api-gateway:8080 ""POST /rest/vxm/internal/do/v1/vc/api/esx/settings/depots/online HTTP/1.1"" 400 556"
2022-05-16-07:12:08 microservice.vlcm "2022-05-16 07:12:08,278 - INFO - do(112) - Call do response status code: 400. "
2022-05-16-07:12:08 microservice.vlcm "2022-05-16 07:12:08,278 - INFO - online_depot_service(42) - INVALID_ARGUMENT"
2022-05-16-07:12:08 microservice.vlcm "2022-05-16 07:12:08,278 - ERROR - app(1892) - Exception on /vlcm/v1/depot/online [POST]"
2022-05-16-07:12:08 microservice.vlcm "Traceback (most recent call last):"
2022-05-16-07:12:08 microservice.vlcm " File ""/usr/local/venv/lib64/python3.6/site-packages/flask/app.py"", line 1950, in full_dispatch_request"
2022-05-16-07:12:08 microservice.vlcm " rv = self.dispatch_request()"
2022-05-16-07:12:08 microservice.vlcm " File ""/usr/local/venv/lib64/python3.6/site-packages/flask/app.py"", line 1936, in dispatch_request"
2022-05-16-07:12:08 microservice.vlcm " return self.view_functions[rule.endpoint](**req.view_args)"
2022-05-16-07:12:08 microservice.vlcm " File ""/usr/local/venv/lib64/python3.6/site-packages/flask_restplus/api.py"", line 325, in wrapper"
2022-05-16-07:12:08 microservice.vlcm " resp = resource(*args, **kwargs)"
2022-05-16-07:12:08 microservice.vlcm " File ""/usr/local/venv/lib64/python3.6/site-packages/flask/views.py"", line 89, in view"
2022-05-16-07:12:08 microservice.vlcm " return self.dispatch_request(*args, **kwargs)"
2022-05-16-07:12:08 microservice.vlcm " File ""/usr/local/venv/lib64/python3.6/site-packages/flask_restplus/resource.py"", line 44, in dispatch_request"
2022-05-16-07:12:08 microservice.vlcm " resp = meth(*args, **kwargs)"
2022-05-16-07:12:08 microservice.vlcm " File ""/home/app/api/vlcm_api.py"", line 64, in post"
2022-05-16-07:12:08 microservice.vlcm " depot_service.add_depot(depot_url)"
2022-05-16-07:12:08 microservice.vlcm " File ""/home/app/services/online_depot_service.py"", line 46, in add_depot"
2022-05-16-07:12:08 microservice.vlcm " raise Exception(response.content)"
2022-05-16-07:12:08 microservice.vlcm "Exception: b'{""error_type"":""INVALID_ARGUMENT"",""messages"":[{""args"":[""https://<vxm_ip_or_fqdn>/vlcm/depot/vlcm/components/ESXi-7.0.3_19482537-8f99f6c1/index.xml""],""default_message"":""Online Depot URL \'https://<vxm_ip_or_fqdn>/vlcm/depot/vlcm/components/ESXi-7.0.3_19482537-8f99f6c1/index.xml\' is not valid or cannot be reached now."",""localized"":""Online Depot URL \'https://<vxm_ip_or_fqdn>/vlcm/depot/vlcm/components/ESXi-7.0.3_19482537-8f99f6c1/index.xml\' is not valid or cannot be reached now."",""id"":""com.vmware.vcIntegrity.lifecycle.depots.online.Invalid""}]}'"
Cause
通常,VxRail Manager 的 SSL 憑證會更換為自我簽署憑證。
Resolution
請遵循以下步驟,更換由 VMCA 簽署的 VxRail Manager SSL 憑證。
- 在 VxRail Manager 上執行以下指令檔,重新產生 VxRail Manager SSL 憑證:
python /etc/vmware-marvin/scripts/lcm/scripts/import_vc_generated_vxm_cert_enhance.py -c <vc_fqdn> -u <vc_mgmt_account> -p <vc_mgmt_passwd> -s root -t <vc_root_passwd> -x <vxm_ip> -n <vxm_fqdn>
Note: Add a ' at the beginning and end of the passwords IF you have special characters in the password.
python /etc/vmware-marvin/scripts/lcm/scripts/import_vc_generated_vxm_cert_enhance.py -c <vc_fqdn> -u <vc_mgmt_account> -p <vc_mgmt_passwd> -s root -t <vc_root_passwd> -x <vxm_ip> -n <vxm_fqdn> -v
Note: Add a ' at the beginning and end of the passwords IF you have special characters in the password.
python /etc/vmware-marvin/scripts/lcm/scripts/import_vc_generated_vxm_cert_enhance.py -h Usage: import_vc_generated_vxm_cert_enhance.py [options] Options: -h, --help show this help message and exit -c ADDRESS, --address=ADDRESS Address of VC to connect to -u MGT_USERNAME, --management-username=MGT_USERNAME user name to use when connecting to VC -p MGT_PASSWORD, --management-password=MGT_PASSWORD Password to use when connecting to VC -s GUEST_USERNAME, --guest-username=GUEST_USERNAME user name to use when running guest OS operation on VC -t GUEST_PASSWORD, --guest-password=GUEST_PASSWORD Password to use when running guest OS operation to VC -x VXM_IP, --vxm-ip=VXM_IP The VXM IP, must provide it when this script is not running on VXM server -n VXM_FQDN, --vxm-fqdn=VXM_FQDN The VXM fqdn, must provide it when this script is not running on VXM server -v, --external-vc The external vc flag, must set true if VC is external. -e, --encrypted-auth The flag to claim provided auth are encrypted, need to decrypt when script running. -m, --m2m-enabled The flag to claim if hosts enable m2m
以下是在使用內部 vCenter 的實驗室中的範例輸出:
vxm:/home/mystic # python /etc/vmware-marvin/scripts/lcm/scripts/import_vc_generated_vxm_cert_enhance.py -c vcluster101-vcsa.vv009.local -u administrator@vsphere.local -p 'Testvxrail123!' -s root -t Testvxrail123! -x 172.16.10.200 -n vcluster101-vxm.vv009.local
以下為指令檔輸出:
Start to connect VC: vcluster101-vcsa.vv009.local
Succeed to connect VC
VXM ip: 172.16.10.200, VXM fqdn: vcluster101-vxm.vv009.local, VC fqdn: vcluster101-vcsa.vv009.local
Getting VM via ip: vcluster101-vcsa.vv009.local
VC VM: vcluster101-vcsa.vv009.local 172.16.10.201
Create temporary SSL cert folder on VC: /tmp/ssl_cert-172.16.10.200-5203
Generate private key for vxm cert
Generate certtool config file
Generate vxm cert
Backup root cert
Generate import ssl cert API script
#!/bin/sh
# Import cert
get_cert_data() {
while read -r line; do
# first line and not empty line
if [[ -z $x ]] && [[ -n $line ]]; then
x=$x"$line"
# other line and not empty line
elif [[ ! -z $line ]]; then
x=$x"\n""$line"
fi
done <$1
echo $x
}
generate_post_data() {
cat <<EOF
{"cert":"${vxm_cert}","primary_key":"${vxm_private_key}","root_cert_chain":"${root_cert}","password":"testpassword"}
EOF
}
vxm_key_file=$1
vxm_crt_file=$2
root_crt_file=$3
vxm_key_filter_file=$4
cat $vxm_key_file | sed -n -e "/BEGIN RSA PRIVATE KEY/, $"p >$vxm_key_filter_file
vxm_cert=$(get_cert_data $vxm_crt_file)
echo "vxm cert: $vxm_cert"
echo ""
root_cert=$(get_cert_data $root_crt_file)
echo "root cert: $root_cert"
echo ""
vxm_private_key=$(get_cert_data $vxm_key_filter_file)
echo "vxm private key: $vxm_private_key"
echo ""
echo $(generate_post_data)
i=1
while [[ $i -le 5 ]]; do
result=`curl -k -X POST -H "Content-Type:application/json" -H "Authorization: Basic $5" https://$6/rest/vxm/$7/certificates/import-vxm --data "$(generate_post_data)"`
echo "result: $result"
echo ""
message=$(echo $result | grep "message")
if [[ $message =~ "successfully" ]]; then
echo "Successfully"
exit 0
else
echo "Import cert retry: $i"
echo ""
let i++
sleep 60
fi
done
if [[ $i -eq 5 ]]; then
echo "Fail"
exit 1
fi
Run import SSL cert API script
Finish to import VC generated cert to VXM.
- 重試升級。
"Failed to decode JSON object: Invalid control character at: line 1 column 4421 (char 4420)". 必須手動執行憑證更新 API,以解決問題:
- 用於呼叫「更新 VxRail Manager 憑證 (版本 2)」的 API
- VxRail 端點:
https://<< vxrm ip>>/rest/vxm/api-doc.html?#/operations/v2_certificates_import_vxm_post - 路徑:存取 VxRail API 網頁並瀏覽至「憑證 -> 更新 VxRail Manager 憑證 (版本 2)」

範例代碼段:
curl --request POST \
--url https://172.xxx.xx.xx/rest/vxm/v2/certificates/import-vxm \
--header 'Authorization: Basic YWRtaW5pc3RyYXRvckB2c3BoZXJlLmxvY2FsOlRlc3R2eHJhaWwxMjMh' \
--header 'Content-Type: application/json' \
--data '{
"cert": "-----BEGIN CERTIFICATE-----...",
"primary_key": "-----BEGIN ENCRYPTED PRIVATE KEY-----...",
"root_cert_chain": "-----BEGIN CERTIFICATE-----...",
"password": "testPassword"
}'
將三個點替換為 PEM 檔的完整原始文字內容,格式為帶有換行符 (\n) 的單行字串。(新憑證的內容必須為 PEM 格式)