ECS: Container is permanently down on node: Symptom Code: 2021

요약: This knowledge article explains why the ECS has reported that a container is permanently down on a node.

이 문서는 다음에 적용됩니다. 이 문서는 다음에 적용되지 않습니다. 이 문서는 특정 제품과 관련이 없습니다. 모든 제품 버전이 이 문서에 나와 있는 것은 아닙니다.
기타 리소스 확인

증상

The system has dialed home with the following alert: 
Clarify Id: APMxxxxxxxx
Site Name: UNKNOWN
Vendor: EMC
DeviceType: ElasticCloudStorageApp
Model: ElasticCloudStorage
SerialNumber: APMxxxxxxxx
WWN: APMxxxxxxxx
Platform: platform
OS: SLES
OS_VER: 12.4
EmbedLevel: 2
InternalMaxSize: 512800
Comment: Fabric
Ucode_Ver: 3.7.0.6-7700.ed29023b
ConnectType: ESRS
IP_Address: Not Available
IP_Name: hostname.domainname.net
ConnectNum: 169.254.1.1
Port: 22

SymptomCode: 2021
Category: Status
Severity: Critical
Status: Failed
Component: Node
ComponentID: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
SubComponent: Service
SubComponentID: <docker container name>
CallHome: true
FirstTime: 2023-12-09T07:48:20.232Z
Description: Container <container> is permanently down on node <node>

원인

The container stopped or paused or not started at all for at least 10 minutes.

해결

Docker containers (object-main, fabric-lifecycle, fabric-zookeeper, fabric-registry) stopped or paused or not started at all for at least 10 minutes. Use the following procedure to determine the failing container:
  1. Based on Component ID or Node ID determine the node in the ECS cluster which registered the failure. Example: Component ID
4ca42022-46ed-475e-8ab7-6ef9141e5415
sudo /opt/emc/caspian/fabric/cli/bin/fcli lifecycle node.network --id 4ca42022-46ed-475e-8ab7-6ef9141e5415
{
  "network": {
    "hostname": "hostname.domainname.net",  << Hostname
    "private_ip": "169.254.1.3",       << NAN IP
    "mgmt_ip": "10.2.3.4",     << Management IP 
    "public_ip": "10.241.207.59",
    "data_ip": "10.241.207.73",
    "replication_ip": "10.241.207.59",
    "public_interface_name": "public",
    "private_interface_name": "private.4",
    "mgmt_interface_name": "public",
    "data_interface_name": "public:data",
    "replication_interface_name": "public"
  },
  "status": "OK",
  "etag": 50
}
  1.  ssh to the target node using management IP, private IP, or hostname.
 
  1. Verify that the docker server is operational with proper parameters
# ps -ef | grep docker
root      50062      1  0 Jun02 ?        00:02:11 /usr/bin/docker daemon -H fd:// --insecure-registry=0.0.0.0/0 --log-level=warn
  1. Next we have to validate which container (NAMES Column) was stopped or not started at all for some reason (look at STATUS column):
# sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED                STATUS              PORTS               NAMES
7bf16df0ef15        464b97154c24        "/opt/vipr/boot/boot."   3 days ago          Up 3 days                                    object-main
0ef5cc422543        24d9d6008893        "./boot.sh lifecycle"    3 days ago          Up 3 days                                    fabric-lifecycle
87d6c77d98ca        32cce433c3dc        "./boot.sh 3 1=169.25"   3 days ago          Up 3 days                                    fabric-zookeeper
  1. Verify that the fabric service is running. Fabric agent attempts to revive problem container automatically
# sudo service fabric-agent status
fabric-agent.service - fabric agent
   Loaded: loaded (/usr/lib/systemd/system/fabric-agent.service; enabled)
   Active: active (running) since Thu 2016-06-02 17:56:39 UTC; 3 days ago
  Process: 50643 ExecStartPre=/bin/rm -f /var/run/fabric-agent.pid (code=exited, status=0/SUCCESS)
 Main PID: 50645 (java)
   CGroup: /system.slice/fabric-agent.service
  1. View the stopped/failed container status 
# sudo docker inspect fabric-zookeeper | grep -A12 State
    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 80462,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2016-06-06T17:29:12.968133861Z",
        "FinishedAt": "2016-06-06T17:29:12.882812946Z"
    },
 
If we are still uncertain if an issue exists or not, consult with ECS technical support for further assistance.

해당 제품

ECS Appliance Software without Encryption

제품

ECS Appliance, ECS Appliance Software with Encryption, ECS Appliance Software without Encryption
문서 속성
문서 번호: 000064491
문서 유형: Solution
마지막 수정 시간: 17 12월 2025
버전:  6
다른 Dell 사용자에게 질문에 대한 답변 찾기
지원 서비스
디바이스에 지원 서비스가 적용되는지 확인하십시오.