ECS: xDoctor RAP014: Yapı Yaşam Döngüsü Hizmeti Sağlıklı Değil | Lifecycle Jetty sunucusu 9241 numaralı bağlantı noktasında çalışmıyor

Sayı #1:
ECS de sürüm 3.0.X veya önceki bir sürümden 3.1 veya sonraki bir sürüme yükseltme çalıştırıldıktan sonra hizmet konsolunda aşağıdaki çıktı görülür:

20180309 01:49:28.456: |   |   | PASS (21 min 29 sec)
20180309 01:49:28.462: |   | PASS (21 min 29 sec)
20180309 01:49:28.463: |   Run Keyword If
20180309 01:49:28.464: |   |   Node Service Upgrade
    Initializing...

    Executing Program: NODE_SERVICE_UPGRADE

    |-Disable CallHome
    |   +-[0.0.0.0] SetCallHomeEnabled                                  PASS (1/7, 1 sec)
    |-Push Service Image To Registries
    |   |-Push Service Image to Head Registry
    |   |   |-[169.254.1.1] LoadImage                                   PASS (2/7, 1 sec)
    |   |   +-[169.254.1.1] PushImage                                   PASS (3/7)
    |   +-Push Service Image to Remote Registries
    |-Upgrade Object On Specified Nodes
    |   +-Initiate Object Upgrade if Required
    |       +-[0.0.0.0] UpdateApplicationOnNodes                        PASS (4/7, 1 sec)
    |-Update Services Ownership To Lifecycle Manager on Specified Nodes
    |   +-Update Ownership For Object
    |       +-[169.254.1.1] UpdateOwnership                             PASS (5/7)
    |-Post-check Services Health
    |   +-Validate Object Service on Specified Nodes
    |       +-[169.254.1.1] ServiceHealth                               PASS (6/7, 21 sec)
    +-Enable CallHome
        +-[0.0.0.0] SetCallHomeEnabled                                  PASS (7/7, 3 sec)
    Elapsed time is 30 sec.

    NODE_SERVICE_UPGRADE completed successfully

    Collecting data from cluster
    Information has been written to the
    Information has been written to the
Executing /configure.sh --start action in object-main container which may take up to 600 seconds.
20180309 01:52:51.711: |   |   | PASS (3 min 23 sec)
20180309 01:52:51.720: |   | PASS (3 min 23 sec)
20180309 01:52:51.722: |   Run Keyword If
20180309 01:52:51.724: |   |   Update manifest file
[ERROR] On node 169.254.1.1, Lifecycle Jetty server is not up and running on port 9241!
20180309 01:58:45.068: |   |   | FAIL (5 min 53 sec)
20180309 01:58:45.071: |   | FAIL (5 min 53 sec)
20180309 01:58:45.072: | FAIL (45 min 43 sec)
20180309 01:58:45.075: Service Console Teardown
20180309 01:58:46.973: | PASS (1 sec)
================================================================================
Status: FAIL
Time Elapsed: 45 min 56 sec
Debug log: /
HTML log: /
================================================================================
Messages:
fabric-lifecycle service should be up and running
================================================================================

Sayı #2:
xDoctor aşağıdakileri bildirebilir:

- xDoctor reports the following:
Timestamp = 2015-09-25_092907
Category = health
Source = fcli
Severity = WARNING
Message = Fabric Lifecycle Service not Healthy
Extra =

Yapı Yaşam Döngüsü Hizmeti'nin "sudo docker ps -a" kullanılarak izlenmesi, hizmetin yeniden başlatıldığını gösterir:

venus2:~ # docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7995f18ba27f ip.ip.ip.ip:5000/emcvipr/object:2.0.1.0-62267.db4d4a8 "/opt/vipr/boot/boot 4 weeks ago Up 21 hours object-main
73f00ed0b6df ip.ip.ip.ip:5000/caspian/fabric:1.1.1.0-1998.1391e7e "./boot.sh lifecycle 4 weeks ago Up 3 seconds fabric-lifecycle
ba19a3c95151 ip.ip.ip.ip:5000/caspian/fabric-zookeeper:1.1.0.0-54.54a204e "./boot.sh 2 1=169.2 4 weeks ago Up 21 hours fabric-zookeeper
venus2:~ # docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7995f18ba27f ip.ip.ip.ip:5000/emcvipr/object:2.0.1.0-62267.db4d4a8 "/opt/vipr/boot/boot 4 weeks ago Up 21 hours object-main
73f00ed0b6df ip.ip.ip.ip:5000/caspian/fabric:1.1.1.0-1998.1391e7e "./boot.sh lifecycle 4 weeks ago Exited (1) 2 seconds ago fabric-lifecycle
ba19a3c95151 ip.ip.ip.ip:5000/caspian/fabric-zookeeper:1.1.0.0-54.54a204e "./boot.sh 2 1=169.2 4 weeks ago Up 21 hours fabric-zookeeper

Neden Sorun #1:
ZooKeeper kapsayıcısı, anlık görüntü boyutu nedeniyle düzgün başlatılamadı.

Neden Sorun #2:
ECS IP'si yanlış bir ana bilgisayar adına çözümleniyor.

Çözüm Sorunu #1:
ECS sürüm 3.0
da çözüldü ECS 3.0, sıkıştırmayı iyileştirir ve ZK mesajları için saklamayı etkinleştirir.

Bu sorun görülürse ECS destek birimine başvurun.

Bu sorun nasıl doğrulanır:
Komutu çalıştırın:

# viprexec 'cat /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log | grep "GC overhead limit exceeded"'

Örnek çıktı:

admin@:~> viprexec 'cat /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log | grep "GC overhead limit exceeded"'

Output from host : 192.168.219.4
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

Output from host : 192.168.219.5
java.lang.OutOfMemoryError: GC overhead limit exceeded

Output from host : 192.168.219.3
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

Output from host : 192.168.219.7
cat: /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log: No such file or directory

Output from host : 192.168.219.2

Output from host : 192.168.219.8
cat: /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log: No such file or directory

Output from host : 192.168.219.6
cat: /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log: No such file or directory

Output from host : 192.168.219.1

adm@:in~>

Bu mesaj ZooKeeper günlük dosyalarında görülür:

OutOfMemoryError

java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
at org.apache.jute.BinaryOutputArchive.writeLong(BinaryOutputArchive.java:59)
at org.apache.zookeeper.data.Stat.serialize(Stat.java:129)
at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
at org.apache.zookeeper.proto.GetDataResponse.serialize(GetDataResponse.java:49)
at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1067)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)

Çözüm Sorunu #2:
nslookup'ı kullanarak DNS'yi ve yaşam döngüsü yeniden başlatılan ECS düğümünün IP adresini doğrulayın.

# nslookup <ip of ecs node>

DNS'niz doğruysa ve yaşam döngüsüyle ilgili sorunlar devam ediyorsa ECS Desteği ile iletişime geçin.