ECS：xDoctor RAP014：光纖生命週期服務不健全 |Lifecycle Jetty 伺服器未在連接埠 9241 上啟動且執行

問題 #1：
在 ECS 上執行從版本 3.0.X 或更早版本至 3.1 版或更新版本的升級後，會在服務控制台中看到以下輸出：

20180309 01:49:28.456: |   |   | PASS (21 min 29 sec)
20180309 01:49:28.462: |   | PASS (21 min 29 sec)
20180309 01:49:28.463: |   Run Keyword If
20180309 01:49:28.464: |   |   Node Service Upgrade
    Initializing...

    Executing Program: NODE_SERVICE_UPGRADE

    |-Disable CallHome
    |   +-[0.0.0.0] SetCallHomeEnabled                                  PASS (1/7, 1 sec)
    |-Push Service Image To Registries
    |   |-Push Service Image to Head Registry
    |   |   |-[169.254.1.1] LoadImage                                   PASS (2/7, 1 sec)
    |   |   +-[169.254.1.1] PushImage                                   PASS (3/7)
    |   +-Push Service Image to Remote Registries
    |-Upgrade Object On Specified Nodes
    |   +-Initiate Object Upgrade if Required
    |       +-[0.0.0.0] UpdateApplicationOnNodes                        PASS (4/7, 1 sec)
    |-Update Services Ownership To Lifecycle Manager on Specified Nodes
    |   +-Update Ownership For Object
    |       +-[169.254.1.1] UpdateOwnership                             PASS (5/7)
    |-Post-check Services Health
    |   +-Validate Object Service on Specified Nodes
    |       +-[169.254.1.1] ServiceHealth                               PASS (6/7, 21 sec)
    +-Enable CallHome
        +-[0.0.0.0] SetCallHomeEnabled                                  PASS (7/7, 3 sec)
    Elapsed time is 30 sec.

    NODE_SERVICE_UPGRADE completed successfully

    Collecting data from cluster
    Information has been written to the
    Information has been written to the
Executing /configure.sh --start action in object-main container which may take up to 600 seconds.
20180309 01:52:51.711: |   |   | PASS (3 min 23 sec)
20180309 01:52:51.720: |   | PASS (3 min 23 sec)
20180309 01:52:51.722: |   Run Keyword If
20180309 01:52:51.724: |   |   Update manifest file
[ERROR] On node 169.254.1.1, Lifecycle Jetty server is not up and running on port 9241!
20180309 01:58:45.068: |   |   | FAIL (5 min 53 sec)
20180309 01:58:45.071: |   | FAIL (5 min 53 sec)
20180309 01:58:45.072: | FAIL (45 min 43 sec)
20180309 01:58:45.075: Service Console Teardown
20180309 01:58:46.973: | PASS (1 sec)
================================================================================
Status: FAIL
Time Elapsed: 45 min 56 sec
Debug log: /
HTML log: /
================================================================================
Messages:
fabric-lifecycle service should be up and running
================================================================================

問題 #2：
xDoctor 可能會報告以下情況：

- xDoctor reports the following:
Timestamp = 2015-09-25_092907
Category = health
Source = fcli
Severity = WARNING
Message = Fabric Lifecycle Service not Healthy
Extra =

使用「sudo docker ps -a」監控光纖生命週期服務時，顯示服務正在重新開機：

venus2:~ # docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7995f18ba27f ip.ip.ip.ip:5000/emcvipr/object:2.0.1.0-62267.db4d4a8 "/opt/vipr/boot/boot 4 weeks ago Up 21 hours object-main
73f00ed0b6df ip.ip.ip.ip:5000/caspian/fabric:1.1.1.0-1998.1391e7e "./boot.sh lifecycle 4 weeks ago Up 3 seconds fabric-lifecycle
ba19a3c95151 ip.ip.ip.ip:5000/caspian/fabric-zookeeper:1.1.0.0-54.54a204e "./boot.sh 2 1=169.2 4 weeks ago Up 21 hours fabric-zookeeper
venus2:~ # docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7995f18ba27f ip.ip.ip.ip:5000/emcvipr/object:2.0.1.0-62267.db4d4a8 "/opt/vipr/boot/boot 4 weeks ago Up 21 hours object-main
73f00ed0b6df ip.ip.ip.ip:5000/caspian/fabric:1.1.1.0-1998.1391e7e "./boot.sh lifecycle 4 weeks ago Exited (1) 2 seconds ago fabric-lifecycle
ba19a3c95151 ip.ip.ip.ip:5000/caspian/fabric-zookeeper:1.1.0.0-54.54a204e "./boot.sh 2 1=169.2 4 weeks ago Up 21 hours fabric-zookeeper

原因問題 #1：
由於快照大小，ZooKeeper 容器無法正確啟動。

原因問題 #2：
ECS IP 解析到不正確的主機名稱。

解決方案問題 #1：
ECS 3.0
版中已解決，ECS 3.0 改進了 ZK 消息的壓縮並啟用了保留功能。

如果出現此問題，請聯絡 ECS 支援。

如何驗證此問題：
執行命令：

# viprexec 'cat /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log | grep "GC overhead limit exceeded"'

範例輸出：

admin@:~> viprexec 'cat /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log | grep "GC overhead limit exceeded"'

Output from host : 192.168.219.4
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

Output from host : 192.168.219.5
java.lang.OutOfMemoryError: GC overhead limit exceeded

Output from host : 192.168.219.3
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

Output from host : 192.168.219.7
cat: /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log: No such file or directory

Output from host : 192.168.219.2

Output from host : 192.168.219.8
cat: /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log: No such file or directory

Output from host : 192.168.219.6
cat: /opt/emc/caspian/fabric/agent/services/fabric/zookeeper/log/zookeeper.log: No such file or directory

Output from host : 192.168.219.1

adm@:in~>

此訊息會在 ZooKeeper 記錄檔中顯示：

OutOfMemoryError

java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
at org.apache.jute.BinaryOutputArchive.writeLong(BinaryOutputArchive.java:59)
at org.apache.zookeeper.data.Stat.serialize(Stat.java:129)
at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
at org.apache.zookeeper.proto.GetDataResponse.serialize(GetDataResponse.java:49)
at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1067)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)

解決方案問題#2：
使用 nslookup 驗證 DNS，以及已重新啟動生命週期的 ECS 節點的 IP 位址。

# nslookup <ip of ecs node>

如果您的 DNS 正確，但生命週期仍然存在問題，請聯繫 ECS 支援。