This is a known issue where IDM agent keeps crashing repeatedly. View the similar entries in log files of PSC:
Vmware-sts-idmd.log
eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[org.apache.logging.log4j.core.impl.MutableLogEvent@4b53f538], detailText=[null], corelationId=[IDM Startup], timestamp=[1581503914883]
[2020-02-12T10:38:34.884Z IDM Startup INFO ] [IdmServer] IDM Server has started
[2020-02-12T10:41:18.285Z IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2020-02-12T10:41:18.285Z IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
[2020-02-12T10:41:54.772Z IDM Startup INFO ] [IdmServer] Starting IDM Server...
[2020-02-12T10:41:54.775Z IDM Startup INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[], eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[org.apache.logging.log4j.core.impl.MutableLogEvent@4b53f538], detailText=[null], corelationId=[IDM Startup], timestamp=[1581504114775]
[2020-02-12T10:41:54.776Z IDM Startup INFO ] [IdmServer] IDM Server has started
[2020-02-12T11:26:08.340Z IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2020-02-12T11:26:08.340Z IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
[2020-02-12T11:26:41.855Z IDM Startup INFO ] [IdmServer] Starting IDM Server...
[2020-02-12T11:26:41.858Z IDM Startup INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[], eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[org.apache.logging.log4j.core.impl.MutableLogEvent@4b53f538], detailText=[null], corelationId=[IDM Startup], timestamp=[1581506801858]
[2020-02-12T11:26:41.859Z IDM Startup INFO ] [IdmServer] IDM Server has started
[2020-02-12T12:59:17.964Z IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2020-02-12T12:59:17.964Z IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
[2020-02-12T12:59:54.032Z IDM Startup INFO ] [IdmServer] Starting IDM Server...
[2020-02-12T12:59:54.038Z IDM Startup INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[], eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[org.apache.logging.log4j.core.impl.MutableLogEvent@4b53f538], detailText=[null], corelationId=[IDM Startup], timestamp=[1581512394038]
[2020-02-12T12:59:54.038Z IDM Startup INFO ] [IdmServer] IDM Server has started
Vmware-stsd.err.log:
INFO: Initialization processed in 1648 ms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Feb 07, 2020 2:32:51 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 13741 ms
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=160m; support was removed in 8.0
Feb 10, 2020 6:49:55 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1538 ms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-slf4j-impl-2.11.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.26.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
The following may also be seen in /storage/core when partition is 100% full due to repeated crash of IDM agent:
/dev/mapper/core_vg-core 10181944 7752212 1889472 100 % /storage/core
VMware vCenter 6.7 U2 release notes describes the same symptoms:
https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-vcenter-server-67u2-release-notes.html
vCenter Server, vSphere Web Client, and vSphere Client Issues.
- Unable to log in to a vCenter Sever system due to a failure of the VMware Security Token Service service (vmware-stsd).
- The vmware-stsd service fails in certain user environments if the Active Directory Integrated Windows Authentication (IWA) is added as an identity source. The addition of IWA as an identity source may generate core dumps that fill up the /storage/core directory and eventually may cause log in failure to the vCenter Server system.
- In the vmware-sts-idmd.log log, entries similar to the below may be seen:
[2018-11-02T13:28:42.168-07:00 IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2018-11-02T13:28:42.523-07:00 IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
[2018-11-02T13:29:38.270-07:00 IDM Startup INFO ] [IdmServer] Starting IDM Server...
[2018-11-02T13:29:38.272-07:00 IDM Startup INFO ] [IdmServer] IDM Server has started
[2018-11-02T13:39:40.913-07:00 IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2018-11-02T13:39:40.913-07:00 IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
- In the /var/log/vmware/sso/utils/vmware-stsd.err log, entries similar to the below are seen:
Nov 02, 2018 1:29:40 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 663 ms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-slf4jimpl-
2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-
1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Nov 02, 2018 1:29:50 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 10097 ms
Service killed by signal 11
Refer to VMware KB article 60161:
https://kb.vmware.com/s/article/60161
Workaround:
To workaround this issue, remove the vCenter Server system from the Active Directory domain and add the LDAP Server as identity source. For more information, see VMware KB article 60161:
https://kb.vmware.com/s/article/60161.
- Take snapshot of PSC and VCSA.
- Remove the IWA Identity Source configuration and re-add it as AD over LDAP or AD over LDAPS
- Remove the PSC from the Active Directory domain.
- Reboot the PSC.
Note: If /storage/core partition is full, clean it up first by deleting old core dumps from /storage/core partition.
Try to log in with FQDN/username. For example: abdc.com\username