Could you see if there are any error or failure messages in /opt/storageos/authsvc.log - and if so, post them here?

That may give us a better idea of where things are failing in the authentication process.

I believe these may be of use (I can provide the full log if necessary) - this error appears when the logon request fails:

015-09-03 19:17:17,988 [ProviderConfigUpdater]  INFO  CustomAuthenticationManager.java (line 509) Starting authn provider config reload, lastNotificationTime = 1441304836797, lastReloadTime = 1441304836952

2015-09-03 19:17:17,988 [ProviderConfigUpdater]  INFO  CustomAuthenticationManager.java (line 442) Reading authentication providers from the database

2015-09-03 19:17:17,988 [ProviderConfigUpdater]  INFO  TCPClient.java (line 65) request ac1f22fe:14f947646a2:3:5 is being routed to data node localhost

2015-09-03 19:17:18,128 [ProviderConfigUpdater] ERROR  CustomAuthenticationManager.java (line 450) Failed to read authentication providers from database, {}

com.emc.storageos.objcontrol.object.exception.ObjectAccessException: exception while listing auth providers

  at com.emc.storageos.objcontrol.object.exception.ObjectAccessException.createExceptionForAPI(ObjectAccessException.java:232)

  at com.emc.storageos.data.object.ipc.protocol.impl.ObjectAccessExceptionParser.parseFrom(ObjectAccessExceptionParser.java:61)

  at com.emc.storageos.data.object.ipc.protocol.impl.ObjectAccessExceptionParser.parseFrom(ObjectAccessExceptionParser.java:28)

  at com.emc.storageos.data.object.ipc.protocol.DefaultExceptionHandler.parseFrom(DefaultExceptionHandler.java:51)

  at com.emc.storageos.ipc.ClientLib.client.impl.AbstractBasicAPIClient.sendRequestAndWait(AbstractBasicAPIClient.java:48)

  at com.emc.storageos.ipc.ClientLib.auth.impl.AuthProviderAPIImpl.sendRequest(AuthProviderAPIImpl.java:130)

  at com.emc.storageos.ipc.ClientLib.auth.impl.AuthProviderAPIImpl.getAuthProviders(AuthProviderAPIImpl.java:118)

  at com.emc.storageos.ipc.ClientLib.auth.impl.AuthProviderAPIImpl.list(AuthProviderAPIImpl.java:105)

  at com.emc.storageos.auth.impl.CustomAuthenticationManager$ProviderConfigUpdater.getProvidersFromDB(CustomAuthenticationManager.java:444)

  at com.emc.storageos.auth.impl.CustomAuthenticationManager$ProviderConfigUpdater.run(CustomAuthenticationManager.java:513)

  at java.lang.Thread.run(Thread.java:745)

And the Docker Logs:

09032015 18:26:15 : ############################### boot operation begin ############################

09032015 18:26:15 : exporting environment variable VIPR_DEPLOYMENT_TYPE to fabric

Starting syslog services..done

Launching container fiendly cron.

Launching container fiendly cron.

Launching container fiendly cron.

rm: cannot remove `/opt/storageos/logs': Device or resource busy

09032015 18:26:16 : Parsing /host/data/network.json

09032015 18:26:16 : VERSION: 1.2.0.0.60101.42a66cf private iface: eth0 public iface: eth0 START VIPR:

09032015 18:26:16 : StorageServer JSON configuration file is /data/partitions.json

09032015 18:26:16 : expecting data partitions to be mounted under /disks

09032015 18:26:16 : setting partitionroot=/disks in /opt/storageos/conf/storageserver.conf

sed: can't read /opt/storageos/conf/storageserver.conf: No such file or directory

09032015 18:26:16 : partitions.json already exists - will not generate it

09032015 18:26:16 : validating config.iso and fabric version file

09032015 18:26:16 : config.iso is missing in the /data location

09032015 18:26:16 : fetching network information

09032015 18:26:16 : IP address of the machine: 172.31.34.254

09032015 18:26:16 : NETMASK found: 255.255.240.0

09032015 18:26:16 : Created /opt/storageos/conf/network

09032015 18:26:16 : IPADDR='172.31.34.254'

09032015 18:26:16 : All validations are complete. Going ahead with the configuration steps.

09032015 18:26:16 : populating vipr shared library path

09032015 18:26:16 : creating scsi device nodes

09032015 18:26:21 : creating volumes skeleton containers

09032015 18:26:21 : populating vipr network info in fabric identifier

09032015 18:26:21 : fixing svcuser account permissions

09032015 18:26:21 : updating storageos property to datanode

09032015 18:26:21 : fixing file permissions

09032015 18:26:21 : fixing storageos and data ownership

09032015 18:26:24 : creating place holders file

09032015 18:26:24 : preparing vipr configuration information

cp: cannot stat `/data/config.iso': No such file or directory

09032015 18:26:24 : registering supporting services

insserv: script storageos-dataservice-standalone: service storageos-dataservice already provided!

boot.ovfenv               0:off  1:off  2:off  3:off  4:off  5:off  6:off  B:on

insserv: script storageos-dataservice-standalone: service storageos-dataservice already provided!

connectemc                0:off  1:off  2:off  3:on   4:off  5:on   6:off

insserv: script storageos-dataservice-standalone: service storageos-dataservice already provided!

nginx                     0:off  1:off  2:off  3:on   4:off  5:on   6:off

insserv: FATAL: service boot.clock has to be enabled to use service storageos-dataservice

insserv: script storageos-dataservice-standalone: service storageos-dataservice already provided!

insserv: exiting now!

/sbin/insserv failed, exit code 1

storageos-dataservice     0:off  1:off  2:off  3:off  4:off  5:off  6:off

09032015 18:26:25 : starting boot.ovfenv service

Initializing configuration properties

..done

Generating configuration files

/etc/systool: Warning: Fabric deployment. Skipping _bootfs_mount_ro()

..failed

/etc/init.d/storageos-dataservice: line 98: /proc/sys/kernel/core_pattern: Read-only file system

Setting up SSL certificates  .  .  .  ...done

Starting ViPR servicesgrep: /proc/scsi/sg/device_strs: No such file or directory

..done

Starting nginx service

..done

Starting ConnectEMC service..done

I am also seeing this issue in our setup. I have a 4 node ECS CE docker environment running. If I build it fresh, everything works until the weekend. Then after the weekend, I cannot login to the admin interface (using root) anymore and Cyberduck cannot login as a user to the S3 space. Here is the authsvc.log entries relevant to the auth efforts.

----------authsvc.log snip start---------------

2015-09-08 16:25:35,982 [ProviderConfigUpdater]  INFO  CustomAuthenticationManager.java (line 509) Starting authn provider config reload, lastNotificationTime = 1441392728487, lastReloadTime = 1441392728489

2015-09-08 16:25:35,982 [ProviderConfigUpdater]  INFO  CustomAuthenticationManager.java (line 442) Reading authentication providers from the database

2015-09-08 16:25:35,982 [ProviderConfigUpdater]  INFO  TCPClient.java (line 65) request 0a71033d:14f99b332bf:7:231 is being routed to data node localhost

2015-09-08 16:25:36,131 [ProviderConfigUpdater] ERROR  CustomAuthenticationManager.java (line 450) Failed to read authentication providers from database, {}

com.emc.storageos.objcontrol.object.exception.ObjectAccessException: exception while listing auth providers

        at com.emc.storageos.objcontrol.object.exception.ObjectAccessException.createExceptionForAPI(ObjectAccessException.java:232)

        at com.emc.storageos.data.object.ipc.protocol.impl.ObjectAccessExceptionParser.parseFrom(ObjectAccessExceptionParser.java:61)

        at com.emc.storageos.data.object.ipc.protocol.impl.ObjectAccessExceptionParser.parseFrom(ObjectAccessExceptionParser.java:28)

        at com.emc.storageos.data.object.ipc.protocol.DefaultExceptionHandler.parseFrom(DefaultExceptionHandler.java:51)

        at com.emc.storageos.ipc.ClientLib.client.impl.AbstractBasicAPIClient.sendRequestAndWait(AbstractBasicAPIClient.java:48)

        at com.emc.storageos.ipc.ClientLib.auth.impl.AuthProviderAPIImpl.sendRequest(AuthProviderAPIImpl.java:130)

        at com.emc.storageos.ipc.ClientLib.auth.impl.AuthProviderAPIImpl.getAuthProviders(AuthProviderAPIImpl.java:118)

        at com.emc.storageos.ipc.ClientLib.auth.impl.AuthProviderAPIImpl.list(AuthProviderAPIImpl.java:105)

        at com.emc.storageos.auth.impl.CustomAuthenticationManager$ProviderConfigUpdater.getProvidersFromDB(CustomAuthenticationManager.java:444)

        at com.emc.storageos.auth.impl.CustomAuthenticationManager$ProviderConfigUpdater.run(CustomAuthenticationManager.java:513)

        at java.lang.Thread.run(Thread.java:745)

----------authsvc.log snip stop---------------

Followed shortly by...

----------authsvc.log snip start---------------

2015-09-08 16:26:23,439 [qtp813694535-209021] ERROR  CustomAuthenticationManager.java (line 151) could not write audit event for Login

com.emc.storageos.objcontrol.object.exception.ObjectAccessException: Hit an exception while creating audit

        at com.emc.storageos.objcontrol.object.exception.ObjectAccessException.createExceptionForAPI(ObjectAccessException.java:232)

        at com.emc.storageos.data.object.ipc.protocol.impl.ObjectAccessExceptionParser.parseFrom(ObjectAccessExceptionParser.java:61)

        at com.emc.storageos.data.object.ipc.protocol.impl.ObjectAccessExceptionParser.parseFrom(ObjectAccessExceptionParser.java:28)

        at com.emc.storageos.data.object.ipc.protocol.DefaultExceptionHandler.parseFrom(DefaultExceptionHandler.java:51)

        at com.emc.storageos.ipc.ClientLib.client.impl.AbstractBasicAPIClient.sendRequestAndWait(AbstractBasicAPIClient.java:48)

        at com.emc.storageos.ipc.ClientLib.monitoring.impl.MonitoringAPIImpl.sendRequest(MonitoringAPIImpl.java:157)

        at com.emc.storageos.ipc.ClientLib.monitoring.impl.MonitoringAPIImpl.createAudit(MonitoringAPIImpl.java:107)

        at com.emc.storageos.auth.impl.CustomAuthenticationManager.CreateAudit(CustomAuthenticationManager.java:146)

        at com.emc.storageos.auth.impl.CustomAuthenticationManager.authenticate(CustomAuthenticationManager.java:124)

        at com.emc.storageos.auth.service.impl.resource.AuthenticationResource.authenticateUser(AuthenticationResource.java:582)

        at com.emc.storageos.auth.service.impl.resource.AuthenticationResource.tryLogin(AuthenticationResource.java:558)

        at com.emc.storageos.auth.service.impl.resource.AuthenticationResource.getLoginToken(AuthenticationResource.java:254)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)

        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1483)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1414)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1363)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1353)

        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:414)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:288)

        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:115)

        at com.emc.storageos.data.security.authentication.AbstractAuthenticationFilter.forwardToService(AbstractAuthenticationFilter.java:50)

        at com.emc.storageos.auth.impl.CustomTokenBasedAuthenticationFilter.doFilter(CustomTokenBasedAuthenticationFilter.java:109)

        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)

        at com.emc.storageos.data.security.authentication.AbstractAuthenticationFilter.doFilter(AbstractAuthenticationFilter.java:84)

        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)

        at com.emc.storageos.data.security.audit.RequestAuditFilter.doFilter(RequestAuditFilter.java:106)

        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)

        at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:82)

        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)

        at org.eclipse.jetty.server.Server.handle(Server.java:348)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)

        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)

        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)

        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)

        at java.lang.Thread.run(Thread.java:745)

2015-09-08 16:26:23,446 [qtp813694535-209021]  INFO  TCPClient.java (line 65) request 0a71033d:14f99b332bf:10:2 is being routed to data node localhost

----------authsvc.log snip stop---------------

And then finally...

----------authsvc.log snip start---------------

2015-09-08 16:27:23,625 [qtp813694535-209021] ERROR  AuthTokenManager.java (line 266) Failed to get user namespace...

2015-09-08 16:27:23,626 [qtp813694535-209021]  INFO  TCPClient.java (line 65) request 0a71033d:14f99b332bf:10:3 is being routed to data node localhost

2015-09-08 16:27:23,779 [qtp813694535-209021]  INFO  TCPClient.java (line 65) request 0a71033d:14f99b332bf:10:4 is being routed to data node localhost

2015-09-08 16:27:23,946 [qtp813694535-209021]  INFO  TCPClient.java (line 65) request 0a71033d:14f99b332bf:10:5 is being routed to data node localhost

2015-09-08 16:27:24,187 [qtp813694535-209021]  INFO  TCPClient.java (line 65) request 0a71033d:14f99b332bf:10:6 is being routed to data node localhost

2015-09-08 16:27:24,342 [qtp813694535-209021]  INFO  TCPClient.java (line 65) request 0a71033d:14f99b332bf:10:7 is being routed to data node localhost

2015-09-08 16:27:24,686 [qtp813694535-209021]  INFO  RequestStatTracker.java (line 265) Request response time: 301864 ms

2015-09-08 16:27:24,686 [qtp813694535-209021]  WARN  AlertsLogger.java (line 80) Request took an unusually long time to complete: 301,864ms

2015-09-08 16:27:24,687 [qtp813694535-209021]  INFO  RequestAuditFilter.java (line 123) Response headers: HTTP/1.1 200

X-SDS-AUTH-TOKEN:  ** masked **

Content-Type: application/xml

----------authsvc.log snip stop---------------

I am also seeing the same issue in our ECS single node. I'd tried to reboot the container docker and linux reboot, but now we can't do login as root.

Has anyone been able to fix it already?

We're continuing to update our installation scripts in order to iron out various issues in the process. In regards to the specific issue resulting in the message "ERROR  CustomAuthenticationManager.java (line 450) Failed to read authentication providers from database, {}" within the authsvc.log, we've updated some of the configuration that happens as part of the step1 script.

Please pull the latest version of the installation scripts from git and rebuild your docker image to resolve this issue (as of 9/29, you may need to checkout the "develop" branch using git to receive the latest version). If you've previously installed an ECSCE docker image on the host system, in order to remove some of the existing data in vnest-main that contributes to the authentication failure, you may need to run the following command on the host system after restarting:

rm -rf /data/vnest/vnest-main/*

Hans,

Since the step1 script requires an unpartitioned data disk, after removing the old docker image you need to restore your data disk to an unpartitioned state so that it can be re-used for a new installation. One of the easiest ways to do this is using the "fdisk" command.

Unfortunately, login failure messages at the GUI aren't very descriptive at the moment - is the issue specific to the GUI, or can you not successfully log in at all, such as with:

curl -i -k https://XXX.XXX.XXX.XXX:4443/login -u root:ChangeMe

In either case, it may help to take a look at the logs as suggested in above posts, as that's currently our most reliable method of diagnosing issues with the CE.

- Aaron

You're missing a '-u' in your command - at the moment, you're treating "root" as a hostname.

If that fails, as well, you may wish to look at /opt/storageos/logs/authsvc.log for any errors in your authentication service and, if you're on a single-node install, attempt the following to check the health of your directory tables:

curl http://10.249.237.144:9101/stats/dt/DTInitStat/

Hans, you might fork to git clone -b develop --single-branch https://github.com/bottkars/ECS-CommunityEdition.git

Wich has the latest develop build with my commit to get rid of that error.

It is currently not merged, but the changes that are in there are important to build the singlenode !

I have not looked into the login issue, but might e some cert issue I have also seen in coprhd.