Highlighted
fjfjfan
1 Copper

Hive/HBase and some other components service check failed after move from HDFS to ECS

Hello, I am trying the integration of BigInsights(use Ambari) and ECS now, the guide I used to set up the integration env is https://www.emc.com/collateral/TechnicalDocument/docu79368.pdf, page 135 - 171, especially page 147-152 and 164-171

After that, the HDFS/MapReduce Service check pass,

but

1. Hive Service check ran by user ambari-qa failed with error:

<-------

2017-03-03 01:06:57,169 ERROR [HiveServer2-Handler-Pool: Thread-43]: vipr.ViPRFileSystemClientBase (ViPRFileSystemClientBase.java:checkResponse(967)) - Permissions failure for request: User: anonymous (auth:SIMPLE), host: bucket1.ns1.Site1, namespace: ns1, bucket: bucket1

2017-03-03 01:06:57,171 ERROR [HiveServer2-Handler-Pool: Thread-43]: vipr.ViPRFileSystemClientBase (ViPRFileSystemClientBase.java:checkResponse(969)) - Request message sent: MkDirRequestMessage[kind=MKDIR_REQUEST,namespace=ns1,bucket=bucket1,path=/tmp/hive/anonymous,hdfsTrustedStatus=HDFS_USER_NOT_TRUSTED,permissions=rwx------,createParent=true]

2017-03-03 01:06:57,172 WARN  [HiveServer2-Handler-Pool: Thread-43]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(311)) - Error opening session:

org.apache.hive.service.cli.HiveSQLException: Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.security.AccessControlException: ERROR_INSUFFICIENT_PERMISSIONS

        at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:266)

        at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:204)

--------->

even after I add "ambari-qa" as the Object owner and add it to the User ACLs of the bucket.

I also tried to modify the fs.viprfs.auth.anonymous_translation from LOCAL_USER to NONE, but did not work.

Also tried to add ambari-qa as the local user of ECS and the host of docker container which runs ECS, still did not work.

Anybody know why ECS treats ambari-qa as anonymous?

2. HBase Service check failed with error:

<-------

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2330)

at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2335)

at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2544)

at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1536)

at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:471)

at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

at java.lang.Thread.run(Thread.java:745)

--------->

I searched and seems it related to there is something wrong with the hbase.rootdir which I set to viprfs://bucket1.ns1.Site1/apps/hbase/data, any other operations I missed except modifying the hbase.rootdir?


3. Oozie Service Check failed with "Error: E0904 : E0904: Scheme [viprfs] not supported in uri [viprfs://bucket1.ns1.Site1/user/ambari-qa/examples/apps/no-op] Invalid sub-command: Missing argument for option: info"


4. Slider Service Check failed with Exception: java.lang.ClassNotFoundException: Class com.emc.hadoop.fs.vipr.ViPRFileSystem not found


5. Titan Service Check failed with Exception: "Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.emc.hadoop.fs.vipr.ViPRFileSystem not found"


For 3#, does ECS support Oozie?

For 4# 5#, does ECS support Slider/Titan, if so, seems I also need to add hdfsclient jar of ECS to their classpath, where to do that?


Thanks!

Tags (3)
0 Kudos
4 Replies
coneryj
1 Copper

Re: Hive/HBase and some other components service check failed after move from HDFS to ECS

I'm not sure, but what about your core-site.xml file? Could you attach that?

most specifically the fs.viprfs.auth.anonymous_translation property

0 Kudos
fjfjfan
1 Copper

Re: Hive/HBase and some other components service check failed after move from HDFS to ECS

coneryj  Thanks for you reply, the core-site.xml file:

  <configuration>

   

    <property>

      <name>fs.AbstractFileSystem.viprfs.impl</name>

      <value>com.emc.hadoop.fs.vipr.ViPRAbstractFileSystem</value>

    </property>

   

    <property>

      <name>fs.defaultFS</name>

      <value>viprfs://bucket1.ns1.Site1/</value>

      <final>true</final>

    </property>

   

    <property>

      <name>fs.permissions.umask-mode</name>

      <value>022</value>

    </property>

   

    <property>

      <name>fs.trash.interval</name>

      <value>360</value>

    </property>

   

    <property>

      <name>fs.vipr.installation.Site1.hosts</name>

      <value>9.30.104.219</value>

    </property>

   

    <property>

      <name>fs.vipr.installation.Site1.resolution</name>

      <value>dynamic</value>

    </property>

   

    <property>

      <name>fs.vipr.installation.Site1.resolution.dynamic.time_to_live_ms</name>

      <value>900000</value>

    </property>

   

    <property>

      <name>fs.vipr.installations</name>

      <value>Site1</value>

    </property>

   

    <property>

      <name>fs.viprfs.auth.anonymous_translation</name>

      <value>LOCAL_USER</value>

    </property>

   

    <property>

      <name>fs.viprfs.auth.identity_translation</name>

      <value>NONE</value>

    </property>

   

    <property>

      <name>fs.viprfs.impl</name>

      <value>com.emc.hadoop.fs.vipr.ViPRFileSystem</value>

    </property>

   

    <property>

      <name>ha.failover-controller.active-standby-elector.zk.op.retries</name>

      <value>120</value>

    </property>

   

    <property>

      <name>hadoop.http.authentication.simple.anonymous.allowed</name>

      <value>true</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.bigsql.groups</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.bigsql.hosts</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.hcat.groups</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.hcat.hosts</name>

      <value>bigaperf194.svl.ibm.com</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.hdfs.groups</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.hdfs.hosts</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.hive.groups</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.hive.hosts</name>

      <value>bigaperf194.svl.ibm.com</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.oozie.groups</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.oozie.hosts</name>

      <value>bigaperf194.svl.ibm.com</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.root.groups</name>

      <value>*</value>

    </property>

   

    <property>

      <name>hadoop.proxyuser.root.hosts</name>

      <value>bigaperf193.svl.ibm.com</value>

    </property>

   

    <property>

      <name>hadoop.security.auth_to_local</name>

      <value>DEFAULT</value>

    </property>

   

    <property>

      <name>hadoop.security.authentication</name>

      <value>simple</value>

    </property>

   

    <property>

      <name>hadoop.security.authorization</name>

      <value>false</value>

    </property>

   

    <property>

      <name>io.compression.codecs</name>

      <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value>

    </property>

   

    <property>

      <name>io.file.buffer.size</name>

      <value>131072</value>

    </property>

   

    <property>

      <name>io.serializations</name>

      <value>org.apache.hadoop.io.serializer.WritableSerialization</value>

    </property>

   

    <property>

      <name>ipc.client.connect.max.retries</name>

      <value>50</value>

    </property>

   

    <property>

      <name>ipc.client.connection.maxidletime</name>

      <value>30000</value>

    </property>

   

    <property>

      <name>ipc.client.idlethreshold</name>

      <value>8000</value>

    </property>

   

    <property>

      <name>ipc.server.tcpnodelay</name>

      <value>true</value>

    </property>

   

    <property>

      <name>mapreduce.jobtracker.webinterface.trusted</name>

      <value>false</value>

    </property>

   

    <property>

      <name>net.topology.script.file.name</name>

      <value>/etc/hadoop/conf/topology_script.py</value>

    </property>

   

  </configuration>

0 Kudos
claudio_fahey
1 Copper

Re: Re: Hive/HBase and some other components service check failed after move from HDFS to ECS

To further investigate the Hive error, you will need to review the error logs on each of the ECS nodes. Run the following command on any ECS node immediately after the failure to gather the logs.

admin@ecs-1:~> viprexec -i -c "tail -100 /opt/storageos/logs/dataheadsvc-error.log" > /tmp/error.log


For HBase, you will need to delete the HBase metadata in Zookeeper.

hbase zkcli

[zk] rmr /hbase-unsecure

Oozie, Slider, and Titan are not supported with ECS.

fjfjfan
1 Copper

Re: Hive/HBase and some other components service check failed after move from HDFS to ECS

Hello Claudio,

Thanks for your reply, the HBase suggestion does resolve the HBase problem. And also the info "Oozie, Slider, and Titan are not supported with ECS" is important for us.

And for Hive, the error log on ECS node is

bigaperf197:/opt/storageos/tools # vi /opt/storageos/logs/dataheadsvc-error.log

2017-03-30T21:38:07,653 [pool-90-thread-14587-091e68db:15a88fc4efd:a6a6:38] ERROR  FileSystemAc

cessHelper.java (line 1939) nfsProcessOperation : for method nfsCreateDirectory failed to proce

ss path : tmp/hive/anonymous

2017-03-30T21:38:07,654 [pool-90-thread-14587] ERROR  RealBlobEngine.java (line 357) Error crea

ting directory: 'tmp/hive/anonymous'. '[code=ERROR_ACCESS_DENIED, message='ERROR_ACCESS_DENIED'

]'

Seems it's back to my original question that:

even after I add "ambari-qa" as the Object owner and add it to the User ACLs of the bucket.

tried to modify the fs.viprfs.auth.anonymous_translation from LOCAL_USER to NONE,

Also tried to add ambari-qa as the local user of ECS and the host of docker container which runs ECS, still did not work.


Why ECS treats ambari-qa as anonymous? any suggestion? Thanks in advance!

0 Kudos