Unsolved
28 Posts
0
1379
Isilon + Cloudera + Kerberos
Hi folks,
after enabling kerberos in a Cloudera deployment the mapreduce jobs launched by users are failing. The HDFS log in Isilon displays this error:
vlab-1: 2019-06-12T19:56:19+02:00 <30.6> vlab-1 hdfs[72426]: [hdfs] RPC V9 user: vlabuser1 exception: org.apache.hadoop.security.authorize.AuthorizationException cause: Username: 'vlabuser1' not found. Make sure your client's username exists on the cluster
The client host has been configured for using Kerberos. I can list the HDFS filesystem after getting Kerberos ticket:
[vlabuser1@m02 ~]$ kinit
Password for vlabuser1@VLAB.LOCAL:
[vlabuser1@m02 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_10001
Default principal: vlabuser1@VLAB.LOCAL
Valid starting Expires Service principal
06/12/2019 21:06:40 06/13/2019 07:06:40 krbtgt/VLAB.LOCAL@VLAB.LOCAL
renew until 06/19/2019 21:06:38
[vlabuser1@m02 ~]$ hadoop fs -ls /
Found 5 items
-rw-r--r-- 3 root wheel 0 2019-06-03 16:44 /THIS_IS_ISILON_cdh.txt
drwxr-xr-x - hbase hbase 0 2019-06-03 16:48 /hbase
drwxrwxr-x - solr solr 0 2019-06-03 16:48 /solr
drwxrwxrwt - hdfs supergroup 0 2019-06-12 20:30 /tmp
drwxr-xr-x - hdfs supergroup 0 2019-06-03 22:31 /user
But I tried to launch a mapreduce job and it fails:
[vlabuser1@m02 ~]$ hadoop jar /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen 1000 /user/vlabuser1/test_teragen_1 19/06/12 20:52:15 INFO client.RMProxy: Connecting to ResourceManager at m02.vlab.local/192.168.110.102:8032 19/06/12 20:52:15 INFO hdfs.DFSClient: Created token for vlabuser1: HDFS_DELEGATION_TOKEN owner=vlabuser1@VLAB.LOCAL, renewer=yarn/m02.vlab.local@VLAB.LOCAL, realUser=, issueDate=1560365535579, maxDate=1560970335579, sequenceNumber=0, masterKeyId=0 on 192.168.110.33:8020 19/06/12 20:52:15 INFO security.TokenCache: Got dt for hdfs://hdfs.vlab.local:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.110.33:8020, Ident: (token for vlabuser1: HDFS_DELEGATION_TOKEN owner=vlabuser1@VLAB.LOCAL, renewer=yarn/m02.vlab.local@VLAB.LOCAL, realUser=, issueDate=1560365535579, maxDate=1560970335579, sequenceNumber=0, masterKeyId=0) 19/06/12 20:52:50 WARN hdfs.DFSClient: Slow waitForAckedSeqno took 34051ms (threshold=30000ms). File being written: /user/vlabuser1/.staging/job_1560193495532_0008/job.jar, block: isi_hdfs_pool:blk_4308534315_1000, Write pipeline datanodes: [DatanodeInfoWithStorage[192.168.110.32:585,isilon,DISK], DatanodeInfoWithStorage[192.168.110.31:585,isilon,DISK], DatanodeInfoWithStorage[192.168.110.33:585,isilon,DISK]] 19/06/12 20:52:51 INFO terasort.TeraGen: Generating 1000 using 2 19/06/12 20:52:51 INFO mapreduce.JobSubmitter: number of splits:2 19/06/12 20:52:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560193495532_0008 19/06/12 20:52:51 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.110.33:8020, Ident: (token for vlabuser1: HDFS_DELEGATION_TOKEN owner=vlabuser1@VLAB.LOCAL, renewer=yarn/m02.vlab.local@VLAB.LOCAL, realUser=, issueDate=1560365535579, maxDate=1560970335579, sequenceNumber=0, masterKeyId=0) 19/06/12 20:52:52 INFO impl.YarnClientImpl: Submitted application application_1560193495532_0008 19/06/12 20:52:52 INFO mapreduce.Job: The url to track the job: http://m02.vlab.local:8088/proxy/application_1560193495532_0008/ 19/06/12 20:52:52 INFO mapreduce.Job: Running job: job_1560193495532_0008 19/06/12 20:53:02 INFO mapreduce.Job: Job job_1560193495532_0008 running in uber mode : false 19/06/12 20:53:02 INFO mapreduce.Job: map 0% reduce 0% 19/06/12 20:53:02 INFO mapreduce.Job: Job job_1560193495532_0008 failed with state FAILED due to: Application application_1560193495532_0008 failed 2 times due to AM Container for appattempt_1560193495532_0008_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://m02.vlab.local:8088/proxy/application_1560193495532_0008/Then, click on links to logs of each attempt. Diagnostics: Application application_1560193495532_0008 initialization failed (exitCode=255) with output: main : command provided 0 main : run as user is vlabuser1 main : requested yarn user is vlabuser1 User vlabuser1 not found Failing this attempt. Failing the application. 19/06/12 20:53:02 INFO mapreduce.Job: Counters: 0
Any suggestion would be appreciated.
Many thanks for your help,
Jorge.