Skip to main content
Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

quanghoc

40 Posts

6341

October 25th, 2017 16:00

Isilon 8.0.0.5 Simulator Username: 'hdfs' not found with HDP 2.5.3.0

I'm deploying the 8.0.0.5 Simulator for HDP 2.5.3.0 (Ambari 2.5.2.0) and had trouble when starting YARN. The installation from Ambari went fine and HDFS service was green.

Screen Shot 2017-10-25 at 4.35.33 PM.png

Below is the log in Ambari

stderr:   /var/lib/ambari-agent/data/errors-53.txt






Traceback (most recent call last):


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 94, in 
  


    ApplicationTimelineServer().execute()


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute


    method(env)


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 44, in start


    self.configure(env) # FOR SECURITY


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 119, in locking_configure


    original_configure(obj, *args, **kw)


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 55, in configure


    yarn(name='apptimelineserver')


  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk


    return fn(*args, **kwargs)


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 356, in yarn


    mode=0755


  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__


    self.env.run()


  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run


    self.run_action(resource, action)


  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action


    provider_action()


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 604, in action_create_on_execute


    self.action_delayed("create")


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 601, in action_delayed


    self.get_hdfs_resource_executor().action_delayed(action_name, self)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 328, in action_delayed


    self._assert_valid()


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 287, in _assert_valid


    self.target_status = self._get_file_status(target)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 430, in _get_file_status


    list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 177, in run_command


    return self._run_command(*args, **kwargs)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 248, in _run_command


    raise WebHDFSCallException(err_msg, result_dict)


resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://zone1.demo.local:8082/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=401. 


{
  


  "RemoteException": {
  


    "exception": "AuthorizationException", 


    "javaClassName": "org.apache.hadoop.security.authorize.AuthorizationException", 


    "message": "Username: 'hdfs' not found. Make sure your client's username exists on the cluster"


  }


}






stdout:   /var/lib/ambari-agent/data/output-53.txt






2017-10-25 15:23:49,100 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37


2017-10-25 15:23:49,111 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf


2017-10-25 15:23:49,561 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37


2017-10-25 15:23:49,562 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf


User Group mapping (user_group) is missing in the hostLevelParams


2017-10-25 15:23:49,569 - Group['hadoop'] {}


2017-10-25 15:23:49,576 - Group['users'] {}


2017-10-25 15:23:49,577 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}


2017-10-25 15:23:49,587 - call['/var/lib/ambari-agent/tmp/changeUid.sh zookeeper'] {}


2017-10-25 15:23:49,631 - call returned (0, '512')


2017-10-25 15:23:49,632 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 512}


2017-10-25 15:23:49,633 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}


2017-10-25 15:23:49,634 - call['/var/lib/ambari-agent/tmp/changeUid.sh ams'] {}


2017-10-25 15:23:49,670 - call returned (0, '524')


2017-10-25 15:23:49,670 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 524}


2017-10-25 15:23:49,672 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users'], 'uid': None}


2017-10-25 15:23:49,673 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}


2017-10-25 15:23:49,676 - call['/var/lib/ambari-agent/tmp/changeUid.sh hdfs'] {}


2017-10-25 15:23:49,708 - call returned (0, '501')


2017-10-25 15:23:49,709 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 501}


2017-10-25 15:23:49,711 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}


2017-10-25 15:23:49,713 - call['/var/lib/ambari-agent/tmp/changeUid.sh yarn'] {}


2017-10-25 15:23:49,743 - call returned (0, '503')


2017-10-25 15:23:49,744 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 503}


2017-10-25 15:23:49,745 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}


2017-10-25 15:23:49,746 - call['/var/lib/ambari-agent/tmp/changeUid.sh mapred'] {}


2017-10-25 15:23:49,776 - call returned (0, '502')


2017-10-25 15:23:49,776 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 502}


2017-10-25 15:23:49,778 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}


2017-10-25 15:23:49,781 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}


2017-10-25 15:23:49,805 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if


2017-10-25 15:23:49,806 - Group['hdfs'] {}


2017-10-25 15:23:49,806 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}


2017-10-25 15:23:49,807 - FS Type: 


2017-10-25 15:23:49,807 - Directory['/etc/hadoop'] {'mode': 0755}


2017-10-25 15:23:49,855 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}


2017-10-25 15:23:49,855 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}


2017-10-25 15:23:49,900 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}


2017-10-25 15:23:50,014 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}


2017-10-25 15:23:50,016 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}


2017-10-25 15:23:50,016 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}


2017-10-25 15:23:50,039 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}


2017-10-25 15:23:50,052 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}


2017-10-25 15:23:50,084 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}


2017-10-25 15:23:50,130 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}


2017-10-25 15:23:50,131 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}


2017-10-25 15:23:50,132 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}


2017-10-25 15:23:50,156 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}


2017-10-25 15:23:50,172 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}


2017-10-25 15:23:50,716 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf


2017-10-25 15:23:50,717 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37


2017-10-25 15:23:50,720 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager'] {'timeout': 20}


2017-10-25 15:23:50,784 - call returned (0, 'hadoop-yarn-resourcemanager - 2.5.3.0-37')


2017-10-25 15:23:50,789 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf


2017-10-25 15:23:50,816 - Directory['/var/log/hadoop-yarn/nodemanager/recovery-state'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}


2017-10-25 15:23:50,824 - Directory['/var/run/hadoop-yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}


2017-10-25 15:23:50,825 - Directory['/var/run/hadoop-yarn/yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}


2017-10-25 15:23:50,826 - Directory['/var/log/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}


2017-10-25 15:23:50,826 - Directory['/var/run/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}


2017-10-25 15:23:50,827 - Directory['/var/run/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}


2017-10-25 15:23:50,831 - Directory['/var/log/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}


2017-10-25 15:23:50,834 - Directory['/var/log/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}


2017-10-25 15:23:50,834 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn', 'group': 'hadoop', 'ignore_failures': True, 'create_parents': True, 'cd_access': 'a'}


2017-10-25 15:23:50,837 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'fs.defaultFS': 'true'}}, 'owner': 'hdfs', 'configurations': ...}


2017-10-25 15:23:50,858 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml


2017-10-25 15:23:50,859 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}


2017-10-25 15:23:50,901 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'dfs.support.append': 'true', 'dfs.datanode.data.dir': 'true', 'dfs.namenode.http-address': 'true', 'dfs.namenode.name.dir': 'true', 'dfs.webhdfs.enabled': 'true', 'dfs.datanode.failed.volumes.tolerated': 'true'}}, 'owner': 'hdfs', 'configurations': ...}


2017-10-25 15:23:50,919 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml


2017-10-25 15:23:50,920 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}


2017-10-25 15:23:51,113 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}


2017-10-25 15:23:51,134 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml


2017-10-25 15:23:51,134 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}


2017-10-25 15:23:51,220 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 502 to yarn


2017-10-25 15:23:51,220 - XmlConfig['yarn-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}


2017-10-25 15:23:51,238 - Generating config: /usr/hdp/current/hadoop-client/conf/yarn-site.xml


2017-10-25 15:23:51,238 - File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}


2017-10-25 15:23:51,463 - XmlConfig['capacity-scheduler.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}


2017-10-25 15:23:51,477 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml


2017-10-25 15:23:51,478 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}


2017-10-25 15:23:51,500 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 501 to yarn


2017-10-25 15:23:51,501 - Directory['/hadoop/yarn/timeline'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}


2017-10-25 15:23:51,501 - Directory['/hadoop/yarn/timeline'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}


2017-10-25 15:23:51,502 - HdfsResource['/ats/done'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://zone1.demo.local:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'change_permissions_for_parents': True, 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0755}


2017-10-25 15:23:51,505 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://zone1.demo.local:8082/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpKY8eVN 2>/tmp/tmpOto4VM''] {'logoutput': None, 'quiet': False}


2017-10-25 15:23:51,634 - call returned (0, 'su: warning: cannot change directory to /home/hdfs: No such file or directory')






Command failed after 1 tries

if I access the Ambari client to su - hdfs then hdfs dfs -ls /, I got this:

17/10/25 16:18:21 WARN retry.RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over null. Not retrying because try once and fail.


org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Username: 'hdfs' not found. Make sure your client's username exists on the cluster


at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)


at org.apache.hadoop.ipc.Client.call(Client.java:1496)


at org.apache.hadoop.ipc.Client.call(Client.java:1396)


at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)


at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)


at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)


at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)


at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


at java.lang.reflect.Method.invoke(Method.java:498)


at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)


at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)


at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)


at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)


at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)


at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)


at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)


at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)


at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)


at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)


at org.apache.hadoop.fs.Globber.glob(Globber.java:265)


at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674)


at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)


at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)


at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)


at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)


at org.apache.hadoop.fs.shell.Command.run(Command.java:165)


at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)


at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)


at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)


at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)


ls: Username: 'hdfs' not found. Make sure your client's username exists on the cluster

When I do  id hdfs. I got this which seems correct:

uid=501(hdfs) gid=538(hadoop) groups=538(hadoop),501(hdfs)

Here is the output from Isilon as root (obviously the hdfs user does exists):

isilon1-1# isi auth users view hdfs --zone=zone1


                    Name: hdfs


                      DN: CN=hdfs,CN=Users,DC=ISILON1


              DNS Domain: -


                  Domain: ISILON1


                Provider: lsa-local-provider:zone1


        Sam Account Name: hdfs


                     UID: 501


                     SID: S-1-5-21-837597287-2823100475-1132252091-1038


                 Enabled: No


                 Expired: No


                  Expiry: -


                  Locked: No


                   Email: -


                   GECOS: -


           Generated GID: No


           Generated UID: No


           Generated UPN: Yes


           Primary Group


                          ID: GID:501


                        Name: hdfs


          Home Directory: /ifs/zone1/hdp/home/hdfs


        Max Password Age: 4W


        Password Expired: No


         Password Expiry: 2017-11-22T21:53:46


       Password Last Set: 2017-10-25T21:53:46


        Password Expires: Yes


                   Shell: /bin/zsh


                     UPN: hdfs@ISILON1


User Can Change Password: Yes

When I installed, before register Isilon in Ambari, I used below content for uid and groups which seem correct also:

isilon1-1# cat zone1.passwd


# use this file to add to the passwd file of your clients


hdfs:x:501:501:hadoop-svc-account:/home/hdfs:/bin/bash


mapred:x:502:502:hadoop-svc-account:/home/mapred:/bin/bash


yarn:x:503:503:hadoop-svc-account:/home/yarn:/bin/bash


hbase:x:504:504:hadoop-svc-account:/home/hbase:/bin/bash


storm:x:505:505:hadoop-svc-account:/home/storm:/bin/bash


falcon:x:506:506:hadoop-svc-account:/home/falcon:/bin/bash


tracer:x:507:507:hadoop-svc-account:/home/tracer:/bin/bash


tez:x:508:508:hadoop-svc-account:/home/tez:/bin/bash


hive:x:509:509:hadoop-svc-account:/home/hive:/bin/bash


hcat:x:510:510:hadoop-svc-account:/home/hcat:/bin/bash


oozie:x:511:511:hadoop-svc-account:/home/oozie:/bin/bash


zookeeper:x:512:512:hadoop-svc-account:/home/zookeeper:/bin/bash


ambari-qa:x:513:513:hadoop-svc-account:/home/ambari-qa:/bin/bash


flume:x:514:514:hadoop-svc-account:/home/flume:/bin/bash


hue:x:515:515:hadoop-svc-account:/home/hue:/bin/bash


accumulo:x:516:516:hadoop-svc-account:/home/accumulo:/bin/bash


hadoopqa:x:517:517:hadoop-svc-account:/home/hadoopqa:/bin/bash


sqoop:x:518:518:hadoop-svc-account:/home/sqoop:/bin/bash


spark:x:519:519:hadoop-svc-account:/home/spark:/bin/bash


mahout:x:520:520:hadoop-svc-account:/home/mahout:/bin/bash


ranger:x:521:521:hadoop-svc-account:/home/ranger:/bin/bash


kms:x:522:522:hadoop-svc-account:/home/kms:/bin/bash


atlas:x:523:523:hadoop-svc-account:/home/atlas:/bin/bash


ams:x:524:524:hadoop-svc-account:/home/ams:/bin/bash


kafka:x:525:525:hadoop-svc-account:/home/kafka:/bin/bash


zeppelin:x:526:526:hadoop-svc-account:/home/zeppelin:/bin/bash


livy:x:527:527:hadoop-svc-account:/home/livy:/bin/bash


logsearch:x:528:528:hadoop-svc-account:/home/logsearch:/bin/bash


infra-solr:x:529:529:hadoop-svc-account:/home/infra-solr:/bin/bash


activity_analyzer:x:530:530:hadoop-svc-account:/home/activity_analyzer:/bin/bash


activity_explorer:x:531:531:hadoop-svc-account:/home/activity_explorer:/bin/bash


HTTP:x:532:532:hadoop-svc-account:/home/HTTP:/bin/bash


knox:x:533:533:hadoop-svc-account:/home/knox:/bin/bash


ambari-server:x:534:534:hadoop-svc-account:/home/ambari-server:/bin/bash


druid:x:535:535:hadoop-svc-account:/home/druid:/bin/bash


anonymous:x:536:536:hadoop-svc-account:/home/anonymous:/bin/bash


admin:x:537:537:hadoop-svc-account:/home/admin:/bin/bash






isilon1-1# cat zone1.group


# use this file to add to the group file of your clients


hdfs:x:501:


mapred:x:502:


yarn:x:503:


hbase:x:504:


storm:x:505:


falcon:x:506:


tracer:x:507:


tez:x:508:


hive:x:509:


hcat:x:510:


oozie:x:511:


zookeeper:x:512:


ambari-qa:x:513:


flume:x:514:


hue:x:515:


accumulo:x:516:


sqoop:x:518:


spark:x:519:


mahout:x:520:


ranger:x:521:


kms:x:522:


atlas:x:523:


ams:x:524:


kafka:x:525:


zeppelin:x:526:


livy:x:527:


logsearch:x:528:


infra-solr:x:529:


activity_analyzer:x:530:


activity_explorer:x:531:


HTTP:x:532:


knox:x:533:


ambari-server:x:534:


druid:x:535:


anonymous:x:536:


admin:x:537:


hadoopqa:x:517: hadoop:x:538:hdfs,mapred,yarn,hbase,storm,falcon,tracer,tez,hive,hcat,oozie,zookeeper,ambari-qa,flume,hue,accumulo,hadoopqa,sqoop,spark,mahout,ranger,kms,atlas,ams,kafka,zeppelin,livy,logsearch,infra-solr,activity_analyzer,activity_explorer,HTTP,knox,ambari-server,druid,

So frankly I am not sure what caused username hdfs (as well as all other users) to be not found.

Any idea to troubleshoot?

BernieC

76 Posts

October 26th, 2017 08:00

I assume you have a SmartConnect zone set up on a pool that is tied to your zone1 access zone. And I would expect that you have configured your Hadoop cluster to connect to that SmartConnect zone. If the Hadoop cluster is for some reason connecting through the system zone this error would be expected since the user account likely doesn't exist in that zone.

Alternatively, you could increase auth and/or hdfs service logging to try to find out how the Isilon cluster is interpreting the incoming request. One of those should give details about the zone where the request was received, but may do it by zone ID which can be found with isi zone zones list -v.    

dfelt1

34 Posts

October 25th, 2017 18:00

You might want to check out the compatibility chart in the Hadoop area as OneFS 8.0.0.5 should be used with HDP 2.6.

Hadoop Info Hub: https://community.emc.com/docs/DOC-39529

Hadoop Distributions and products supported by OneFS versions: https://community.emc.com/docs/DOC-37101

quanghoc

40 Posts

October 26th, 2017 00:00

I just restored the snapshot to original and installed this stack here

Screen Shot 2017-10-26 at 12.08.42 AM.png

I still have the same problem:

Screen Shot 2017-10-26 at 12.07.35 AM.png

Traceback (most recent call last):


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 94, in 


    ApplicationTimelineServer().execute()


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute


    method(env)


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 44, in start


    self.configure(env) # FOR SECURITY


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 119, in locking_configure


    original_configure(obj, *args, **kw)


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 55, in configure


    yarn(name='apptimelineserver')


  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk


    return fn(*args, **kwargs)


  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 356, in yarn


    mode=0755


  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__


    self.env.run()


  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run


    self.run_action(resource, action)


  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action


    provider_action()


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 604, in action_create_on_execute


    self.action_delayed("create")


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 601, in action_delayed


    self.get_hdfs_resource_executor().action_delayed(action_name, self)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 328, in action_delayed


    self._assert_valid()


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 287, in _assert_valid


    self.target_status = self._get_file_status(target)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 430, in _get_file_status


    list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 177, in run_command


    return self._run_command(*args, **kwargs)


  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 248, in _run_command


    raise WebHDFSCallException(err_msg, result_dict)


resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://zone1.demo.local:8082/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=401.


{


  "RemoteException": {


    "exception": "AuthorizationException",


    "javaClassName": "org.apache.hadoop.security.authorize.AuthorizationException",


    "message": "Username: 'hdfs' not found. Make sure your client's username exists on the cluster"


  }


}

I don't know if this is a problem with the isilon_create_users.sh as it created two files: zone1.group and zone1.group.bak. The last line is different

hdfs:x:501:


mapred:x:502:


yarn:x:503:


hbase:x:504:


storm:x:505:


falcon:x:506:


tracer:x:507:


tez:x:508:


hive:x:509:


hcat:x:510:


oozie:x:511:


zookeeper:x:512:


ambari-qa:x:513:


flume:x:514:


hue:x:515:


accumulo:x:516:


hadoopqa:x:517:


sqoop:x:518:


spark:x:519:


mahout:x:520:


ranger:x:521:


kms:x:522:


atlas:x:523:


ams:x:524:


kafka:x:525:


zeppelin:x:526:


livy:x:527:


logsearch:x:528:


infra-solr:x:529:


activity_analyzer:x:530:


activity_explorer:x:531:


HTTP:x:532:


knox:x:533:


ambari-server:x:534:


druid:x:535:


anonymous:x:536:


admin:x:537:


hadoop:x:538:

Output seems good though:

isilon1-1# isi auth users view hdfs --zone=zone1


                    Name: hdfs


                      DN: CN=hdfs,CN=Users,DC=ISILON1


              DNS Domain: -


                  Domain: ISILON1


                Provider: lsa-local-provider:zone1


        Sam Account Name: hdfs


                     UID: 501


                     SID: S-1-5-21-209712790-4098821363-1044942985-1038


                 Enabled: No


                 Expired: No


                  Expiry: -


                  Locked: No


                   Email: -


                   GECOS: -


           Generated GID: No


           Generated UID: No


           Generated UPN: Yes


           Primary Group


                          ID: GID:501


                        Name: hdfs


          Home Directory: /ifs/zone1/hdp/home/hdfs


        Max Password Age: 4W


        Password Expired: No


         Password Expiry: 2017-11-23T04:29:14


       Password Last Set: 2017-10-26T04:29:14


        Password Expires: Yes


                   Shell: /bin/zsh


                     UPN: hdfs@ISILON1


User Can Change Password: Yes

Please share me some more idea.

bergec

275 Posts

October 26th, 2017 00:00

Hello

Could it be because your hdfs user is not enabled?

Regards, Claude

quanghoc

40 Posts

October 26th, 2017 07:00

I am not sure if I have to enable hdfs user. I have enabled and licensed HDFS. I also just tried to enable users in the GUI and still same problem:

Screen Shot 2017-10-26 at 8.08.38 AM.png

Here are my CLIs in the Isilon

isi zone zones create --name=zone1 --path=/ifs/zone1/hdp --create-path


mkdir -p /ifs/zone1/hdp/hadoop


isi network pools create --id=groupnet0:subnet0:hadoop-pool-hdp --ranges=192.168.94.17-192.168.94.21 --access-zone=zone1 --alloc-method=dynamic --ifaces=1:ext-1  --sc-subnet=subnet0 --sc-dns-zone=zone1.demo.local --description=hadoop


isi hdfs settings modify --zone=zone1 --root-directory=/ifs/zone1/hdp/hadoop


isi hdfs settings modify --zone=zone1 --ambari-namenode=zone1.demo.local


isi hdfs settings modify --zone=zone1 --ambari-server=bt.demo.local


touch /ifs/zone1/hdp/hadoop/THIS_IS_ISILON-hdp.txt


isi auth settings acls modify --group-owner-inheritance=parent


cd /ifs/scripts/isilon


wget https://raw.githubusercontent.com/Isilon/isilon_hadoop_tools/master/isilon_create_users.sh --no-check-certificate


wget https://raw.githubusercontent.com/Isilon/isilon_hadoop_tools/master/isilon_create_directories.sh --no-check-certificate


chmod u+x *


bash isilon_create_users.sh --dist hwx --startgid 501 --startuid 501 --zone zone1


isi zone zones modify --user-mapping-rules="hdfs=>root" --zone=zone1


bash isilon_create_directories.sh --dist hwx --fixperm --zone zone1

Below are what in /var/log/hdfs.log

2017-10-24T20:23:32Z <30.6> isilon1-1 hdfs[12760]: [hdfs] WebHDFS exception: org.apache.hadoop.securi


ty.authorize.AuthorizationException cause: Username: 'hdfs' not found. Make sure your client's userna


me exists on the cluster


2017-10-24T20:32:15Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hdfs exception: org.apache.ha


doop.security.authorize.AuthorizationException cause: Username: 'hdfs' not found. Make sure your clie


nt's username exists on the cluster


2017-10-24T20:53:10Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hive exception: org.apache.ha


doop.security.authorize.AuthorizationException cause: Username: 'hive' not found. Make sure your clie


nt's username exists on the cluster


2017-10-24T21:12:14Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hdfs exception: org.apache.ha


doop.security.authorize.AuthorizationException cause: Username: 'hdfs' not found. Make sure your clie


nt's username exists on the cluster


2017-10-24T21:34:12Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hdfs exception: org.apache.ha


doop.security.authorize.AuthorizationException cause: Username: 'hdfs' not found. Make sure your clie


nt's username exists on the cluster


2017-10-24T21:39:06Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hdfs exception: org.apache.ha


doop.security.authorize.AuthorizationException cause: Username: 'hdfs' not found. Make sure your clie


nt's username exists on the cluster


2017-10-24T21:43:55Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hive exception: org.apache.ha


doop.security.authorize.AuthorizationException cause: Username: 'hive' not found. Make sure your clie


nt's username exists on the cluster


2017-10-24T21:47:55Z <30.6> isilon1-1 hdfs[12760]: [hdfs] WebHDFS exception: org.apache.hadoop.securi


ty.authorize.AuthorizationException cause: Username: 'hdfs' not found. Make sure your client's userna


me exists on the cluster


2017-10-24T21:48:08Z <30.6> isilon1-1 hdfs[12760]: [hdfs] Ambari: Sending the heartbeat to the Ambari


 server for zone 2 failed. Agent will reset and restart


2017-10-24T21:48:35Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hdfs exception: org.apache.ha


doop.security.authorize.AuthorizationException cause: Username: 'hdfs' not found. Make sure your clie


nt's username exists on the cluster


2017-10-24T21:55:02Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hduser1 exception: org.apache


.hadoop.security.authorize.AuthorizationException cause: Username: 'hduser1' not found. Make sure you


r client's username exists on the cluster


2017-10-24T22:00:26Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user: hduser2 exception: org.apache


.hadoop.security.authorize.AuthorizationException cause: Username: 'hduser2' not found. Make sure you


r client's username exists on the cluster


2017-10-24T22:10:26Z <30.6> isilon1-1 hdfs[12760]: [hdfs] RPC V9  user:  exception: org.apache.hadoop


.security.authorize.AuthorizationException cause: Unknown protocol: org.apache.hadoop.tools.GetUserMa


ppingsProtocol


2017-10-24T22:22:49Z <30.6> isilon1-1 hdfs[12760]: [hdfs] Ambari: Sending the heartbeat to the Ambari


 server for zone 2 failed. Agent will reset and restart


2017-10-24T22:23:49Z <30.4> isilon1-1 hdfs[12760]: [hdfs] Ambari: Agent for zone 2 could not connect


to its Ambari server. Agent will attempt to connect again later.


2017-10-24T22:24:49Z <30.4> isilon1-1 hdfs[12760]: [hdfs] Ambari: Agent for zone 2 could not connect


to its Ambari server. Agent will attempt to connect again later.


2017-10-24T22:25:49Z <30.4> isilon1-1 hdfs[12760]: [hdfs] Ambari: Agent for zone 2 could not connect


to its Ambari server. Agent will attempt to connect again later.


2017-10-24T22:26:49Z <30.4> isilon1-1 hdfs[12760]: [hdfs] Ambari: Agent for zone 2 could not connect


to its Ambari server. Agent will attempt to connect again later.


2017-10-24T22:27:49Z <30.4> isilon1-1 hdfs[12760]: [hdfs] Ambari: Agent for zone 2 could not connect

BernieC

76 Posts

October 26th, 2017 08:00

BTW, the hdfs user in my environment is also disabled and requests work fine with it against both Kerberized and non-Kerberized Hadoop clusters.    

quanghoc

40 Posts

October 31st, 2017 10:00

Bernie Case Thanks for the hint as I found out that in fact my Hortonworks connecting through System Zone due to wrong DNS IP. Phew...

But just a side question: how do I "see" or troubleshoot a request into Isilon in term of how Isilon received it? Is there a way to do that or watch traffic? etc...

BernieC

76 Posts

October 31st, 2017 11:00

Hi there Hoc Phan,

Couple of ways to go about this. Every node has /var/log/hdfs.log. If you change the log level to verbose (isi hdfs log-level modify --set=verbose and don't forget to set it back to warning when done) you will see more descriptive information when connects are made, but it won't always include IPs.

I find that I rely on the service logs from the Hadoop clients to see what IPs they were connecting to or having trouble with before I go and turn up logging on Isilon. In addition, I use isi statistics to look at incoming active connections to see what they're doing. For example:

isi statistics client list --protocols=hdfs --format=top --long

Would that work, or are you looking for something more?

quanghoc

40 Posts

November 1st, 2017 11:00

Bernie Case This is great. Thank you!

View More

View All

No Events found!

Top