This post is more than 5 years old
40 Posts
0
6341
Isilon 8.0.0.5 Simulator Username: 'hdfs' not found with HDP 2.5.3.0
I'm deploying the 8.0.0.5 Simulator for HDP 2.5.3.0 (Ambari 2.5.2.0) and had trouble when starting YARN. The installation from Ambari went fine and HDFS service was green.
Below is the log in Ambari
stderr: /var/lib/ambari-agent/data/errors-53.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 94, in
ApplicationTimelineServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 44, in start
self.configure(env) # FOR SECURITY
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 119, in locking_configure
original_configure(obj, *args, **kw)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 55, in configure
yarn(name='apptimelineserver')
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 356, in yarn
mode=0755
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 604, in action_create_on_execute
self.action_delayed("create")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 601, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 328, in action_delayed
self._assert_valid()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 287, in _assert_valid
self.target_status = self._get_file_status(target)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 430, in _get_file_status
list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 177, in run_command
return self._run_command(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 248, in _run_command
raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://zone1.demo.local:8082/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=401.
{
"RemoteException": {
"exception": "AuthorizationException",
"javaClassName": "org.apache.hadoop.security.authorize.AuthorizationException",
"message": "Username: 'hdfs' not found. Make sure your client's username exists on the cluster"
}
}
stdout: /var/lib/ambari-agent/data/output-53.txt
2017-10-25 15:23:49,100 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37
2017-10-25 15:23:49,111 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-10-25 15:23:49,561 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37
2017-10-25 15:23:49,562 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2017-10-25 15:23:49,569 - Group['hadoop'] {}
2017-10-25 15:23:49,576 - Group['users'] {}
2017-10-25 15:23:49,577 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-10-25 15:23:49,587 - call['/var/lib/ambari-agent/tmp/changeUid.sh zookeeper'] {}
2017-10-25 15:23:49,631 - call returned (0, '512')
2017-10-25 15:23:49,632 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 512}
2017-10-25 15:23:49,633 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-10-25 15:23:49,634 - call['/var/lib/ambari-agent/tmp/changeUid.sh ams'] {}
2017-10-25 15:23:49,670 - call returned (0, '524')
2017-10-25 15:23:49,670 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 524}
2017-10-25 15:23:49,672 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users'], 'uid': None}
2017-10-25 15:23:49,673 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-10-25 15:23:49,676 - call['/var/lib/ambari-agent/tmp/changeUid.sh hdfs'] {}
2017-10-25 15:23:49,708 - call returned (0, '501')
2017-10-25 15:23:49,709 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 501}
2017-10-25 15:23:49,711 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-10-25 15:23:49,713 - call['/var/lib/ambari-agent/tmp/changeUid.sh yarn'] {}
2017-10-25 15:23:49,743 - call returned (0, '503')
2017-10-25 15:23:49,744 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 503}
2017-10-25 15:23:49,745 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-10-25 15:23:49,746 - call['/var/lib/ambari-agent/tmp/changeUid.sh mapred'] {}
2017-10-25 15:23:49,776 - call returned (0, '502')
2017-10-25 15:23:49,776 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': 502}
2017-10-25 15:23:49,778 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-10-25 15:23:49,781 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-10-25 15:23:49,805 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2017-10-25 15:23:49,806 - Group['hdfs'] {}
2017-10-25 15:23:49,806 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2017-10-25 15:23:49,807 - FS Type:
2017-10-25 15:23:49,807 - Directory['/etc/hadoop'] {'mode': 0755}
2017-10-25 15:23:49,855 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2017-10-25 15:23:49,855 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2017-10-25 15:23:49,900 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2017-10-25 15:23:50,014 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2017-10-25 15:23:50,016 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2017-10-25 15:23:50,016 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2017-10-25 15:23:50,039 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2017-10-25 15:23:50,052 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2017-10-25 15:23:50,084 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-10-25 15:23:50,130 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2017-10-25 15:23:50,131 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2017-10-25 15:23:50,132 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2017-10-25 15:23:50,156 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2017-10-25 15:23:50,172 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2017-10-25 15:23:50,716 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-10-25 15:23:50,717 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37
2017-10-25 15:23:50,720 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager'] {'timeout': 20}
2017-10-25 15:23:50,784 - call returned (0, 'hadoop-yarn-resourcemanager - 2.5.3.0-37')
2017-10-25 15:23:50,789 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-10-25 15:23:50,816 - Directory['/var/log/hadoop-yarn/nodemanager/recovery-state'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2017-10-25 15:23:50,824 - Directory['/var/run/hadoop-yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-10-25 15:23:50,825 - Directory['/var/run/hadoop-yarn/yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-10-25 15:23:50,826 - Directory['/var/log/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2017-10-25 15:23:50,826 - Directory['/var/run/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-10-25 15:23:50,827 - Directory['/var/run/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-10-25 15:23:50,831 - Directory['/var/log/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2017-10-25 15:23:50,834 - Directory['/var/log/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2017-10-25 15:23:50,834 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn', 'group': 'hadoop', 'ignore_failures': True, 'create_parents': True, 'cd_access': 'a'}
2017-10-25 15:23:50,837 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'fs.defaultFS': 'true'}}, 'owner': 'hdfs', 'configurations': ...}
2017-10-25 15:23:50,858 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2017-10-25 15:23:50,859 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-10-25 15:23:50,901 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'dfs.support.append': 'true', 'dfs.datanode.data.dir': 'true', 'dfs.namenode.http-address': 'true', 'dfs.namenode.name.dir': 'true', 'dfs.webhdfs.enabled': 'true', 'dfs.datanode.failed.volumes.tolerated': 'true'}}, 'owner': 'hdfs', 'configurations': ...}
2017-10-25 15:23:50,919 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2017-10-25 15:23:50,920 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-10-25 15:23:51,113 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2017-10-25 15:23:51,134 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2017-10-25 15:23:51,134 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-10-25 15:23:51,220 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 502 to yarn
2017-10-25 15:23:51,220 - XmlConfig['yarn-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2017-10-25 15:23:51,238 - Generating config: /usr/hdp/current/hadoop-client/conf/yarn-site.xml
2017-10-25 15:23:51,238 - File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-10-25 15:23:51,463 - XmlConfig['capacity-scheduler.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2017-10-25 15:23:51,477 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2017-10-25 15:23:51,478 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2017-10-25 15:23:51,500 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 501 to yarn
2017-10-25 15:23:51,501 - Directory['/hadoop/yarn/timeline'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2017-10-25 15:23:51,501 - Directory['/hadoop/yarn/timeline'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2017-10-25 15:23:51,502 - HdfsResource['/ats/done'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://zone1.demo.local:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'change_permissions_for_parents': True, 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0755}
2017-10-25 15:23:51,505 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://zone1.demo.local:8082/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpKY8eVN 2>/tmp/tmpOto4VM''] {'logoutput': None, 'quiet': False}
2017-10-25 15:23:51,634 - call returned (0, 'su: warning: cannot change directory to /home/hdfs: No such file or directory')
Command failed after 1 tries
if I access the Ambari client to su - hdfs then hdfs dfs -ls /, I got this:
17/10/25 16:18:21 WARN retry.RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over null. Not retrying because try once and fail.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Username: 'hdfs' not found. Make sure your client's username exists on the cluster
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:265)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)
ls: Username: 'hdfs' not found. Make sure your client's username exists on the cluster
When I do id hdfs. I got this which seems correct:
uid=501(hdfs) gid=538(hadoop) groups=538(hadoop),501(hdfs)
Here is the output from Isilon as root (obviously the hdfs user does exists):
isilon1-1# isi auth users view hdfs --zone=zone1
Name: hdfs
DN: CN=hdfs,CN=Users,DC=ISILON1
DNS Domain: -
Domain: ISILON1
Provider: lsa-local-provider:zone1
Sam Account Name: hdfs
UID: 501
SID: S-1-5-21-837597287-2823100475-1132252091-1038
Enabled: No
Expired: No
Expiry: -
Locked: No
Email: -
GECOS: -
Generated GID: No
Generated UID: No
Generated UPN: Yes
Primary Group
ID: GID:501
Name: hdfs
Home Directory: /ifs/zone1/hdp/home/hdfs
Max Password Age: 4W
Password Expired: No
Password Expiry: 2017-11-22T21:53:46
Password Last Set: 2017-10-25T21:53:46
Password Expires: Yes
Shell: /bin/zsh
UPN: hdfs@ISILON1
User Can Change Password: Yes
When I installed, before register Isilon in Ambari, I used below content for uid and groups which seem correct also:
isilon1-1# cat zone1.passwd
# use this file to add to the passwd file of your clients
hdfs:x:501:501:hadoop-svc-account:/home/hdfs:/bin/bash
mapred:x:502:502:hadoop-svc-account:/home/mapred:/bin/bash
yarn:x:503:503:hadoop-svc-account:/home/yarn:/bin/bash
hbase:x:504:504:hadoop-svc-account:/home/hbase:/bin/bash
storm:x:505:505:hadoop-svc-account:/home/storm:/bin/bash
falcon:x:506:506:hadoop-svc-account:/home/falcon:/bin/bash
tracer:x:507:507:hadoop-svc-account:/home/tracer:/bin/bash
tez:x:508:508:hadoop-svc-account:/home/tez:/bin/bash
hive:x:509:509:hadoop-svc-account:/home/hive:/bin/bash
hcat:x:510:510:hadoop-svc-account:/home/hcat:/bin/bash
oozie:x:511:511:hadoop-svc-account:/home/oozie:/bin/bash
zookeeper:x:512:512:hadoop-svc-account:/home/zookeeper:/bin/bash
ambari-qa:x:513:513:hadoop-svc-account:/home/ambari-qa:/bin/bash
flume:x:514:514:hadoop-svc-account:/home/flume:/bin/bash
hue:x:515:515:hadoop-svc-account:/home/hue:/bin/bash
accumulo:x:516:516:hadoop-svc-account:/home/accumulo:/bin/bash
hadoopqa:x:517:517:hadoop-svc-account:/home/hadoopqa:/bin/bash
sqoop:x:518:518:hadoop-svc-account:/home/sqoop:/bin/bash
spark:x:519:519:hadoop-svc-account:/home/spark:/bin/bash
mahout:x:520:520:hadoop-svc-account:/home/mahout:/bin/bash
ranger:x:521:521:hadoop-svc-account:/home/ranger:/bin/bash
kms:x:522:522:hadoop-svc-account:/home/kms:/bin/bash
atlas:x:523:523:hadoop-svc-account:/home/atlas:/bin/bash
ams:x:524:524:hadoop-svc-account:/home/ams:/bin/bash
kafka:x:525:525:hadoop-svc-account:/home/kafka:/bin/bash
zeppelin:x:526:526:hadoop-svc-account:/home/zeppelin:/bin/bash
livy:x:527:527:hadoop-svc-account:/home/livy:/bin/bash
logsearch:x:528:528:hadoop-svc-account:/home/logsearch:/bin/bash
infra-solr:x:529:529:hadoop-svc-account:/home/infra-solr:/bin/bash
activity_analyzer:x:530:530:hadoop-svc-account:/home/activity_analyzer:/bin/bash
activity_explorer:x:531:531:hadoop-svc-account:/home/activity_explorer:/bin/bash
HTTP:x:532:532:hadoop-svc-account:/home/HTTP:/bin/bash
knox:x:533:533:hadoop-svc-account:/home/knox:/bin/bash
ambari-server:x:534:534:hadoop-svc-account:/home/ambari-server:/bin/bash
druid:x:535:535:hadoop-svc-account:/home/druid:/bin/bash
anonymous:x:536:536:hadoop-svc-account:/home/anonymous:/bin/bash
admin:x:537:537:hadoop-svc-account:/home/admin:/bin/bash
isilon1-1# cat zone1.group
# use this file to add to the group file of your clients
hdfs:x:501:
mapred:x:502:
yarn:x:503:
hbase:x:504:
storm:x:505:
falcon:x:506:
tracer:x:507:
tez:x:508:
hive:x:509:
hcat:x:510:
oozie:x:511:
zookeeper:x:512:
ambari-qa:x:513:
flume:x:514:
hue:x:515:
accumulo:x:516:
sqoop:x:518:
spark:x:519:
mahout:x:520:
ranger:x:521:
kms:x:522:
atlas:x:523:
ams:x:524:
kafka:x:525:
zeppelin:x:526:
livy:x:527:
logsearch:x:528:
infra-solr:x:529:
activity_analyzer:x:530:
activity_explorer:x:531:
HTTP:x:532:
knox:x:533:
ambari-server:x:534:
druid:x:535:
anonymous:x:536:
admin:x:537:
hadoopqa:x:517: hadoop:x:538:hdfs,mapred,yarn,hbase,storm,falcon,tracer,tez,hive,hcat,oozie,zookeeper,ambari-qa,flume,hue,accumulo,hadoopqa,sqoop,spark,mahout,ranger,kms,atlas,ams,kafka,zeppelin,livy,logsearch,infra-solr,activity_analyzer,activity_explorer,HTTP,knox,ambari-server,druid,
So frankly I am not sure what caused username hdfs (as well as all other users) to be not found.
Any idea to troubleshoot?
BernieC
76 Posts
1
October 26th, 2017 08:00
I assume you have a SmartConnect zone set up on a pool that is tied to your zone1 access zone. And I would expect that you have configured your Hadoop cluster to connect to that SmartConnect zone. If the Hadoop cluster is for some reason connecting through the system zone this error would be expected since the user account likely doesn't exist in that zone.
Alternatively, you could increase auth and/or hdfs service logging to try to find out how the Isilon cluster is interpreting the incoming request. One of those should give details about the zone where the request was received, but may do it by zone ID which can be found with isi zone zones list -v.
dfelt1
34 Posts
0
October 25th, 2017 18:00
You might want to check out the compatibility chart in the Hadoop area as OneFS 8.0.0.5 should be used with HDP 2.6.
Hadoop Info Hub: https://community.emc.com/docs/DOC-39529
Hadoop Distributions and products supported by OneFS versions: https://community.emc.com/docs/DOC-37101
quanghoc
40 Posts
0
October 26th, 2017 00:00
I just restored the snapshot to original and installed this stack here
I still have the same problem:
I don't know if this is a problem with the isilon_create_users.sh as it created two files: zone1.group and zone1.group.bak. The last line is different
Output seems good though:
Please share me some more idea.
bergec
275 Posts
0
October 26th, 2017 00:00
Hello
Could it be because your hdfs user is not enabled?
Regards, Claude
quanghoc
40 Posts
0
October 26th, 2017 07:00
I am not sure if I have to enable hdfs user. I have enabled and licensed HDFS. I also just tried to enable users in the GUI and still same problem:
Here are my CLIs in the Isilon
Below are what in /var/log/hdfs.log
BernieC
76 Posts
1
October 26th, 2017 08:00
BTW, the hdfs user in my environment is also disabled and requests work fine with it against both Kerberized and non-Kerberized Hadoop clusters.
quanghoc
40 Posts
0
October 31st, 2017 10:00
Bernie Case Thanks for the hint as I found out that in fact my Hortonworks connecting through System Zone due to wrong DNS IP. Phew...
But just a side question: how do I "see" or troubleshoot a request into Isilon in term of how Isilon received it? Is there a way to do that or watch traffic? etc...
BernieC
76 Posts
2
October 31st, 2017 11:00
Hi there Hoc Phan,
Couple of ways to go about this. Every node has /var/log/hdfs.log. If you change the log level to verbose (isi hdfs log-level modify --set=verbose and don't forget to set it back to warning when done) you will see more descriptive information when connects are made, but it won't always include IPs.
I find that I rely on the service logs from the Hadoop clients to see what IPs they were connecting to or having trouble with before I go and turn up logging on Isilon. In addition, I use isi statistics to look at incoming active connections to see what they're doing. For example:
isi statistics client list --protocols=hdfs --format=top --long
Would that work, or are you looking for something more?
quanghoc
40 Posts
1
November 1st, 2017 11:00
Bernie Case This is great. Thank you!