OneFS and Ambari 2.4, HDP 2.5


OneFS and Ambari 2.4, HDP 2.5




Hortonworks recently launched HDP 2.5 which includes a number of new services and Apache Ambari 2.4.

OneFS versions 8.0.1.0 and 8.0.0.3 will support Ambari 2.4. Right now, you can install patch 180735 on top of 8.0.0.2 to add Ambari 2.4 support.

Here are additional considerations for compatibility with HDP 2.5 services and upgrading to HDP 2.5.


HDP Upgrade

If you are upgrading HDP, you need to set the target version in OneFS so that it replies correctly to the upgrade request by Ambari. For the initial release of 2.5, use this command:

isi hdfs settings modify --odp-version=2.5.0.0-1237 --zone=spiral

In the OneFS Admin UI, the setting is in Protocols > Hadoop (HDFS) > Settings.



Users

There are some new services in HDP that need new service users. They are all in the user script. Here is the list, in case you need to add them manually to the hadoop service account group on OneFS:

  • zeppelin
  • livy
  • logsearch
  • infra-solr
  • activity_analyzer
  • activity_explorer

For each service, you can run the following command:

isi auth users create zeppelin --primary-group=hadoop --zone=spiral

If you choose to configure a different username for Spark2, rather than Spark, be sure to add that as well.


Kerberos

To tie OneFS with Ambari and HDP in a Kerberized environment, you need to be careful about principal names, proxy users, and a few other settings.

The Hive service check now tests whether the HTTP service account can proxy as a user. In a non-Kerberized environment, there is no impact. However, for proper configuration in a kerberized environment, be sure that an HTTP account exists on OneFS (covered in the current user script), and that it has a proxyuser setting to act as ambari-qa (minimum bar for the service check) and the groups that your users are in.

isi auth users create HTTP --primary-group=hadoop --zone=spiral
isi hdfs proxyusers create --proxyuser=HTTP --add-user=ambari-qa --add-group=users --zone=spiral

Also, Spark's component Livy (a REST API for Spark) requires a proxy user configured on OneFS. For example, when using Livy to access Spark from Zeppelin, Livy needs to proxy as the zeppelin user. Modify these to match the group for your hadoop service accounts and any user accounts that need to use Livy.

isi hdfs proxyusers create --proxyuser=livy --add-group=hadoop --zone=spiral
isi hdfs proxyusers modify --proxyuser=livy --add-group=users --zone=spiral



As with services in earlier HDP releases, the principals need to match users on OneFS. Zeppelin and Spark2 both have the cluster name added to the end by default. To match the sample user names above, the cluster name should be removed.

Finally, Livy has its own concept of superusers, which has a separate configuration in Ambari. Livy needs the zeppelin user included in its superuser list. The default setting is zeppelin-{cluster-name}. If you change Zeppelin's username, Ambari does not automatically change the Livy setting. You can find the Livy superuser list and correct it at Kerberos > Advanced > Spark, "livy.superusers".


Article ID: SLN319404

Last Date Modified: 03/11/2020 08:15 PM

Rate this article

Accurate
Useful
Easy to understand
Was this article helpful?
0/3000 characters
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\
characters left.