OneFS and Ambari 2.5, HDP 2.6


OneFS and Ambari 2.5, HDP 2.6




Note: This topic is part of the Using Hadoop with OneFS - PowerScale Info Hub.

As with prior launches of Hortonworks Data Platform (HDP) and Apache Ambari, there are a few required changes in OneFS integration.

New Ambari version: 2.5.x
New HDP version: 2.6.x

Minimum OneFS versions for current release families. Future .0 releases will include support:
8.0.0.5 (GA 6/19/2017)
8.0.1.2 (upcoming)
8.1.0.1 (upcoming)


HDP Upgrade

If you're moving to HDP 2.6 from a prior release, be sure to take a look through our upgrade guide.

The key is to find the full HDP version string including package identifier. Then update OneFS's odp-version setting for your access zone.

The HDP release notes Repositories page include the full string for the current release. For 2.6.0.3 that is 2.6.0.3-8.

isi hdfs settings modify --odp-version=2.6.0.3-8 --zone=<your-zone>


Users

One new service has a new user account: Druid. The default service account name is simply druid. If you're using the current user creation script it is already included. Otherwise be sure to add the druid user before installing Druid.


Kerberos

Two new issues have cropped up with Kerberos integration between OneFS and Apache Hadoop. Both should be resolved in imminent releases of Ambari. In case you have an early release though, here are the details.


Principal-suffix

This new setting is good news overall, but hampered by a bug that will be fixed in an upcoming release of Ambari (still present in 2.5.1.0).

Apache 2.5 introduced a common property for all of the -${cluster_name|toLower()} chunks in default principal names in the Kerberos wizard. This was implemented in JIRA AMBARI-19430. The new property is called principal_suffix. The Dell EMC installation guide is updated with new steps to reflect that many principal names no longer need to be changed for a default install.

The bug in Ambari 2.5.0.3 and 2.5.1.0 is that the principal_suffix could not be blank. With these releases, you may need to modify each principal to remove $ if you do not want it. A fix is now available but was not included in Ambari 2.5.1.0 despite what JIRA AMBARI-20693 says. Hopefully it is fixed in 2.5.2.


Yarn HA state store

A more problematic issue has been introduced with the yarn.resourcemanager.zk-acl setting.

In 2.5, Ambari now configures an ACL-controlled ZooKeeper table to store the state of Yarn Resource Manager for High Availability. It records which host is the current Active resource manager. Only the resource manager process has read or write access to this table.

This is controlled with a user name in the midst of the string. In Ambari 2.5.0 and 2.5.1 at least that setting is `sasl:rm:rwcda`. The middle value is the user name, which must be changed to match the yarn.resourcemanager.principal root when integrating with OneFS.

This must be set properly before completing the Kerberos wizard. If not, the resource manager principal will not be able to access the state store. It will traceback and stop immediately after starting. There's no documented work around for this. Yarn may need to be reinstalled.

A change to an upcoming version of Apache Ambari should include a usabilty change for zk-acl to avoid this. That may be 2.5.2. You can track the status in JIRA AMBARI-20877.


In the Wizard

To address these two issues, find them on both tabs of the Configure Identities page in the Ambari Kerberos configuration wizard.

Principal Suffix is in the General tab, Global dropdown:
Image
zk-acl is in the Advanced tab under the Yarn drop down. Note the last setting shown here, yarn.resourcemanager.zk-acl. The middle value, yarn, matches the root of the top property, yarn.resourcemanager.principal.

Image
Oozie

We have great news about Oozie in a Kerberized environment with HDP 2.6. Hortonworks has included the patch for OOZIE-2940 in 2.6, so Oozie will now respect the use-ip setting. Oozie will now work when Yarn is properly configured for PowerScale Kerberos integration.

The patch was also included in HDP 2.5.3.0.


Article ID: SLN319157

Last Date Modified: 07/08/2020 06:57 PM

Rate this article

Accurate
Useful
Easy to understand
Was this article helpful?
0/3000 characters
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\
characters left.