HDP 2.3 Upgrade with OneFS

HDP 2.3 Upgrade with OneFS

Note: This topic is part of the Using Hadoop with OneFS - Isilon Info Hub.

This guide describes how to upgrade HDP from version 2.2 to 2.3. It covers upgrades on Ambari 2.1.

See the table at the top of the Ever Better Upgrades post to choose the HDP upgrade guide that is right for you.


There are three current OneFS versions affected:,, and The upcoming release will be similarly impacted. and would only allow upgrade from one release of HDP 2.2 to another maintenance release, eg 2.2.6 to 2.2.8. OneFS introduced compatibility with Ambari 2.1.0 and HDP 2.3, which is the target of this post.

These steps walk through manual upgrade, based on the HDP upgrade guide. HDP 2.3.0 and 2.3.2 are both possibilities, but because of compatibility requirements of OneFS, Ambari 2.1 must be used. Also the steps can be applied to upgrades to maintenance releases such as 2.2.6 to 2.2.8, but they are specifically written for an upgrade from 2.2 to 2.3 and align with those HDP upgrade steps. The examples assume that postgresql is used for the Ambari database, and the steps were confirmed on Ambari Server installed on CentOS 6.6

The procedures in this article use the following variables:
  • $ONEFSHOST - This is the host name of your OneFS cluster, as registered in Ambari Server.
  • $STACK - This is the 2-number stack of HDP that you are upgrading to. This is "2.3" currently.
  • $HDPVERSION-PACKAGE - This is the 4-number release version and 4 digit package version of HDP. The two HDP releases of 2.3 that are available today are and
  • $VERSIONDISPLAYNAME - This is the text that appears in bold type in Ambari Server's version page. For example, "HDP-".
  • $CLUSTERNAME - This is the name that you assigned to your cluster during installation, which appears on the top left of Ambari's web UI.

Ensure Correct Pre-Install State

Because OneFS is not sending expected information to Ambari Server, in some cases no entry will be created during installation for it in the database table that tracks the state of HDP versions per host. In this sequence you will check if this is true and correct it if so.

  • Enter the Ambari database. From the command line of your Ambari Server machine, run the following command: psql -U ambari
  • When prompted for database password input "bigdata"
  • Check to make sure that the OneFS host has a CURRENT entry in the host_version table.
SELECT * from host_version LEFT OUTER JOIN hosts ON host_version.host_id = hosts.host_id WHERE hosts.host_name = '$ONEFSHOST';

In the results, there should be a line with the OneFS host and the state set to CURRENT. This is what it would look like:

  • If you do not see a CURRENT entry for OneFS, then create a new entry in the host_version table. Choose an ID that is not in use, indicated by $NEWID.

insert into host_version (id, repo_version_id, host_id, state) values ('$NEWID', (select repo_version_id from repo_version where version = '$HDPVERSION-PACKAGE'), (select host_id from hosts where host_name = '$ONEFSHOST'), 'CURRENT');

  • Exit psql by typing "\q",

Install Packages

In this section, you will have Ambari install packages on all of the HDP hosts, and then edit the database to reflect the state as if OneFS had provided installation results to Ambari Server.

Begin the HDP manual upgrade steps from the Hortonworks upgrade guide.

Before the final step of "Installing a New Version" (2.2 step 6), the UI will be hung with the status "Installing" and the progress bar will be full but blue. Clicking on "Installing" will show that all hosts are complete and green. Ambari Server is waiting to hear version information about successful installation on the OneFS host. It will look like this:

Stop Ambari Server. From the command line: ambari-server stop

Set the OneFS host and the cluster to INSTALLED. Enter the database as described above. Then enter the following queries:

update host_version set state = 'INSTALLED' where state in ('INSTALLING', 'INSTALL_FAILED'); update cluster_version set state = 'INSTALLED' where state in ('INSTALLING', 'INSTALL_FAILED');

Exit psql as described above and then start Ambari Server with this command:

ambari-server start

Installation in the Ambari Server version page should complete at this point, transitioning to a "Perform Upgrade" button. Do not press it.

Upgrade HDP

In this section you will follow the "Preparing to Upgrade" section of the HDP upgrade guide, skipping a number of irrelevant steps for HDFS. The Ambari database will need to be updated as if OneFS has completed upgrade. The normal step to complete upgrade will not work, so that is replaced with further database edits and an ambari-server command line action.

  • Begin 2.3, "Preparing to Upgrade"

  • Complete through 2.3.5 step 1

  • Skip 2.3.5 steps 2 to 8.

  • Resume from 2.3.5 step 9.

  • Do not perform 2.4.1 on OneFS, but do it on other HDP hosts.

  • At 2.4.4 step 1, move OneFS components to the new version. First stop Ambari Server from the command line then enter the database as described above. Then enter the following queries:

update hostcomponentstate set upgrade_state = 'COMPLETE' where host_id in (select host_id FROM hosts where host_name = '$ONEFSHOST') ;
update hostcomponentstate set version = '$HDPVERSION-PACKAGE' where component_name in ('SECONDARY_NAMENODE', 'NAMENODE', 'DATANODE');

  • Exit psql and start Ambari Server again.

  • In 2.4.4, skip steps 1 to 7.

  • At 8(b), perform Start instead of Restart All.

  • 8(c) is irrelevant. Skip steps 9 to 13.

  • Resume and complete service upgrade tasks.

  • Skip 2.4.20.

  • After 2.4.21 step 1, set all components to the current stack, move OneFS host and cluster state to UPGRADED.

update hostcomponentstate set current_stack_id = subquery.stack_id from (select stack_id from stack where stack_version = '$STACK') as subquery;
update hostcomponentstate set upgrade_state = 'NONE';
update host_version set state = 'UPGRADED' where state IN ('UPGRADING', 'INSTALLED', 'INSTALLING');
update cluster_version set state = 'UPGRADED' where state IN ('UPGRADING', 'INSTALLED', 'INSTALLING');

  • Set the new version to CURRENT.

ambari-server set-current --cluster-name=$CLUSTER --version-display-name=$VERSIONDISPLAYNAME

Your upgrade is now complete.

Article ID: SLN319419

Last Date Modified: 02/25/2020 03:58 PM

Rate this article

Easy to understand
Was this article helpful?
0/3000 characters
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\
characters left.