Rolling Upgrade of HDP 2.3 with OneFS


Rolling Upgrade of HDP 2.3 with OneFS




Note: This topic is part of the Using Hadoop with OneFS - Isilon Info Hub.

This article describes Rolling Upgrades on Ambari 2.1. See the table at the top of Ever Better Upgrades to see which HDP upgrade guide is right for you.

OneFS supports rolling upgrades of Hortonworks Data Platform (HDP) in releases 8.0.0.0 and 7.2.1.2. The functionality is also available in patch 164059 for 7.2.1.1. This guide describes the rolling upgrade process.


Background

This procedure assumes that you have installed the patch on 7.2.1.1, or are running a higher build in the same family. There are both manual upgrade and rolling upgrade steps, based on the HDP upgrade guide. Any build of HDP 2.2 or 2.3 can be upgraded from and to. The steps were confirmed on Ambari 2.1.0 with the server installed on CentOS 6.6.

The steps in this article use the following variables:
  • $HDP_VERSION_WITH_BUILD - This is the 4-number release version and 4-digit build number of HDP. The three HDP releases of 2.3 that are available today are 2.3.0.0-2557, 2.3.2.0-2950, and 2.3.4.0-3485. This can be determined from the release notes of the HDP build that you are upgrading to, or by running the command 'hadoop version' on an existing deployment.
  • $ZONE - This is the name of the OneFS zone where HDFS data is stored.
  • $ONEFSHOST - This is the host name of your OneFS cluster, as registered in Ambari Server.
  • $AMBARIHOSTNAME - This is the host name of the system running Ambari Server.
  • $CLUSTERNAME - This is the name that you assigned to your HDP cluster during installation, which appears on the top left of Ambari's web UI.


Configure OneFS

Ambari Server expects each agent to send it the full installed HDP package number after completing some commands. Previously OneFS was unable to send that. Starting with this patch and from 7.2.1.2, OneFS has a new setting which stores the value to send to Ambari Server. This is "hdfs-odp-version". In the OneFS CLI, set the value with this command:

isi zone zones modify --hdfs-odp-version=$HDP_VERSION_WITH_BUILD --zone $ZONE

In some cases Ambari's database is in an invalid state. No entry exists for OneFS in the database table that tracks the state of HDP versions per host. If you have just completed installation of the patch or fixed version of OneFS, first stop and start the HDFS service in Ambari before continuing. Stopping and starting the HDFS service triggers Ambari to refresh the version for hosts with HDFS components.


Rolling Upgrade

Rolling upgrade does not require intervention on each Hadoop cluster machine, and is also non-disruptive. This makes it the easiest way to upgrade the cluster. However, there are a few steps that are necessary because a cluster with OneFS is not configured with the prerequisite NameNode High Availability (NNHA). OneFS provides continuous availability of data and HDFS streams, but does not communicate to Ambari Server as if it has NNHA.

  1. Begin setup and complete installation of packages.
  2. Click the Perform Upgrade button (it may have changed to a link named Upgrade: In Progress; if so, click this), and observe the list of failed pre-upgrade checks. A NameNode High Availability failure is expected. If there is anything else, resolve those issues before continuing.
  3. To skip the NameNode High Availability check, navigate to the Ambari Server experimental features WebUI at http://$AMBARIHOSTNAME:8080/#/experimental/.
  4. Uncheck the "preUpgradeCheck" option.

  5. Complete through steps 1.4 or 3.5 of the HDP upgrade guide. Continue with steps 1.5 or 3.5
  6. After upgrade starts, run the following commands on the Ambari Server host's command line, or any machine that has access to it:

    curl http://$AMBARIHOSTNAME:8080/api/v1/clusters/$CLUSTERNAME/hosts/$ONEFSHOST/host_components/SECONDARY_NAMENODE -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop SNN via REST"}, "Body": {"HostRoles": {"state": "INSTALLED"}}}'


    curl http://$AMBARIHOSTNAME:8080/api/v1/clusters/$CLUSTERNAME/hosts/$ONEFSHOST/host_components/SECONDARY_NAMENODE -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Start SNN via REST"}, "Body": {"HostRoles": {"state": "STARTED"}}}'
  7. After the upgrade completes, click the finalized button.
Your upgrade is now complete.

Notes:
  1. If you need to stop the upgrade mid-stream, a downgrade process is required. First, set the odp-version in OneFS back to the old version. Then complete the downgrade process in HDP.
  2. If you waited too long to execute the curl commands, they are Accepted but enter a paused state in Ambari Server. When you click the Finalize button, it results in a failure such as "Failed on: Save Cluster state". Click the dismiss button on this dialog, and then click on the "op" balloon on the top of the Ambari UI. You may see the operations that were paused, and they will complete without any further interaction. Choose the "Upgrade: Action Required" button in the top middle of the Ambari UI, and then the Retry button.

Manual Upgrade

There are situations where you must manually upgrade your cluster. For those cases, here are the new, simpler steps. This procedure does not require the use of the Ambari experimental feature, but it is more labor intensive on each HDP host.

For maintenance releases, eg upgrading from 2.3.0.0 to 2.3.2.0:

  1. Complete through 4.4 step 5 of the HDP upgrade guide.

  2. Do not perform 4.4 step 6 on OneFS, but do it on other Hadoop hosts.

  3. Complete the remaining tasks including starting each service.

For major version upgrades, eg from 2.2.6 to 2.3.0:

  1. Complete through 2.3.5 step 1.

  2. Skip 2.3.5 steps 2 to 8.

  3. Resume from 2.3.5 step 9.

  4. Do not perform 2.4.1 on OneFS, but do it on other Hadoop hosts.

  5. In 2.4.4, skip steps 1 to 7. 8(c) is irrelevant. Skip steps 9 to 13.

  6. Resume and complete service upgrade tasks.

  7. Skip 2.4.20.

  8. Complete the upgrade.

Your upgrade is now complete.

Isilon plans to continue working with the Apache community to improve the upgrade experience as new features, such as Express Upgrade, are released.



Article ID: SLN319416

Last Date Modified: 02/25/2020 03:47 PM

Rate this article

Accurate
Useful
Easy to understand
Was this article helpful?
0/3000 characters
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\
characters left.