ECS 3.6.2 Data Access Guide

PDF

Hadoop core-site.xml properties for ECS HDFS

When configuring the Hadoop core-site.xml file, use this table as a reference for the properties and their related values.

Table 1. Hadoop core-site.xml propertiesThe table lists the Hadoop core-site.xml properties details
Property	Description
File system implementation properties
fs.viprfs.impl	<property> <name>fs.viprfs.impl</name> <value>com.emc.hadoop.fs.vipr.ViPRFileSystem</value> </property>
fs.AbstractFileSystem.viprfs.impl	<property> <name>fs.AbstractFileSystem.viprfs.impl</name> <value>com.emc.hadoop.fs.vipr.ViPRAbstractFileSystem</value> </property>
Properties that define the authority section of the ECS HDFS file system URI
fs.vipr.installations	A comma-separated list of names. The names are further defined by the fs.vipr.installation.[federation].hosts property to uniquely identify sets of ECS data nodes. The names are used as a component of the authority section of the ECS HDFS file system URI. For example: <property> <name>fs.vipr.installations</name> <value><federation>,<site1>,<testsite></value> </property>
fs.vipr.installation.[federation].hosts	The IP addresses of the ECS cluster's data nodes or the load balancers for each name listed in the fs.vipr.installations property. Specify the value in the form of a comma-separated list of IP addresses or FQDNs. For example: <property> <name>fs.vipr.installation.<federation>.hosts</name> <value>203.0.113.10,203.0.113.11,203.0.113.12</value> </property>
fs.vipr.installation. [installation_name].resolution	Specifies how the ECS HDFS software knows how to access the ECS data nodes. Values are: dynamic: Use this value when accessing ECS data nodes directly without a load balancer. fixed: Use this value when accessing ECS data nodes through a load balancer. <property> <name>fs.vipr.installation.<federation>.resolution</name> <value>dynamic</value> </property>
fs.vipr.installation.[installation_name].resolution.dynamic.time_to_live_ms	When the fs.vipr.installation.[installation_name].resolution property is set to dynamic, this property specifies how often to query ECS for the list of active nodes. Values are in milliseconds. The default is 10 minutes. <property> <name>fs.vipr.installation.`<federation>`.resolution.dynamic.time_to_live_ms</name> <value>600000</value> </property>
ECS file system URI
fs.defaultFS	A standard Hadoop property that specifies the URI to the default file system. Setting this property to the ECS HDFS file system is optional. If you do not set it to the ECS HDFS file system, you must specify the full URI on each file system operation. The ECS HDFS file system URI has this format: viprfs://[bucket_name].[namespace].[federation] `bucket_name`: The name of the HDFS-enabled bucket that contains the data you want to use when you run Hadoop jobs. `namespace`: The tenant namespace associated with the HDFS-enabled bucket. `federation`: The name associated with the set of ECS data nodes that Hadoop can use to access ECS data. The value of this property must match one of the values specified in the fs.vipr.installations property. For example: <property> <name>fs.defaultFS</name> <value>viprfs://testbucket.s3.federation1</value> </property>
umask property
fs.permissions.umask-mode	This standard Hadoop property specifies how ECS HDFS should compute permissions on objects. Permissions are computed by applying a umask on the input permissions. The recommended value for both simple and Kerberos configurations is: 022. For example: <property> <name>fs.permissions.umask-mode</name> <value>022</value> </property>
Identity translation properties
fs.viprfs.auth.identity_translation	This property specifies how the ECS HDFS client determines what Kerberos realm a particular user belongs to if one is not specified. ECS data nodes store file owners as username@REALM, while Hadoop stores file owners as just the username. The possible values are: NONE: Default. Users are not mapped to a realm. Use this setting with a Hadoop cluster that uses simple security. With this setting ECS HDFS does not perform realm translation. CURRENT_USER_REALM: Valid when Kerberos is present. The user's realm is auto-detected, and it is the realm of the currently signed in user. In the example below, the realm is EMC.COM because sally is in the EMC.COM realm. The file ownership is changed john@EMC.COM. # kinit sally@EMC.COM # hdfs dfs -chown john /path/to/file Realms provided at the command line takes precedence over the property settings. <property> <name>fs.viprfs.auth.identity_translation </name> <value>CURRENT_USER_REALM</value> </property> NOTE: FIXED_REALM is now deprecated.
fs.viprfs.auth.realm	The realm assigned to users when the fs.viprfs.auth.identity_translation property is set to FIXED_REALM. This is now deprecated.
fs.viprfs.auth.anonymous_translation	This property is used to determine how users and groups are assigned to newly created files. NOTE: This property was used to determine what happened to files that had no owner. These files were said to be owned by anonymous. Files and directories are no longer anonymously owned. The values are: LOCAL_USER: Use this setting with a Hadoop cluster that uses simple security. Assigns the Unix user and group of the Hadoop cluster to newly created files and directories. CURRENT_USER: Use this setting for a Hadoop cluster that uses Kerberos. Assigns the Kerberos principal (user@REALM.COM) as the file or directory owner, and uses the group that has been assigned as the default for the bucket. NONE: (Deprecated) Previously indicated that no mapping from the anonymously owned objects to the current user should be performed. <property> <name>fs.viprfs.auth.anonymous_translation</name> <value>CURRENT_USER</value> </property>
Kerberos realm and service principal properties
viprfs.security.principal	This property specifies the ECS service principal. This property tells the KDC about the ECS service. This value is specific to your configuration. The principal name can include _HOST which is automatically replaced by the actual data node FQDN at run time. For example: <property> <name>viprfs.security.principal</name> <value>vipr/_HOST@example.com</value> </property>