- Notes, cautions, and warnings
- S3
- Amazon S3 API support in ECS
- S3 API supported and unsupported features
- Bucket policy support
- Object Tagging
- S3 Object Lock
- Object lifecycle management
- S3 Extensions
- Metadata Search
- S3 and Swift Interoperability
- Create and manage secret keys
- Authenticating with the S3 service
- Using s3curl with ECS
- Use SDKs to access the S3 service
- ECS S3 error codes
- Hadoop S3A for ECS
- Enabling data2 IP in ECS S3
- Cloud DVR
- ECS IAM for S3
- OpenStack Swift
- EMC Atmos
- CAS
- Setting up CAS support in ECS
- Cold Storage
- Compliance
- CAS retention in ECS
- Advanced retention for CAS applications: event-based retention, litigation hold, and the min/max governor
- Set up namespace retention policies
- Create and set up a bucket for a CAS user
- Set up a CAS object user
- Set up bucket ACLs for CAS
- ECS Management APIs that support CAS users
- Content Addressable Storage (CAS) SDK API support
- ECS CAS error codes
- Enabling data2 IP in CAS
- ECS Management REST API
- ECS HDFS
- ECS HDFS Introduction
- Configuring Hadoop to use ECS HDFS
- Hadoop authentication modes
- Migration from a simple to a Kerberos Hadoop cluster
- File system interaction
- Supported Hadoop applications
- Integrate a simple Hadoop cluster with ECS HDFS
- Install Hortonworks HDP using Ambari
- Create a bucket for HDFS using the ECS Portal
- Plan the ECS HDFS and Hadoop integration
- Obtain the ECS HDFS installation and support package
- Deploy the ECS HDFS Client Library
- Configure ECS client properties
- Set up Hive
- Verify Hadoop access to ECS
- Secure the bucket
- Relocate the default file system from HDFS to an ECS bucket
- Integrate a secure Hadoop cluster with ECS HDFS
- Verify that AD/LDAP is correctly configured with a secure Hadoop cluster
- Pig test fails: unable to obtain Kerberos principal
- Permission denied for AD user
- Permissions errors
- Failed to process request
- Enable Kerberos client-side logging and debugging
- Debug Kerberos on the KDC
- Eliminate clock skew
- Configure one or more new ECS nodes with the ECS service principal
- Workaround for Yarn directory does not exist error
- Set up the Kerberos KDC
- Configure AD user authentication for Kerberos
- Secure bucket metadata
- Hadoop core-site.xml properties for ECS HDFS
- Hadoop core-site.xml properties for ECS S3
- External key management
- Document feedback
ECS 3.6.2 Data Access Guide
Configuring Hadoop to use ECS HDFS
Hadoop stores system configuration information in several files, including core-site.xml, hdfs-site.xml and hive-site.xml. The ECS HDFS configuration requires you to edit core-site.xml.
Add, or modify several types of properties in the core-site.xml file, including:
- ECS HDFS Java classes: This set of properties defines the ECS HDFS implementation classes that are contained in the ECS HDFS Client Library.
- File system location properties: These properties define the file system URI (scheme and authority) to use when running Hadoop jobs, and the IP addresses or FQDNs of the ECS data nodes for a specific ECS file system.
- Kerberos realm and service principal properties: These properties are required only in a Hadoop environment where Kerberos is present. These properties map Hadoop and ECS HDFS users.
The core-site.xml file resides on each node in the Hadoop cluster. Add the same properties to each instance of core-site.xml.
NOTE:
When modifying configuration files, you should use the management interface (Ambari) rather than manually editing files. Changes that you make using the Ambari management interface are persisted across the cluster.
NOTE: HDFS now supports automatic trash removal for HDFS files that a user removes. In traditional Hadoop, the
fs.rash.interval hadoop setting defines the minimum time that is taken to clean up the removed files from a users trash directory, and
fs.trash.checkpoint.interval
defines the time interval between active and inactive period, the trash cleanup thread takes before checking for removal of the candidates. In this release of ECS, the storage administrator manages the trash removal on the ECS. Use the
cf_client command, the bucket, namespace of a Hadoop file system, and maintenance interval settings to define maintenance intervals for all trash folders.
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\