Using Alluxio
- Configuring an Underlying Storage System
- Accessing Alluxio Using a Data Application
- Common Operations of Alluxio
Using CarbonData (for Versions Earlier Than MRS 3.x)
- Using CarbonData from Scratch
- About CarbonData Table
- Creating a CarbonData Table
- Deleting a CarbonData Table
Using CarbonData (for MRS 3.x or Later)
- Overview
  - CarbonData Overview
  - Main Specifications of CarbonData
- Common CarbonData Parameters
- CarbonData Operation Guide
- CarbonData Performance Tuning
- CarbonData Access Control
- CarbonData Syntax Reference
- CarbonData Troubleshooting
  - Filter Result Is not Consistent with Hive when a Big Double Type Value Is Used in Filter
  - Query Performance Deterioration
- CarbonData FAQ
Using ClickHouse
- Using ClickHouse from Scratch
- ClickHouse Table Engine Overview
- Creating a ClickHouse Table
- Common ClickHouse SQL Syntax
- Migrating ClickHouse Data
- User Management and Authentication
- Backing Up and Restoring ClickHouse Data Using a Data File
- ClickHouse Log Overview
- ClickHouse FAQ
  - How Do I Do If the Disk Status Displayed in the System.disks Table Is fault or abnormal?
Using DBService
- DBService Log Overview
Using Flink
- Using Flink from Scratch
- Viewing Flink Job Information
- Configuring Flink Service Parameters
- Configuring Flink Security Features
- Configuring and Developing a Flink Visualization Job
- Flink Log Overview
- Flink Performance Tuning
- Common Flink Shell Commands
- Reference
  - Example of Issuing a Certificate
- Flink Restart Policy
Using Flume
- Using Flume from Scratch
- Overview
- Installing the Flume Client
  - Installing the Flume Client on Clusters of Versions Earlier Than MRS 3.x
  - Installing the Flume Client on MRS 3.x or Later Clusters
- Viewing Flume Client Logs
- Stopping or Uninstalling the Flume Client
- Using the Encryption Tool of the Flume Client
- Flume Service Configuration Guide
- Flume Configuration Parameter Description
- Using Environment Variables in the properties.properties File
- Non-Encrypted Transmission
- Encrypted Transmission
  - Configuring the Encrypted Transmission
  - Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
- Viewing Flume Client Monitoring Information
- Connecting Flume to Kafka in Security Mode
- Configuring the Flume Service Model
  - Overview
  - Service Model Configuration Guide
- Introduction to Flume Logs
- Flume Client Cgroup Usage Guide
- Secondary Development Guide for Flume Third-Party Plug-ins
- Common Issues About Flume
Using HBase
- Using HBase from Scratch
- Using an HBase Client
- Creating HBase Roles
- Configuring HBase Replication
- Configuring HBase Parameters
- Enabling Cross-Cluster Copy
- Using the ReplicationSyncUp Tool
- Using HIndex
- Configuring HBase DR
- Configuring HBase Data Compression and Encoding
- Performing an HBase DR Service Switchover
- Performing an HBase DR Active/Standby Cluster Switchover
- Community BulkLoad Tool
- Configuring Secure HBase Replication
- Configuring Region In Transition Recovery Chore Service
- Using a Secondary Index
- HBase Log Overview
- HBase Performance Tuning
- Common Issues About HBase
Using HDFS
- Using Hadoop from Scratch
- Configuring Memory Management
- Creating an HDFS Role
- Using the HDFS Client
- Running the DistCp Command
- Overview of HDFS File System Directories
- Changing the DataNode Storage Directory
- Configuring HDFS Directory Permission
- Configuring NFS
- Planning HDFS Capacity
- Configuring ulimit for HBase and HDFS
- Configuring HDFS DataNode Data Balancing
- Configuring Replica Replacement Policy for Heterogeneous Capacity Among DataNodes
- Configuring the Number of Files in a Single HDFS Directory
- Configuring the Recycle Bin Mechanism
- Setting Permissions on Files and Directories
- Setting the Maximum Lifetime and Renewal Interval of a Token
- Configuring the Damaged Disk Volume
- Configuring Encrypted Channels
- Reducing the Probability of Abnormal Client Application Operation When the Network Is Not Stable
- Configuring the NameNode Blacklist
- Optimizing HDFS NameNode RPC QoS
- Optimizing HDFS DataNode RPC QoS
- Configuring Reserved Percentage of Disk Usage on DataNodes
- Configuring HDFS NodeLabel
- Configuring HDFS Mover
- Using HDFS AZ Mover
- Configuring HDFS DiskBalancer
- Configuring the Observer NameNode to Process Read Requests
- Performing Concurrent Operations on HDFS Files
- Introduction to HDFS Logs
- HDFS Performance Tuning
  - Improving Write Performance
  - Improving Read Performance Using Client Metadata Cache
  - Improving the Connection Between the Client and NameNode Using Current Active Cache
- FAQ
  - NameNode Startup Is Slow
  - DataNode Is Normal but Cannot Report Data Blocks
  - HDFS WebUI Cannot Properly Update Information About Damaged Data
  - Why Do DistCp Commands Fail to Run in a Security Cluster and What Should I Do When Exceptions Are Thrown?
  - What Should I Do If DataNode Fails to Be Started When the Number of Disks Defined in dfs.datanode.data.dir Equals the Value of dfs.datanode.failed.volumes.tolerated?
  - Failed to Calculate the Capacity of a DataNode when Multiple data.dir Directories Are Configured in a Disk Partition
  - Standby NameNode Fails to Be Restarted When the System Is Powered off During Metadata (Namespace) Storage
  - What Should I Do If Data in the Cache Is Lost When the System Is Powered Off During Small File Storage?
  - Why Does Array Border-crossing Occur During FileInputFormat Split?
  - Why Is the Storage Type of File Copies DISK When the Tiered Storage Policy Is LAZY_PERSIST?
  - How Do I Handle the Problem that HDFS Client Is Irresponsive When the NameNode Is Overloaded for a Long Time?
  - Can I Delete or Modify the Data Storage Directory in DataNode?
  - Blocks Miss on the NameNode UI After the Successful Rollback
  - Why Is "java.net.SocketException: No buffer space available" Reported When Data Is Written to HDFS
  - Why are There Two Standby NameNodes After the active NameNode Is Restarted?
  - When Does a Balance Process in HDFS, Shut Down and Fail to be Executed Again?
  - "This page can't be displayed" Is Displayed When Internet Explorer Fails to Access the Native HDFS UI
  - NameNode Fails to Be Restarted Due to EditLog Discontinuity
Using Hive
- Using Hive from Scratch
- Configuring Hive Parameters
- Hive SQL
- Permission Management
  - Hive Permission
  - Creating a Hive Role
  - Configuring Permissions for Hive Tables, Columns, or Databases
  - Configuring Permissions to Use Other Components for Hive
- Using a Hive Client
- Using HDFS Colocation to Store Hive Tables
- Using the Hive Column Encryption Function
- Customizing Row Separators
- Configuring Hive on HBase in Across Clusters with Mutual Trust Enabled
- Deleting Single-Row Records from Hive on HBase
- Configuring HTTPS/HTTP-based REST APIs
- Enabling or Disabling the Transform Function
- Access Control of a Dynamic Table View on Hive
- Specifying Whether the ADMIN Permissions Is Required for Creating Temporary Functions
- Using Hive to Read Data in a Relational Database
- Supporting Traditional Relational Database Syntax in Hive
- Creating User-Defined Hive Functions
- Enhancing beeline Reliability
- Viewing Table Structures Using the show create Statement as Users with the select Permission
- Writing a Directory into Hive with the Old Data Removed to the Recycle Bin
- Inserting Data to a Directory That Does Not Exist
- Creating Databases and Tables in the Default Database Only as the Hive Administrator
- Disabling of Specifying the location Keyword When Creating an Internal Hive Table
- Enabling the Function of Creating a Foreign Table in a Directory That Can Only Be Read
- Authorizing Over 32 Roles in Hive
- Restricting the Maximum Number of Maps for Hive Tasks
- HiveServer Lease Isolation
- Switching the Hive Execution Engine to Tez
- Hive Log Overview
- Hive Performance Tuning
  - Creating Table Partitions
  - Optimizing Join
  - Optimizing Group By
  - Optimizing Data Storage
  - Optimizing SQL Statements
  - Optimizing the Query Function Using Hive CBO
- Common Issues About Hive
  - How Do I Delete UDFs on Multiple HiveServers at the Same Time?
  - Why Cannot the DROP operation Be Performed on a Backed-up Hive Table?
  - How to Perform Operations on Local Files with Hive User-Defined Functions
  - How Do I Forcibly Stop MapReduce Jobs Executed by Hive?
  - How Do I Monitor the Hive Table Size?
  - How Do I Prevent Key Directories from Data Loss Caused by Misoperations of the insert overwrite Statement?
  - Why Is Hive on Spark Task Freezing When HBase Is Not Installed?
  - Error Reported When the WHERE Condition Is Used to Query Tables with Excessive Partitions in FusionInsight Hive
  - Why Cannot I Connect to HiveServer When I Use IBM JDK to Access the Beeline Client?
  - Description of Hive Table Location (Either Be an OBS or HDFS Path)
  - Why Cannot Data Be Queried After the MapReduce Engine Is Switched After the Tez Engine Is Used to Execute Union-related Statements?
  - Why Does Hive Not Support Concurrent Data Writing to the Same Table or Partition?
  - Why Does Hive Not Support Vectorized Query?
  - Why Does Metadata Still Exist When the HDFS Data Directory of the Hive Table Is Deleted by Mistake?
  - How Do I Disable the Logging Function of Hive?
  - Why Hive Tables in the OBS Directory Fail to Be Deleted?
  - Hive Configuration Problems
  - How Do I Optimize the INSERT OVERWRITE for Reading and Writing in Same Table?
Using Hudi
- Getting Started
- Basic Operations
- Hudi Performance Tuning
- Common Issues About Hudi
Using Hue (Versions Earlier Than MRS 3.x)
- Using Hue from Scratch
- Accessing the Hue Web UI
- Hue Common Parameters
- Using HiveQL Editor on the Hue Web UI
- Using the Metadata Browser on the Hue Web UI
- Using File Browser on the Hue Web UI
- Using Job Browser on the Hue Web UI
Using Hue (MRS 3.x or Later)
- Using Hue from Scratch
- Accessing the Hue Web UI
- Hue Common Parameters
- Using HiveQL Editor on the Hue Web UI
- Using the SparkSql Editor on the Hue Web UI
- Using the Metadata Browser on the Hue Web UI
- Using File Browser on the Hue Web UI
- Using Job Browser on the Hue Web UI
- Using HBase on the Hue Web UI
- Typical Scenarios
- Hue Log Overview
- Common Issues About Hue
Using Impala
- Using Impala from Scratch
- Common Impala Parameters
- Accessing the Impala Web UI
- Using Impala to Operate Kudu
- Interconnecting Impala with External LDAP
- Enabling and Configuring a Dynamic Resource Pool for Impala
- Impala FAQ
  - Does Impala Support Disk Hot Swapping?
Using Kafka
- Using Kafka from Scratch
- Managing Kafka Topics
- Querying Kafka Topics
- Managing Kafka User Permissions
- Managing Messages in Kafka Topics
- Synchronizing Binlog-based MySQL Data to the MRS Cluster
- Creating a Kafka Role
- Kafka Common Parameters
- Safety Instructions on Using Kafka
- Kafka Specifications
- Using the Kafka Client
- Configuring Kafka HA and High Reliability Parameters
- Changing the Broker Storage Directory
- Checking the Consumption Status of Consumer Group
- Kafka Balancing Tool Instructions
- Balancing Data After Kafka Node Scale-Out
- Kafka Token Authentication Mechanism Tool Usage
- Introduction to Kafka Logs
- Performance Tuning
  - Kafka Performance Tuning
- Kafka Feature Description
- Migrating Data Between Kafka Nodes
- Common Issues About Kafka
  - How Do I Solve the Problem that Kafka Topics Cannot Be Deleted?
Using KafkaManager
- Introduction to KafkaManager
- Accessing the KafkaManager Web UI
- Managing Kafka Clusters
- Kafka Cluster Monitoring Management
Using Loader
- Using Loader from Scratch
- How to Use Loader
- Common Loader Parameters
- Creating a Loader Role
- Loader Link Configuration
- Managing Loader Links (Versions Earlier Than MRS 3.x)
- Managing Loader Links (MRS 3.x and Later Versions)
- Source Link Configurations of Loader Jobs
- Destination Link Configurations of Loader Jobs
- Managing Loader Jobs
- Preparing a Driver for MySQL Database Link
- Importing Data
- Exporting Data
- Managing Jobs
- Operator Help
- Client Tools
- Loader Log Overview
- Example: Using Loader to Import Data from OBS to HDFS
- Common Issues About Loader
  - Data Cannot Be Saved in Internet Explorer 10 or 11
  - Differences Among Connectors Used During the Process of Importing Data from the Oracle Database to HDFS
Using Kudu
- Using Kudu from Scratch
- Accessing the Kudu Web UI
Using MapReduce
- Configuring the Log Archiving and Clearing Mechanism
- Reducing Client Application Failure Rate
- Transmitting MapReduce Tasks from Windows to Linux
- Configuring the Distributed Cache
- Configuring the MapReduce Shuffle Address
- Configuring the Cluster Administrator List
- Introduction to MapReduce Logs
- MapReduce Performance Tuning
- Common Issues About MapReduce
Using Oozie
- Using Oozie from Scratch
- Using the Oozie Client
- Using Oozie Client to Submit an Oozie Job
- Using Hue to Submit an Oozie Job
- Oozie Log Overview
- Common Issues About Oozie
Using Presto
- Accessing the Presto Web UI
- Using a Client to Execute Query Statements
Using Ranger (MRS 3.x)
- Logging In to the Ranger Web UI
- Enabling Ranger Authentication
- Configuring Component Permission Policies
- Viewing Ranger Audit Information
- Configuring a Security Zone
- Viewing Ranger Permission Information
- Adding a Ranger Access Permission Policy for HDFS
- Adding a Ranger Access Permission Policy for HBase
- Adding a Ranger Access Permission Policy for Hive
- Adding a Ranger Access Permission Policy for Yarn
- Adding a Ranger Access Permission Policy for Spark2x
- Adding a Ranger Access Permission Policy for Kafka
- Adding a Ranger Access Permission Policy for Storm
- Ranger Log Overview
- Common Issues About Ranger
Using Spark
- Precautions
- Getting Started with Spark
- Getting Started with Spark SQL
- Using the Spark Client
- Accessing the Spark Web UI
- Interconnecting Spark with OpenTSDB
Using Spark2x
- Precautions
- Basic Operation
- Spark2x Logs
- Obtaining Container Logs of a Running Spark Application
- Small File Combination Tools
- Using CarbonData for First Query
- Spark2x Performance Tuning
- Common Issues About Spark2x
Using Sqoop
- Using Sqoop from Scratch
- Adapting Sqoop 1.4.7 to MRS 3.x Clusters
- Common Sqoop Commands and Parameters
- Common Issues About Sqoop
Using Storm
- Using Storm from Scratch
- Using the Storm Client
- Submitting Storm Topologies on the Client
- Accessing the Storm Web UI
- Managing Storm Topologies
- Querying Storm Topology Logs
- Storm Common Parameters
- Configuring a Storm Service User Password Policy
- Migrating Storm Services to Flink
- Storm Log Introduction
- Performance Tuning
  - Storm Performance Tuning
Using Tez
- Precautions
- Common Tez Parameters
- Accessing TezUI
- Log Overview
- Common Issues
Using YARN
- Common YARN Parameters
- Creating Yarn Roles
- Using the YARN Client
- Configuring Resources for a NodeManager Role Instance
- Changing NodeManager Storage Directories
- Configuring Strict Permission Control for Yarn
- Configuring Container Log Aggregation
- Using CGroups with YARN
- Configuring the Number of ApplicationMaster Retries
- Configure the ApplicationMaster to Automatically Adjust the Allocated Memory
- Configuring the Access Channel Protocol
- Configuring Memory Usage Detection
- Configuring the Additional Scheduler WebUI
- Configuring Yarn Restart
- Configuring ApplicationMaster Work Preserving
- Configuring the Localized Log Levels
- Configuring Users That Run Tasks
- Yarn Log Overview
- Yarn Performance Tuning
  - Preempting a Task
  - Setting the Task Priority
  - Optimizing Node Configuration
- Common Issues About Yarn
  - Why Mounted Directory for Container is Not Cleared After the Completion of the Job While Using CGroups?
  - Why the Job Fails with HDFS_DELEGATION_TOKEN Expired Exception?
  - Why Are Local Logs Not Deleted After YARN Is Restarted?
  - Why the Task Does Not Fail Even Though AppAttempts Restarts for More Than Two Times?
  - Why Is an Application Moved Back to the Original Queue After ResourceManager Restarts?
  - Why Does Yarn Not Release the Blacklist Even All Nodes Are Added to the Blacklist?
  - Why Does the Switchover of ResourceManager Occur Continuously?
  - Why Does a New Application Fail If a NodeManager Has Been in Unhealthy Status for 10 Minutes?
  - Why Does an Error Occur When I Query the ApplicationID of a Completed or Non-existing Application Using the RESTful APIs?
  - Why May A Single NodeManager Fault Cause MapReduce Task Failures in the Superior Scheduling Mode?
  - Why Are Applications Suspended After They Are Moved From Lost_and_Found Queue to Another Queue?
  - How Do I Limit the Size of Application Diagnostic Messages Stored in the ZKstore?
  - Why Does a MapReduce Job Fail to Run When a Non-ViewFS File System Is Configured as ViewFS?
  - Why Do Reduce Tasks Fail to Run in Some OSs After the Native Task Feature is Enabled?
Using ZooKeeper
- Using ZooKeeper from Scratch
- Common ZooKeeper Parameters
- Using a ZooKeeper Client
- Configuring the ZooKeeper Permissions
- ZooKeeper Log Overview
- Common Issues About ZooKeeper
Appendix
- Modifying Cluster Service Configuration Parameters
- Accessing Manager
  - Accessing MRS Manager (Versions Earlier Than MRS 3.x)
  - Accessing FusionInsight Manager (MRS 3.x or Later)
- Using an MRS Client

User Management and Authentication