HDFS Capacity Reaches 100%, Causing Unavailable Upper-Layer Services Such as HBase and Spark

Issue

Symptom

Cause Analysis

Procedure

1. Log in to any Master node in the cluster.
2. Run the source /opt/client/bigdata_env command to initialize environment variables.
   Note

   If it is a security cluster, run the kinit -kt <keytab file> <Principal name> command for authentication.

3. Run the hdfs dfs -put ./startDetail.log /tmp command to check whether HDFS fails to write files.

   19/05/12 10:07:32 WARN hdfs.DataStreamer: DataStreamer Exception
   org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/startDetail.log._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and no node(s) are excluded in this operation.
   
   

4. Run the hdfs dfsadmin -report command to check the used HDFS capacity. The command output shows that the HDFS capacity usage has reached 100%.

   Configured Capacity: 5389790579100 (4.90 TB)
   Present Capacity: 5067618628404 (4.61 TB)
   DFS Remaining: 133350196 (127.17 MB)
   DFS Used: 5067485278208 (4.61 TB)
   DFS Used%: 100.00%
   Under replicated blocks: 10
   Blocks with corrupt replicas: 0
   Missing blocks: 0
   Missing blocks (with replication factor 1): 0
   Pending deletion blocks: 0
   
   

5. When the HDFS capacity usage reaches 100%, change the percentage of data disk space by setting HDFS parameter dfs.datanode.du.reserved.percentage.
   1. Go to the service configuration page.
      • MRS Manager: Log in to MRS Manager and choose Services > HDFS > Configuration.
      • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Services > HDFS > Configurations.
   2. Click All Configurations and search for dfs.datanode.du.reserved.percentage.
   3. Change the value of this parameter to 10.

6. After the modification, increase the number of disks of the Core node.