CPU Usage of DataNodes Is Close to 100% Occasionally, Causing Node Loss

Symptom

Figure 1 DataNode CPU usage close to 100%

Cause Analysis

1. A lot of write failure logs exist on DataNodes.

   Figure 2 DataNode write failure log

   
   
2. A large number of files are written in a short time, causing insufficient DataNode memory.

   Figure 3 Insufficient DataNode memory

   
   

Solution

1. Check DataNode memory configuration and whether the remaining server memory is sufficient.
2. Increase DataNode memory and restart the DataNode.