How Can I Locate a Fault That Occurs with a Node?

Figure 1 Performing a self-check

1. Log in to the CCE console.
2. Click the cluster name to access the cluster console. Choose Nodes in the navigation pane. In the right pane, click the Nodes tab.
3. Locate the row containing the target node, choose More > View YAML in the Operation column, and check the Status field of the node.

   The node is in the NotReady state.

   • Check the node status and verify whether the value of PIDPressure, DiskPressure, or MemoryPressure becomes True. If any of them becomes True, you can find the appropriate solution based on the exception keyword.
   • Check the key components on the node and the logs of these components. The key components on a node include a kubelet and the node runtime (Docker or containerd). For details, see Checking Key Components of the Node.
     • Check the kubelet.

       Check whether the kubelet and its logs are normal. If there is an exception, see Abnormal kubelet.

     • Check the runtime (Docker or containerd).
       • Check the runtime of the node. If you are not sure whether the runtime is Docker or containerd, log in to the CCE console and view the runtime of the node.
       • If there is an exception, see Abnormal Runtime.
     • Check the NTP.
       • Check whether the NTP, its logs, and the configurations are normal.
       • If there is an exception, see Abnormal NTP.
   • Check the node monitoring data and see whether the CPU, memory, and network resources of the node are normal. If there is an exception, rectify the fault by referring to Memory Pressure.

   The node is in the Unknown state.

   • Log in to the ECS console and check whether the node is present in the ECS list.
   • Check whether the node is running properly.
   • Check the key components on the node and the logs of these components. The key components on a node include a kubelet and the node runtime (Docker or containerd). For details, see Checking Key Components of the Node.
     • Check the kubelet.
       • Check whether the kubelet and its logs are normal. If there is an exception, see Abnormal kubelet.
   • Check the network connectivity of the node.

1. Log in to the CCE console.
2. Click the name of the target cluster to access the cluster console.
3. In the navigation pane, choose Nodes. In the right pane, click the Nodes tab, locate the row containing the unavailable node, and view its status. (If NPD 1.6.10 or a later version is installed in the cluster, you will see a message indicating that the metrics for the unavailable node are abnormal. In this case, you can move the cursor to the upper part to view the specific problem. If the add-on is not installed, you can rectify the fault by referring to the check items.)

1. Log in to the CCE console.
2. Click the name of the target cluster to access the cluster console.
3. In the navigation pane, choose Nodes. In the right pane, click the Nodes tab, locate the row containing the abnormal node, and click Monitor in the Operation column.

   On the top of the displayed page, click More Monitoring Data to go to the AOM console and view historical monitoring records. If the CPU or memory usage of the node is too high, it can lead to high network latency or trigger system OOM, causing the node to be marked as unavailable.

1. Log in to the CCE console.
2. Click the name of the target cluster to access the cluster console.
3. In the navigation pane, choose Nodes. In the right pane, click the Nodes tab, locate the row containing the abnormal node, click View Events in the Operation column, and check whether any abnormal event is reported. (The NPD add-on must be installed.)

1. Log in to the CCE console, click the name of the target cluster to access the cluster console, and view the name of the unavailable node.
2. Log in to the ECS console, search for the node, and check the ECS status.
   • If the ECS has been deleted, go back to the CCE console, delete the node from the node list, and create another one.
   • If the ECS is stopped or frozen, restore it first. It takes about 3 minutes to restore the node.
   • If the ECS is faulty, restart it to rectify the fault.
   • If the ECS is available, rectify the fault by referring to Checking Key Components of the Node.

1. Log in to the ECS console.
2. Check whether the node name displayed on the ECS console is the same as that on the VM and whether the password or key can be used to log in to the node.

   If the node names are inconsistent and the password or key cannot be used to log in to the node, Cloud-Init problems occurred when the ECS was created. In this case, you can restart the node and then submit a service ticket to the ECS personnel to locate the root cause.

• The node security group was changed.
  1. Log in to the VPC console. In the navigation pane, choose Access Control > Security Groups and find the master node security group of the cluster.
  2. Search for the name of the security group that contains the cluster name and -cce-control-. The name of a master node security group is in the format of cluster-name-cce-control-random-ID.
  3. Check whether the security group rules have been changed. For details about security groups, see How Can I Configure a Security Group Rule for a Cluster?
• The node security group rules must contain a policy that allows the communication between the master nodes and the worker nodes.
  • Check whether such a security group policy is present.
  • When adding a node to the cluster, add the security group rules listed in Table 2 to the cluster-name-cce-control-random-ID security group to ensure the availability of the added node. This is necessary if a secondary CIDR block is added to the VPC of the node subnet and the subnet is in the secondary CIDR block. However, if a secondary CIDR block has already been added to the VPC during cluster creation, this step is not required.
  • For details about security groups, see How Can I Configure a Security Group Rule for a Cluster?

    Table 2 Security group rules to be added

    Protocol and Port	Type	Source IP Address
    TCP port 8445	IPv4	The new secondary CIDR block where the subnet is in
    TCP port 9443	IPv4	The new secondary CIDR block where the subnet is in
    TCP port 5444	IPv4	The new secondary CIDR block where the subnet is in

• By default, a 100-GiB data disk is attached to a node for runtime purposes. You have the option to attach additional data disks to the node if needed. If the data disk is detached or damaged, the runtime becomes abnormal and the node becomes unavailable.

• You need to check whether the data disks of the node are detached from it. If they are, you are advised to create a node and delete the unavailable node. (To minimize risks, you are not advised to perform operations on the CCE nodes through the ECS console.)

• Check the kubelet.

  Log in to the target node, run the following command on it, and check the kubelet:

  systemctl status kubelet
  
  

  The following shows an example of the expected output.

  

• View the kubelet logs.

  Log in to the target node, run the following command on it, and check the kubelet:

  journalctl -u kubelet
  
  

• Docker
  • Check the Docker runtime.

    Log in to the target node, run the following command on it, and check the Docker process:

    systemctl status docker
    
    

    The following shows an example of the expected output.

    

  • View the Docker logs.

    Log in to the target node, run the following command on it, and check the Docker logs:

    journalctl -u docker
    
    

• containerd
  • Check the containerd runtime.

    Log in to the target node, run the following command on it, and check the containerd process:

    systemctl status containerd
    
    

    The following shows an example of the expected output.

    

  • View the containerd logs.

    Log in to the target node, run the following command on it, and check the containerd logs:

    journalctl -u containerd
    
    

• NTP
  • Check whether the NTP is normal.

    Log in to the target node, run the following command on it, and check the chronyd process:

    systemctl status chronyd
    
    

    The following shows an example of the expected output.

    

  • View the NTP logs.

    Log in to the target node, run the following command on it, and check the NTP logs:

    journalctl -u chronyd
    
    

• Log in to the target node and check whether any domain name resolution failure is recorded in /var/log/cloud-init-output.log:

  cat /var/log/cloud-init-output.log | grep resolv
  
  

  If information similar to the following is displayed, the domain name cannot be resolved:

  Could not resolve host: xxx ; Unknown error
  
  

• Ping the domain name that cannot be resolved on the node:

  ping xxx
  
  

  If the domain name cannot be pinged, the DNS cannot resolve the IP address. You need to verify if the DNS address in the /etc/resolv.conf file matches the configuration on the VPC subnet. In most cases, the DNS address in the file is improperly configured, leading to the inability to resolve domain names.

1. Check the maximum number of PIDs on the node and the processes that use the most PIDs:

   sysctl kernel.pid_max # Check the maximum number of PIDs.
   ps -eLf|awk '{print $2}' | sort -rn| head -n 1 #: Check the processes that use the most PIDs on the node.
   
   

2. Check the top five processes that use the most PIDs:

   ps -elT | awk '{print $4}' | sort | uniq -c | sort -k1 -g | tail -5
   
   

   The following shows an example of the expected output:

   17 1211619
   18 3739112
   18 5299
   24 964
   25 3739756
   
   

   The first column shows the number of PIDs used by each process, while the second column displays the current process IDs. You can locate the process and associated pods with the given process ID, analyze the reason for excessive PID usage, and optimize the relevant code accordingly.

3. Reduce the load of the node.
4. To restart the node, go to the ECS console and restart it. (Be cautious when restarting the node because it may cause interruptions to your services.)

• If the number of available memory resources on a node is lower than the specified value of memory.available, the MemoryPressure of the node will be set to True, resulting in the eviction of pods running on that node. For details about node eviction, see Node-pressure Eviction.
• If the memory of a node is not enough, the following message will show:
  • The value of MemoryPressure becomes True.
  • When some pods on the node are evicted:
    • You can see "The node was low on resource: memory" in the events of the evicted pods.
    • You can see "attempting to reclaim memory" in the node events.
  • The system OOM may behave abnormally. If such an error occurs, you can see "System OOM" in the node events.

• Check the node memory usage through the node monitoring data, identify the time when the exception occurs, and verify if there is any memory leak in the processes on the node. For details, see Checking the Node Monitoring Data.
• Reduce the load of the node.
• To restart the node, go to the ECS console and restart it. (Be cautious when restarting the node because it may cause interruptions to your services.)

The root file system, image file system, or container file system on the node is consuming excessive disk space and inodes, surpassing the eviction threshold. This leads to the nodefs.available, nodefs.inodesFree, imagefs.available, or imagefs.inodesFree metric met the eviction threshold, causing disk pressure. The table below shows the default values for these parameters.

• If the available disk space on the node is less than the value of imagefs.available, the value of DiskPressure of the node becomes True.
• If the available disk space is less than the value of nodefs.available, all pods on the node will be evicted. For details about node eviction, see Node-pressure Eviction.
• If the disk space on the node is insufficient, the following message will show:
  • The value of DiskPressure becomes True.
  • If the disk space remains insufficient to meet the healthy threshold (defaulted at 80%) even after the image reclaim policy is triggered, you can see "failed to garbage collect required amount of images" in the node events.
  • When some pods on the node are evicted:
    • You can see "The node was low on resource: [DiskPressure]" in the events of the evicted pods.
    • You can see "attempting to reclaim ephemeral-storage" or "attempting to reclaim nodefs" in the node events.

• View the node disk usage through the node monitoring data, identify the time when the exception occurs, and check if processes on the node are consuming excessive disk space. For details, see Checking the Node Monitoring Data.
• If a large number of files are not deleted from the node disks, delete these files.
• Restrict the ephemeral-storage configurations of the pods based on service requirements.
• Use cloud storage services instead of hostPath volumes.
• Expand the capacity of the node disks.
• Reduce the load of the node.

1. Log in to the abnormal node and restart kubelet: (Restarting kubelet does not affect the running containers.)

   systemctl restart kubelet
   
   

2. Check whether the kubelet status becomes normal:

   systemctl status kubelet
   
   

3. If the kubelet status is still abnormal after the restart, log in to the node and view the kubelet logs:

   journalctl -u kubelet
   
   

   • If there are error messages in the logs, find the cause by looking for specific keywords associated with the error.
   • If there is an issue with the kubelet configuration, find the node pool that the node belongs to, click Manage in the Operation column, and make changes to the kubelet configuration.

• Docker
  • Docker is inactive.
  • Docker is active and running, but the node is experiencing issues and not functioning properly, resulting in abnormal behavior. In this case, the docker ps or docker exec command fails to be executed.
  • The value of RuntimeOffline of the node becomes True.
• containerd
  • containerd is inactive.
  • The value of RuntimeOffline of the node becomes True.

1. Log in to the abnormal node.
2. Restart the runtime:

   # Docker
   systemctl restart docker
   # Containerd
   systemctl restart containerd
   
   

3. After the command is executed, check whether the running status is normal.

   # Docker
   systemctl status docker
   # Containerd
   systemctl status containerd
   
   

4. If the component status is still abnormal after the restart, check the component logs:

   # Docker
   journalctl -u docker
   # containerd
   journalctl -u containerd
   
   

• chronyd is inactive.
• The value of NTPProblem becomes True.

1. Log in to the abnormal node and restart chronyd:

   systemctl restart chronyd
   
   

2. After the restart, check whether the chronyd status becomes normal:

   systemctl status chronyd
   
   

3. If the chronyd status is still abnormal after the restart, log in to the node and view the chronyd logs:

   journalctl -u chronyd
   
   

1. Check the time when the node was restarted:

   last reboot
   
   

   The following shows an example of the expected output.

   

2. View the node monitoring data and locate the abnormal resource based on the restart time of the node. For details, see Checking the Node Monitoring Data.
3. Check the kernel logs and locate the fault based on the restart time.

• The node cannot be logged in to.
• The node is in the Unknown state.

• If you cannot log in to the node, take the following steps to locate the fault:
  • Check whether the node is running on the ECS console.
  • Check whether the fault is caused by the execution failure of Cloud-Init of the ECS. For details, see Verifying Whether the ECS Can Be Logged In.
  • Check the security group configuration of the node. For details, see Checking the Node Security Group.
• If the network load of the node is too high, perform the following operations:
  • View the node networking through the node monitoring data and check whether the pods on the node are consuming excessive network bandwidth.
  • Use network policies to control network traffic of the pods on the node.

• The node is in the NotReady state.
• You can see the following information in the kubelet logs:

  skipping pod synchronization - PLEG is not healthy: pleg was last seen active 3m17.028393648s ago; threshold is 3m0s.
  
  

• Restart Docker or containerd and kubelet in sequence and then check whether the node is restored.
• If the node is not restored after the restart of the key components, restart the node. (Be cautious when restarting the node because it may cause interruptions to your services.)

If there are not enough scheduling resources available on the node, pod scheduling will fail and the following error information will be displayed: (Only errors related to common resources are listed.)

• Insufficient CPUs in a cluster: 0/2 nodes are available: 2 Insufficient cpu
• Insufficient memory in a cluster: 0/2 nodes are available: 2 Insufficient memory
• Insufficient temporary storage space in a cluster: 0/2 nodes are available: 2 Insufficient ephemeral-storage

• Whether the CPUs of a node are insufficient: Total CPUs requested by a pod > (Total allocatable CPUs on the node - Total CPUs that have been allocated to the pods on the node)
• Whether the memory of a node is insufficient: Total memory requested by a > (Total allocatable memory on the node - Total memory that has been allocated to the pods on the node)
• Whether the temporary storage space of a node is insufficient: Temporary storage space requested by a pod > (Total allocatable temporary storage space on the node - Total temporary storage space that has been allocated to the pods on the node)

kubectl describe node $nodeName

Allocatable:
  cpu:                1930m
  ephemeral-storage:  94576560382
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  localssd:           0
  localvolume:        0
  memory:             2511096Ki
pods:               20
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1255m (65%)   4600m (238%)
  memory             1945Mi (79%)  3876Mi (158%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-1Gi      0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
  localssd           0             0
  localvolume        0             0

• Allocatable: specifies the total number of allocatable resources like CPUs, memory, and temporary storage on a node.
• Allocated resources: specifies the total number of resources like CPUs, memory, and temporary storage that have been allocated to the pods on a node.

If the resources on a node are not enough for pod scheduling, reduce the node load through either of the following ways:

• Delete unnecessary pods.
• Restrict the resource configurations of pods based on service requirements.
• Add more nodes to the cluster.

1. If there are other normal nodes in the cluster, run the ping command to check the network connectivity between containers on different nodes.

   Create a container for testing. In the following example, {node_ip} indicates the IP address of the abnormal node.

   kind: Pod
   apiVersion: v1
   metadata:
     name: nginx
     namespace: default
   spec:
     containers:
       - name: container-1
         image: nginx:latest
         imagePullPolicy: IfNotPresent
     imagePullSecrets:
       - name: default-secret
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
             - matchExpressions:
                 - key: kubernetes.io/hostname
                   operator: In
                   values:
                     - {node_ip}
     schedulerName: default-scheduler
     tolerations:
       - key: node.kubernetes.io/route-unreachabe
         operator: Exists
         effect: NoSchedule
   
   

   After the container is started, log in to another normal node and ping the IP address of the container. If the communication is abnormal, go to the next step.

2. On the Route Tables page of the VPC console, check whether the node route has been added, whether the next hop type is cloud container, and whether the next hop is the node name. If the node route is present in the route table but the node is still unable to connect to the network, it suggests a potential problem with the underlying network. In such situations, submit a service ticket to the networking team for assistance.
3. If the fault persists, submit a service ticket to CCE for troubleshooting.

kubectl describe node {nodeName}

• Reset the faulty node.
• Delete unnecessary pods.
• Restrict the resource configurations of pods based on service requirements.
• Add more nodes to the cluster.