If a pod is in the Pending state and the events contain the information that indicates a pod scheduling failure, you can locate the cause based on the events. For details about how to view events, see How Can I Locate the Root Cause If a Workload Is Abnormal?
Determine the cause based on the events, as listed in Table 1.
Event | Cause and Solution |
|---|---|
no nodes available to schedule pods. | There are not any available nodes in the cluster. |
0/2 nodes are available: 2 Insufficient cpu. 0/2 nodes are available: 2 Insufficient memory. | The resources (CPUs and memory) on the node are insufficient. Check Item 2: Whether Node Resources (CPUs and Memory) Are Sufficient |
0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity rules, 1 node(s) didn't match pod affinity/anti-affinity. | The node and pod affinity configurations are mutually exclusive. No node meets the pod requirements. Check Item 3: Affinity and Anti-Affinity Configuration of the Workload |
0/2 nodes are available: 2 node(s) had volume node affinity conflict. | The EVS volume mounted to the pod and the node are not in the same AZ. Check Item 4: Whether the Workload's Volume and the Node Are in the Same AZ |
0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. | There are some taints on the node, and the pod cannot tolerate these taints. |
0/7 nodes are available: 7 Insufficient ephemeral-storage. | The ephemeral storage space on the node is insufficient. |
0/1 nodes are available: 1 everest driver not found at node | The everest-csi-driver on the node is not in the running state. Check Item 7: Whether the CCE Container Storage (Everest) Add-on Works Properly |
Failed to create pod sandbox: ... Create more free space in thin pool or use dm.min_free_space option to change behavior | The node thin pool space is insufficient. |
0/1 nodes are available: 1 Too many pods. | The number of pods scheduled to the node exceeds the maximum allowed. Check Item 9: Whether the Node Has Too Many Pods Scheduled onto It |
UnexpectedAdmissionError Allocate failed due to not enough cpus available to satisfy request, which is unexpected. | The kubelet static CPU pinning is abnormal due to a known community issue. Check Item 10: Whether the Static CPU Pinning of kubelet Is Abnormal |
You can log in to the CCE console and check whether the node status is Available. You can also use the following command to check whether the node status is Ready:
$ kubectl get nodeNAME STATUS ROLES AGE VERSION192.168.0.37 Ready <none> 21d v1.19.10-r1.0.0-source-121-gb9675686c54267192.168.0.71 Ready <none> 21d v1.19.10-r1.0.0-source-121-gb9675686c54267
If the status of all nodes is Not Ready, it means that there are no available nodes in the cluster.
Solution
0/2 nodes are available: 2 Insufficient cpu. indicates that the CPUs are insufficient.
0/2 nodes are available: 2 Insufficient memory. indicates that the memory is insufficient.
If the resources requested by the pod exceed the allocatable resources on the node where the pod will run, the pod scheduling onto the node will definitely fail due to insufficient node resources.
If there are fewer allocatable resources on the node than the resources that a pod requests, the pod scheduling will fail.
Solution
Add more nodes to the cluster. Scale-out is the common solution to insufficient resources.
Inappropriate affinity policies will cause the pod scheduling to fail.
For example, an anti-affinity policy is configured for workload 1 and workload 2. They run on node 1 and node 2, respectively.
If you try to configure an affinity policy for workload 3 and workload 2 and then deploy workload 3 on a node different from one hosting workload 2, such as node 1, it will cause a conflict and lead to the workload deployment failure.
0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity rules, 1 node(s) didn't match pod affinity/anti-affinity.
Solution
No nodes are available that match all of the following predicates: MatchNode Selector, NodeNotSupportsContainer
If the value is false, the pod scheduling will fail.
0/2 nodes are available: 2 node(s) had volume node affinity conflict. indicates that an affinity conflict occurs between the volume mounted to the pod and the host node. As a result, the pod scheduling fails.
This is because EVS disks cannot be attached to nodes in different AZs from the EVS disks. For example, a workload pod with an EVS volume that is in AZ 1 cannot be scheduled to a node in AZ 2.
The EVS volumes created on CCE have affinity settings by default, as shown below.
kind: PersistentVolumeapiVersion: v1metadata:name: pvc-c29bfac7-efa3-40e6-b8d6-229d8a5372acspec:...nodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: failure-domain.beta.kubernetes.io/zoneoperator: Invalues:-
Solution
In the AZ where the workload's node resides, create a volume. Alternatively, create an identical workload and select an automatically assigned cloud storage volume.
0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. indicates that there are some taints on the node, and the pod cannot tolerate these taints.
In this case, you can check the taints on the node. If information similar to the following is displayed, there are some taints on the node:
$ kubectl describe node 192.168.0.37Name: 192.168.0.37...Taints: key1=value1:NoSchedule...
In some cases, the system automatically adds a taint to a node. The built-in taints include:
Solution
To schedule the pod to the node, use either of the following methods:
apiVersion: v1kind: Podmetadata:name: nginxspec:containers:- name: nginximage: nginx:alpinetolerations:- key: "key1"operator: "Equal"value: "value1"effect: "NoSchedule"
0/7 nodes are available: 7 Insufficient ephemeral-storage. indicates that there are not enough ephemeral storage space on the node.
In this case, you can check whether the space of the ephemeral volume is limited by the pod. If the ephemeral volume space required by the application exceeds the existing capacity on the node, the application cannot be scheduled to that node. To solve this problem, change the space of the ephemeral volume or expand the disk capacity on the node.
apiVersion: v1kind: Podmetadata:name: frontendspec:containers:- name: appimage: images.my-company.example/app:v4resources:requests:ephemeral-storage: "2Gi"limits:ephemeral-storage: "4Gi"volumeMounts:- name: ephemeralmountPath: "/tmp"volumes:- name: ephemeralemptyDir: {}
To obtain the total capacity (Capacity) and available capacity (Allocatable) of the temporary volumes on the node, run the kubectl describe node command and check the memory request and limit of the allocated temporary volume on the node.
The following is an example of the output:
...Capacity:cpu: 4ephemeral-storage: 61607776Kihugepages-1Gi: 0hugepages-2Mi: 0localssd: 0localvolume: 0memory: 7614352Kipods: 40Allocatable:cpu: 3920mephemeral-storage: 56777726268hugepages-1Gi: 0hugepages-2Mi: 0localssd: 0localvolume: 0memory: 6180752Kipods: 40...Allocated resources:(Total limits may be over 100 percent, i.e., overcommitted.)Resource Requests Limits-------- -------- ------cpu 1605m (40%) 6530m (166%)memory 2625Mi (43%) 5612Mi (92%)ephemeral-storage 0 (0%) 0 (0%)hugepages-1Gi 0 (0%) 0 (0%)hugepages-2Mi 0 (0%) 0 (0%)localssd 0 0localvolume 0 0Events: <none>
0/1 nodes are available: 1 everest driver not found at node. indicates that everest-csi-driver of CCE Container Storage (Everest) is not started properly on the node.
In this case, you can check the daemon named everest-csi-driver in the kube-system namespace and check whether the pod is started properly. If it is not, delete the pod. The daemon will restart another pod.
A data disk dedicated for kubelet and the container runtime will be attached to a new node. If the data disk space is insufficient, the pod cannot be created on the node.
Solution 1: Clearing images
Perform the following operations to clear unused images: Do not delete system images, such as the cce-pause image. Otherwise, the pod creation may fail.
Solution 2: Expanding the disk capacity
To expand a disk's capacity, perform the following operations:
Only the storage capacity of EVS disks can be expanded. You need to perform the following operations to expand the capacity of logical volumes and file systems.
A data disk is divided depending on the container storage Rootfs:
0/1 nodes are available: 1 Too many pods. indicates excessive number of pods have been scheduled to the node.
When creating a node, configure Max. Pods in the Advanced Settings area to specify the maximum number of pods that can run properly on the node. The default value varies with the node flavor. You can change the value as needed.
On the Nodes page, obtain the Pods (Allocated/Total Available Addresses/Total) value of the node, and check whether the number of pods scheduled onto the node has reached the upper limit. If so, add nodes or change the maximum number of pods.
To change the maximum number of pods that can run on a node, do as follows:
If a pod has an init container with a CPU request that is different from the main container settings, and it is assigned a Guaranteed QoS class while the kubelet uses static CPU pinning (with cpuManagerPolicy set to static), the pod scheduling could fail, resulting in the error UnexpectedAdmissionError.
Community-related issue: https://github.com/kubernetes/kubernetes/issues/112228
Solution
Set the CPU request of the init container to a decimal value that matches the CPU limit and avoid using CPU pinning.
For example: the main container: {"limits":{"cpu":"7","memory":"60G"},"requests":{"cpu":"7","memory":"60G"}}; the init container: {"limits":{"cpu":"6.9","memory":"60G"},"requests":{"cpu":"6.9","memory":"60G"}}