CCE Cluster Autoscaler

Introduction

How the Add-on Works

• Auto scale-out

  You can choose either of the following methods:

  • If a pod cannot be scheduled due to insufficient resources of worker nodes, CCE will add more nodes to the cluster. The new nodes have the same resource quotas as those configured for the node pools that the new nodes are in.

    Auto scale-out will be performed when:

    • Node resources are insufficient.
    • No node affinity policy is set in the scheduling configurations of the pod. If the pod is configured affinity for a node, the system will not automatically add more nodes in the cluster. For details about how to configure node affinity policies, see Configuring Node Affinity Scheduling (nodeAffinity).

  • When the cluster meets the node scaling policy, cluster scale-out is also triggered. For details, see Creating a Node Auto Scaling Policy.
  Note

  The add-on follows the "No Less, No More" policy. For example, if three cores are required for creating a pod and the system supports four-core and eight-core nodes, Autoscaler will preferentially create a four-core node.

• Auto scale-in

  If a node's pre-allocated CPUs and memory remain below the scale-in threshold for a sustained period (10 minutes by default), the cluster initiates a scale-in operation, automatically deleting the underutilized node. However, a node cannot be deleted from a cluster if the following pods exist:

  • Pods that do not meet specific requirements set in Pod Disruption Budgets (PodDisruptionBudget)
  • Pods that cannot be scheduled to other nodes due to constraints such as affinity and anti-affinity policies
  • Pods that have the cluster-autoscaler.kubernetes.io/safe-to-evict: 'false' annotation
  • Pods (except those created by DaemonSets in the kube-system namespace) that exist in the kube-system namespace
  • Pods that are not created by the controllers (such as Deployments, ReplicaSets, jobs, and StatefulSets)
  Note

  • When a node meets the scale-in conditions, Autoscaler adds the DeletionCandidateOfClusterAutoscaler taint to the node in advance to prevent pods from being scheduled to the node. After the CCE Cluster Autoscaler add-on is uninstalled, if the taint still exists on the node, manually delete it.
  • To ensure system stability and efficient resource utilization, CCE Cluster Autoscaler uses a conservative policy. Nodes that are not entirely idle are drained one at a time. When these nodes host pods configured with graceful termination, the draining process can be prolonged. As a result, the overall scale-in process may take more time to complete.

Notes and Constraints

• There must be enough resources in the cluster during the add-on installation.
• This add-on supports VM nodes and does not support PM nodes.
• The default node pool does not support auto scaling. For details, see Description of DefaultPool.
• Node scale-in will cause PVC/PV data loss for the local PVs associated with the node. These PVCs and PVs cannot be restored or used again. In a node scale-in, a pod that uses the local PV will be evicted from the node. A new pod will be created, but it remains in a pending state because the label of the PVC bound to it conflicts with the node label.
• When CCE Cluster Autoscaler is used, some taints or annotations may affect auto scaling. Therefore, do not use the following taints or annotations in clusters:
  • ignore-taint.cluster-autoscaler.kubernetes.io: The taint works on nodes. Kubernetes-native Autoscaler supports protection against abnormal scale-outs and periodically evaluates the proportion of available nodes in the cluster. When the proportion of non-ready nodes exceeds 45%, protection will be triggered. In this case, all nodes with this taint in the cluster are filtered out from the Autoscaler template and recorded as non-ready nodes, which affect cluster scaling.
  • cluster-autoscaler.kubernetes.io/enable-ds-eviction: The annotation works on pods, which determines whether DaemonSet pods can be evicted by Autoscaler. For details, see Well-Known Labels, Annotations and Taints.
• CCE Cluster Autoscaler earlier than 1.27.205, 1.28.172, 1.29.134, 1.30.100, 1.32.38, 1.33.31, and 1.31.62 does not support deployment of both Snt9 and CPU nodes in the same cluster. If both kinds of nodes run in the same cluster, Snt9 nodes will be considered as non-ready nodes. When the proportion of non-ready nodes exceeds 45% of the total, the protection system is triggered. This will affect the scaling of CPU node pools in the cluster.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Add-ons. Locate CCE Cluster Autoscaler on the right and click Install.
3. On the Install Add-on page, configure the specifications as needed.

   There are three types of preset specifications based on the cluster scale. You can select one as required. The system will configure the number of pods and resource quotas for the add-on based on the selected preset specifications. You can see the configurations on the console.

   If your cluster is large and the preset specifications do not meet your needs, you can customize the resource specifications and estimate the number of pods in the cluster to better determine the memory usage of the add-on. The recommended memory request and limit in typical large clusters can be calculated as follows:

   • Memory request = Number of pods × Size of the pod YAML files (KB)/10000 × 0.28 GiB + 1 GiB
   • Memory limit =Memory request + 2 GiB

   For example, if there are 20,000 pods and the YAML file size of each pod is 10 KB, the memory request would be 6.6 GiB (2 × 10 × 0.28 GiB + 1 GiB) and the memory limit would be 8.6 GiB (6.6 GiB + 2 GiB). (The calculated values may differ from the recommendations listed in Table 1. You can refer to that table or use these formulas.)

   Table 1 Recommended add-on memory size in large-scale scenarios

   Number of Pods (10 KB for Each Pod YAML)	Recommended Memory Request	Recommended Memory Limit
   10000	4 GiB	6 GiB
   30000	8 GiB	10 GiB
   50000	16 GiB	18 GiB
   80000	24 GiB	26 GiB
   100000	28 GiB	30 GiB

4. Configure deployment policies for the add-on pods.
   Note

   • Scheduling policies do not take effect on add-on pods of the DaemonSet type.
   • When configuring multi-AZ deployment or node affinity, ensure that there are nodes meeting the scheduling policy and that resources are sufficient in the cluster. Otherwise, the add-on cannot run.

   Table 2 Configurations for add-on scheduling

   Parameter	Description
   Multi-AZ Deployment	Preferred: Deployment pods of the add-on will be preferentially scheduled to nodes in different AZs. If all the nodes in the cluster are deployed in the same AZ, the pods will be scheduled to different nodes in that AZ. Equivalent mode: Deployment pods of the add-on are evenly scheduled to the nodes in the cluster in each AZ. If a new AZ is added, you are advised to increase add-on pods for cross-AZ HA deployment. With the Equivalent multi-AZ deployment, the difference between the number of add-on pods in different AZs will be less than or equal to 1. If resources in one of the AZs are insufficient, pods cannot be scheduled to that AZ. Forcible: Deployment pods of the add-on are forcibly scheduled to nodes in different AZs. There can be at most one pod in each AZ. If nodes in a cluster are not in different AZs, some add-on pods cannot run properly. If a node is faulty, add-on pods on it may fail to be migrated.
   Node Affinity	Not configured: Node affinity is disabled for the add-on. Specify node: Specify the nodes where the add-on is deployed. If you do not specify the nodes, the add-on will be randomly scheduled based on the default cluster scheduling policy. Specify node pool: Specify the node pool where the add-on is deployed. If you do not specify the node pools, the add-on will be randomly scheduled based on the default cluster scheduling policy. Customize affinity: Enter the labels of the nodes where the add-on is to be deployed for more flexible scheduling policies. If you do not specify node labels, the add-on will be randomly scheduled based on the default cluster scheduling policy. If multiple custom affinity policies are configured, ensure that there are nodes that meet all the affinity policies in the cluster. Otherwise, the add-on cannot run.
   Toleration	Using both taints and tolerations allows (not forcibly) the add-on Deployment to be scheduled to a node with the matching taints, and controls the Deployment eviction policies after the node where the Deployment is located is tainted. The add-on adds the default tolerance policy for the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints, respectively. The tolerance time window is 60s. For details, see Configuring Tolerance Policies.

5. After the configuration is complete, click Install.

Components