Upgrading the Version of an OpenSearch Cluster

Upgrade Impact

• Upgrade process and performance impact

  The nodes of a cluster are upgraded one at a time to ensure service continuity. However, data migration during the upgrade consumes I/O performance, and taking individual nodes offline still has some impact on the overall cluster performance. To minimize this impact, it is advisable to adjust the data migration speed based on the cluster's traffic cycle: increase the data migration speed during off-peak hours to accelerate the upgrade, and decrease it during peak hours to ensure optimal cluster performance.

  The data migration speed is determined by the indices.recovery.max_bytes_per_sec parameter. The default value is Number of CPU cores x 32, in MB (per second). You can change this parameter as needed within the range of 40 MB to 1000 MB.

  PUT /_cluster/settings  
  {  
    "transient": {  
      "indices.recovery.max_bytes_per_sec": "1000mb"  
    }  
  }
  
  

• Node replacement and request handling

  During the upgrade, replacing a node may lead to request failures. To avoid this problem, you are advised to use the VPC Endpoint Service or a dedicated load balancer to handle cluster access requests, and also add a request retry logic to the client code. Additionally, you are advised to perform the upgrade during off-peak hours to further minimize potential impact.

  For details about how to configure the VPC Endpoint Service or a dedicated load balancer, see Configuring VPC Endpoint Service for an OpenSearch Cluster and Configuring a Dedicated Load Balancer for an OpenSearch Cluster.

• Rebuilding OpenSearch Dashboards and Cerebro

  OpenSearch Dashboards and Cerebro will be rebuilt during the upgrade, making them temporarily unavailable. Additionally, due to cross-version compatibility issues, OpenSearch Dashboards may become unavailable during the upgrade. These problems will go away once the upgrade is completed.

• Upgrade task management

  Once started, an upgrade task cannot be stopped until it succeeds or fails. An upgrade failure affects individual nodes only. It will not affect services as long as there are replicas for such nodes.

Constraints

• A maximum of 20 clusters can be upgraded at the same time. You are advised to perform the upgrade during off-peak hours.
• Clusters that have ongoing tasks cannot be upgraded.

Pre-Upgrade Check

Creating an Upgrade Task

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters to display the cluster list.
3. Click the target cluster name. The cluster information page is displayed.
4. In the navigation pane on the left, choose Cluster Snapshots, and create snapshots to back up all index data. For details, see Manually Creating a Snapshot.

   When creating an upgrade task, you can choose to check whether the full index data has been backed up using snapshots. This helps to prevent data loss in case of an upgrade failure.

5. In the navigation pane on the left, choose Version Upgrade.
6. On the displayed page, set upgrade parameters.

   Table 2 Upgrade parameters

   Parameter	Description
   Upgrade Type	Same-version upgrade: upgrade kernel patches to the latest images within the current cluster version. Cross-version upgrade: upgrade a cluster to the latest image of the target version.
   Target Image	Image of the target version. After you select an image, the image name and target version details are displayed below. The supported target versions are displayed in the drop-down list of Target Image. If no target image is available, possible causes are as follows: The current cluster is of the latest version. The current cluster is created before 2023 and has vector indexes. The new version images have not been added at the current region. The current cluster does not support the upgrade type you have selected.

7. After setting the parameters, click Submit. Determine whether to enable Check full index snapshot and Perform cluster load detection and click OK.

   If a cluster is overloaded, the upgrade task may suspend or fail. Enabling Cluster load detection can effectively avoid failures.

   If any of the following situations occurs during the detection, wait or reduce the load. If you urgently need to upgrade the version and you have understood the upgrade failure risks, you can disable the Cluster load detection function. The cluster load check items are as follows:

   • nodes.thread_pool.search.queue < 1000: Check whether the maximum number of search queues is less than 1000.
   • nodes.thread_pool.write.queue < 200: Check whether the maximum number of write queues is less than 200.
   • nodes.process.cpu.percent < 90: Check whether the maximum CPU usage is less than 90%.
   • nodes.os.cpu.load_average/Number of CPU cores < 80%: Check whether the ratio of the maximum load to the number of CPU cores is less than 80%.
8. View the upgrade task in the task list. If the task status is Running, you can expand the task list and click View Progress to view the upgrade progress.

   If the task status is Failed, you can retry or terminate the task.

   • Retry the task: Click Retry in the Operation column.
   • Terminate the task: Click Terminate in the Operation column.
     Notice

     • Same version upgrade: If the upgrade task status is Failed, you can terminate the upgrade task.
     • Cross version upgrade: You can stop an upgrade task only when the task status is Failed and no node has been upgraded.

     After an upgrade task is terminated, the Task Status of the cluster is rolled back to the status before the upgrade, and other tasks in the cluster are not affected.