Configuring Read/Write Splitting Between Two Elasticsearch Clusters

Scenario

1. Data writes: Users send write requests, and the leader cluster handles these requests.
2. Data synchronization.

   The leader synchronizes data changes to the follower through a REST API. Two synchronization methods are supported:

   • Exact match: synchronization of a specified index
   • Match by wildcard: batch synchronization of indexes matching a wildcard expression

3. Query processing: Users send query requests. The follower cluster handles these requests and returns the results.

Figure 1 How read/write splitting works

• Read and write split when both clusters are available (left): The leader handles writes, and the follower handles queries.
• Leader-to-follower switchover when the leader fails (right): If the leader becomes unavailable, the follower automatically upgrades to leader to ensure service continuity. For details, see Switching the Roles of the Leader and Follower Clusters.

Constraints

• Only Elasticsearch 7.6.2 and Elasticsearch 7.10.2 clusters support read/write splitting.
• The versions of the leader and follower clusters must be kept consistent, or errors may occur.

Prerequisites

Logging In to Kibana

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

Connecting the Leader and Follower Clusters

1. Run the following command to configure information about the leader cluster in the follower cluster:

   PUT /_cluster/settings
   {
     "persistent" : {
       "cluster" : {
         "remote.rest" : {
           "leader1" : {
             "seeds" : [
               "http://10.0.0.1:9200",
               "http://10.0.0.2:9200",
               "http://10.0.0.3:9200"
             ] ,
               "username": "elastic",
               "password": "*****"
           }
         }
       }
     }
   }
   
   

   Table 1 Request body parameters

   Parameter	Description
   leader1	Name of the leader cluster configuration task, which is user-defined and will be used for configuring read/write splitting later.
   seeds	Address for accessing the leader cluster. When HTTPS is enabled for the cluster, the URL schema must use HTTPS.
   username	Username of the leader cluster. This parameter is required only when security mode is enabled for the leader cluster.
   password	Password of the leader cluster. This parameter is required only when security mode is enabled for the leader cluster.

   Example response:

   {
     "acknowledged" : true,  //Whether the operation is successful
     "persistent" : {
       "cluster" : {
         "remote" : {
           "rest" : {
             "leader1" : {
               "seeds" : [
               "http://10.0.0.1:9200",
               "http://10.0.0.2:9200",
               "http://10.0.0.3:9200"
               ] ,
               "username": "elastic", 
                "password": "*****"
             }
           }
         }
       }
     },
     "transient" : { }
   }
   
   

2. After the configuration is complete, run the following command in the follower cluster to check the connection between the follower and leader clusters:

   GET _remote/rest/info
   
   

   Example response:

   {
     "leader1" : {
       "connected" : true  //The two clusters are connected.
     }
   }
   
   

Index Synchronization

Synchronizing Specified Indexes

• Run the following command in the follower cluster to synchronize a single index from the leader cluster to the follower cluster without modifying index settings:

  PUT start_remote_sync
  {
    "remote_cluster": "leader1",
    "remote_index": "data1_leader",
    "local_index": "data1_follower"
  }
  
  

• Run the following command in the follower cluster to synchronize a single index from the leader cluster to the follower cluster while modifying some of the index settings—enabling synchronization of index settings:

  PUT start_remote_sync
  {
    "remote_cluster": "leader1",
    "remote_index": "data1_leader",
    "local_index": "data1_follower",
    "settings": {
      "number_of_replicas": 4
    },
    "settings_sync_enable": true,
    "settings_sync_patterns": ["*"],
    "settings_sync_exclude_patterns": ["index.routing.allocation.*"],
    "alias_sync_enable": true,
    "state_sync_enable": true
  }
  
  

  Note

  The following index configuration items cannot be modified: number_of_shards, version.created, uuid, creation_date, and soft_deletes.enabled.

Matching Index Synchronization

• Run the following command in the follower cluster to create a pattern-matching index synchronization policy, which synchronizes matched indexes from the leader cluster to the follower cluster:

  PUT auto_sync/pattern/${PATTERN}
  {
   "remote_cluster": "leader1",
   "remote_index_patterns": "log*",
   "local_index_pattern": "{{remote_index}}-sync",
   "apply_exist_index": true
  }
  
  

• Run the following command in the follower cluster to create a pattern-matching index synchronization policy, which synchronizes matched indexes from the leader cluster to the follower cluster, with some of the index settings modified—enabling synchronization of index settings:

  PUT auto_sync/pattern/${PATTERN}
  {
   "remote_cluster": "leader1",
   "remote_index_patterns": "log*",
   "local_index_pattern": "{{remote_index}}-sync",
   "apply_exist_index": true,
   "settings": {
     "number_of_replicas": 4
   },
   "settings_sync_enable": true,
   "settings_sync_patterns": ["*"],
   "settings_sync_exclude_patterns": ["index.routing.allocation.*"],
   "alias_sync_enable": true,
   "state_sync_enable": true
  }
  
  

  Note

  The following index configuration items cannot be modified: number_of_shards, version.created, uuid, creation_date, and soft_deletes.enabled.

Stopping Index Synchronization

PUT log*/stop_remote_sync

Querying and Deleting Created Patterns

1. Run the following command in the follower cluster to query created patterns:
   • Query the list of patterns.

     GET auto_sync/pattern
     
     

   • Query a specified pattern by name.

     GET auto_sync/pattern/{PATTERN}
     
     

   The following is an example of the response:

   {
     "patterns" : [
       {
         "name" : "pattern1",
         "pattern" : {
           "remote_cluster" : "leader",
           "remote_index_patterns" : [
             "log*"
           ],
           "local_index_pattern" : "{{remote_index}}-sync",
           "settings" : { }
         }
       }
     ]
   }
   
   

2. Run the following command in the follower cluster to delete a specified pattern by name:

   DELETE auto_sync/pattern/{PATTERN}
   
   

Enabling Forcible Synchronization

PUT _cluster/settings
{
  "persistent": {
    "remote_sync.force_synchronize": true
  }
}

Changing the Synchronization Interval

PUT {index_name}/_settings
{
  "index.remote_sync.sync_interval": "2s"
}

Changing the Synchronization Speed

PUT _cluster/settings
{
  "persistent": {
    "remote_sync.chunk_size": "2MB",
    "remote_sync.max_concurrent_file_chunks": 20,
    "remote_sync.max_bytes_per_sec": "100MB"
  }
}

Querying Index Synchronization Status

GET {index_name}/sync_stats

{
  "indices" : {
    "data1_follower" : {
      "shards" : {
        "0" : [
          {
            "primary" : false,             // Whether it is a primary shard
            "total_synced_times" : 27,    // Total synchronization times
            "total_empty_times" : 25,     // Total number of times when no data is synchronized between the leader and follower clusters because they have identical shards and data
            "total_synced_files" : 4,    // Number of synchronized files
            "total_synced_bytes" : 3580, // Total size of synchronized files
            "total_paused_nanos" : 0,   //Duration of synchronization pauses due to traffic throttling
            "total_paused_times" : 0,   //Number of synchronization pauses due to traffic throttling
            "current" : {
              "files_count" : 0, //Number of files that are being synchronized
              "finished_files_count" : 0, //Number of files that have been synchronized
              "bytes" : 0, //Size of files that are being synchronized
              "finished_bytes" : 0 //Size of files that have been synchronized
            }
          },
          {
            "primary" : true,             // Whether it is a primary shard
            "total_synced_times" : 28,    // Total synchronization times
            "total_empty_times" : 26,     // Total number of times when no data is synchronized between the leader and follower clusters because they have identical shards and data
            "total_synced_files": 20,    // Number of synchronized files
            "total_synced_bytes": 17547, // Total size of synchronized files
            "total_paused_nanos" : 0,     //Duration of synchronization pauses due to traffic throttling
            "total_paused_times" : 0,     //Number of synchronization pauses due to traffic throttling
            "current" : {
              "files_count" : 0,          //Number of files that are being synchronized
              "finished_files_count" : 0, //Number of files that have been synchronized
              "bytes" : 0,                //Size of files that are being synchronized
              "finished_bytes" : 0        //Size of files that have been synchronized
            }
          }
        ]
      }
    }
  }
}

Switching the Roles of the Leader and Follower Clusters

1. Determine the index synchronization method between the leader and follower clusters. Check whether pattern-matching index synchronization policies have been configured in the follower cluster. For the command to use, see Querying and Deleting Created Patterns.
   • If there are no such policies, synchronization is performed for specified indexes between the leader and follower clusters. In this case, go to 3.
   • If there are such policies, index synchronization between the leader and follower clusters is based on index patterns. In this case, go to 2.
2. Delete pattern-matching index synchronization policies in the follower cluster. For the command to use, see Querying and Deleting Created Patterns.
3. Perform Stopping Index Synchronization in the follower cluster. Then redirect read and write traffic to it. If the leader and follower clusters synchronize indexes based on index patterns, use a wildcard to match indexes when running the command that stops index synchronization.
4. After the leader cluster recovers, configure information about the follower cluster in the leader cluster, and connect the leader and follower clusters again. For details, see Connecting the Leader and Follower Clusters.
5. Under the leader cluster, perform Index Synchronization to synchronize data from the follower cluster to the leader cluster, and then perform a leader/follower switchover to switch back.