Configuring Large Query Isolation for an Elasticsearch Cluster

Scenario

Large query isolation can effectively solve the following problems and improve the search performance of clusters:

• A small number of queries occupy large chunks of node heap memory, resulting in frequent Garbage Collection (GC) and even out of memory (OOM) exceptions.
• Frequent GC causes node disconnections. As a result, queries cannot get response and may fail.
• The CPU usage is high due to heavy query load, affecting online services.

Constraints

Logging In to Kibana

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

Enabling Large Query Isolation

Run the following commands to enable large query isolation and global query timeout:

PUT _cluster/settings
{
  "persistent": {
    "search.isolator.enabled": true,
    "search.isolator.time.enabled": true
  }
}

Configuring Large Query Isolation Thresholds

• Run the following commands to configure thresholds for defining a large query task:

  PUT _cluster/settings
  {
    "persistent": {
      "search.isolator.memory.task.limit": "50MB",
      "search.isolator.time.management": "10s"
    }
  }
  
  

  Table 2 Parameters for configuring large query isolation thresholds

  Parameter	Type	Description
  search.isolator.memory.task.limit	String	Threshold of the memory requested by a query task to perform aggregation or other operations. If the requested memory exceeds the threshold, the task will be isolated and observed. Value range: 0b to the maximum heap memory of a node Default value: 50MB
  search.isolator.time.management	String	Threshold of the duration of a query (started when cluster resources are used for query). If the duration of a query exceeds the threshold, it will be isolated and observed. Value range: ≥ 0ms Default value: 10s

• Configure the resource usage thresholds for triggering the isolation of query tasks.

  PUT _cluster/settings
  {
    "persistent": {
      "search.isolator.memory.pool.limit": "50%",
      "search.isolator.memory.heap.limit": "90%",
      "search.isolator.count.limit": 1000
    }
  }
  
  

  Table 3 Parameters for configuring isolation pool thresholds

  Parameter	Type	Description
  search.isolator.memory.pool.limit	String	Threshold of the heap memory percentage of the current node. If the total memory requested by large query tasks in the isolation pool exceeds the threshold, the interrupt control program will be triggered to cancel one of the tasks. Value range: 0.0 to 100.0% Default value: 50%
  search.isolator.memory.heap.limit	String	Heap memory threshold of the current node. If the heap memory of the node exceeds the threshold, the interrupt control program will be triggered to cancel a large query task in the isolation pool. Value range: 0.0 to 100.0% Default value: 90%
  search.isolator.count.limit	Integer	Threshold of the number of large query tasks in the current node isolation pool. If the number of observed query tasks exceeds the threshold, the interrupt control program will be triggered to stop accepting new large queries. New large query requests will be directly canceled. Value range: 10–50000 Default value: 1000

  Note

  In addition to search.isolator.memory.pool.limit and search.isolator.count.limit parameters, you can configure search.isolator.memory.task.limit and search.isolator.time.management to control the number of query tasks that enter the isolation pool.

• Configure a policy for selecting query tasks to pause in the isolation pool.

  PUT _cluster/settings
  {
    "persistent": {
      "search.isolator.strategy": "fair",
      "search.isolator.strategy.ratio": "0.5%"
    }
  }
  
  

  Table 4 Parameters for configuring a query cancelation policy

  Parameter	Type	Description
  search.isolator.strategy	String	Policy for selecting large queries when the interrupt control program is triggered. The selected queries will be interrupted. NOTE: The large query isolation pool is checked every second until the heap memory is within the safe range. Values: fair, mem-first, or time-first mem-first: The query task that uses the most heap memory in the isolation pool is interrupted. time-first: The query task that has been running for the longest time in the isolation pool is interrupted. fair: If the difference between the heap memory of shard queries is smaller than Maximum_heap_memory x search.isolator.strategy.ratio, the query that takes the longest time should be interrupted. Otherwise, the query that uses the most heap memory is interrupted. Default value: fair
  search.isolator.strategy.ratio	String	Threshold of the fair policy. This parameter takes effect only if search.isolator.strategy is set to fair. If the difference between the memory usage of large query tasks does not exceed the threshold, the query that takes the longest time should be interrupted. If the difference between the memory usage of large query tasks exceeds the threshold, the query that uses the most memory is interrupted. Value range: 0.0 to 100.0% Default value: 1%

Configuring the Global Query Timeout

Run the following command to set the global timeout of query tasks:

PUT _cluster/settings
{
  "persistent": {
    "search.isolator.time.limit": "120s"
  }
}

Configuring the Maximum Number of Log Records for Canceled Query Requests

Run the following command to set the maximum number of log records kept for canceled query requests:

PUT _cluster/settings
{
  "persistent": {
    "search.isolator.log.count": "100"
  }
}