Configuring Enhanced Aggregation

Enhanced aggregation works by pre-sorting and physically clustering data using carefully selected sorting and clustering keys upon ingestion, thereby minimizing data scanning and computational overheads during aggregation.

• Sorting key: A field (such as timestamps) used to physically order documents on disk, ensuring that documents with identical or similar sorting key values are stored contiguously.
• Clustering key: A subset of fields from the sorting key that groups related documents into contiguous physical blocks. This way, aggregations can process contiguous data blocks, instead of scattered documents.

For example, for an index that contains the region, host, and value fields, documents can be sorted by these three fields upon ingestion, and then clustered and stored based on the region and host fields. In this case, documents with the same region or host are stored together, and are then sorted by the value field within each cluster.

When an aggregation query is performed, data is queried by clustering key (region and host), or by the value field within each cluster. This improves query efficiency.

Only Elasticsearch 7.10.2 supports enhanced aggregation.

Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

Generally, low-cardinality fields use grouping as a way of aggregation. With appropriate sorting keys, grouping prepares the data for batch processing.

For example, to aggregate two low-cardinality fields city and product, perform the following steps:

1. Run the following command to set the index testindex:

   PUT testindex
   {
     "mappings": {
       "properties": {
         "date": {
           "type": "date",
           "format": "yyyy-MM-dd"
         },
         "city": {
           "type": "keyword"
         },
         "product": {
           "type": "keyword"
         },
         "trade": {
           "type": "double"
         }
       }
     },
     "settings": {
       "index": {
         "search": {
           "turbo": {
             "enabled": "true" 
           }
         },
         "sort": {
           "field": [
             "city",
             "product"
           ]
         },
         "cluster": {
           "field": [
             "city",
             "product"
           ]
         }
       }
     }
   }
   
   

   Table 2 Parameters for low-cardinality field aggregation

   Parameter	Type	Default Value	Description
   index.search.turbo.enabled	Boolean	false	Whether to enable enhanced aggregation. Normally, enhanced aggregation must be enabled where aggregations are used. The value can be: false: Disable enhanced aggregation. true: Enable enhanced aggregation.
   index.sort.field	Array of strings	N/A	Specify sorting keys. Sorting keys are fields used to sequence or rank documents. You can specify one or multiple fields as sorting keys. When multiple fields are specified, they will apply in the sequence in which they are specified. Documents are first ranked by the first field, then the initial result set is ranked by the second field, and so on. Value range: The value must be fields contained in the index.
   index.cluster.field	Array of strings	N/A	Specify clustering keys. Clustering keys determine which documents are collected into the same clusters. During an aggregation operation, documents in the same cluster can be processed in batches, significantly enhancing aggregation performance. Constraint: The clustering keys must be a subset of the sorting keys. Value range: The value must be fields contained in the index.

2. Run the following command to import sample data.

   PUT /_bulk
   { "index": { "_index": "testindex", "_id": "1" } }
   { "date": "2025-01-01", "city": "cityA", "product": "books", "trade": 3000.0}
   { "index": { "_index": "testindex", "_id": "2" } }
   { "date": "2025-01-02", "city": "cityA", "product": "books", "trade": 1000.0}
   { "index": { "_index": "testindex", "_id": "3" } }
   { "date": "2025-01-01", "city": "cityA", "product": "bottles", "trade": 100.0}
   { "index": { "_index": "testindex", "_id": "4" } }
   { "date": "2025-01-02", "city": "cityA", "product": "bottles", "trade": 300.0}
   { "index": { "_index": "testindex", "_id": "5" } }
   { "date": "2025-01-01", "city": "cityB", "product": "books", "trade": 7000.0}
   { "index": { "_index": "testindex", "_id": "6" } }
   { "date": "2025-01-02", "city": "cityB", "product": "books", "trade": 1000.0}
   
   

3. Run the following query command to aggregate low-cardinality fields.

   For example, query the average trade value for each product in different cities.

   POST testindex/_search
   {
     "size": 0,
     "aggs": {
       "groupby_city": {
         "terms": {
           "field": "city"
         },
         "aggs": {
           "groupby_product": {
             "terms": {
               "field": "product"
             },
             "aggs": {
               "avg_trade": {
                 "avg": {
                   "field": "trade"
                 }
               }
             }
           }
         }
       }
     }
   }
   
   

High-cardinality fields commonly use sorting keys for histogram aggregation, which facilitates data processing per range or scope.

For example, to aggregate the typical high-cardinality field date, perform the following steps:

1. Run the following command to set the index testindex:

   PUT testindex
   {
     "mappings": {
       "properties": {
         "date": {
           "type": "date",
           "format": "yyyy-MM-dd"
         },
         "score": {
           "type": "double"
         }
       }
     },
     "settings": {
       "index": {
         "search": {
           "turbo": {
             "enabled": "true" 
           }
         },
         "sort": {
           "field": [
             "date"
           ]
         }
       }
     }
   }
   
   

   Table 3 Parameters for high-cardinality field aggregation

   Parameter	Type	Default Value	Description
   index.search.turbo.enabled	Boolean	false	Whether to enable enhanced aggregation. Normally, enhanced aggregation must be enabled where aggregations are used. The value can be: false: Disable enhanced aggregation. true: Enable enhanced aggregation.
   index.sort.field	Array of strings	N/A	Specify sorting keys. Sorting keys are fields used to sequence or rank documents. You can specify one or multiple fields as sorting keys. When multiple fields are specified, they will apply in the sequence in which they are specified. Documents are first ranked by the first field, then the initial result set is ranked by the second field, and so on. Value range: The value must be fields contained in the index.

2. Run the following command to import sample data.

   PUT /_bulk
   { "index": { "_index": "testindex", "_id": "1" } }
   { "date": "2025-01-01", "score": "12.0"}
   { "index": { "_index": "testindex", "_id": "2" } }
   { "date": "2025-01-02","score": "24.0"}
   { "index": { "_index": "testindex", "_id": "3" } }
   { "date": "2025-01-01","score": "53.0"}
   { "index": { "_index": "testindex", "_id": "4" } }
   { "date": "2025-01-02", "score": "22.0"}
   { "index": { "_index": "testindex", "_id": "5" } }
   { "date": "2025-01-01", "score": "99.0"}
   { "index": { "_index": "testindex", "_id": "6" } }
   { "date": "2025-01-02","score": "26.0"}
   
   

3. Run the following query command to aggregate high-cardinality fields.

   This query groups the date field using a histogram and then calculates the average score.

   POST testindex/_search?pretty
   {
     "size": 0,
     "aggs": {
       "groupbytime": {
         "date_histogram": {
           "field": "date",
           "calendar_interval": "day"
         },
         "aggs": {
           "avg_score": {
             "avg": {
               "field": "score"
             }
           }
         }
       }
     }
   }
   
   

Where low-cardinality and high-cardinality fields are mixed, first group and aggregate low-cardinality fields using a clustering key, then create a histogram for high-cardinality fields using a sorting key.

For example, to first group the low-cardinality field city, then group the low-cardinality field product, and then group the high-cardinality field date into a histogram, perform the following steps:

1. Run the following command to set the index testindex:

   PUT testindex
   {
     "mappings": {
       "properties": {
         "date": {
           "type": "date",
           "format": "yyyy-MM-dd"
         },
         "city": {
           "type": "keyword"
         },
         "product": {
           "type": "keyword"
         },
         "trade": {
           "type": "double"
         }
       }
     },
     "settings": {
       "index": {
         "search": {
           "turbo": {
             "enabled": "true"
           }
         },
         "sort": { 
           "field": [
             "city",
             "product",
             "date"
           ]
         },
         "cluster": {
           "field": [
             "city",
             "product"
           ]
         }
       }
     }
   }
   
   

   Table 4 Parameters for hybrid aggregation of low- and high-cardinality fields

   Parameter	Type	Default Value	Description
   index.search.turbo.enabled	Boolean	false	Whether to enable enhanced aggregation. Normally, enhanced aggregation must be enabled where aggregations are used. The value can be: false: Disable enhanced aggregation. true: Enable enhanced aggregation.
   index.sort.field	Array of strings	N/A	Specify sorting keys. Sorting keys are fields used to sequence or rank documents. You can specify one or multiple fields as sorting keys. When multiple fields are specified, they will apply in the sequence in which they are specified. Documents are first ranked by the first field, then the initial result set is ranked by the second field, and so on. Constraint: High-cardinality fields must be among the sorting keys, and must follow the last low-cardinality field. Value range: The value must be fields contained in the index.
   index.cluster.field	Array of strings	N/A	Specify clustering keys. Clustering keys determine which documents are collected into the same clusters. During an aggregation operation, documents in the same cluster can be processed in batches, significantly enhancing aggregation performance. Constraint: The clustering keys must be a subset of the sorting keys. Value range: The value must be fields contained in the index.

2. Run the following command to import sample data.

   PUT /_bulk
   { "index": { "_index": "testindex", "_id": "1" } }
   { "date": "2025-01-01", "city": "cityA", "product": "books", "trade": 3000.0}
   { "index": { "_index": "testindex", "_id": "2" } }
   { "date": "2025-01-02", "city": "cityA", "product": "books", "trade": 1000.0}
   { "index": { "_index": "testindex", "_id": "3" } }
   { "date": "2025-01-01", "city": "cityA", "product": "bottles", "trade": 100.0}
   { "index": { "_index": "testindex", "_id": "4" } }
   { "date": "2025-01-02", "city": "cityA", "product": "bottles", "trade": 300.0}
   { "index": { "_index": "testindex", "_id": "5" } }
   { "date": "2025-01-01", "city": "cityB", "product": "books", "trade": 7000.0}
   { "index": { "_index": "testindex", "_id": "6" } }
   { "date": "2025-01-02", "city": "cityB", "product": "books", "trade": 1000.0}
   
   

3. Run the following query command to perform a hybrid aggregation of low- and high-cardinality fields.

   For example, query the average trade value of each product in different cities on each day specified by the date field.

   POST testindex/_search
   {
     "size": 0,
     "aggs": {
       "groupby_region": {
         "terms": {
           "field": "city"
         },
         "aggs": {
           "groupby_host": {
             "terms": {
               "field": "product"
             },
             "aggs": {
               "groupby_timestamp": {
                 "date_histogram": {
                   "field": "date",
                   "interval": "day"
                 },
                 "aggs": {
                   "avg_score": {
                     "avg": {
                       "field": "trade"
                     }
                   }
                 }
               }
             }
           }
         }
       }
     }
   }
   
   

Test environment

• Dataset: esrally nyc_taxis
• Cluster specifications: 4U16G, 100 GB, high I/O x 3 nodes

Test Procedure

1. Create an index template in the cluster, specify sorting keys, and disable enhanced aggregation.

   PUT /_template/nyc_taxis
   {
     "template": "nyc_taxis*",
     "settings": {
       "index.search.turbo.enabled": false,
       "index.sort.field": "dropoff_datetime",
       "number_of_shards": 3,
       "number_of_replicas": 0
     }
   }
   
   

2. Use esrally to test the nyc_taxis dataset and obtain the result when enhanced aggregation is disabled.
3. Create another index template in the same cluster, specify sorting keys, and enable enhanced aggregation.

   PUT /_template/nyc_taxis
   {
     "template": "nyc_taxis*",
     "settings": {
       "index.search.turbo.enabled": true,
       "index.sort.field": "dropoff_datetime",
       "number_of_shards": 3,
       "number_of_replicas": 0
     }
   }
   
   

4. Use esrally to test the nyc_taxis dataset and obtain the result when enhanced aggregation is enabled.

Test Result

This test focuses on the query result of dropoff_datetime aggregation, that is, the results of tasks autohisto_agg and date_histogram_agg. The following table compares the test results between when enhanced aggregation is disabled and when it is enabled.

Test Conclusion

Given the same cluster configuration, aggregation performance improves significantly when enhanced aggregation is enabled. Query throughput improves by more than 2.5 times, and latency decreases by more than 60%.