Querying Logs

Query recently generated logs that are not yet archived.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
4. Choose Logs > Log Search. The Log Search page is displayed.

   You can search records by log type, node, log level, or keyword. For a detailed description of each type of logs, see Log Types.

   Caution

   When a log file reaches 128 MB or when the time reaches 00:00 UTC, the system automatically compresses and archives it. Only unarchived logs appear on the log search page, while archived logs remain accessible through the log backup function. For more information, see Backing Up Logs to OBS.

Log4j2 is used as the log component in Elasticsearch clusters. Multiple log levels (ERROR, WARN, INFO, DEBUG, and TRACE) are supported. The default log level is INFO. To facilitate troubleshooting and debugging, you can dynamically adjust the log levels.

• INFO is the default log level. The levels, in order of increasing detail, are: ERROR, WARN, INFO, DEBUG, and TRACE. When INFO is set, you will see logs for itself and all higher-severity levels (ERROR and WARN), while more verbose levels (DEBUG and TRACE) are excluded.
• You can change the log level of a specified module in real time via an Elasticsearch API.

1. Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.
   1. Log in to the CSS management console.
   2. In the navigation pane on the left, choose Clusters > Elasticsearch.
   3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
   4. In the left navigation pane, choose Dev Tools.

      The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

2. Run the following command to change the log level, for example, to DEBUG:

   PUT _cluster/settings
   {
     "persistent": {
       "logger": {
         "org.elasticsearch.action": "DEBUG"
       }
     }
   }
   
   

3. Refresh the log query page and check whether DEBUG logs have been generated. You should always restore the log level to the default setting once you finish the task at hand.
4. Run the following command to restore the default log level INFO:

   PUT _cluster/settings
   {
     "persistent": {
       "logger": {
         "org.elasticsearch.action": null
       }
     }
   }
   
   

To analyze communication details on the HTTP or transport layer, you can enable trace logging for the HTTP or Transport module to obtain detailed log records.

1. Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.
   1. Log in to the CSS management console.
   2. In the navigation pane on the left, choose Clusters > Elasticsearch.
   3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
   4. In the left navigation pane, choose Dev Tools.

      The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

2. Run the following command to enable trace logging:

   PUT _cluster/settings
   {
     "transient": {
       "logger.org.elasticsearch.transport.TransportService.tracer": "trace",
       "transport.tracer.include": "",
       "http.tracer.include": "",
       "logger.org.elasticsearch.http.HttpTracer": "trace"
     }
   }
   
   

   Caution

   Enabling trace logging is a non-persistent configuration and will be disabled upon a cluster restart. It is typically used for emergency packet capture and analysis during troubleshooting.

3. Go to the log details page to view trace logs.
   1. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
   2. Choose Logs > Log Search. The Log Search page is displayed.
   3. Select all log levels (mandatory) and view trace logs.

• Run log description

  Run logs record the cluster status and key information about write and query operations. For example, the log record below indicates that an index named test was created and later the cluster status changed from YELLOW to GREEN.

  Figure 1 A sample of run logs

  
  

  Log content:

  • 1. Log generation time
  • 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
  • 3. Log-generating module
  • 4. Name of the log-generating node
  • 5. Log content

• Slow indexing log description

  Slow indexing logs record indexing operations that took a long time to complete. For example, the log record below shows an indexing request that lasted longer than the preset threshold. The log contains the index name, duration, and request content.

  Figure 2 A sample of slow indexing logs

  
  

  Log content:

  • 1. Log generation time

  • 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
  • 3. Log-generating module
  • 4. Name of the log-generating node
  • 5. Index name and ID
  • 6. Log content. In this example, the log recorded the request execution duration, index type, and index request body.

• Slow query log description

  Slow query logs record search requests that took a long time to complete. For example, the log record below shows a search request that lasted longer than the preset threshold. The log contains the index name, duration, and request content.

  Figure 3 A sample of slow query logs

  
  

  Log content:

  • 1. Log generation time
  • 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
  • 3. Log-generating module
  • 4. Name of the log-generating node
  • 5. Index name and shard ID
  • 6. Log content. In this example, the log recorded the query duration, number of hits, and query request body.

• Deprecation log description

  Deprecation logs record deprecation warnings. For example, the log record below indicates that GET /_cat/master has been deprecated and should be replaced with GET /_cat/cluster_manager.

  Figure 4 A sample of deprecation logs

  
  

  Log content:

  • 1. Log generation time
  • 2. Log level, which can only be DEPRECATION.
  • 3. Log-generating module
  • 4. Name of the log-generating node
  • 5. Log content

• Access log description

  Access logs record cluster access requests and source addresses. For example, the log record below has recorded source information for the /_snapshot/my_backup/my_snapshot/_restore?pretty=true operation.

  Figure 5 A sample of access logs

  
  

  Log content:

  • 1. Log generation time
  • 2. Name of the log-generating node
  • 3. Name of the log-generating thread
  • 4. Log level, which can be DEBUG, INFO, WARN, or ERROR
  • 5. Log request method
  • 6. Request path
  • 7. Source and destination addresses of the request

How Do I Set Slow Query Log Thresholds for an Elasticsearch Cluster of CSS?