Querying and Managing Logstash Cluster Logs

Prerequisites

• Storage Class: Standard.
• Region: the same as that of the cluster.

Querying Logs

1. Log in to the CSS management console.
2. Choose Clusters in the navigation pane. On the Clusters page, click the cluster whose logs you want to query. The cluster information page is displayed.
3. In the navigation pane on the left, choose Log Management.
4. Query logs on the log management page.
5. Select the target node, and click . The query result is displayed.

   The latest 10,000 logs can be queried, and a maximum of 100 records are displayed.

   When a log file reaches 128 MB, the system automatically compresses and archives it. Only unarchived logs appear on the log query page, while archived logs remain accessible through the log backup function.

Backing up logs

1. Log in to the CSS management console.
2. Choose Clusters in the navigation pane. On the Clusters page, click the name of the target cluster. The cluster information page is displayed.
3. Click the Logs tab and toggle on the Log Management switch.
4. In the Edit Log Backup Configuration dialog box, set the parameters.

   If log backup has been enabled for the cluster, you can also click the edit icon next to Log Backup Configuration to update the settings.

   Table 1 Log backup settings

   Parameter	Description
   OBS Bucket	Select an OBS bucket for storing log backups from the drop-down list box. If no OBS buckets meet your requirements, click Create Bucket to go to the OBS console and create one. Select an OBS bucket that meets the following requirements: Storage Class: Standard. Region: the same as that of the cluster. To grant an IAM user access to an OBS bucket, you need to grant the GetBucketStoragePolicy, GetBucketLocation, ListBucket, and ListAllMyBuckets permissions to that user.
   Backup Path	Set the log storage location in the OBS bucket. The backup path cannot: Contain the following characters: \:*?"<>|'{} Start with a slash (/). Start or end with a period (.). Contain more than two consecutive slashes (/) or periods (.). Exceed 512 characters.
   IAM Agency	Select an IAM agency to grant the current account the permission to access and use OBS. To store logs to an OBS bucket, you must have the required OBS access permissions. The IAM agency must meet the following requirements: Agency Type must be Cloud service. Set Cloud Service to Elasticsearch or CSS. Mandatory policies: Tenant Administrator If no agency is available, contact the CSS administrator to create one. WARNING: The agency name can contain only letters (case-sensitive), digits, underscores (_), and hyphens (-). Otherwise, the backup will fail.

5. Back up logs.
   • Automatically backing up logs

     Click the icon on the right of Auto Backup to enable automatic log backup.

     After automatic backup is enabled, set the backup start time in the Configure Auto Backup dialog box. When the scheduled time arrives, the system will back up logs automatically.

     After Automatic Snapshot Creation is enabled, you can click the Edit icon on the right of the parameter to change the backup start time.

   • Manually backing up logs

     On the Log Backup tab page, click Back Up. In the displayed dialog box, click Yes to start backup.

     If Task Status in the log backup list is Successful, the backup is successful.

6. View log files.

   Logs are backed up incrementally. After the backup is successful, you can access the target OBS bucket to obtain the full log files. Table 2 lists the log types, where clustername indicates the cluster name.

   Table 2 Log types

   Log Name	Description
   clustername_deprecation.log	Deprecation log file
   clustername-plain.log	Run log file

Log Introduction

• Run log description

  Run logs record a cluster's node and pipeline status. For example, the log record below indicates that the destination cluster could not be reached. You need to check whether the cluster address is correct and whether the cluster status is normal.

  Figure 1 A sample of run logs

  
  

  Log content:

  • 1. Log generation time
  • 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
  • 3. Log-generating module
  • 4. Name of the log-generating node
  • 5. Log content

• Deprecation log description

  Deprecation logs record deprecation warnings.

  Figure 2 A sample of deprecation logs

  
  

  Log content:

  • 1. Log generation time
  • 2. Log level, which can be DEBUG, INFO, WARN, ERROR, or DEPRECATION
  • 3. Log-generating module
  • 4. Log content. The log record shown in the figure above indicates that the ECS compatibility mode was not explicitly declared for a plugin when Logstash was started.