When using Logstash for large-scale data migration or real-time log ingestion, errors such as incorrect pipeline configurations, broken connections between the source and destination, or erroneous data processing logic may interrupt data flows. CSS provides a log query feature that aggregates underlying server logs to the console, where you can query them by node and keyword in seconds. By analyzing recent logs, you can gain real-time insight into each phase of the pipelines and accurately locate the root causes of exceptions such as connection timeouts or configuration syntax errors.
Querying Recent Logs
Query recently generated logs that are not yet archived.
- Log in to the CSS management console.
- In the navigation pane on the left, choose Clusters > Logstash.
- In the cluster list, click the name of the target cluster. The cluster information page is displayed.
- Choose Logs > Log Search. The Log Search page is displayed.
You can search log records by node or keyword. For a detailed description of each type of logs, see Log Types.
CautionWhen a log file reaches 128 MB or when the time reaches 00:00 UTC, the system automatically compresses and archives it. Only unarchived logs appear on the log search page, while archived logs remain accessible through the log backup function. For more information, see Backing Up Logs to OBS.
Log Types
Log Type | Description | Purpose |
|---|---|---|
Run logs | Run logs record a cluster's node and pipeline status, such as source-destination connectivity, pipeline creation or modification, and pipeline running errors. | Check run logs to troubleshoot pipeline errors. |
Deprecation logs | Deprecation logs record deprecation warnings. Deprecation warnings are written to this log when you use APIs, configurations, or functions that are marked for removal in future versions. You cannot check deprecation logs on the console. To check them, you need to back them up to an OBS bucket first. | Check for APIs or features that are about to expire in future versions. |
- Run log description
Run logs record a cluster's node and pipeline status. For example, the log record below indicates that the destination cluster could not be reached. You need to check whether the cluster address is correct and whether the cluster status is normal.
Figure 1 A sample of run logs
Log content:
- 1. Log generation time
- 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
- 3. Log-generating module
- 4. Name of the log-generating node
- 5. Log content
- Deprecation log description
Deprecation logs record deprecation warnings.
Figure 2 A sample of deprecation logs
Log content:
- 1. Log generation time
- 2. Log level, which can only be DEPRECATION.
- 3. Log-generating module
- 4. Log content. The log record shown in the figure above indicates that the ECS compatibility mode was not explicitly declared for a plugin when Logstash was started.