When using Logstash for large-scale data migration or log ingestion, you may need to suspend or terminate pipelines to allow for policy adjustment or configuration optimization, or upon task completion. However, inappropriately terminating pipelines may cause ingestion checkpoint loss or data inconsistency between the source and destination. CSS provides two methods for stopping Logstash pipelines: Stop All and Hot Stop. Each method is designed for specific task scenarios. By selecting the appropriate method, you can ensure data migration and log ingestion tasks are stopped safely.
Stop Option | Stop All | Hot Stop |
|---|---|---|
Pipeline Status | The pipeline status changes from Running to Stopped. | The pipeline is removed. |
How It Works | A global signal is sent to the Logstash process to block the pipeline execution logic without deregistering the configuration metadata. | A dynamic unloading operation is performed on the selected pipeline to disassociate the configuration file. The Logstash process will no longer load this pipeline. |
Data integrity | Stopping a pipeline may cause brief latency for data in transit or trigger an offset reset. Ensure that the source data maintains traceability, for example, via incremental change-tracking columns in the source database. Additionally, it will immediately interrupt data migration. Exercise caution. | |
When to Use | Use this option when you are preparing your cluster for a comprehensive maintenance, or when you need to release resources during peak hours. With this option, you can suspend pipelines while retaining their configuration to allow for a one-click restoration later. | Use this option when you want to permanently remove a pipeline that is no longer needed. This releases resources without interrupting other pipelines. |
There are Running pipeline tasks in the configuration center of the Logstash cluster.
Suspend all running pipelines in the cluster in one click.
Remove a specific pipeline without interrupting other pipelines.