Starting Migration Tasks

There can be up to 50 Running configuration files in the pipeline list of a Logstash cluster.

Before starting a migration task, test the connectivity between Logstash and both the data source and destination. This helps prevent task failures caused by connectivity issues.

How a connectivity test works:

• IP layer test: Run the fping command to check whether the target IP address is reachable.
• Transport layer test: Run the telnet command to check whether the TCP port associated with the target address is reachable.

Prerequisites for the connectivity test

Depending on the locations of the Logstash cluster, data source, and destination, the following conditions must be met:

• If they are in the same VPC: The security group used by the Logstash cluster allows outbound traffic for all protocols, and inbound traffic for the ICMP protocol as well as the TCP ports associated with the peer addresses. For details, see section "Adding a Security Group Rule" in the Virtual Private Cloud User Guide.
• If they are in different VPCs: The security group used by the Logstash cluster allows outbound traffic for all protocols, and inbound traffic for the ICMP protocol as well as the TCP ports associated with the peer addresses. In addition, you must configure a route between the Logstash cluster and the peer VPCs. For details, see .
• If they are in different LANs (they must communicate across the public network): Configure a public IP address for the Logstash cluster to ensure it can be accessed from the Internet. For details, see Configuring Public Network Access.

1. Go to the Configuration Center page.
   1. Log in to the CSS management console.
   2. In the navigation pane on the left, choose Clusters > Logstash.
   3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
   4. Click the Configuration Center tab.
2. On the Configuration Center page, click Test Connectivity.
3. In the Test Connectivity dialog box, enter the IP addresses and port numbers of the data source and destination, and click Test.

   You can test a maximum of 10 IP addresses at a time. You can click Add to add more IP addresses and click Test at the bottom to test connectivity to multiple IP addresses at a time.

   If Available is returned, the network is connected. If Unavailable is returned, troubleshoot by following What Should I Do If the Result of a Logstash Connectivity Test Is Unavailable?.

Start a Logstash pipeline to migrate data from the source to the destination.

1. Check whether there are pipelines whose status is Running in the pipeline list.
   • Yes: Perform 3 to hot-start a configuration file.
   • No: Perform 2 to start a configuration file.
2. Start a configuration file.
   1. In the configuration file list, select 1 to 50 configuration files and click Start Logstash above.
   2. In the displayed dialog box, choose whether to select Keepalive.

      Select Keepalive for long-term data streams in production environments. If the Logstash service stops due to an unexpected error, the system will automatically try to restart it.

      Deselect Keepalive for one-time offline data migration tasks. Note that if the source has no data, enabling Keepalive may cause the pipeline to restart repeatedly, which may lead to errors.

   3. Click OK to activate Logstash pipelines.
3. Hot-start a configuration file.
   1. In the configuration file list, select one configuration file, and click Hot Start above.
   2. In the displayed dialog box, confirm the Keepalive setting. The Keepalive setting is inherited from existing pipelines in the cluster. It cannot be modified.
   3. Click OK to activate the Logstash pipeline.
4. Verify the result. When the status of the new pipeline changes to Running in the pipeline list, the data migration task has started and is running properly. The Events column will be dynamically updated to show the number of processed documents.

After starting data migration pipelines, you can monitor them in the following ways:

• Check their metrics.

  In the pipeline list, locate the target pipeline, and click Metric Monitoring in the Operation column to go to the Cloud Eye console, where you can check the metrics to evaluate its status and performance.

  When the Events data of a pipeline changes dynamically, the monitoring data changes accordingly. When a pipeline is being started or stopped, or the Events data is stable, the monitoring data remains unchanged.

  For details about the metrics supported, see Logstash Pipeline Monitoring Metrics. For how to configure alarms, see Using Cloud Eye to Monitor Clusters.

• Check run logs.

  Above the pipeline list, click Run Logs to check the logs of the Logstash process, which you can use to troubleshoot data parsing errors.

• Check operation records.

  Above the pipeline list, click View to check the start and stop records of the pipelines. This helps track historical operations and facilitate troubleshooting.