/ DOCS
Advanced
Тема интерфейса

Using Kafka from Scratch

MRS provides Hadoop-based high-performance big data components, such as Spark, HBase, Kafka, and Storm.

This section uses a cluster with Kerberos authentication disabled as an example to describe how to generate and consume messages in a Kafka topic.

You can get started by reading the following steps:

  1. Buying a Cluster
  2. Installing the Kafka Client
  3. Logging In to a Master Node Using VNC
  4. Creating a Topic Using the Kafka Client
  5. Managing Messages in Kafka Topics

Buying a Cluster

  1. Create a cluster.

    1. Log in to the management console.
    2. Choose EI Enterprise Intelligence > MapReduce Service. The MRS management console is displayed.
    3. Click Buy Cluster, the Buy Cluster page is displayed.
    4. Click the Custom Config tab.

  2. Configure software.

    1. Region: Select a region as required.
    2. Billing Mode: Select Pay-per-use.
    3. Cluster Name: Enter mrs_demo or use another name following the naming rules.
    4. Cluster Type: Select Streaming cluster.
    5. Cluster Version: Select MRS 3.1.0.
    6. Component: Select all components required for a streaming cluster.
    7. Click Next.

  3. Configure hardware.

    1. AZ: Select AZ2.
    2. Enterprise Project: Select default.
    3. VPC and Subnet: Retain their default values or click View VPC and View Subnet to create new ones.
    4. Subnet: Retain the default value.
    5. Security Group: Use the default option Auto create.
    6. EIP: Retain the default option Bind later.
    7. Cluster Nodes: Retain the default settings. Do not add new Task nodes.
    8. Click Next.

  4. Set advanced options.

    1. Kerberos Authentication: Trun off this function.
    2. Username: admin is used by default.
    3. Password and Confirm Password: Set them to the password of the FusionInsight Manager administrator.
    4. Login Mode: Select Password. Enter a password and confirm the password for user root.
    5. Hostname Prefix: Retain the default value.
    6. Select Configure for Set Advanced Options, and set Agency to MRS_ECS_DEFAULT_AGENCY.
    7. Click Next.

  5. Confirm the configuration.

    1. Configure: Confirm the settings configured on the Configure Software, Configure Hardware, and Set Advanced Options pages.
    2. Enable Secure Communications.
    3. Click Buy Now. A page is displayed showing that the task has been submitted.
    4. Click Back to Cluster List. You can view the status of the newly created cluster on the Active Clusters page. Wait for the cluster creation to complete. The initial status of the cluster is Starting. After the cluster is created, the cluster status becomes Running.

Installing the Kafka Client

  1. Choose Active Clusters in the navigation pane on the left. On the Active Clusters page, click the cluster named mrs_demo to go to its details page.
  2. Click Access Manager next to MRS Manager. On the page that is displayed, configure the EIP information and click OK. Enter the username and password to access FusionInsight Manager.
  3. Choose Cluster > Services > Kafka. On the page displayed, choose More > Download Client. In the Download Cluster Client dialog box, select Complete Client for Select Client Type, select a platform type, select Save to Path, and click OK. The Kafka client software package, for example, FusionInsight_Cluster_1_Kafka_Client.tar, is downloaded.

  4. Log in to the Master1 node as user root.
  5. Go to the directory where the software package is stored and run the following commands to decompress and verify the software package, and decompress the obtained installation file:

    cd /tmp/FusionInsight-Client

    tar -xvf FusionInsight_Cluster_1_Kafka_Client.tar

    sha256sum -c FusionInsight_Cluster_1_Kafka_ClientConfig.tar.sha256

    tar -xvf FusionInsight_Cluster_1_Kafka_ClientConfig.tar

  6. Go to the directory where the installation package is stored, and run the following command to install the client to a specified directory (absolute path), for example, /opt/hadoopclient:

    cd /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Kafka_ClientConfig

    Run the ./install.sh /opt/hadoopclient command and wait until the client installation is complete.

  7. Check whether the client is installed.

    cd /opt/hadoopclient

    source bigdata_env

    Run the klist command to query and confirm authentication details. If the command is executed, the Kafka client is installed.

Logging In to a Master Node Using VNC

  1. Choose Active Clusters in the navigation pane on the left. On the Active Clusters page that is displayed, click the cluster named mrs_demo. On the cluster details page that is displayed, click the Nodes tab. On this tab page, locate the node whose type is Master1 and click the node name to go to the ECS details page.

    Figure 1 Remotely logging in to the Master1 node


  2. Click Remote Login in the upper right corner of the page to remotely log in to the master node. Log in using the username root and the password configured during cluster creation.

Creating a Topic Using the Kafka Client

  1. Configure environment variables. For example, if the Kafka client installation directory is /opt/hadoopclient, run the following command:

    source /opt/hadoopclient/bigdata_env

  2. Choose Active Clusters in the navigation pane on the left. On the Active Clusters page, click the cluster named mrs_demo to go to the Dashboard tab page. On this page, click Synchronize next to IAM User Sync.
  3. After the synchronization is complete, click the Components tab. On this tab page, select ZooKeeper. On the page that is displayed, click the Instances tab. Record the IP address of any ZooKeeper instance,
  4. Run the following command to create a Kafka topic:

    kafka-topics.sh --create --zookeeper <IP address of the node where the ZooKeeper instance resides:2181/kafka> --partitions 2 --replication-factor 2 --topic <Topic name>

Managing Messages in Kafka Topics

  1. Click the Components tab. On this tab page, select Kafka. On the page that is displayed, click the Instances tab. On the Instances tab page, view the IP addresses of Kafka instances. Record the IP address of any Kafka instance,
  2. Log in to the master node and run the following command to generate messages in a topic test:

    cd /opt/hadoopclient

    source bigdata_env

    kafka-console-producer.sh --broker-list <IP address of the node where the Kafka instance resides:9092> --topic <Topic name> --producer.config /opt/hadoopclient/Kafka/kafka/config/producer.properties

    <Topic name> indicates the name of the topic created Creating a Topic Using the Kafka Client.

    Enter the specified content as the messages generated by the producer and press Enter to send the messages. To stop generating messages, press Ctrl+C to exit.

  3. Consume messages in the topic test.

    kafka-console-consumer.sh --topic <Topic name> --bootstrap-server <IP address of the node where the Kafka instance resides:9092> --consumer.config /opt/hadoopclient/Kafka/kafka/config/consumer.properties

Parent topic: Getting Started
Предыдущая статья
Using Hadoop from Scratch
Следующая статья
Using HBase from Scratch
Была ли эта статья полезна?
Поддержка Юридические документы
© 2025 Cloud.ru