Introduction
This section describes metrics reported by DMS for Kafka to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms, or view Kafka instance metrics on the Monitoring Details page of the DMS for Kafka console.
Cloud Eye supports a maximum of four hierarchical dimensions that are numbered from 0 to 3. For example, if the dimension information of a monitoring metric is "kafka_instance_id,kafka_broker", the dimension of "kafka_instance_id" is numbered 0 and that of "kafka_broker" is numbered 1.
Namespace
SYS.DMS
Instance Metrics
Metric ID | Metric Name | Description | Value Range | Unit | Conversion Rule | Dimension | Monitoring Period (Raw Data) |
|---|---|---|---|---|---|---|---|
current_partitions | Partitions | Number of used partitions in the instance | 0–100,000 | Count | N/A | kafka_instance_id | 1 minute |
current_topics | Topics | Number of created topics in the instance | 0–100,000 | Count | N/A | kafka_instance_id | 1 minute |
group_msgs | Accumulated Messages | Total number of accumulated messages in all consumer groups of the instance | 0–1,000,000,000 | Count | N/A | kafka_instance_id | 1 minute |
Broker Metrics
Metric ID | Metric Name | Description | Value Range | Unit | Conversion Rule | Dimension | Monitoring Period (Raw Data) |
|---|---|---|---|---|---|---|---|
broker_data_size | Message Size | Total size of messages in the broker | 0–5,000,000,000,000 | Byte | 1024(IEC) | kafka_instance_id,kafka_broker | 1 minute |
broker_messages_in_rate | Message Creation Rate | Number of messages created per second | 0–500,000 | Count/s | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_bytes_out_rate | Message Retrieval | Number of bytes retrieved per second | 0–500,000,000 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_broker | 1 minute |
broker_bytes_in_rate | Message Creation | Number of bytes created per second | 0–500,000,000 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_broker | 1 minute |
broker_fetch_mean | Average Message Retrieval Processing Duration | Average time that the broker spends processing message retrieval requests | 0–10,000 | ms | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_produce_mean | Average Message Creation Processing Duration | Average time that the broker spends processing message creation requests | 0–10,000 | ms | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_cpu_core_load | Average Load per CPU Core | Average load of each CPU core of the Kafka VM | 0–20 | N/A | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_disk_usage | Disk Capacity Usage | Disk usage of the Kafka VM | 0–100 | % | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_memory_usage | Memory Usage | Memory usage of the Kafka VM | 0–100 | % | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_heap_usage | JVM Heap Memory Usage of JVM | Heap memory usage of the Kafka JVM | 0–100 | % | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_alive | Broker Alive | Whether the Kafka broker is alive |
| N/A | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_connections | Connections | Total number of TCP connections on the Kafka broker | 0–65,535 | Count | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_cpu_usage | CPU Usage | CPU usage of the Kafka VM | 0–100 | % | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_disk_read_await | Average Disk Read Time | Average time for each disk I/O read in the monitoring period | > 0 | ms | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_disk_write_await | Average Disk Write Time | Average time for each disk I/O write in the monitoring period | > 0 | ms | N/A | kafka_instance_id,kafka_broker | 1 minute |
broker_total_bytes_in_rate | Inbound Traffic | Inbound traffic per second | 0–1,000,000,000 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_broker | 1 minute |
broker_total_bytes_out_rate | Outbound Traffic | Outbound traffic per second | 0–1,000,000,000 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_broker | 1 minute |
broker_disk_read_rate | Disk Read Speed | Read traffic on the disk | ≥ 0 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_broker | 1 minute |
broker_disk_write_rate | Disk Write Speed | Write traffic on the disk | ≥ 0 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_broker | 1 minute |
network_bandwidth_usage | Network Bandwidth Usage | Network bandwidth usage | 0–100 | % | N/A | kafka_instance_id,kafka_broker | 1 minute |
Topic Metrics
Metric ID | Metric Name | Description | Value Range | Unit | Conversion Rule | Dimension | Monitoring Period (Raw Data) |
|---|---|---|---|---|---|---|---|
topic_bytes_in_rate | Message Creation | Number of bytes created per second This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. | 0–500,000,000 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_topics | 1 minute |
topic_bytes_out_rate | Message Retrieval | Number of bytes retrieved per second This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. | 0–500,000,000 | Byte/s | 1024(IEC) | kafka_instance_id,kafka_topics | 1 minute |
topic_data_size | Message Size | Total size of messages in the queue This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. | 0–5,000,000,000,000 | Byte | 1024(IEC) | kafka_instance_id,kafka_topics | 1 minute |
topic_messages | Total Messages | Total number of messages in the queue This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. | ≥ 0 | Count | N/A | kafka_instance_id,kafka_topics | 1 minute |
topic_messages_in_rate | Message Creation Rate | Number of messages created per second This metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page. | 0–500,000 | Count/s | N/A | kafka_instance_id,kafka_topics | 1 minute |
partition_messages | Partition Messages | Total number of messages in the partition This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page. | ≥ 0 | Count | N/A | kafka_instance_id,kafka_topics,kafka_partitions | 1 minute |
produced_messages | Created Messages | Number of messages that have been created This metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page. | ≥ 0 | Count | N/A | kafka_instance_id,kafka_topics,kafka_partitions | 1 minute |
Consumer Group Metrics
Metric ID | Metric Name | Description | Value Range | Unit | Conversion Rule | Dimension | Monitoring Period (Raw Data) |
|---|---|---|---|---|---|---|---|
messages_consumed | Retrieved Messages | Number of messages that have been retrieved in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. | ≥ 0 | Count | N/A | kafka_instance_id,kafka_groups,kafka_groups_topics,kafka_groups_partitions | 1 minute |
messages_remained | Available Messages | Number of messages that can be retrieved in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. | ≥ 0 | Count | N/A | kafka_instance_id,kafka_groups,kafka_groups_topics,kafka_groups_partitions | 1 minute |
topic_messages_remained | Topic Available Messages | Number of remaining messages that can be retrieved from the specified topic in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. | 0 to 263–1 | Count | N/A | kafka_instance_id,kafka_groups,kafka_groups_topics | 1 minute |
topic_messages_consumed | Topic Retrieved Messages | Number of messages that have been retrieved from the specified topic in the consumer group This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. | 0 to 263–1 | Count | N/A | kafka_instance_id,kafka_groups,kafka_groups_topics | 1 minute |
consumer_messages_remained | Consumer Available Messages | Number of remaining messages that can be retrieved in the consumer group This metric is available only when Topic is set to All topics on the By Consumer Group tab page. | 0 to 263–1 | Count | N/A | kafka_instance_id,kafka_groups | 1 minute |
consumer_messages_consumed | Consumer Retrieved Messages | Number of messages that have been retrieved in the consumer group This metric is available only when Topic is set to All topics on the By Consumer Group tab page. | 0 to 263–1 | Count | N/A | kafka_instance_id,kafka_groups | 1 minute |
messages_consumed_per_min | Partition Consumption Rate | Number of messages consumed from the specified queue partition in the consumer group every minute This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page. Some instances do not support this metric. Check whether your instance supports it on the console. | 0–30,000,000 | Count/min | N/A | kafka_instance_id,kafka_groups,kafka_groups_topics,kafka_groups_partitions | 1 minute |
topic_messages_consumed_per_min | Queue Consumption Rate | Number of messages consumed from the specified queue in the consumer group every minute This metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page. Some instances do not support this metric. Check whether your instance supports it on the console. | 0–30,000,000 | Count/min | N/A | kafka_instance_id,kafka_groups,kafka_groups_topics | 1 minute |
consumer_messages_consumed_per_min | Consumer Group Consumption Rate | Number of messages consumed from the consumer group every minute This metric is available only when Topic is set to All topics on the By Consumer Group tab page. Some instances do not support this metric. Check whether your instance supports it on the console. | 0–30,000,000 | Count/min | N/A | kafka_instance_id,kafka_groups | 1 minute |
If a monitored object has multiple dimensions, the dimensional level of specific metrics is required when you use APIs to query the metrics.
For example, to query the disk capacity usage (broker_disk_usage) of a Kafka broker, its dimension is "kafka_instance_id,kafka_broker", indicating that kafka_instance_id is numbered 0 and kafka_broker is numbered 1.
- To query a single metric by calling the API, the kafka_broker dimension is used as follows:dim.0=kafka_instance_id,6a0df74a-fxxx-497bfdda6c8e&dim.1=kafka_broker,broker-0
6a0df74a-fxxx-497bfdda6c8e and broker-0 are the dimension values of kafka_instance_id and kafka_broker. For details about how to obtain them, see the guide in the Dimension table.
- To batch query metrics by calling the API, the kafka_broker dimension is used as follows:"dimensions": [{"name": "kafka_instance_id","value": "6a0df74a-fxxx-497bfdda6c8e"},{"name": "kafka_broker","value": "broker-0"}]
6a0df74a-fxxx-497bfdda6c8e and broker-0 are the dimension values of kafka_instance_id and kafka_broker. For details about how to obtain them, see the guide in the Dimension table.
Dimension
Key | Value |
|---|---|
kafka_instance_id | Kafka instance ID, for example, 6a0df74a-fxxx-497bfdda6c8e. |
kafka_broker | Kafka instance broker. The broker ID starts from 0 and increases in ascending order. For example, if an instance has three brokers, they are broker-0, broker-1, and broker-2. |
kafka_topics | Topic name, for example, topic-01. |
kafka_partitions | Partition number in a topic, for example, 0. |
kafka_groups_partitions | Partition number in a topic consumed by a consumer group, for example, 0. |
kafka_groups_topics | Topic name consumed by a consumer group, for example, topic-01. |
kafka_groups | Consumer group name of a Kafka instance, for example, group-01. |