Advanced
Тема интерфейса

ClickHouse Cluster Monitoring Metrics

Description

Monitoring is critical to ensure CloudTable reliability, availability, and performance. You can monitor the running status of CloudTable servers. This section describes the metrics that can be monitored by Cloud Eye as well as their namespaces and dimensions.

Namespace

SYS.CloudTable

Monitoring metrics

Table 1 Monitoring metrics

Metric

Name

Description

Value Range

Metric Unit

Conversion Rule

Monitored Object (Dimension)

Monitoring Period

cmdProcessCPU

CPU Utilization Rate

CPU usage of the monitored object

0~100

%

N/A

CloudTable instance node

60s

memory_usage

Memory Usage

Memory usage of each node.

[0, Node memory capacity]

MB

N/A

CloudTable instance node

60s

cmdProcessMem

Memory Utilization Rate

Memory usage of the monitored object

0~100

%

N/A

CloudTable instance node

60s

disk_usage

Disk Usage

Disk space used by each node.

[0, Node disk capacity]

MB

N/A

CloudTable instance node

60s

cmdForUsedStorageRate

Used Storage Rate

Ratio of the used storage space to the total storage space in the cluster

>=0

%

N/A

CloudTable instance node

60s

network_throughput_inbound_rate

Inbound Throughput

Inbound data volume over network of each node per second.

>=0

KB/s

N/A

CloudTable instance node

60s

network_throughput_outgoing_rate

Outbound Throughput

Outbound data volume over network of each node per second.

>=0

KB/s

N/A

CloudTable instance node

60s

number_of_mutations

Mutations

Number of running modification tasks on each node.

>=0

Count

N/A

CloudTable instance node

60s

number_of_run_merges

Running Merges

Number of running merge tasks.

>=0

Count

N/A

CloudTable instance node

60s

number_of_merges_thread

Merge Threads

Number of threads being merged.

[0, Number of CPU cores]

Count

N/A

CloudTable instance node

60s

number_of_zookeeper_requests

ZooKeeper Requests

Number of ZooKeeper requests on the node at the current time.

>=0

Count

N/A

CloudTable instance node

60s

number_of_zk_session

ZooKeeper Sessions

Number of connected ZooKeeper sessions on the node at the current time.

>=0

Count

N/A

CloudTable instance node

60s

number_of_zookeeper_watches

ZooKeeper Watches

Number of connected ZooKeeper watches on the node at the current time.

>=0

Count

N/A

CloudTable instance node

60s

number_of_zookeeper_nodes

ZooKeeper Nodes

Number of temporary nodes created in ZooKeeper.

>=0

Count

N/A

CloudTable instance node

60s

number_of_parts

Parts

Number of parts.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_holding_read_locks

Threads Holding the Read Lock

Number of threads that hold read locks.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_waiting_to_be_read

Threads Waiting to Be Read

Number of threads waiting to be read.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_waiting_to_be_written

Number of Threads Waiting to Be Written

Number of threads waiting to be written.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_holding_write_locks

Threads Holding the Write Lock

Number of threads that hold write locks.

>=0

Count

N/A

CloudTable instance node

60s

maximum_number_of_active_blocks

Max. Active Blocks

Maximum number of active blocks of the ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_consistency_checkdata_blocks

Data Blocks for Consistency Check

Number of data blocks for consistency check on a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

extracting_the_number_of_data_chunks_from_the_replica

Data Blocks Being Extracted from a Replica

Number of data blocks that are being extracted from a replica by a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_data_chunks_being_sent_to_the_replica

Data Blocks Being Sent to a Replica

Number of data blocks on a ClickHouse instance that are being sent to a replica.

>=0

Count

N/A

CloudTable instance node

60s

number_of_readonly_replicas

Read-Only Replicas

Number of read-only replicas of a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_inserted_into_all_tables

Inserted Rows

Number of rows inserted into all tables in a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_open_read_files

Opened Read Files

Number of opened read files on a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_open_write_files

Opened Write Files

Number of opened write files on a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

total_file_opens

Opened Files

Number of opened files on a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_read_file_descriptors

Read File Descriptors

Number of file descriptors that have been read.

>=0

Count

N/A

CloudTable instance node

60s

number_of_written_file_descriptors

Write File Descriptors

Number of file descriptors that have been written.

>=0

Count

N/A

CloudTable instance node

60s

instance_process_running_duration

Instance Process Running Duration

Running duration of the process on a ClickHouse instance.

>=0

s

N/A

CloudTable instance node

60s

number_of_threads_running_in_global_thread_pool

Threads of Running Tasks in Global Thread Pool

Number of threads of running tasks in the global thread pool of a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_in_global_thread_pool

Threads in Global Thread Pool

Number of threads in the global thread pool of a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_in_local_thread_pool

Threads in Local Thread Pool

Number of threads in the local thread pool of a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_waiting_locked

Threads to Lock in Context

Number of threads waiting to be locked in context on a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_threads_running_in_local_thread_pool

Threads of Running Tasks in Local Thread Pool

Number of threads of running tasks in the local thread pool of a ClickHouse instance.

>=0

Count

N/A

CloudTable instance node

60s

number_of_tcp_connections

TCP Connections

Number of TCP connections.

[0,4096]

Count

N/A

CloudTable instance node

60s

number_of_http_connections

HTTP Connections

Number of HTTP connections.

[0,4096]

Count

N/A

CloudTable instance node

60s

number_of_databases

Databases

Number of databases on each node.

[0, 5000]

Count

N/A

CloudTable instance node

60s

number_of_tables

Tables

Number of local tables on each node.

[0, 5000]

Count

N/A

CloudTable instance node

60s

write_size_per_second

Write Size per Second

Amount of data written to each node per unit time.

>=0

Byte, KiB, MiB, GiB, TiB, and PiB

1024(IEC)

CloudTable instance node

60s

number_of_running_queries

Running Queries

Number of running queries on each node.

>=0

Count

N/A

CloudTable instance node

60s

number_of_query_threads

Query Threads

Number of running queries on each node.

>=0

Count

N/A

CloudTable instance node

60s

number_of_distributed_ddls

Distributed DDLs

Number of distributed DDL statements.

>=0

Count

N/A

CloudTable instance node

60s

number_of_distributed_table_files

Distributed Tables

Number of tables to which data is to be inserted.

>=0

Count

N/A

CloudTable instance node

60s

number_of_concurrency

Concurrency of Each ClickHouse Node

Concurrency of each ClickHouse node.

[0,100]

Count

N/A

CloudTable instance node

60s

zookeeper_quota_percentage

ZooKeeper Quantity Percentage

Quota of the ZooKeeper nodes used by ClickHouse.

0~100

%

N/A

CloudTable instance node

60s

zookeeper_capacity_quota_percentage

ZooKeeper Capacity Percentage

Capacity quota of ZooKeeper directory used by ClickHouse.

0~100

%

N/A

CloudTable instance node

60s

zookeeper_process_is_alive

ZooKeeper Process Survival Status

ZooKeeper heartbeat check. 1 indicates that the ZooKeeper heartbeat is normal, and 0 indicates that it is abnormal.

[0, 1]

Count

N/A

CloudTable instance node

60s

number_of_global_sessions

Global Sessions

Number of global sessions.

>=0

Count

N/A

CloudTable instance node

60s

number_of_rejected_connections

Refused Connections

Number of connections refused by Zookeeper per unit time.

>=0

Count

N/A

CloudTable instance node

60s

number_of_request_submission_queues

Requests to Submit in a Queue

Number of requests waiting to be submitted in a queue at the current time.

>=0

Count

N/A

CloudTable instance node

60s

waiting_time_of_the_preprocessing_queue

Wait Time of Request in Preprocessing Queue

Time that a request in the preprocessing queue waits to be processed per unit time.

>=0

s

N/A

CloudTable instance node

60s

number_of_zk_watches

ZooKeeper Watches

Number of ZooKeeper watches at the current time.

>=0

Count

N/A

CloudTable instance node

60s

zk_heapmemory_usage

Heap Memory Used

Heap memory size used by ZooKeeper process.

>=0

Byte, KiB, MiB, GiB, TiB, and PiB

1024(IEC)

CloudTable instance node

60s

zk_directmemory_usage

Used Direct Memory

Direct memory size used by ZooKeeper process.

>=0

Byte, KiB, MiB, GiB, TiB, and PiB

1024(IEC)

CloudTable instance node

60s

zk_heapmemory_usage_ratio

Heap Memory Usage

Heap memory usage of the ZooKeeper service.

0~100

%

N/A

CloudTable instance node

60s

zk_directmemory_usage_ratio

Direct Memory Usage

Direct memory usage of the ZooKeeper service.

0~100

%

N/A

CloudTable instance node

60s

cluster_number_of_sent_packets

ZooKeeper Sent Packets

Number of packets sent by a ZooKeeper node.

>=0

Count

N/A

CloudTable instance node

60s

cluster_number_of_received_packets

ZooKeeper Received Packets

Number of packets received by a ZooKeeper node.

>=0

Count

N/A

CloudTable instance node

60s

cluster_preprocessing_time

Preprocessing Duration

Preprocessing time.

>=0

s

N/A

CloudTable instance node

60s

rows_written_per_second

Rows Written per Second

Number of rows written to each node per unit time.

>=0

Count

N/A

CloudTable instance node

60s

number_of_failed_queries

Failed Queries

Number of failed query statements.

>=0

Count

N/A

CloudTable instance node

60s

number_of_failed_insert_queries

Failed Insert Queries

Number of failed insert statements.

>=0

Count

N/A

CloudTable instance node

60s

number_of_failed_select_query_messages

Failed Select Queries

Number of failed select statements.

>=0

Count

N/A

CloudTable instance node

60s

number_of_delayed_inserts

Delayed Inserts

Number of delayed insertions on each node.

>=0

Count

N/A

CloudTable instance node

60s

number_of_rows

Rows

Number of mergetree engine fields on each node.

>=0

Count

N/A

CloudTable instance node

60s

disk_throughput_read_rate

Disk Read Throughput

Disk read throughput.

>=0

Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, and PiB/s

1024(IEC)

CloudTable instance node

60s

disk_throughput_write_rate

Disk Write Throughput

Disk write throughput.

>=0

Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, and PiB/s

1024(IEC)

CloudTable instance node

60s

Dimension

Key

Value

cluster_id

CloudTable cluster ID.

instance_name

Name of a CloudTable cluster node.