ClickHouse Cluster Monitoring Metrics
Description
Monitoring is critical to ensure CloudTable reliability, availability, and performance. You can monitor the running status of CloudTable servers. This section describes the metrics that can be monitored by Cloud Eye as well as their namespaces and dimensions.
Namespace
SYS.CloudTable
Monitoring metrics
Metric | Name | Description | Value Range | Metric Unit | Conversion Rule | Monitored Object (Dimension) | Monitoring Period |
|---|---|---|---|---|---|---|---|
cmdProcessCPU | CPU Utilization Rate | CPU usage of the monitored object | 0~100 | % | N/A | CloudTable instance node | 60s |
memory_usage | Memory Usage | Memory usage of each node. | [0, Node memory capacity] | MB | N/A | CloudTable instance node | 60s |
cmdProcessMem | Memory Utilization Rate | Memory usage of the monitored object | 0~100 | % | N/A | CloudTable instance node | 60s |
disk_usage | Disk Usage | Disk space used by each node. | [0, Node disk capacity] | MB | N/A | CloudTable instance node | 60s |
cmdForUsedStorageRate | Used Storage Rate | Ratio of the used storage space to the total storage space in the cluster | >=0 | % | N/A | CloudTable instance node | 60s |
network_throughput_inbound_rate | Inbound Throughput | Inbound data volume over network of each node per second. | >=0 | KB/s | N/A | CloudTable instance node | 60s |
network_throughput_outgoing_rate | Outbound Throughput | Outbound data volume over network of each node per second. | >=0 | KB/s | N/A | CloudTable instance node | 60s |
number_of_mutations | Mutations | Number of running modification tasks on each node. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_run_merges | Running Merges | Number of running merge tasks. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_merges_thread | Merge Threads | Number of threads being merged. | [0, Number of CPU cores] | Count | N/A | CloudTable instance node | 60s |
number_of_zookeeper_requests | ZooKeeper Requests | Number of ZooKeeper requests on the node at the current time. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_zk_session | ZooKeeper Sessions | Number of connected ZooKeeper sessions on the node at the current time. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_zookeeper_watches | ZooKeeper Watches | Number of connected ZooKeeper watches on the node at the current time. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_zookeeper_nodes | ZooKeeper Nodes | Number of temporary nodes created in ZooKeeper. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_parts | Parts | Number of parts. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_holding_read_locks | Threads Holding the Read Lock | Number of threads that hold read locks. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_waiting_to_be_read | Threads Waiting to Be Read | Number of threads waiting to be read. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_waiting_to_be_written | Number of Threads Waiting to Be Written | Number of threads waiting to be written. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_holding_write_locks | Threads Holding the Write Lock | Number of threads that hold write locks. | >=0 | Count | N/A | CloudTable instance node | 60s |
maximum_number_of_active_blocks | Max. Active Blocks | Maximum number of active blocks of the ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_consistency_checkdata_blocks | Data Blocks for Consistency Check | Number of data blocks for consistency check on a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
extracting_the_number_of_data_chunks_from_the_replica | Data Blocks Being Extracted from a Replica | Number of data blocks that are being extracted from a replica by a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_data_chunks_being_sent_to_the_replica | Data Blocks Being Sent to a Replica | Number of data blocks on a ClickHouse instance that are being sent to a replica. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_readonly_replicas | Read-Only Replicas | Number of read-only replicas of a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_inserted_into_all_tables | Inserted Rows | Number of rows inserted into all tables in a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_open_read_files | Opened Read Files | Number of opened read files on a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_open_write_files | Opened Write Files | Number of opened write files on a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
total_file_opens | Opened Files | Number of opened files on a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_read_file_descriptors | Read File Descriptors | Number of file descriptors that have been read. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_written_file_descriptors | Write File Descriptors | Number of file descriptors that have been written. | >=0 | Count | N/A | CloudTable instance node | 60s |
instance_process_running_duration | Instance Process Running Duration | Running duration of the process on a ClickHouse instance. | >=0 | s | N/A | CloudTable instance node | 60s |
number_of_threads_running_in_global_thread_pool | Threads of Running Tasks in Global Thread Pool | Number of threads of running tasks in the global thread pool of a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_in_global_thread_pool | Threads in Global Thread Pool | Number of threads in the global thread pool of a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_in_local_thread_pool | Threads in Local Thread Pool | Number of threads in the local thread pool of a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_waiting_locked | Threads to Lock in Context | Number of threads waiting to be locked in context on a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_threads_running_in_local_thread_pool | Threads of Running Tasks in Local Thread Pool | Number of threads of running tasks in the local thread pool of a ClickHouse instance. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_tcp_connections | TCP Connections | Number of TCP connections. | [0,4096] | Count | N/A | CloudTable instance node | 60s |
number_of_http_connections | HTTP Connections | Number of HTTP connections. | [0,4096] | Count | N/A | CloudTable instance node | 60s |
number_of_databases | Databases | Number of databases on each node. | [0, 5000] | Count | N/A | CloudTable instance node | 60s |
number_of_tables | Tables | Number of local tables on each node. | [0, 5000] | Count | N/A | CloudTable instance node | 60s |
write_size_per_second | Write Size per Second | Amount of data written to each node per unit time. | >=0 | Byte, KiB, MiB, GiB, TiB, and PiB | 1024(IEC) | CloudTable instance node | 60s |
number_of_running_queries | Running Queries | Number of running queries on each node. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_query_threads | Query Threads | Number of running queries on each node. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_distributed_ddls | Distributed DDLs | Number of distributed DDL statements. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_distributed_table_files | Distributed Tables | Number of tables to which data is to be inserted. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_concurrency | Concurrency of Each ClickHouse Node | Concurrency of each ClickHouse node. | [0,100] | Count | N/A | CloudTable instance node | 60s |
zookeeper_quota_percentage | ZooKeeper Quantity Percentage | Quota of the ZooKeeper nodes used by ClickHouse. | 0~100 | % | N/A | CloudTable instance node | 60s |
zookeeper_capacity_quota_percentage | ZooKeeper Capacity Percentage | Capacity quota of ZooKeeper directory used by ClickHouse. | 0~100 | % | N/A | CloudTable instance node | 60s |
zookeeper_process_is_alive | ZooKeeper Process Survival Status | ZooKeeper heartbeat check. 1 indicates that the ZooKeeper heartbeat is normal, and 0 indicates that it is abnormal. | [0, 1] | Count | N/A | CloudTable instance node | 60s |
number_of_global_sessions | Global Sessions | Number of global sessions. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_rejected_connections | Refused Connections | Number of connections refused by Zookeeper per unit time. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_request_submission_queues | Requests to Submit in a Queue | Number of requests waiting to be submitted in a queue at the current time. | >=0 | Count | N/A | CloudTable instance node | 60s |
waiting_time_of_the_preprocessing_queue | Wait Time of Request in Preprocessing Queue | Time that a request in the preprocessing queue waits to be processed per unit time. | >=0 | s | N/A | CloudTable instance node | 60s |
number_of_zk_watches | ZooKeeper Watches | Number of ZooKeeper watches at the current time. | >=0 | Count | N/A | CloudTable instance node | 60s |
zk_heapmemory_usage | Heap Memory Used | Heap memory size used by ZooKeeper process. | >=0 | Byte, KiB, MiB, GiB, TiB, and PiB | 1024(IEC) | CloudTable instance node | 60s |
zk_directmemory_usage | Used Direct Memory | Direct memory size used by ZooKeeper process. | >=0 | Byte, KiB, MiB, GiB, TiB, and PiB | 1024(IEC) | CloudTable instance node | 60s |
zk_heapmemory_usage_ratio | Heap Memory Usage | Heap memory usage of the ZooKeeper service. | 0~100 | % | N/A | CloudTable instance node | 60s |
zk_directmemory_usage_ratio | Direct Memory Usage | Direct memory usage of the ZooKeeper service. | 0~100 | % | N/A | CloudTable instance node | 60s |
cluster_number_of_sent_packets | ZooKeeper Sent Packets | Number of packets sent by a ZooKeeper node. | >=0 | Count | N/A | CloudTable instance node | 60s |
cluster_number_of_received_packets | ZooKeeper Received Packets | Number of packets received by a ZooKeeper node. | >=0 | Count | N/A | CloudTable instance node | 60s |
cluster_preprocessing_time | Preprocessing Duration | Preprocessing time. | >=0 | s | N/A | CloudTable instance node | 60s |
rows_written_per_second | Rows Written per Second | Number of rows written to each node per unit time. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_failed_queries | Failed Queries | Number of failed query statements. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_failed_insert_queries | Failed Insert Queries | Number of failed insert statements. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_failed_select_query_messages | Failed Select Queries | Number of failed select statements. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_delayed_inserts | Delayed Inserts | Number of delayed insertions on each node. | >=0 | Count | N/A | CloudTable instance node | 60s |
number_of_rows | Rows | Number of mergetree engine fields on each node. | >=0 | Count | N/A | CloudTable instance node | 60s |
disk_throughput_read_rate | Disk Read Throughput | Disk read throughput. | >=0 | Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, and PiB/s | 1024(IEC) | CloudTable instance node | 60s |
disk_throughput_write_rate | Disk Write Throughput | Disk write throughput. | >=0 | Byte/s, KiB/s, MiB/s, GiB/s, TiB/s, and PiB/s | 1024(IEC) | CloudTable instance node | 60s |
Dimension
Key | Value |
|---|---|
cluster_id | CloudTable cluster ID. |
instance_name | Name of a CloudTable cluster node. |
- Description
- Namespace
- Monitoring metrics
- Dimension