Volcengine Kafka¶

Volcengine Kafka metrics include Broker online rate, CPU utilization, memory utilization, disk utilization, message production and consumption rates, and request latency. These metrics reflect the operating status, resource usage, and performance of Kafka instances.

Configuration¶

Install Func¶

We recommend enabling managed Func under Guance Integration - Extensions. All prerequisites are installed automatically; proceed with the script installation.

To deploy Func yourself, see Self-hosted Func.

Install the Script¶

Note: Prepare a Volcengine AK that meets the requirements in advance (for simplicity, you can grant the global read-only permission ReadOnlyAccess).

Log in to the Func console, click [Script Market] to open the Guance Script Market, and search for integration_volcengine_kafka.
Click [Install], then enter the required parameters: Volcengine AK ID, AK Secret, and account name.
Click [Deploy Startup Script]. The system automatically creates a Startup script set and configures the corresponding startup script.
Once enabled, the corresponding automatic trigger configuration appears under [Management / Automatic Trigger Configuration]. Click [Execute] to run it immediately without waiting for the scheduled time. After a short wait, check the task execution records and logs.

Verification¶

Under [Management / Automatic Trigger Configuration], confirm that the task has a corresponding automatic trigger configuration. Check its task records and logs for errors.
In Guance, check for asset information under [Infrastructure / Custom].
In Guance, check for the corresponding monitoring data under [Metrics].

Metrics¶

After configuring Volcengine Cloud Monitoring, the default metrics are listed below. You can configure additional metrics for collection; see Volcengine Cloud Monitoring Metric Details.

Note: Install the monitoring plugin in the volcengine Kafka console.

`MetricName`	`Subnamespace`	Metric Name	MetricUnit	Dimension
`MetricName`	`SubNamespace`	Metric Name	MetricUnit	Dimension
`broker_online_rate`	`Instance`	Node Online Rate	Percent	ResourceID
`cap_util`	`Instance`	Capacity Utilization	Percent	ResourceID
`connections_count`	`Instance`	Instance Connections	Count	ResourceID
`connections_per_ip`	`Instance`	Connections per IP	Count	ClientIP,ResourceID
`consumption_rate_percent`	`Instance`	Consumption Bandwidth Percentage	Percent	ResourceID
`consumption_request_mean_time`	`Instance`	Average Message Consumption Latency	Millisecond	ResourceID
`consumption_request_p99_time`	`Instance`	P99 Message Consumption Latency	Millisecond	ResourceID
`cpu_util`	`Instance`	CPU Utilization	Percent	ResourceID
`disk_util`	`Broker`	Disk Utilization	Percent	ResourceID,Broker
`dsik_skewness`	`Instance`	Disk Skewness Ratio	Percent	ResourceID
`group_consumption_rate`	`Group`	Consumer Group Consumption Rate	Count	Group,ResourceID
`group_msg_accum`	`Group`	Consumer Group Message Backlog	Count	Group,ResourceID
`group_rebalance_count`	`Group`	Consumer Group Rebalance Count	Count	Group,ResourceID
`instance_available_group`	`Instance`	Remaining Available Consumer Groups	Count	ResourceID
`instance_available_partition`	`Instance`	Remaining Available Partitions	Count	ResourceID
`max_disk_util`	`Instance`	Maximum Node Disk Utilization	Percent	ResourceID
`mem_util`	`Instance`	Memory Utilization	Percent	ResourceID
`msg_consumption_rate`	`Instance`	Message Consumption Rate	Bytes/Second(SI)	ResourceID
`msg_production_count`	`Instance`	Messages Produced	Count/Second	ResourceID
`msg_production_rate`	`Instance`	Message Production Rate	Bytes/Second(SI)	ResourceID
`partition_msg_accum`	`Group_Partition`	Partition Message Backlog	Count	Group,Partition,ResourceID,Topic
`produce_request_mean_time`	`Broker`	Average Message Production Latency	Millisecond	ResourceID,Broker
`production_rate_percent`	`Instance`	Production Bandwidth Percentage	Percent	ResourceID
`production_request_mean_time`	`Instance`	Average Message Production Latency	Millisecond	ResourceID
`production_request_p99_time`	`Instance`	P99 Message Production Latency	Millisecond	ResourceID
`req_consumption_count`	`Instance`	Message Consumption Count	Count/Second	ResourceID
`req_production_count`	`Instance`	Message Production Count	Count/Second	ResourceID
`topic_consumption_rate`	`Group_Topic`	Topic Consumption Rate	Count	Group,ResourceID,Topic
`topic_data_size`	`Topic`	Topic Data Size	Bits(IEC)	ResourceID,Topic
`topic_msg_accum`	`Group_Topic`	Topic Message Backlog Size	Count	Group,ResourceID,Topic

Objects¶

The collected Volcengine Kafka object is named volcengine_kafka. You can view its data and structure under [Infrastructure - Custom].