Kafka
采集 Kafka 指标和日志上报到观测云，帮助你监控分析 Kafka 各种异常情况。
配置¶
前置条件¶
安装或下载 Jolokia。DataKit 安装目录下的 data 目录中已经有下载好的 Jolokia jar 包。
Jolokia 是作为 Kafka 的 Java agent，基于 HTTP 协议提供了一个使用 JSON 作为数据格式的外部接口，提供给 DataKit 使用。 Kafka 启动时，先配置 KAFKA_OPTS 环境变量：(port 可根据实际情况修改成可用端口）
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/local/datakit/data/jolokia-jvm-agent.jar=host=*,port=8080"
另外，也可以单独启动 Jolokia，将其指向 Kafka 进程 PID：
Note
- Jolokia 不允许运行过程中修改端口号。如果发现通过
--port命令无法修改端口号，就是这个原因。若想修改 Jolokia 端口号必须先退出 Jolokia 再启动才能成功。 - 退出 Jolokia 命令是：
java -jar </path/to/jolokia-jvm-agent.jar> --quiet stop <Kafka-PID>。更多 Jolokia 命令信息可参考这里。
采集器配置¶
进入 DataKit 安装目录下的 conf.d/samples 目录，复制 kafka.conf.sample 并命名为 kafka.conf。示例如下：
[[inputs.kafka]]
# default_tag_prefix = ""
# default_field_prefix = ""
# default_field_separator = "."
# username = ""
# password = ""
# response_timeout = "5s"
## Optional TLS config
# tls_ca = "/var/private/ca.pem"
# tls_cert = "/var/private/client.pem"
# tls_key = "/var/private/client-key.pem"
# insecure_skip_verify = false
## Monitor Interval
# interval = "60s"
election = true
# Add agents URLs to query
urls = ["http://localhost:8080/jolokia"]
## Add metrics to read
[[inputs.kafka.metric]]
name = "kafka_controller"
mbean = "kafka.controller:name=*,type=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_replica_manager"
mbean = "kafka.server:name=*,type=ReplicaManager"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_zookeeper"
mbean = "kafka.server:type=ZooKeeperClientMetrics,name=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_purgatory"
mbean = "kafka.server:delayedOperation=*,name=*,type=DelayedOperationPurgatory"
field_name = "#1.#2"
[[inputs.kafka.metric]]
name = "kafka_client"
mbean = "kafka.server:client-id=*,type=*"
tag_keys = ["client-id", "type"]
[[inputs.kafka.metric]]
name = "kafka_request"
mbean = "kafka.network:name=*,request=*,type=RequestMetrics"
field_prefix = "#1."
tag_keys = ["request"]
[[inputs.kafka.metric]]
name = "kafka_request_handler"
mbean = "kafka.server:type=KafkaRequestHandlerPool,name=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_network"
mbean = "kafka.network:type=*,name=*"
field_name = "#2"
tag_keys = ["type"]
[[inputs.kafka.metric]]
name = "kafka_topics"
mbean = "kafka.server:name=*,type=BrokerTopicMetrics"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_topic"
mbean = "kafka.server:name=*,topic=*,type=BrokerTopicMetrics"
field_prefix = "#1."
tag_keys = ["topic"]
[[inputs.kafka.metric]]
name = "kafka_partition"
mbean = "kafka.log:name=*,partition=*,topic=*,type=Log"
field_name = "#1"
tag_keys = ["topic", "partition"]
[[inputs.kafka.metric]]
name = "kafka_log"
mbean = "kafka.log:type=*,name=*"
field_name = "#2"
tag_keys = ["type"]
[[inputs.kafka.metric]]
name = "kafka_partition"
mbean = "kafka.cluster:name=UnderReplicated,partition=*,topic=*,type=Partition"
field_name = "UnderReplicatedPartitions"
tag_keys = ["topic", "partition"]
# # The following metrics are available on consumer instances.
# [[inputs.kafka.metric]]
# name = "kafka_consumer"
# mbean = "kafka.consumer:type=*,client-id=*"
# tag_keys = ["client-id", "type"]
# # The following metrics are available on producer instances.
# [[inputs.kafka.metric]]
# name = "kafka_producer"
# mbean = "kafka.producer:type=*,client-id=*"
# tag_keys = ["client-id", "type"]
# # The following metrics are available on connector instances.
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*"
# tag_keys = ["type"]
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*,connector=*"
# tag_keys = ["type", "connector"]
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*,connector=*,task=*"
# tag_keys = ["type", "connector", "task"]
# [inputs.kafka.log]
# files = []
# #grok pipeline script path
# pipeline = "kafka.p"
[inputs.kafka.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
配置好后，重启 DataKit 即可。
目前可以通过 ConfigMap 方式注入采集器配置来开启采集器。
指标¶
以下所有数据采集，默认会追加全局选举 tag，也可以在配置中通过 [inputs.kafka.tags] 指定其它标签：
kafka_controller¶
In Kafka cluster mode, a unique controller node will be elected, and only the controller node will receive valid metrics.
| Tags & Fields | Description |
|---|---|
| ActiveControllerCount.Value | Type: int Unit: count |
| AutoLeaderBalanceRateAndTimeMs.50thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.75thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.95thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.98thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.999thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.99thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Count | Type: int Unit: count |
| AutoLeaderBalanceRateAndTimeMs.EventType | Type: string Unit: - |
| AutoLeaderBalanceRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Max | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Mean | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.MeanRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Min | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.RateUnit | Type: string Unit: - |
| AutoLeaderBalanceRateAndTimeMs.StdDev | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.Count | Type: int Unit: count |
| ControlledShutdownRateAndTimeMs.EventType | Type: string Unit: - |
| ControlledShutdownRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ControlledShutdownRateAndTimeMs.Max | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.Mean | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.MeanRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.Min | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.RateUnit | Type: string Unit: - |
| ControlledShutdownRateAndTimeMs.StdDev | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.Count | Type: int Unit: count |
| ControllerChangeRateAndTimeMs.EventType | Type: string Unit: - |
| ControllerChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ControllerChangeRateAndTimeMs.Max | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.Mean | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.Min | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| ControllerChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.Count | Type: int Unit: count |
| ControllerShutdownRateAndTimeMs.EventType | Type: string Unit: - |
| ControllerShutdownRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ControllerShutdownRateAndTimeMs.Max | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.Mean | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.MeanRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.Min | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.RateUnit | Type: string Unit: - |
| ControllerShutdownRateAndTimeMs.StdDev | Type: float Unit: - |
| ControllerState.Value | Type: int Unit: - |
| EventQueueSize.Value | Type: int Unit: - |
| EventQueueTimeMs.50thPercentile | Type: float Unit: - |
| EventQueueTimeMs.75thPercentile | Type: float Unit: - |
| EventQueueTimeMs.95thPercentile | Type: float Unit: - |
| EventQueueTimeMs.98thPercentile | Type: float Unit: - |
| EventQueueTimeMs.999thPercentile | Type: float Unit: - |
| EventQueueTimeMs.99thPercentile | Type: float Unit: - |
| EventQueueTimeMs.Count | Type: int Unit: count |
| EventQueueTimeMs.EventType | Type: string Unit: - |
| EventQueueTimeMs.FifteenMinuteRate | Type: float Unit: - |
| EventQueueTimeMs.FiveMinuteRate | Type: float Unit: - |
| EventQueueTimeMs.LatencyUnit | Type: string Unit: - |
| EventQueueTimeMs.Max | Type: float Unit: - |
| EventQueueTimeMs.Mean | Type: float Unit: - |
| EventQueueTimeMs.MeanRate | Type: float Unit: - |
| EventQueueTimeMs.Min | Type: float Unit: - |
| EventQueueTimeMs.OneMinuteRate | Type: float Unit: - |
| EventQueueTimeMs.RateUnit | Type: string Unit: - |
| EventQueueTimeMs.StdDev | Type: float Unit: - |
| GlobalPartitionCount.Value | Type: int Unit: count |
| GlobalTopicCount.Value | Type: int Unit: count |
| IsrChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.Count | Type: int Unit: count |
| IsrChangeRateAndTimeMs.EventType | Type: string Unit: - |
| IsrChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| IsrChangeRateAndTimeMs.Max | Type: float Unit: - |
| IsrChangeRateAndTimeMs.Mean | Type: float Unit: - |
| IsrChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.Min | Type: float Unit: - |
| IsrChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| IsrChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.50thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.75thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.95thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.98thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.999thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.99thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Count | Type: int Unit: count |
| LeaderAndIsrResponseReceivedRateAndTimeMs.EventType | Type: string Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Max | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Mean | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.MeanRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Min | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.RateUnit | Type: string Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.StdDev | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.50thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.75thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.95thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.98thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.999thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.99thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.Count | Type: int Unit: count |
| LeaderElectionRateAndTimeMs.EventType | Type: string Unit: - |
| LeaderElectionRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| LeaderElectionRateAndTimeMs.Max | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.Mean | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.MeanRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.Min | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.RateUnit | Type: string Unit: - |
| LeaderElectionRateAndTimeMs.StdDev | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Count | Type: int Unit: count |
| ListPartitionReassignmentRateAndTimeMs.EventType | Type: string Unit: - |
| ListPartitionReassignmentRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Max | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Mean | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.MeanRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Min | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.RateUnit | Type: string Unit: - |
| ListPartitionReassignmentRateAndTimeMs.StdDev | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.Count | Type: int Unit: count |
| LogDirChangeRateAndTimeMs.EventType | Type: string Unit: - |
| LogDirChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| LogDirChangeRateAndTimeMs.Max | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.Mean | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.Min | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| LogDirChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Count | Type: int Unit: count |
| ManualLeaderBalanceRateAndTimeMs.EventType | Type: string Unit: - |
| ManualLeaderBalanceRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Max | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Mean | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.MeanRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Min | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.RateUnit | Type: string Unit: - |
| ManualLeaderBalanceRateAndTimeMs.StdDev | Type: float Unit: - |
| OfflinePartitionsCount.Value | Type: int Unit: count |
| PartitionReassignmentRateAndTimeMs.50thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.75thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.95thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.98thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.999thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.99thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.Count | Type: int Unit: count |
| PartitionReassignmentRateAndTimeMs.EventType | Type: string Unit: - |
| PartitionReassignmentRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| PartitionReassignmentRateAndTimeMs.Max | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.Mean | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.MeanRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.Min | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.RateUnit | Type: string Unit: - |
| PartitionReassignmentRateAndTimeMs.StdDev | Type: float Unit: - |
| PreferredReplicaImbalanceCount.Value | Type: int Unit: count |
| ReplicasIneligibleToDeleteCount.Value | Type: int Unit: count |
| ReplicasToDeleteCount.Value | Type: int Unit: count |
| TopicChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.Count | Type: int Unit: count |
| TopicChangeRateAndTimeMs.EventType | Type: string Unit: - |
| TopicChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| TopicChangeRateAndTimeMs.Max | Type: float Unit: - |
| TopicChangeRateAndTimeMs.Mean | Type: float Unit: - |
| TopicChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.Min | Type: float Unit: - |
| TopicChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| TopicChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.50thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.75thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.95thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.98thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.999thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.99thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.Count | Type: int Unit: count |
| TopicDeletionRateAndTimeMs.EventType | Type: string Unit: - |
| TopicDeletionRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| TopicDeletionRateAndTimeMs.Max | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.Mean | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.MeanRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.Min | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.RateUnit | Type: string Unit: - |
| TopicDeletionRateAndTimeMs.StdDev | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.50thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.75thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.95thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.98thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.999thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.99thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Count | Type: int Unit: count |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.EventType | Type: string Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Max | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Mean | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.MeanRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Min | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.RateUnit | Type: string Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.StdDev | Type: float Unit: - |
| TopicsIneligibleToDeleteCount.Value | Type: int Unit: count |
| TopicsToDeleteCount.Value | Type: int Unit: count |
| TotalQueueSize.Value | Type: int Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.50thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.75thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.95thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.98thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.999thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.99thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Count | Type: int Unit: count |
| UncleanLeaderElectionEnableRateAndTimeMs.EventType | Type: string Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Max | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Mean | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.MeanRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Min | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.RateUnit | Type: string Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.StdDev | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.50thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.75thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.95thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.98thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.999thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.99thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.Count | Type: int Unit: count |
| UncleanLeaderElectionsPerSec.EventType | Type: string Unit: - |
| UncleanLeaderElectionsPerSec.FifteenMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.FiveMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.LatencyUnit | Type: string Unit: - |
| UncleanLeaderElectionsPerSec.Max | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.Mean | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.MeanRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.Min | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.OneMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.RateUnit | Type: string Unit: - |
| UncleanLeaderElectionsPerSec.StdDev | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.50thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.75thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.95thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.98thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.999thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.99thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.Count | Type: int Unit: count |
| UpdateFeaturesRateAndTimeMs.EventType | Type: string Unit: - |
| UpdateFeaturesRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| UpdateFeaturesRateAndTimeMs.Max | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.Mean | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.MeanRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.Min | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.RateUnit | Type: string Unit: - |
| UpdateFeaturesRateAndTimeMs.StdDev | Type: float Unit: - |
kafka_replica_manager¶
| Tags & Fields | Description |
|---|---|
| AtMinIsrPartitionCount.Value | Type: int Unit: count |
| FailedIsrUpdatesPerSec.Count | Type: int Unit: count |
| FailedIsrUpdatesPerSec.EventType | Type: string Unit: - |
| FailedIsrUpdatesPerSec.FifteenMinuteRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.FiveMinuteRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.MeanRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.OneMinuteRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.RateUnit | Type: string Unit: - |
| IsrExpandsPerSec.Count | Type: int Unit: count |
| IsrExpandsPerSec.EventType | Type: string Unit: - |
| IsrExpandsPerSec.FifteenMinuteRate | Type: float Unit: - |
| IsrExpandsPerSec.FiveMinuteRate | Type: float Unit: - |
| IsrExpandsPerSec.MeanRate | Type: float Unit: - |
| IsrExpandsPerSec.OneMinuteRate | Type: float Unit: - |
| IsrExpandsPerSec.RateUnit | Type: string Unit: - |
| IsrShrinksPerSec.Count | Type: int Unit: count |
| IsrShrinksPerSec.EventType | Type: string Unit: - |
| IsrShrinksPerSec.FifteenMinuteRate | Type: float Unit: - |
| IsrShrinksPerSec.FiveMinuteRate | Type: float Unit: - |
| IsrShrinksPerSec.MeanRate | Type: float Unit: - |
| IsrShrinksPerSec.OneMinuteRate | Type: float Unit: - |
| IsrShrinksPerSec.RateUnit | Type: string Unit: - |
| LeaderCount.Value | Type: int Unit: count |
| OfflineReplicaCount.Value | Type: int Unit: count |
| PartitionCount.Value | Type: int Unit: count |
| ReassigningPartitions.Value | Type: int Unit: count |
| UnderMinIsrPartitionCount.Value | Type: int Unit: count |
| UnderReplicatedPartitions.Value | Type: int Unit: count |
kafka_purgatory¶
| Tags & Fields | Description |
|---|---|
| AlterAcls.NumDelayedOperations | Type: int Unit: - |
| AlterAcls.PurgatorySize | Type: int Unit: - |
| DeleteRecords.NumDelayedOperations | Type: int Unit: - |
| DeleteRecords.PurgatorySize | Type: int Unit: - |
| ElectLeader.NumDelayedOperations | Type: int Unit: - |
| ElectLeader.PurgatorySize | Type: int Unit: - |
| Fetch.NumDelayedOperations | Type: int Unit: - |
| Fetch.PurgatorySize | Type: int Unit: - |
| Heartbeat.NumDelayedOperations | Type: int Unit: - |
| Heartbeat.PurgatorySize | Type: int Unit: - |
| Produce.NumDelayedOperations | Type: int Unit: - |
| Produce.PurgatorySize | Type: int Unit: - |
| Rebalance.NumDelayedOperations | Type: int Unit: - |
| Rebalance.PurgatorySize | Type: int Unit: - |
| topic.NumDelayedOperations | Type: int Unit: - |
| topic.PurgatorySize | Type: int Unit: - |
kafka_request¶
| Tags & Fields | Description |
|---|---|
| LocalTimeMs.50thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.75thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.95thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.98thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.999thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.99thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.Count | Type: int Unit: count |
| LocalTimeMs.Max | Type: float Unit: time,ms |
| LocalTimeMs.Mean | Type: float Unit: time,ms |
| LocalTimeMs.Min | Type: float Unit: time,ms |
| LocalTimeMs.StdDev | Type: float Unit: time,ms |
| RemoteTimeMs.50thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.75thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.95thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.98thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.999thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.99thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.Count | Type: int Unit: count |
| RemoteTimeMs.Max | Type: float Unit: time,ms |
| RemoteTimeMs.Mean | Type: float Unit: time,ms |
| RemoteTimeMs.Min | Type: float Unit: time,ms |
| RemoteTimeMs.StdDev | Type: float Unit: time,ms |
| RequestBytes.50thPercentile | Type: float Unit: time,ms |
| RequestBytes.75thPercentile | Type: float Unit: time,ms |
| RequestBytes.95thPercentile | Type: float Unit: time,ms |
| RequestBytes.98thPercentile | Type: float Unit: time,ms |
| RequestBytes.999thPercentile | Type: float Unit: time,ms |
| RequestBytes.99thPercentile | Type: float Unit: time,ms |
| RequestBytes.Count | Type: int Unit: count |
| RequestBytes.Max | Type: float Unit: time,ms |
| RequestBytes.Mean | Type: float Unit: time,ms |
| RequestBytes.Min | Type: float Unit: time,ms |
| RequestBytes.StdDev | Type: float Unit: time,ms |
| RequestQueueTimeMs.50thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.75thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.95thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.98thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.999thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.99thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.Count | Type: int Unit: count |
| RequestQueueTimeMs.Max | Type: float Unit: time,ms |
| RequestQueueTimeMs.Mean | Type: float Unit: time,ms |
| RequestQueueTimeMs.Min | Type: float Unit: time,ms |
| RequestQueueTimeMs.StdDev | Type: float Unit: time,ms |
| ResponseQueueTimeMs.50thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.75thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.95thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.98thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.999thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.99thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.Count | Type: int Unit: count |
| ResponseQueueTimeMs.Max | Type: float Unit: time,ms |
| ResponseQueueTimeMs.Mean | Type: float Unit: time,ms |
| ResponseQueueTimeMs.Min | Type: float Unit: time,ms |
| ResponseQueueTimeMs.StdDev | Type: float Unit: time,ms |
| ResponseSendTimeMs.50thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.75thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.95thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.98thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.999thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.99thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.Count | Type: int Unit: count |
| ResponseSendTimeMs.Max | Type: float Unit: time,ms |
| ResponseSendTimeMs.Mean | Type: float Unit: time,ms |
| ResponseSendTimeMs.Min | Type: float Unit: time,ms |
| ResponseSendTimeMs.StdDev | Type: float Unit: time,ms |
| ThrottleTimeMs.50thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.75thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.95thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.98thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.999thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.99thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.Count | Type: int Unit: count |
| ThrottleTimeMs.Max | Type: float Unit: time,ms |
| ThrottleTimeMs.Mean | Type: float Unit: time,ms |
| ThrottleTimeMs.Min | Type: float Unit: time,ms |
| ThrottleTimeMs.StdDev | Type: float Unit: time,ms |
| TotalTimeMs.50thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.75thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.95thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.98thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.999thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.99thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.Count | Type: int Unit: count |
| TotalTimeMs.Max | Type: float Unit: time,ms |
| TotalTimeMs.Mean | Type: float Unit: time,ms |
| TotalTimeMs.Min | Type: float Unit: time,ms |
| TotalTimeMs.StdDev | Type: float Unit: time,ms |
kafka_topics¶
| Tags & Fields | Description |
|---|---|
| BytesInPerSec.Count | Type: int Unit: count |
| BytesInPerSec.EventType | Type: string Unit: - |
| BytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesInPerSec.MeanRate | Type: float Unit: - |
| BytesInPerSec.OneMinuteRate | Type: float Unit: - |
| BytesInPerSec.RateUnit | Type: string Unit: - |
| BytesOutPerSec.Count | Type: int Unit: count |
| BytesOutPerSec.EventType | Type: string Unit: - |
| BytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesOutPerSec.MeanRate | Type: float Unit: - |
| BytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| BytesOutPerSec.RateUnit | Type: string Unit: - |
| BytesRejectedPerSec.Count | Type: int Unit: count |
| BytesRejectedPerSec.EventType | Type: string Unit: - |
| BytesRejectedPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesRejectedPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesRejectedPerSec.MeanRate | Type: float Unit: - |
| BytesRejectedPerSec.OneMinuteRate | Type: float Unit: - |
| BytesRejectedPerSec.RateUnit | Type: string Unit: - |
| FailedFetchRequestsPerSec.Count | Type: int Unit: count |
| FailedFetchRequestsPerSec.EventType | Type: string Unit: - |
| FailedFetchRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.MeanRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.RateUnit | Type: string Unit: - |
| FailedProduceRequestsPerSec.Count | Type: int Unit: count |
| FailedProduceRequestsPerSec.EventType | Type: string Unit: - |
| FailedProduceRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.MeanRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.RateUnit | Type: string Unit: - |
| FetchMessageConversionsPerSec.Count | Type: int Unit: count |
| FetchMessageConversionsPerSec.EventType | Type: string Unit: - |
| FetchMessageConversionsPerSec.FifteenMinuteRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.FiveMinuteRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.MeanRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.OneMinuteRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.RateUnit | Type: string Unit: - |
| InvalidMagicNumberRecordsPerSec.Count | Type: int Unit: count |
| InvalidMagicNumberRecordsPerSec.EventType | Type: string Unit: - |
| InvalidMagicNumberRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.MeanRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.RateUnit | Type: string Unit: - |
| InvalidMessageCrcRecordsPerSec.Count | Type: int Unit: count |
| InvalidMessageCrcRecordsPerSec.EventType | Type: string Unit: - |
| InvalidMessageCrcRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.MeanRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.RateUnit | Type: string Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.Count | Type: int Unit: count |
| InvalidOffsetOrSequenceRecordsPerSec.EventType | Type: string Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.MeanRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.RateUnit | Type: string Unit: - |
| MessagesInPerSec.Count | Type: int Unit: count |
| MessagesInPerSec.EventType | Type: string Unit: - |
| MessagesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| MessagesInPerSec.FiveMinuteRate | Type: float Unit: - |
| MessagesInPerSec.MeanRate | Type: float Unit: - |
| MessagesInPerSec.OneMinuteRate | Type: float Unit: - |
| MessagesInPerSec.RateUnit | Type: string Unit: - |
| NoKeyCompactedTopicRecordsPerSec.Count | Type: int Unit: count |
| NoKeyCompactedTopicRecordsPerSec.EventType | Type: string Unit: - |
| NoKeyCompactedTopicRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.MeanRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.RateUnit | Type: string Unit: - |
| ProduceMessageConversionsPerSec.Count | Type: int Unit: count |
| ProduceMessageConversionsPerSec.EventType | Type: string Unit: - |
| ProduceMessageConversionsPerSec.FifteenMinuteRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.FiveMinuteRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.MeanRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.OneMinuteRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.RateUnit | Type: string Unit: - |
| ReassignmentBytesInPerSec.Count | Type: int Unit: count |
| ReassignmentBytesInPerSec.EventType | Type: string Unit: - |
| ReassignmentBytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.MeanRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.OneMinuteRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.RateUnit | Type: string Unit: - |
| ReassignmentBytesOutPerSec.Count | Type: int Unit: count |
| ReassignmentBytesOutPerSec.EventType | Type: string Unit: - |
| ReassignmentBytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.MeanRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.RateUnit | Type: string Unit: - |
| ReplicationBytesInPerSec.Count | Type: int Unit: count |
| ReplicationBytesInPerSec.EventType | Type: string Unit: - |
| ReplicationBytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReplicationBytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| ReplicationBytesInPerSec.MeanRate | Type: float Unit: - |
| ReplicationBytesInPerSec.OneMinuteRate | Type: float Unit: - |
| ReplicationBytesInPerSec.RateUnit | Type: string Unit: - |
| ReplicationBytesOutPerSec.Count | Type: int Unit: count |
| ReplicationBytesOutPerSec.EventType | Type: string Unit: - |
| ReplicationBytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.MeanRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.RateUnit | Type: string Unit: - |
| TotalFetchRequestsPerSec.Count | Type: int Unit: count |
| TotalFetchRequestsPerSec.EventType | Type: string Unit: - |
| TotalFetchRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.RateUnit | Type: string Unit: - |
| TotalProduceRequestsPerSec.Count | Type: int Unit: count |
| TotalProduceRequestsPerSec.EventType | Type: string Unit: - |
| TotalProduceRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.RateUnit | Type: string Unit: - |
kafka_topic¶
| Tags & Fields | Description |
|---|---|
| BytesInPerSec.Count | Type: int Unit: count |
| BytesInPerSec.EventType | Type: string Unit: - |
| BytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesInPerSec.MeanRate | Type: float Unit: - |
| BytesInPerSec.OneMinuteRate | Type: float Unit: - |
| BytesInPerSec.RateUnit | Type: string Unit: - |
| BytesOutPerSec.Count | Type: int Unit: count |
| BytesOutPerSec.EventType | Type: string Unit: - |
| BytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesOutPerSec.MeanRate | Type: float Unit: - |
| BytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| BytesOutPerSec.RateUnit | Type: string Unit: - |
| MessagesInPerSec.Count | Type: int Unit: count |
| MessagesInPerSec.EventType | Type: string Unit: - |
| MessagesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| MessagesInPerSec.FiveMinuteRate | Type: float Unit: - |
| MessagesInPerSec.MeanRate | Type: float Unit: - |
| MessagesInPerSec.OneMinuteRate | Type: float Unit: - |
| MessagesInPerSec.RateUnit | Type: string Unit: - |
| TotalFetchRequestsPerSec.Count | Type: int Unit: count |
| TotalFetchRequestsPerSec.EventType | Type: string Unit: - |
| TotalFetchRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.RateUnit | Type: string Unit: - |
| TotalProduceRequestsPerSec.Count | Type: int Unit: count |
| TotalProduceRequestsPerSec.EventType | Type: string Unit: - |
| TotalProduceRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.RateUnit | Type: string Unit: - |
kafka_partition¶
| Tags & Fields | Description |
|---|---|
| LogEndOffset | Type: int Unit: - |
| LogStartOffset | Type: int Unit: - |
| NumLogSegments | Type: int Unit: - |
| Size | Type: int Unit: - |
| UnderReplicatedPartitions | Type: int Unit: - |
kafka_zookeeper¶
| Tags & Fields | Description |
|---|---|
| ZooKeeperRequestLatencyMs.50thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.75thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.95thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.98thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.999thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.99thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.Count | Type: int Unit: count |
| ZooKeeperRequestLatencyMs.Max | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.Mean | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.Min | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.StdDev | Type: float Unit: time,ms |
kafka_network¶
| Tags & Fields | Description |
|---|---|
| ControlPlaneExpiredConnectionsKilledCount | Type: int Unit: count |
| ExpiredConnectionsKilledCount | Type: int Unit: count |
| MemoryPoolAvailable | Type: int Unit: count |
| MemoryPoolUsed | Type: int Unit: count |
| NetworkProcessorAvgIdlePercent | Type: float Unit: - |
kafka_log¶
| Tags & Fields | Description |
|---|---|
| DeadThreadCount | Type: int Unit: count |
| OfflineLogDirectoryCount | Type: int Unit: count |
| cleaner_recopy_percent | Type: float Unit: - |
| max_buffer_utilization_percent | Type: float Unit: - |
| max_clean_time_secs | Type: int Unit: time,s |
| max_compaction_delay_secs | Type: int Unit: time,s |
kafka_consumer¶
This metrics needs to be collected on the Consumer instance
| Tags & Fields | Description |
|---|---|
| assigned_partitions | Type: int Unit: count |
| bytes_consumed_rate | Type: float Unit: - |
| bytes_consumed_total | Type: int Unit: count |
| commit_id | Type: string Unit: - |
| commit_rate | Type: float Unit: - |
| commit_total | Type: int Unit: count |
| connection_close_rate | Type: float Unit: - |
| connection_close_total | Type: int Unit: count |
| connection_count | Type: int Unit: count |
| connection_creation_rate | Type: float Unit: - |
| connection_creation_total | Type: int Unit: count |
| count | Type: int Unit: count |
| failed_authentication_rate | Type: float Unit: - |
| failed_authentication_total | Type: int Unit: count |
| failed_reauthentication_rate | Type: float Unit: - |
| failed_reauthentication_total | Type: int Unit: count |
| failed_rebalance_rate_per_hour | Type: float Unit: - |
| failed_rebalance_total | Type: float Unit: - |
| fetch_latency_avg | Type: float Unit: - |
| fetch_latency_max | Type: float Unit: - |
| fetch_rate | Type: float Unit: - |
| fetch_throttle_time_avg | Type: float Unit: - |
| fetch_throttle_time_max | Type: float Unit: - |
| fetch_total | Type: float Unit: - |
| heartbeat_rate | Type: float Unit: - |
| heartbeat_response_time_max | Type: float Unit: - |
| heartbeat_total | Type: int Unit: count |
| incoming_byte_rate | Type: float Unit: - |
| incoming_byte_total | Type: float Unit: - |
| io_ratio | Type: float Unit: - |
| io_time_ns_avg | Type: int Unit: time,ns |
| io_wait_ratio | Type: float Unit: - |
| io_wait_time_ns_avg | Type: int Unit: time,ns |
| io_waittime_total | Type: int Unit: time,ns |
| iotime_total | Type: int Unit: count |
| join_rate | Type: float Unit: - |
| join_total | Type: int Unit: count |
| last_heartbeat_seconds_ago | Type: float Unit: - |
| last_poll_seconds_ago | Type: int Unit: count |
| last_rebalance_seconds_ago | Type: float Unit: - |
| network_io_rate | Type: float Unit: - |
| network_io_total | Type: int Unit: count |
| outgoing_byte_rate | Type: float Unit: - |
| outgoing_byte_total | Type: float Unit: - |
| rebalance_latency_total | Type: int Unit: count |
| rebalance_rate_per_hour | Type: float Unit: - |
| rebalance_total | Type: int Unit: count |
| records_consumed_rate | Type: float Unit: - |
| records_consumed_total | Type: float Unit: - |
| request_rate | Type: float Unit: - |
| request_size_avg | Type: float Unit: - |
| request_size_max | Type: float Unit: - |
| request_total | Type: int Unit: count |
| response_rate | Type: float Unit: - |
| response_total | Type: int Unit: count |
| select_rate | Type: float Unit: - |
| select_total | Type: int Unit: count |
| start_time_ms | Type: int Unit: timeStamp,msec |
| successful_authentication_no_reauth_total | Type: int Unit: count |
| successful_authentication_rate | Type: float Unit: - |
| successful_authentication_total | Type: int Unit: count |
| successful_reauthentication_rate | Type: float Unit: - |
| successful_reauthentication_total | Type: int Unit: count |
| sync_rate | Type: float Unit: - |
| sync_total | Type: int Unit: - |
| version | Type: string Unit: - |
kafka_producer¶
This metrics needs to be collected on the Producer instance
| Tags & Fields | Description |
|---|---|
| batch_split_rate | Type: float Unit: - |
| batch_split_total | Type: int Unit: - |
| buffer_available_bytes | Type: int Unit: - |
| buffer_exhausted_rate | Type: float Unit: - |
| buffer_exhausted_total | Type: float Unit: - |
| buffer_total_bytes | Type: int Unit: - |
| bufferpool_wait_ratio | Type: float Unit: - |
| bufferpool_wait_time_total | Type: int Unit: count |
| commit_id | Type: string Unit: - |
| connection_close_rate | Type: float Unit: - |
| connection_close_total | Type: int Unit: count |
| connection_count | Type: int Unit: count |
| connection_creation_rate | Type: float Unit: - |
| connection_creation_total | Type: int Unit: count |
| count | Type: int Unit: count |
| failed_authentication_rate | Type: float Unit: - |
| failed_authentication_total | Type: int Unit: count |
| failed_reauthentication_rate | Type: float Unit: - |
| failed_reauthentication_total | Type: int Unit: count |
| incoming_byte_rate | Type: float Unit: - |
| incoming_byte_total | Type: int Unit: - |
| io_ratio | Type: float Unit: - |
| io_time_ns_avg | Type: float Unit: time,ns |
| io_wait_ratio | Type: float Unit: - |
| io_wait_time_ns_avg | Type: int Unit: time,ns |
| io_waittime_total | Type: int Unit: time,ns |
| iotime_total | Type: int Unit: time,ns |
| metadata_age | Type: float Unit: - |
| network_io_rate | Type: float Unit: - |
| network_io_total | Type: int Unit: count |
| outgoing_byte_rate | Type: float Unit: - |
| outgoing_byte_total | Type: int Unit: count |
| produce_throttle_time_avg | Type: int Unit: time,ms |
| produce_throttle_time_max | Type: int Unit: time,ms |
| record_error_rate | Type: float Unit: - |
| record_error_total | Type: int Unit: - |
| record_retry_rate | Type: float Unit: - |
| record_retry_total | Type: int Unit: - |
| record_send_rate | Type: float Unit: - |
| record_send_total | Type: int Unit: count |
| request_rate | Type: float Unit: - |
| request_size_avg | Type: float Unit: - |
| request_size_max | Type: int Unit: - |
| request_total | Type: int Unit: count |
| requests_in_flight | Type: int Unit: count |
| response_rate | Type: float Unit: - |
| response_total | Type: int Unit: count |
| select_rate | Type: float Unit: - |
| select_total | Type: int Unit: count |
| start_time_ms | Type: int Unit: timeStamp,msec |
| successful_authentication_no_reauth_total | Type: int Unit: count |
| successful_authentication_rate | Type: float Unit: - |
| successful_authentication_total | Type: int Unit: count |
| successful_reauthentication_rate | Type: float Unit: - |
| successful_reauthentication_total | Type: int Unit: - |
| version | Type: string Unit: - |
| waiting_threads | Type: int Unit: count |
kafka_connect¶
This metrics needs to be collected on the Connect instance
| Tags & Fields | Description |
|---|---|
| batch_size_avg | Type: int Unit: - |
| batch_size_max | Type: int Unit: - |
| commit_id | Type: string Unit: - |
| connector_class | Type: string Unit: - |
| connector_count | Type: int Unit: count |
| connector_destroyed_task_count | Type: int Unit: count |
| connector_failed_task_count | Type: int Unit: count |
| connector_paused_task_count | Type: int Unit: count |
| connector_restarting_task_count | Type: int Unit: count |
| connector_running_task_count | Type: int Unit: count |
| connector_startup_attempts_total | Type: int Unit: count |
| connector_startup_failure_percentage | Type: float Unit: - |
| connector_startup_failure_total | Type: int Unit: count |
| connector_startup_success_percentage | Type: float Unit: - |
| connector_startup_success_total | Type: int Unit: count |
| connector_total_task_count | Type: int Unit: count |
| connector_type | Type: string Unit: - |
| connector_unassigned_task_count | Type: int Unit: count |
| connector_version | Type: string Unit: - |
| count | Type: int Unit: count |
| deadletterqueue_produce_failures | Type: int Unit: count |
| deadletterqueue_produce_requests | Type: int Unit: count |
| last_error_timestamp | The epoch timestamp when this task last encountered an error in millisecond. Type: int Unit: timeStamp,msec |
| offset_commit_avg_time_ms | Type: float Unit: time,ms |
| offset_commit_completion_rate | Type: float Unit: - |
| offset_commit_completion_total | Type: int Unit: count |
| offset_commit_failure_percentage | Type: float Unit: - |
| offset_commit_max_time_ms | Type: float Unit: time,ms |
| offset_commit_seq_no | Type: int Unit: - |
| offset_commit_skip_rate | Type: float Unit: - |
| offset_commit_skip_total | Type: int Unit: count |
| offset_commit_success_percentage | Type: float Unit: - |
| partition_count | Type: int Unit: count |
| pause_ratio | Type: float Unit: - |
| put_batch_avg_time_ms | Type: float Unit: time,ms |
| put_batch_max_time_ms | Type: float Unit: time,ms |
| running_ratio | Type: float Unit: - |
| sink_record_active_count | Type: int Unit: count |
| sink_record_active_count_avg | Type: int Unit: count |
| sink_record_active_count_max | Type: int Unit: count |
| sink_record_read_rate | Type: float Unit: - |
| sink_record_read_total | Type: int Unit: count |
| sink_record_send_rate | Type: float Unit: - |
| sink_record_send_total | Type: int Unit: count |
| source_record_active_count | Type: int Unit: count |
| source_record_poll_rate | Type: float Unit: - |
| source_record_poll_total | Type: int Unit: count |
| source_record_write_rate | Type: float Unit: - |
| source_record_write_total | Type: int Unit: count |
| start_time_ms | Type: int Unit: timeStamp,msec |
| status | Type: string Unit: - |
| task_count | Type: int Unit: count |
| task_startup_attempts_total | Type: int Unit: count |
| task_startup_failure_percentage | Type: float Unit: - |
| task_startup_failure_total | Type: int Unit: count |
| task_startup_success_percentage | Type: float Unit: - |
| task_startup_success_total | Type: int Unit: count |
| total_errors_logged | Type: int Unit: count |
| total_record_errors | Type: int Unit: count |
| total_record_failures | Type: int Unit: count |
| total_records_skipped | Type: int Unit: count |
| total_retries | Type: int Unit: count |
| version | Type: string Unit: - |
日志¶
如需采集 Kafka 的日志，可在 kafka.conf 中 将 files 打开，并写入 kafka 日志文件的绝对路径。比如：
[[inputs.kafka]]
...
[inputs.kafka.log]
files = ["/usr/local/var/log/kafka/error.log","/usr/local/var/log/kafka/kafka.log"]
开启日志采集以后，默认会产生日志来源（source）为 kafka 的日志。
注意：必须将 DataKit 安装在 Kafka 所在主机才能采集 Kafka 日志
切割日志示例：
[2020-07-07 15:04:29,333] DEBUG Progress event: HTTP_REQUEST_COMPLETED_EVENT, bytes: 0 (io.confluent.connect.s3.storage.S3OutputStream:286)
切割后的字段列表如下：
| 字段名 | 字段值 |
|---|---|
| msg | Progress event: HTTP_REQUEST_COMPLETED_EVENT, bytes: 0 |
| name | io.confluent.connect.s3.storage.S3OutputStream:286 |
| status | DEBUG |
| time | 1594105469333000000 |
FAQ¶
为什么看不到 kafka_producer/kafka_consumer/kafka_connect 指标集？¶
在开启 Kafka 服务后，如需采集 Producer/Consumer/Connector 指标，则需分别为其配置 Jolokia。
参考 Kafka Quick Start ，以 Producer 为例，先配置 KAFKA_OPTS 环境变量，示例如下：
export KAFKA_OPTS="-javaagent:/usr/local/datakit/data/jolokia-jvm-agent.jar=host=127.0.0.1,port=8090"
进入 Kafka 目录下启动一个 Producer：
复制出一个 kafka.conf 以开启多个 Kafka 采集器，并配置该 url：
并将采集 Producer 指标部分的字段去掉注释：
# The following metrics are available on producer instances.
[[inputs.kafka.metric]]
name = "kafka_producer"
mbean = "kafka.producer:type=*,client-id=*"
tag_keys = ["client-id", "type"]
重启 DataKit，这时 DataKit 便可采集到 Producer 实例的指标。