Kafka
Collect Kafka indicators and logs and report them to Guance to help you monitor and analyze various abnormal situations of Kafka.
Configuration¶
Requirements¶
Install or download Jolokia. The downloaded Jolokia jar package is already available in the data directory under the DataKit installation directory.
Jolokia is a Java agent of Kafka, which provides an external interface using JSON as data format based on HTTP protocol for DataKit to use. When Kafka starts, first configure the KAFKA_OPTS environment variable: (port can be modified to be available according to the actual situation)
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/local/datakit/data/jolokia-jvm-agent.jar=host=*,port=8080"
Alternatively, you can start Jolokia separately and point it to the Kafka process PID:
Info
- Jolokia not allows change port number in the running state. If found command with
--portcan't change the port, this indicates Jolokia is still in running. If want to change Jolokia port, you must exit Jolokia first and restart it. - Exit Jolokia command:
java -jar </path/to/jolokia-jvm-agent.jar> --quiet stop <Kafka-PID>.
For more Jolokia command information can refer to here.
Collector Configuration¶
Go to the conf.d/samples directory under the DataKit installation directory, copy kafka.conf.sample and name it kafka.conf. Examples are as follows:
[[inputs.kafka]]
# default_tag_prefix = ""
# default_field_prefix = ""
# default_field_separator = "."
# username = ""
# password = ""
# response_timeout = "5s"
## Optional TLS config
# tls_ca = "/var/private/ca.pem"
# tls_cert = "/var/private/client.pem"
# tls_key = "/var/private/client-key.pem"
# insecure_skip_verify = false
## Monitor Interval
# interval = "60s"
election = true
# Add agents URLs to query
urls = ["http://localhost:8080/jolokia"]
## Add metrics to read
[[inputs.kafka.metric]]
name = "kafka_controller"
mbean = "kafka.controller:name=*,type=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_replica_manager"
mbean = "kafka.server:name=*,type=ReplicaManager"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_zookeeper"
mbean = "kafka.server:type=ZooKeeperClientMetrics,name=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_purgatory"
mbean = "kafka.server:delayedOperation=*,name=*,type=DelayedOperationPurgatory"
field_name = "#1.#2"
[[inputs.kafka.metric]]
name = "kafka_client"
mbean = "kafka.server:client-id=*,type=*"
tag_keys = ["client-id", "type"]
[[inputs.kafka.metric]]
name = "kafka_request"
mbean = "kafka.network:name=*,request=*,type=RequestMetrics"
field_prefix = "#1."
tag_keys = ["request"]
[[inputs.kafka.metric]]
name = "kafka_request_handler"
mbean = "kafka.server:type=KafkaRequestHandlerPool,name=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_network"
mbean = "kafka.network:type=*,name=*"
field_name = "#2"
tag_keys = ["type"]
[[inputs.kafka.metric]]
name = "kafka_topics"
mbean = "kafka.server:name=*,type=BrokerTopicMetrics"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_topic"
mbean = "kafka.server:name=*,topic=*,type=BrokerTopicMetrics"
field_prefix = "#1."
tag_keys = ["topic"]
[[inputs.kafka.metric]]
name = "kafka_partition"
mbean = "kafka.log:name=*,partition=*,topic=*,type=Log"
field_name = "#1"
tag_keys = ["topic", "partition"]
[[inputs.kafka.metric]]
name = "kafka_log"
mbean = "kafka.log:type=*,name=*"
field_name = "#2"
tag_keys = ["type"]
[[inputs.kafka.metric]]
name = "kafka_partition"
mbean = "kafka.cluster:name=UnderReplicated,partition=*,topic=*,type=Partition"
field_name = "UnderReplicatedPartitions"
tag_keys = ["topic", "partition"]
# # The following metrics are available on consumer instances.
# [[inputs.kafka.metric]]
# name = "kafka_consumer"
# mbean = "kafka.consumer:type=*,client-id=*"
# tag_keys = ["client-id", "type"]
# # The following metrics are available on producer instances.
# [[inputs.kafka.metric]]
# name = "kafka_producer"
# mbean = "kafka.producer:type=*,client-id=*"
# tag_keys = ["client-id", "type"]
# # The following metrics are available on connector instances.
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*"
# tag_keys = ["type"]
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*,connector=*"
# tag_keys = ["type", "connector"]
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*,connector=*,task=*"
# tag_keys = ["type", "connector", "task"]
# [inputs.kafka.log]
# files = []
# #grok pipeline script path
# pipeline = "kafka.p"
[inputs.kafka.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
Once configured, restart DataKit.
The collector can now be turned on by ConfigMap Injection Collector Configuration.
Metric¶
For all of the following data collections, the global election tags will added automatically, we can add extra tags in [inputs.kafka.tags] if needed:
kafka_controller¶
In Kafka cluster mode, a unique controller node will be elected, and only the controller node will receive valid metrics.
| Tags & Fields | Description |
|---|---|
| ActiveControllerCount.Value | Type: int | (count) Unit: count |
| AutoLeaderBalanceRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Count | Type: int | (count) Unit: count |
| AutoLeaderBalanceRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| AutoLeaderBalanceRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.Count | Type: int | (count) Unit: count |
| ControlledShutdownRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| ControlledShutdownRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| ControlledShutdownRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| ControlledShutdownRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| ControlledShutdownRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.Count | Type: int | (count) Unit: count |
| ControllerChangeRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| ControllerChangeRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| ControllerChangeRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| ControllerChangeRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| ControllerChangeRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.Count | Type: int | (count) Unit: count |
| ControllerShutdownRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| ControllerShutdownRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| ControllerShutdownRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| ControllerShutdownRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| ControllerShutdownRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| ControllerState.Value | Type: int | (gauge) Unit: - |
| EventQueueSize.Value | Type: int | (gauge) Unit: - |
| EventQueueTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.Count | Type: int | (count) Unit: count |
| EventQueueTimeMs.EventType | Type: string | (string) Unit: - |
| EventQueueTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| EventQueueTimeMs.Max | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.Mean | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.Min | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| EventQueueTimeMs.RateUnit | Type: string | (string) Unit: - |
| EventQueueTimeMs.StdDev | Type: float | (gauge) Unit: - |
| GlobalPartitionCount.Value | Type: int | (count) Unit: count |
| GlobalTopicCount.Value | Type: int | (count) Unit: count |
| IsrChangeRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.Count | Type: int | (count) Unit: count |
| IsrChangeRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| IsrChangeRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| IsrChangeRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| IsrChangeRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| IsrChangeRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Count | Type: int | (count) Unit: count |
| LeaderAndIsrResponseReceivedRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.Count | Type: int | (count) Unit: count |
| LeaderElectionRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| LeaderElectionRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| LeaderElectionRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| LeaderElectionRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| LeaderElectionRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Count | Type: int | (count) Unit: count |
| ListPartitionReassignmentRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| ListPartitionReassignmentRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.Count | Type: int | (count) Unit: count |
| LogDirChangeRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| LogDirChangeRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| LogDirChangeRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| LogDirChangeRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| LogDirChangeRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Count | Type: int | (count) Unit: count |
| ManualLeaderBalanceRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| ManualLeaderBalanceRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| OfflinePartitionsCount.Value | Type: int | (count) Unit: count |
| PartitionReassignmentRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.Count | Type: int | (count) Unit: count |
| PartitionReassignmentRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| PartitionReassignmentRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| PartitionReassignmentRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| PartitionReassignmentRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| PartitionReassignmentRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| PreferredReplicaImbalanceCount.Value | Type: int | (count) Unit: count |
| ReplicasIneligibleToDeleteCount.Value | Type: int | (count) Unit: count |
| ReplicasToDeleteCount.Value | Type: int | (count) Unit: count |
| TopicChangeRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.Count | Type: int | (count) Unit: count |
| TopicChangeRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| TopicChangeRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| TopicChangeRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| TopicChangeRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| TopicChangeRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.Count | Type: int | (count) Unit: count |
| TopicDeletionRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| TopicDeletionRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| TopicDeletionRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| TopicDeletionRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| TopicDeletionRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Count | Type: int | (count) Unit: count |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| TopicsIneligibleToDeleteCount.Value | Type: int | (count) Unit: count |
| TopicsToDeleteCount.Value | Type: int | (count) Unit: count |
| TotalQueueSize.Value | Type: int | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Count | Type: int | (count) Unit: count |
| UncleanLeaderElectionEnableRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.50thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.75thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.95thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.98thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.999thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.99thPercentile | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.Count | Type: int | (count) Unit: count |
| UncleanLeaderElectionsPerSec.EventType | Type: string | (string) Unit: - |
| UncleanLeaderElectionsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.LatencyUnit | Type: string | (string) Unit: - |
| UncleanLeaderElectionsPerSec.Max | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.Mean | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.Min | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| UncleanLeaderElectionsPerSec.RateUnit | Type: string | (string) Unit: - |
| UncleanLeaderElectionsPerSec.StdDev | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.50thPercentile | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.75thPercentile | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.95thPercentile | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.98thPercentile | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.999thPercentile | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.99thPercentile | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.Count | Type: int | (count) Unit: count |
| UpdateFeaturesRateAndTimeMs.EventType | Type: string | (string) Unit: - |
| UpdateFeaturesRateAndTimeMs.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.FiveMinuteRate | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.LatencyUnit | Type: string | (string) Unit: - |
| UpdateFeaturesRateAndTimeMs.Max | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.Mean | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.MeanRate | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.Min | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.OneMinuteRate | Type: float | (gauge) Unit: - |
| UpdateFeaturesRateAndTimeMs.RateUnit | Type: string | (string) Unit: - |
| UpdateFeaturesRateAndTimeMs.StdDev | Type: float | (gauge) Unit: - |
kafka_replica_manager¶
| Tags & Fields | Description |
|---|---|
| AtMinIsrPartitionCount.Value | Type: int | (count) Unit: count |
| FailedIsrUpdatesPerSec.Count | Type: int | (count) Unit: count |
| FailedIsrUpdatesPerSec.EventType | Type: string | (string) Unit: - |
| FailedIsrUpdatesPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| FailedIsrUpdatesPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| FailedIsrUpdatesPerSec.MeanRate | Type: float | (gauge) Unit: - |
| FailedIsrUpdatesPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| FailedIsrUpdatesPerSec.RateUnit | Type: string | (string) Unit: - |
| IsrExpandsPerSec.Count | Type: int | (count) Unit: count |
| IsrExpandsPerSec.EventType | Type: string | (string) Unit: - |
| IsrExpandsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| IsrExpandsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| IsrExpandsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| IsrExpandsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| IsrExpandsPerSec.RateUnit | Type: string | (string) Unit: - |
| IsrShrinksPerSec.Count | Type: int | (count) Unit: count |
| IsrShrinksPerSec.EventType | Type: string | (string) Unit: - |
| IsrShrinksPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| IsrShrinksPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| IsrShrinksPerSec.MeanRate | Type: float | (gauge) Unit: - |
| IsrShrinksPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| IsrShrinksPerSec.RateUnit | Type: string | (string) Unit: - |
| LeaderCount.Value | Type: int | (count) Unit: count |
| OfflineReplicaCount.Value | Type: int | (count) Unit: count |
| PartitionCount.Value | Type: int | (count) Unit: count |
| ReassigningPartitions.Value | Type: int | (count) Unit: count |
| UnderMinIsrPartitionCount.Value | Type: int | (count) Unit: count |
| UnderReplicatedPartitions.Value | Type: int | (count) Unit: count |
kafka_purgatory¶
| Tags & Fields | Description |
|---|---|
| AlterAcls.NumDelayedOperations | Type: int | (gauge) Unit: - |
| AlterAcls.PurgatorySize | Type: int | (gauge) Unit: - |
| DeleteRecords.NumDelayedOperations | Type: int | (gauge) Unit: - |
| DeleteRecords.PurgatorySize | Type: int | (gauge) Unit: - |
| ElectLeader.NumDelayedOperations | Type: int | (gauge) Unit: - |
| ElectLeader.PurgatorySize | Type: int | (gauge) Unit: - |
| Fetch.NumDelayedOperations | Type: int | (gauge) Unit: - |
| Fetch.PurgatorySize | Type: int | (gauge) Unit: - |
| Heartbeat.NumDelayedOperations | Type: int | (gauge) Unit: - |
| Heartbeat.PurgatorySize | Type: int | (gauge) Unit: - |
| Produce.NumDelayedOperations | Type: int | (gauge) Unit: - |
| Produce.PurgatorySize | Type: int | (gauge) Unit: - |
| Rebalance.NumDelayedOperations | Type: int | (gauge) Unit: - |
| Rebalance.PurgatorySize | Type: int | (gauge) Unit: - |
| topic.NumDelayedOperations | Type: int | (gauge) Unit: - |
| topic.PurgatorySize | Type: int | (gauge) Unit: - |
kafka_request¶
| Tags & Fields | Description |
|---|---|
| LocalTimeMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.Count | Type: int | (count) Unit: count |
| LocalTimeMs.Max | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.Mean | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.Min | Type: float | (gauge) Unit: time,ms |
| LocalTimeMs.StdDev | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.Count | Type: int | (count) Unit: count |
| RemoteTimeMs.Max | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.Mean | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.Min | Type: float | (gauge) Unit: time,ms |
| RemoteTimeMs.StdDev | Type: float | (gauge) Unit: time,ms |
| RequestBytes.50thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestBytes.75thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestBytes.95thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestBytes.98thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestBytes.999thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestBytes.99thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestBytes.Count | Type: int | (count) Unit: count |
| RequestBytes.Max | Type: float | (gauge) Unit: time,ms |
| RequestBytes.Mean | Type: float | (gauge) Unit: time,ms |
| RequestBytes.Min | Type: float | (gauge) Unit: time,ms |
| RequestBytes.StdDev | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.Count | Type: int | (count) Unit: count |
| RequestQueueTimeMs.Max | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.Mean | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.Min | Type: float | (gauge) Unit: time,ms |
| RequestQueueTimeMs.StdDev | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.Count | Type: int | (count) Unit: count |
| ResponseQueueTimeMs.Max | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.Mean | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.Min | Type: float | (gauge) Unit: time,ms |
| ResponseQueueTimeMs.StdDev | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.Count | Type: int | (count) Unit: count |
| ResponseSendTimeMs.Max | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.Mean | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.Min | Type: float | (gauge) Unit: time,ms |
| ResponseSendTimeMs.StdDev | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.Count | Type: int | (count) Unit: count |
| ThrottleTimeMs.Max | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.Mean | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.Min | Type: float | (gauge) Unit: time,ms |
| ThrottleTimeMs.StdDev | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.Count | Type: int | (count) Unit: count |
| TotalTimeMs.Max | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.Mean | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.Min | Type: float | (gauge) Unit: time,ms |
| TotalTimeMs.StdDev | Type: float | (gauge) Unit: time,ms |
kafka_topics¶
| Tags & Fields | Description |
|---|---|
| BytesInPerSec.Count | Type: int | (count) Unit: count |
| BytesInPerSec.EventType | Type: string | (string) Unit: - |
| BytesInPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.MeanRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.RateUnit | Type: string | (string) Unit: - |
| BytesOutPerSec.Count | Type: int | (count) Unit: count |
| BytesOutPerSec.EventType | Type: string | (string) Unit: - |
| BytesOutPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.MeanRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.RateUnit | Type: string | (string) Unit: - |
| BytesRejectedPerSec.Count | Type: int | (count) Unit: count |
| BytesRejectedPerSec.EventType | Type: string | (string) Unit: - |
| BytesRejectedPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| BytesRejectedPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| BytesRejectedPerSec.MeanRate | Type: float | (gauge) Unit: - |
| BytesRejectedPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| BytesRejectedPerSec.RateUnit | Type: string | (string) Unit: - |
| FailedFetchRequestsPerSec.Count | Type: int | (count) Unit: count |
| FailedFetchRequestsPerSec.EventType | Type: string | (string) Unit: - |
| FailedFetchRequestsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| FailedFetchRequestsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| FailedFetchRequestsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| FailedFetchRequestsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| FailedFetchRequestsPerSec.RateUnit | Type: string | (string) Unit: - |
| FailedProduceRequestsPerSec.Count | Type: int | (count) Unit: count |
| FailedProduceRequestsPerSec.EventType | Type: string | (string) Unit: - |
| FailedProduceRequestsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| FailedProduceRequestsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| FailedProduceRequestsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| FailedProduceRequestsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| FailedProduceRequestsPerSec.RateUnit | Type: string | (string) Unit: - |
| FetchMessageConversionsPerSec.Count | Type: int | (count) Unit: count |
| FetchMessageConversionsPerSec.EventType | Type: string | (string) Unit: - |
| FetchMessageConversionsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| FetchMessageConversionsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| FetchMessageConversionsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| FetchMessageConversionsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| FetchMessageConversionsPerSec.RateUnit | Type: string | (string) Unit: - |
| InvalidMagicNumberRecordsPerSec.Count | Type: int | (count) Unit: count |
| InvalidMagicNumberRecordsPerSec.EventType | Type: string | (string) Unit: - |
| InvalidMagicNumberRecordsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| InvalidMagicNumberRecordsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| InvalidMagicNumberRecordsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| InvalidMagicNumberRecordsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| InvalidMagicNumberRecordsPerSec.RateUnit | Type: string | (string) Unit: - |
| InvalidMessageCrcRecordsPerSec.Count | Type: int | (count) Unit: count |
| InvalidMessageCrcRecordsPerSec.EventType | Type: string | (string) Unit: - |
| InvalidMessageCrcRecordsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| InvalidMessageCrcRecordsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| InvalidMessageCrcRecordsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| InvalidMessageCrcRecordsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| InvalidMessageCrcRecordsPerSec.RateUnit | Type: string | (string) Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.Count | Type: int | (count) Unit: count |
| InvalidOffsetOrSequenceRecordsPerSec.EventType | Type: string | (string) Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.RateUnit | Type: string | (string) Unit: - |
| MessagesInPerSec.Count | Type: int | (count) Unit: count |
| MessagesInPerSec.EventType | Type: string | (string) Unit: - |
| MessagesInPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.MeanRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.RateUnit | Type: string | (string) Unit: - |
| NoKeyCompactedTopicRecordsPerSec.Count | Type: int | (count) Unit: count |
| NoKeyCompactedTopicRecordsPerSec.EventType | Type: string | (string) Unit: - |
| NoKeyCompactedTopicRecordsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| NoKeyCompactedTopicRecordsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| NoKeyCompactedTopicRecordsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| NoKeyCompactedTopicRecordsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| NoKeyCompactedTopicRecordsPerSec.RateUnit | Type: string | (string) Unit: - |
| ProduceMessageConversionsPerSec.Count | Type: int | (count) Unit: count |
| ProduceMessageConversionsPerSec.EventType | Type: string | (string) Unit: - |
| ProduceMessageConversionsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ProduceMessageConversionsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ProduceMessageConversionsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| ProduceMessageConversionsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| ProduceMessageConversionsPerSec.RateUnit | Type: string | (string) Unit: - |
| ReassignmentBytesInPerSec.Count | Type: int | (count) Unit: count |
| ReassignmentBytesInPerSec.EventType | Type: string | (string) Unit: - |
| ReassignmentBytesInPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesInPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesInPerSec.MeanRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesInPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesInPerSec.RateUnit | Type: string | (string) Unit: - |
| ReassignmentBytesOutPerSec.Count | Type: int | (count) Unit: count |
| ReassignmentBytesOutPerSec.EventType | Type: string | (string) Unit: - |
| ReassignmentBytesOutPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesOutPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesOutPerSec.MeanRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesOutPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| ReassignmentBytesOutPerSec.RateUnit | Type: string | (string) Unit: - |
| ReplicationBytesInPerSec.Count | Type: int | (count) Unit: count |
| ReplicationBytesInPerSec.EventType | Type: string | (string) Unit: - |
| ReplicationBytesInPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ReplicationBytesInPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ReplicationBytesInPerSec.MeanRate | Type: float | (gauge) Unit: - |
| ReplicationBytesInPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| ReplicationBytesInPerSec.RateUnit | Type: string | (string) Unit: - |
| ReplicationBytesOutPerSec.Count | Type: int | (count) Unit: count |
| ReplicationBytesOutPerSec.EventType | Type: string | (string) Unit: - |
| ReplicationBytesOutPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| ReplicationBytesOutPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| ReplicationBytesOutPerSec.MeanRate | Type: float | (gauge) Unit: - |
| ReplicationBytesOutPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| ReplicationBytesOutPerSec.RateUnit | Type: string | (string) Unit: - |
| TotalFetchRequestsPerSec.Count | Type: int | (count) Unit: count |
| TotalFetchRequestsPerSec.EventType | Type: string | (string) Unit: - |
| TotalFetchRequestsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.RateUnit | Type: string | (string) Unit: - |
| TotalProduceRequestsPerSec.Count | Type: int | (count) Unit: count |
| TotalProduceRequestsPerSec.EventType | Type: string | (string) Unit: - |
| TotalProduceRequestsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.RateUnit | Type: string | (string) Unit: - |
kafka_topic¶
| Tags & Fields | Description |
|---|---|
| BytesInPerSec.Count | Type: int | (count) Unit: count |
| BytesInPerSec.EventType | Type: string | (string) Unit: - |
| BytesInPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.MeanRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| BytesInPerSec.RateUnit | Type: string | (string) Unit: - |
| BytesOutPerSec.Count | Type: int | (count) Unit: count |
| BytesOutPerSec.EventType | Type: string | (string) Unit: - |
| BytesOutPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.MeanRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| BytesOutPerSec.RateUnit | Type: string | (string) Unit: - |
| MessagesInPerSec.Count | Type: int | (count) Unit: count |
| MessagesInPerSec.EventType | Type: string | (string) Unit: - |
| MessagesInPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.MeanRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| MessagesInPerSec.RateUnit | Type: string | (string) Unit: - |
| TotalFetchRequestsPerSec.Count | Type: int | (count) Unit: count |
| TotalFetchRequestsPerSec.EventType | Type: string | (string) Unit: - |
| TotalFetchRequestsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| TotalFetchRequestsPerSec.RateUnit | Type: string | (string) Unit: - |
| TotalProduceRequestsPerSec.Count | Type: int | (count) Unit: count |
| TotalProduceRequestsPerSec.EventType | Type: string | (string) Unit: - |
| TotalProduceRequestsPerSec.FifteenMinuteRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.FiveMinuteRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.MeanRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.OneMinuteRate | Type: float | (gauge) Unit: - |
| TotalProduceRequestsPerSec.RateUnit | Type: string | (string) Unit: - |
kafka_partition¶
| Tags & Fields | Description |
|---|---|
| LogEndOffset | Type: int | (gauge) Unit: - |
| LogStartOffset | Type: int | (gauge) Unit: - |
| NumLogSegments | Type: int | (gauge) Unit: - |
| Size | Type: int | (gauge) Unit: - |
| UnderReplicatedPartitions | Type: int | (gauge) Unit: - |
kafka_zookeeper¶
| Tags & Fields | Description |
|---|---|
| ZooKeeperRequestLatencyMs.50thPercentile | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.75thPercentile | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.95thPercentile | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.98thPercentile | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.999thPercentile | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.99thPercentile | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.Count | Type: int | (count) Unit: count |
| ZooKeeperRequestLatencyMs.Max | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.Mean | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.Min | Type: float | (gauge) Unit: time,ms |
| ZooKeeperRequestLatencyMs.StdDev | Type: float | (gauge) Unit: time,ms |
kafka_network¶
| Tags & Fields | Description |
|---|---|
| ControlPlaneExpiredConnectionsKilledCount | Type: int | (count) Unit: count |
| ExpiredConnectionsKilledCount | Type: int | (count) Unit: count |
| MemoryPoolAvailable | Type: int | (count) Unit: count |
| MemoryPoolUsed | Type: int | (count) Unit: count |
| NetworkProcessorAvgIdlePercent | Type: float | (gauge) Unit: - |
kafka_log¶
| Tags & Fields | Description |
|---|---|
| DeadThreadCount | Type: int | (count) Unit: count |
| OfflineLogDirectoryCount | Type: int | (count) Unit: count |
| cleaner_recopy_percent | Type: float | (gauge) Unit: - |
| max_buffer_utilization_percent | Type: float | (gauge) Unit: - |
| max_clean_time_secs | Type: int | (gauge) Unit: time,s |
| max_compaction_delay_secs | Type: int | (gauge) Unit: time,s |
kafka_consumer¶
This metrics needs to be collected on the Consumer instance
| Tags & Fields | Description |
|---|---|
| assigned_partitions | Type: int | (count) Unit: count |
| bytes_consumed_rate | Type: float | (gauge) Unit: - |
| bytes_consumed_total | Type: int | (count) Unit: count |
| commit_id | Type: string | (string) Unit: - |
| commit_rate | Type: float | (gauge) Unit: - |
| commit_total | Type: int | (count) Unit: count |
| connection_close_rate | Type: float | (gauge) Unit: - |
| connection_close_total | Type: int | (count) Unit: count |
| connection_count | Type: int | (count) Unit: count |
| connection_creation_rate | Type: float | (gauge) Unit: - |
| connection_creation_total | Type: int | (count) Unit: count |
| count | Type: int | (count) Unit: count |
| failed_authentication_rate | Type: float | (gauge) Unit: - |
| failed_authentication_total | Type: int | (count) Unit: count |
| failed_reauthentication_rate | Type: float | (gauge) Unit: - |
| failed_reauthentication_total | Type: int | (count) Unit: count |
| failed_rebalance_rate_per_hour | Type: float | (gauge) Unit: - |
| failed_rebalance_total | Type: float | (gauge) Unit: - |
| fetch_latency_avg | Type: float | (gauge) Unit: - |
| fetch_latency_max | Type: float | (gauge) Unit: - |
| fetch_rate | Type: float | (gauge) Unit: - |
| fetch_throttle_time_avg | Type: float | (gauge) Unit: - |
| fetch_throttle_time_max | Type: float | (gauge) Unit: - |
| fetch_total | Type: float | (gauge) Unit: - |
| heartbeat_rate | Type: float | (gauge) Unit: - |
| heartbeat_response_time_max | Type: float | (gauge) Unit: - |
| heartbeat_total | Type: int | (count) Unit: count |
| incoming_byte_rate | Type: float | (gauge) Unit: - |
| incoming_byte_total | Type: float | (gauge) Unit: - |
| io_ratio | Type: float | (gauge) Unit: - |
| io_time_ns_avg | Type: int | (gauge) Unit: time,ns |
| io_wait_ratio | Type: float | (gauge) Unit: - |
| io_wait_time_ns_avg | Type: int | (gauge) Unit: time,ns |
| io_waittime_total | Type: int | (gauge) Unit: time,ns |
| iotime_total | Type: int | (count) Unit: count |
| join_rate | Type: float | (gauge) Unit: - |
| join_total | Type: int | (count) Unit: count |
| last_heartbeat_seconds_ago | Type: float | (gauge) Unit: - |
| last_poll_seconds_ago | Type: int | (count) Unit: count |
| last_rebalance_seconds_ago | Type: float | (gauge) Unit: - |
| network_io_rate | Type: float | (gauge) Unit: - |
| network_io_total | Type: int | (count) Unit: count |
| outgoing_byte_rate | Type: float | (gauge) Unit: - |
| outgoing_byte_total | Type: float | (gauge) Unit: - |
| rebalance_latency_total | Type: int | (count) Unit: count |
| rebalance_rate_per_hour | Type: float | (gauge) Unit: - |
| rebalance_total | Type: int | (count) Unit: count |
| records_consumed_rate | Type: float | (gauge) Unit: - |
| records_consumed_total | Type: float | (gauge) Unit: - |
| request_rate | Type: float | (gauge) Unit: - |
| request_size_avg | Type: float | (gauge) Unit: - |
| request_size_max | Type: float | (gauge) Unit: - |
| request_total | Type: int | (count) Unit: count |
| response_rate | Type: float | (gauge) Unit: - |
| response_total | Type: int | (count) Unit: count |
| select_rate | Type: float | (gauge) Unit: - |
| select_total | Type: int | (count) Unit: count |
| start_time_ms | Type: int | (gauge) Unit: timeStamp,msec |
| successful_authentication_no_reauth_total | Type: int | (count) Unit: count |
| successful_authentication_rate | Type: float | (gauge) Unit: - |
| successful_authentication_total | Type: int | (count) Unit: count |
| successful_reauthentication_rate | Type: float | (gauge) Unit: - |
| successful_reauthentication_total | Type: int | (count) Unit: count |
| sync_rate | Type: float | (gauge) Unit: - |
| sync_total | Type: int | (count) Unit: - |
| version | Type: string | (string) Unit: - |
kafka_producer¶
This metrics needs to be collected on the Producer instance
| Tags & Fields | Description |
|---|---|
| batch_split_rate | Type: float | (gauge) Unit: - |
| batch_split_total | Type: int | (gauge) Unit: - |
| buffer_available_bytes | Type: int | (gauge) Unit: - |
| buffer_exhausted_rate | Type: float | (gauge) Unit: - |
| buffer_exhausted_total | Type: float | (gauge) Unit: - |
| buffer_total_bytes | Type: int | (gauge) Unit: - |
| bufferpool_wait_ratio | Type: float | (gauge) Unit: - |
| bufferpool_wait_time_total | Type: int | (count) Unit: count |
| commit_id | Type: string | (string) Unit: - |
| connection_close_rate | Type: float | (gauge) Unit: - |
| connection_close_total | Type: int | (count) Unit: count |
| connection_count | Type: int | (count) Unit: count |
| connection_creation_rate | Type: float | (gauge) Unit: - |
| connection_creation_total | Type: int | (count) Unit: count |
| count | Type: int | (count) Unit: count |
| failed_authentication_rate | Type: float | (gauge) Unit: - |
| failed_authentication_total | Type: int | (count) Unit: count |
| failed_reauthentication_rate | Type: float | (gauge) Unit: - |
| failed_reauthentication_total | Type: int | (count) Unit: count |
| incoming_byte_rate | Type: float | (gauge) Unit: - |
| incoming_byte_total | Type: int | (gauge) Unit: - |
| io_ratio | Type: float | (gauge) Unit: - |
| io_time_ns_avg | Type: float | (gauge) Unit: time,ns |
| io_wait_ratio | Type: float | (gauge) Unit: - |
| io_wait_time_ns_avg | Type: int | (gauge) Unit: time,ns |
| io_waittime_total | Type: int | (gauge) Unit: time,ns |
| iotime_total | Type: int | (gauge) Unit: time,ns |
| metadata_age | Type: float | (gauge) Unit: - |
| network_io_rate | Type: float | (gauge) Unit: - |
| network_io_total | Type: int | (count) Unit: count |
| outgoing_byte_rate | Type: float | (gauge) Unit: - |
| outgoing_byte_total | Type: int | (count) Unit: count |
| produce_throttle_time_avg | Type: int | (gauge) Unit: time,ms |
| produce_throttle_time_max | Type: int | (gauge) Unit: time,ms |
| record_error_rate | Type: float | (gauge) Unit: - |
| record_error_total | Type: int | (gauge) Unit: - |
| record_retry_rate | Type: float | (gauge) Unit: - |
| record_retry_total | Type: int | (gauge) Unit: - |
| record_send_rate | Type: float | (gauge) Unit: - |
| record_send_total | Type: int | (count) Unit: count |
| request_rate | Type: float | (gauge) Unit: - |
| request_size_avg | Type: float | (gauge) Unit: - |
| request_size_max | Type: int | (gauge) Unit: - |
| request_total | Type: int | (count) Unit: count |
| requests_in_flight | Type: int | (count) Unit: count |
| response_rate | Type: float | (gauge) Unit: - |
| response_total | Type: int | (count) Unit: count |
| select_rate | Type: float | (gauge) Unit: - |
| select_total | Type: int | (count) Unit: count |
| start_time_ms | Type: int | (gauge) Unit: timeStamp,msec |
| successful_authentication_no_reauth_total | Type: int | (count) Unit: count |
| successful_authentication_rate | Type: float | (gauge) Unit: - |
| successful_authentication_total | Type: int | (count) Unit: count |
| successful_reauthentication_rate | Type: float | (gauge) Unit: - |
| successful_reauthentication_total | Type: int | (gauge) Unit: - |
| version | Type: string | (string) Unit: - |
| waiting_threads | Type: int | (count) Unit: count |
kafka_connect¶
This metrics needs to be collected on the Connect instance
| Tags & Fields | Description |
|---|---|
| batch_size_avg | Type: int | (gauge) Unit: - |
| batch_size_max | Type: int | (gauge) Unit: - |
| commit_id | Type: string | (string) Unit: - |
| connector_class | Type: string | (string) Unit: - |
| connector_count | Type: int | (count) Unit: count |
| connector_destroyed_task_count | Type: int | (count) Unit: count |
| connector_failed_task_count | Type: int | (count) Unit: count |
| connector_paused_task_count | Type: int | (count) Unit: count |
| connector_restarting_task_count | Type: int | (count) Unit: count |
| connector_running_task_count | Type: int | (count) Unit: count |
| connector_startup_attempts_total | Type: int | (count) Unit: count |
| connector_startup_failure_percentage | Type: float | (gauge) Unit: - |
| connector_startup_failure_total | Type: int | (count) Unit: count |
| connector_startup_success_percentage | Type: float | (gauge) Unit: - |
| connector_startup_success_total | Type: int | (count) Unit: count |
| connector_total_task_count | Type: int | (count) Unit: count |
| connector_type | Type: string | (string) Unit: - |
| connector_unassigned_task_count | Type: int | (count) Unit: count |
| connector_version | Type: string | (string) Unit: - |
| count | Type: int | (count) Unit: count |
| deadletterqueue_produce_failures | Type: int | (count) Unit: count |
| deadletterqueue_produce_requests | Type: int | (count) Unit: count |
| last_error_timestamp | The epoch timestamp when this task last encountered an error in millisecond. Type: int | (gauge) Unit: timeStamp,msec |
| offset_commit_avg_time_ms | Type: float | (gauge) Unit: time,ms |
| offset_commit_completion_rate | Type: float | (gauge) Unit: - |
| offset_commit_completion_total | Type: int | (count) Unit: count |
| offset_commit_failure_percentage | Type: float | (gauge) Unit: - |
| offset_commit_max_time_ms | Type: float | (gauge) Unit: time,ms |
| offset_commit_seq_no | Type: int | (gauge) Unit: - |
| offset_commit_skip_rate | Type: float | (gauge) Unit: - |
| offset_commit_skip_total | Type: int | (count) Unit: count |
| offset_commit_success_percentage | Type: float | (gauge) Unit: - |
| partition_count | Type: int | (count) Unit: count |
| pause_ratio | Type: float | (gauge) Unit: - |
| put_batch_avg_time_ms | Type: float | (gauge) Unit: time,ms |
| put_batch_max_time_ms | Type: float | (gauge) Unit: time,ms |
| running_ratio | Type: float | (gauge) Unit: - |
| sink_record_active_count | Type: int | (count) Unit: count |
| sink_record_active_count_avg | Type: int | (count) Unit: count |
| sink_record_active_count_max | Type: int | (count) Unit: count |
| sink_record_read_rate | Type: float | (gauge) Unit: - |
| sink_record_read_total | Type: int | (count) Unit: count |
| sink_record_send_rate | Type: float | (gauge) Unit: - |
| sink_record_send_total | Type: int | (count) Unit: count |
| source_record_active_count | Type: int | (count) Unit: count |
| source_record_poll_rate | Type: float | (gauge) Unit: - |
| source_record_poll_total | Type: int | (count) Unit: count |
| source_record_write_rate | Type: float | (gauge) Unit: - |
| source_record_write_total | Type: int | (count) Unit: count |
| start_time_ms | Type: int | (gauge) Unit: timeStamp,msec |
| status | Type: string | (string) Unit: - |
| task_count | Type: int | (count) Unit: count |
| task_startup_attempts_total | Type: int | (count) Unit: count |
| task_startup_failure_percentage | Type: float | (gauge) Unit: - |
| task_startup_failure_total | Type: int | (count) Unit: count |
| task_startup_success_percentage | Type: float | (gauge) Unit: - |
| task_startup_success_total | Type: int | (count) Unit: count |
| total_errors_logged | Type: int | (count) Unit: count |
| total_record_errors | Type: int | (count) Unit: count |
| total_record_failures | Type: int | (count) Unit: count |
| total_records_skipped | Type: int | (count) Unit: count |
| total_retries | Type: int | (count) Unit: count |
| version | Type: string | (string) Unit: - |
Log Collection¶
To collect kafka's log, open files in kafka.conf and write to the absolute path of the kafka log file. For example:
[[inputs.kafka]]
...
[inputs.kafka.log]
files = ["/usr/local/var/log/kafka/error.log","/usr/local/var/log/kafka/kafka.log"]
When log collection is turned on, a log with a log source of kafka is generated by default.
Note: DataKit must be installed on Kafka's host to collect Kafka logs.
Example of cutting logs:
[2020-07-07 15:04:29,333] DEBUG Progress event: HTTP_REQUEST_COMPLETED_EVENT, bytes: 0 (io.confluent.connect.s3.storage.S3OutputStream:286)
The list of cut fields is as follows:
| Field Name | Field Value |
|---|---|
| msg | Progress event: HTTP_REQUEST_COMPLETED_EVENT, bytes: 0 |
| name | io.confluent.connect.s3.storage.S3OutputStream:286 |
| status | DEBUG |
| time | 1594105469333000000 |
FAQ¶
Why can't see kafka_producer/kafka_producer/kafka_connect measurements?¶
After Kafka service is started, if you need to collect Producer/Consumer/Connector indicators, you need to configure Jolokia for them respectively.
Referring to Kafka Quick Start, configure the KAFKA_OPTS environment variable for the example of Producer, as follows:
export KAFKA_OPTS="-javaagent:/usr/local/datakit/data/jolokia-jvm-agent.jar=host=127.0.0.1,port=8090"
Go into the Kafka directory and start a Producer:
Copy a Kafka.conf to open multiple Kafka collectors and configure the url:
And remove comments from the fields in the collect producer metrics section:
# The following metrics are available on producer instances.
[[inputs.kafka.metric]]
name = "kafka_producer"
mbean = "kafka.producer:type=*,client-id=*"
tag_keys = ["client-id", "type"]
Restart DataKit, which then collects metrics for the Producer instance.