Kafka
Collect Kafka indicators and logs and report them to Guance to help you monitor and analyze various abnormal situations of Kafka.
Configuration¶
Requirements¶
Install or download Jolokia. The downloaded Jolokia jar package is already available in the data directory under the DataKit installation directory.
Jolokia is a Java agent of Kafka, which provides an external interface using JSON as data format based on HTTP protocol for DataKit to use. When Kafka starts, first configure the KAFKA_OPTS environment variable: (port can be modified to be available according to the actual situation)
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/local/datakit/data/jolokia-jvm-agent.jar=host=*,port=8080"
Alternatively, you can start Jolokia separately and point it to the Kafka process PID:
Info
- Jolokia not allows change port number in the running state. If found command with
--portcan't change the port, this indicates Jolokia is still in running. If want to change Jolokia port, you must exit Jolokia first and restart it. - Exit Jolokia command:
java -jar </path/to/jolokia-jvm-agent.jar> --quiet stop <Kafka-PID>.
For more Jolokia command information can refer to here.
Collector Configuration¶
Go to the conf.d/db directory under the DataKit installation directory, copy kafka.conf.sample and name it kafka.conf. Examples are as follows:
[[inputs.kafka]]
# default_tag_prefix = ""
# default_field_prefix = ""
# default_field_separator = "."
# username = ""
# password = ""
# response_timeout = "5s"
## Optional TLS config
# tls_ca = "/var/private/ca.pem"
# tls_cert = "/var/private/client.pem"
# tls_key = "/var/private/client-key.pem"
# insecure_skip_verify = false
## Monitor Interval
# interval = "60s"
election = true
# Add agents URLs to query
urls = ["http://localhost:8080/jolokia"]
## Add metrics to read
[[inputs.kafka.metric]]
name = "kafka_controller"
mbean = "kafka.controller:name=*,type=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_replica_manager"
mbean = "kafka.server:name=*,type=ReplicaManager"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_zookeeper"
mbean = "kafka.server:type=ZooKeeperClientMetrics,name=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_purgatory"
mbean = "kafka.server:delayedOperation=*,name=*,type=DelayedOperationPurgatory"
field_name = "#1.#2"
[[inputs.kafka.metric]]
name = "kafka_client"
mbean = "kafka.server:client-id=*,type=*"
tag_keys = ["client-id", "type"]
[[inputs.kafka.metric]]
name = "kafka_request"
mbean = "kafka.network:name=*,request=*,type=RequestMetrics"
field_prefix = "#1."
tag_keys = ["request"]
[[inputs.kafka.metric]]
name = "kafka_request_handler"
mbean = "kafka.server:type=KafkaRequestHandlerPool,name=*"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_network"
mbean = "kafka.network:type=*,name=*"
field_name = "#2"
tag_keys = ["type"]
[[inputs.kafka.metric]]
name = "kafka_topics"
mbean = "kafka.server:name=*,type=BrokerTopicMetrics"
field_prefix = "#1."
[[inputs.kafka.metric]]
name = "kafka_topic"
mbean = "kafka.server:name=*,topic=*,type=BrokerTopicMetrics"
field_prefix = "#1."
tag_keys = ["topic"]
[[inputs.kafka.metric]]
name = "kafka_partition"
mbean = "kafka.log:name=*,partition=*,topic=*,type=Log"
field_name = "#1"
tag_keys = ["topic", "partition"]
[[inputs.kafka.metric]]
name = "kafka_log"
mbean = "kafka.log:type=*,name=*"
field_name = "#2"
tag_keys = ["type"]
[[inputs.kafka.metric]]
name = "kafka_partition"
mbean = "kafka.cluster:name=UnderReplicated,partition=*,topic=*,type=Partition"
field_name = "UnderReplicatedPartitions"
tag_keys = ["topic", "partition"]
# # The following metrics are available on consumer instances.
# [[inputs.kafka.metric]]
# name = "kafka_consumer"
# mbean = "kafka.consumer:type=*,client-id=*"
# tag_keys = ["client-id", "type"]
# # The following metrics are available on producer instances.
# [[inputs.kafka.metric]]
# name = "kafka_producer"
# mbean = "kafka.producer:type=*,client-id=*"
# tag_keys = ["client-id", "type"]
# # The following metrics are available on connector instances.
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*"
# tag_keys = ["type"]
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*,connector=*"
# tag_keys = ["type", "connector"]
# [[inputs.kafka.metric]]
# name = "kafka_connect"
# mbean = "kafka.connect:type=*,connector=*,task=*"
# tag_keys = ["type", "connector", "task"]
# [inputs.kafka.log]
# files = []
# #grok pipeline script path
# pipeline = "kafka.p"
[inputs.kafka.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
Once configured, restart DataKit.
The collector can now be turned on by ConfigMap Injection Collector Configuration.
Metric¶
For all of the following data collections, the global election tags will added automatically, we can add extra tags in [inputs.kafka.tags] if needed:
kafka_controller¶
In Kafka cluster mode, a unique controller node will be elected, and only the controller node will receive valid metrics.
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
- Metrics
| Metric | Description |
|---|---|
| ActiveControllerCount.Value | Type: int Unit: count |
| AutoLeaderBalanceRateAndTimeMs.50thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.75thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.95thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.98thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.999thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.99thPercentile | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Count | Type: int Unit: count |
| AutoLeaderBalanceRateAndTimeMs.EventType | Type: string Unit: - |
| AutoLeaderBalanceRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Max | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Mean | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.MeanRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.Min | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| AutoLeaderBalanceRateAndTimeMs.RateUnit | Type: string Unit: - |
| AutoLeaderBalanceRateAndTimeMs.StdDev | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.Count | Type: int Unit: count |
| ControlledShutdownRateAndTimeMs.EventType | Type: string Unit: - |
| ControlledShutdownRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ControlledShutdownRateAndTimeMs.Max | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.Mean | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.MeanRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.Min | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ControlledShutdownRateAndTimeMs.RateUnit | Type: string Unit: - |
| ControlledShutdownRateAndTimeMs.StdDev | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.Count | Type: int Unit: count |
| ControllerChangeRateAndTimeMs.EventType | Type: string Unit: - |
| ControllerChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ControllerChangeRateAndTimeMs.Max | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.Mean | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.Min | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ControllerChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| ControllerChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.Count | Type: int Unit: count |
| ControllerShutdownRateAndTimeMs.EventType | Type: string Unit: - |
| ControllerShutdownRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ControllerShutdownRateAndTimeMs.Max | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.Mean | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.MeanRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.Min | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ControllerShutdownRateAndTimeMs.RateUnit | Type: string Unit: - |
| ControllerShutdownRateAndTimeMs.StdDev | Type: float Unit: - |
| ControllerState.Value | Type: int Unit: - |
| EventQueueSize.Value | Type: int Unit: - |
| EventQueueTimeMs.50thPercentile | Type: float Unit: - |
| EventQueueTimeMs.75thPercentile | Type: float Unit: - |
| EventQueueTimeMs.95thPercentile | Type: float Unit: - |
| EventQueueTimeMs.98thPercentile | Type: float Unit: - |
| EventQueueTimeMs.999thPercentile | Type: float Unit: - |
| EventQueueTimeMs.99thPercentile | Type: float Unit: - |
| EventQueueTimeMs.Count | Type: int Unit: count |
| EventQueueTimeMs.EventType | Type: string Unit: - |
| EventQueueTimeMs.FifteenMinuteRate | Type: float Unit: - |
| EventQueueTimeMs.FiveMinuteRate | Type: float Unit: - |
| EventQueueTimeMs.LatencyUnit | Type: string Unit: - |
| EventQueueTimeMs.Max | Type: float Unit: - |
| EventQueueTimeMs.Mean | Type: float Unit: - |
| EventQueueTimeMs.MeanRate | Type: float Unit: - |
| EventQueueTimeMs.Min | Type: float Unit: - |
| EventQueueTimeMs.OneMinuteRate | Type: float Unit: - |
| EventQueueTimeMs.RateUnit | Type: string Unit: - |
| EventQueueTimeMs.StdDev | Type: float Unit: - |
| GlobalPartitionCount.Value | Type: int Unit: count |
| GlobalTopicCount.Value | Type: int Unit: count |
| IsrChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| IsrChangeRateAndTimeMs.Count | Type: int Unit: count |
| IsrChangeRateAndTimeMs.EventType | Type: string Unit: - |
| IsrChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| IsrChangeRateAndTimeMs.Max | Type: float Unit: - |
| IsrChangeRateAndTimeMs.Mean | Type: float Unit: - |
| IsrChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.Min | Type: float Unit: - |
| IsrChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| IsrChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| IsrChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.50thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.75thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.95thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.98thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.999thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.99thPercentile | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Count | Type: int Unit: count |
| LeaderAndIsrResponseReceivedRateAndTimeMs.EventType | Type: string Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Max | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Mean | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.MeanRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.Min | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.RateUnit | Type: string Unit: - |
| LeaderAndIsrResponseReceivedRateAndTimeMs.StdDev | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.50thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.75thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.95thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.98thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.999thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.99thPercentile | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.Count | Type: int Unit: count |
| LeaderElectionRateAndTimeMs.EventType | Type: string Unit: - |
| LeaderElectionRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| LeaderElectionRateAndTimeMs.Max | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.Mean | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.MeanRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.Min | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| LeaderElectionRateAndTimeMs.RateUnit | Type: string Unit: - |
| LeaderElectionRateAndTimeMs.StdDev | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Count | Type: int Unit: count |
| ListPartitionReassignmentRateAndTimeMs.EventType | Type: string Unit: - |
| ListPartitionReassignmentRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Max | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Mean | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.MeanRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.Min | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ListPartitionReassignmentRateAndTimeMs.RateUnit | Type: string Unit: - |
| ListPartitionReassignmentRateAndTimeMs.StdDev | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.Count | Type: int Unit: count |
| LogDirChangeRateAndTimeMs.EventType | Type: string Unit: - |
| LogDirChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| LogDirChangeRateAndTimeMs.Max | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.Mean | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.Min | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| LogDirChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| LogDirChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.50thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.75thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.95thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.98thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.999thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.99thPercentile | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Count | Type: int Unit: count |
| ManualLeaderBalanceRateAndTimeMs.EventType | Type: string Unit: - |
| ManualLeaderBalanceRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Max | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Mean | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.MeanRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.Min | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| ManualLeaderBalanceRateAndTimeMs.RateUnit | Type: string Unit: - |
| ManualLeaderBalanceRateAndTimeMs.StdDev | Type: float Unit: - |
| OfflinePartitionsCount.Value | Type: int Unit: count |
| PartitionReassignmentRateAndTimeMs.50thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.75thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.95thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.98thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.999thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.99thPercentile | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.Count | Type: int Unit: count |
| PartitionReassignmentRateAndTimeMs.EventType | Type: string Unit: - |
| PartitionReassignmentRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| PartitionReassignmentRateAndTimeMs.Max | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.Mean | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.MeanRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.Min | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| PartitionReassignmentRateAndTimeMs.RateUnit | Type: string Unit: - |
| PartitionReassignmentRateAndTimeMs.StdDev | Type: float Unit: - |
| PreferredReplicaImbalanceCount.Value | Type: int Unit: count |
| ReplicasIneligibleToDeleteCount.Value | Type: int Unit: count |
| ReplicasToDeleteCount.Value | Type: int Unit: count |
| TopicChangeRateAndTimeMs.50thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.75thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.95thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.98thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.999thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.99thPercentile | Type: float Unit: - |
| TopicChangeRateAndTimeMs.Count | Type: int Unit: count |
| TopicChangeRateAndTimeMs.EventType | Type: string Unit: - |
| TopicChangeRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| TopicChangeRateAndTimeMs.Max | Type: float Unit: - |
| TopicChangeRateAndTimeMs.Mean | Type: float Unit: - |
| TopicChangeRateAndTimeMs.MeanRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.Min | Type: float Unit: - |
| TopicChangeRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| TopicChangeRateAndTimeMs.RateUnit | Type: string Unit: - |
| TopicChangeRateAndTimeMs.StdDev | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.50thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.75thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.95thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.98thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.999thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.99thPercentile | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.Count | Type: int Unit: count |
| TopicDeletionRateAndTimeMs.EventType | Type: string Unit: - |
| TopicDeletionRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| TopicDeletionRateAndTimeMs.Max | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.Mean | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.MeanRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.Min | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| TopicDeletionRateAndTimeMs.RateUnit | Type: string Unit: - |
| TopicDeletionRateAndTimeMs.StdDev | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.50thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.75thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.95thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.98thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.999thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.99thPercentile | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Count | Type: int Unit: count |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.EventType | Type: string Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Max | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Mean | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.MeanRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.Min | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.RateUnit | Type: string Unit: - |
| TopicUncleanLeaderElectionEnableRateAndTimeMs.StdDev | Type: float Unit: - |
| TopicsIneligibleToDeleteCount.Value | Type: int Unit: count |
| TopicsToDeleteCount.Value | Type: int Unit: count |
| TotalQueueSize.Value | Type: int Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.50thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.75thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.95thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.98thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.999thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.99thPercentile | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Count | Type: int Unit: count |
| UncleanLeaderElectionEnableRateAndTimeMs.EventType | Type: string Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Max | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Mean | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.MeanRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.Min | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.RateUnit | Type: string Unit: - |
| UncleanLeaderElectionEnableRateAndTimeMs.StdDev | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.50thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.75thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.95thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.98thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.999thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.99thPercentile | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.Count | Type: int Unit: count |
| UncleanLeaderElectionsPerSec.EventType | Type: string Unit: - |
| UncleanLeaderElectionsPerSec.FifteenMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.FiveMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.LatencyUnit | Type: string Unit: - |
| UncleanLeaderElectionsPerSec.Max | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.Mean | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.MeanRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.Min | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.OneMinuteRate | Type: float Unit: - |
| UncleanLeaderElectionsPerSec.RateUnit | Type: string Unit: - |
| UncleanLeaderElectionsPerSec.StdDev | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.50thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.75thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.95thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.98thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.999thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.99thPercentile | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.Count | Type: int Unit: count |
| UpdateFeaturesRateAndTimeMs.EventType | Type: string Unit: - |
| UpdateFeaturesRateAndTimeMs.FifteenMinuteRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.FiveMinuteRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.LatencyUnit | Type: string Unit: - |
| UpdateFeaturesRateAndTimeMs.Max | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.Mean | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.MeanRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.Min | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.OneMinuteRate | Type: float Unit: - |
| UpdateFeaturesRateAndTimeMs.RateUnit | Type: string Unit: - |
| UpdateFeaturesRateAndTimeMs.StdDev | Type: float Unit: - |
kafka_replica_manager¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
- Metrics
| Metric | Description |
|---|---|
| AtMinIsrPartitionCount.Value | Type: int Unit: count |
| FailedIsrUpdatesPerSec.Count | Type: int Unit: count |
| FailedIsrUpdatesPerSec.EventType | Type: string Unit: - |
| FailedIsrUpdatesPerSec.FifteenMinuteRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.FiveMinuteRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.MeanRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.OneMinuteRate | Type: float Unit: - |
| FailedIsrUpdatesPerSec.RateUnit | Type: string Unit: - |
| IsrExpandsPerSec.Count | Type: int Unit: count |
| IsrExpandsPerSec.EventType | Type: string Unit: - |
| IsrExpandsPerSec.FifteenMinuteRate | Type: float Unit: - |
| IsrExpandsPerSec.FiveMinuteRate | Type: float Unit: - |
| IsrExpandsPerSec.MeanRate | Type: float Unit: - |
| IsrExpandsPerSec.OneMinuteRate | Type: float Unit: - |
| IsrExpandsPerSec.RateUnit | Type: string Unit: - |
| IsrShrinksPerSec.Count | Type: int Unit: count |
| IsrShrinksPerSec.EventType | Type: string Unit: - |
| IsrShrinksPerSec.FifteenMinuteRate | Type: float Unit: - |
| IsrShrinksPerSec.FiveMinuteRate | Type: float Unit: - |
| IsrShrinksPerSec.MeanRate | Type: float Unit: - |
| IsrShrinksPerSec.OneMinuteRate | Type: float Unit: - |
| IsrShrinksPerSec.RateUnit | Type: string Unit: - |
| LeaderCount.Value | Type: int Unit: count |
| OfflineReplicaCount.Value | Type: int Unit: count |
| PartitionCount.Value | Type: int Unit: count |
| ReassigningPartitions.Value | Type: int Unit: count |
| UnderMinIsrPartitionCount.Value | Type: int Unit: count |
| UnderReplicatedPartitions.Value | Type: int Unit: count |
kafka_purgatory¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
- Metrics
| Metric | Description |
|---|---|
| AlterAcls.NumDelayedOperations | Type: int Unit: - |
| AlterAcls.PurgatorySize | Type: int Unit: - |
| DeleteRecords.NumDelayedOperations | Type: int Unit: - |
| DeleteRecords.PurgatorySize | Type: int Unit: - |
| ElectLeader.NumDelayedOperations | Type: int Unit: - |
| ElectLeader.PurgatorySize | Type: int Unit: - |
| Fetch.NumDelayedOperations | Type: int Unit: - |
| Fetch.PurgatorySize | Type: int Unit: - |
| Heartbeat.NumDelayedOperations | Type: int Unit: - |
| Heartbeat.PurgatorySize | Type: int Unit: - |
| Produce.NumDelayedOperations | Type: int Unit: - |
| Produce.PurgatorySize | Type: int Unit: - |
| Rebalance.NumDelayedOperations | Type: int Unit: - |
| Rebalance.PurgatorySize | Type: int Unit: - |
| topic.NumDelayedOperations | Type: int Unit: - |
| topic.PurgatorySize | Type: int Unit: - |
kafka_request¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
- Metrics
| Metric | Description |
|---|---|
| LocalTimeMs.50thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.75thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.95thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.98thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.999thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.99thPercentile | Type: float Unit: time,ms |
| LocalTimeMs.Count | Type: int Unit: count |
| LocalTimeMs.Max | Type: float Unit: time,ms |
| LocalTimeMs.Mean | Type: float Unit: time,ms |
| LocalTimeMs.Min | Type: float Unit: time,ms |
| LocalTimeMs.StdDev | Type: float Unit: time,ms |
| RemoteTimeMs.50thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.75thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.95thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.98thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.999thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.99thPercentile | Type: float Unit: time,ms |
| RemoteTimeMs.Count | Type: int Unit: count |
| RemoteTimeMs.Max | Type: float Unit: time,ms |
| RemoteTimeMs.Mean | Type: float Unit: time,ms |
| RemoteTimeMs.Min | Type: float Unit: time,ms |
| RemoteTimeMs.StdDev | Type: float Unit: time,ms |
| RequestBytes.50thPercentile | Type: float Unit: time,ms |
| RequestBytes.75thPercentile | Type: float Unit: time,ms |
| RequestBytes.95thPercentile | Type: float Unit: time,ms |
| RequestBytes.98thPercentile | Type: float Unit: time,ms |
| RequestBytes.999thPercentile | Type: float Unit: time,ms |
| RequestBytes.99thPercentile | Type: float Unit: time,ms |
| RequestBytes.Count | Type: int Unit: count |
| RequestBytes.Max | Type: float Unit: time,ms |
| RequestBytes.Mean | Type: float Unit: time,ms |
| RequestBytes.Min | Type: float Unit: time,ms |
| RequestBytes.StdDev | Type: float Unit: time,ms |
| RequestQueueTimeMs.50thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.75thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.95thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.98thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.999thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.99thPercentile | Type: float Unit: time,ms |
| RequestQueueTimeMs.Count | Type: int Unit: count |
| RequestQueueTimeMs.Max | Type: float Unit: time,ms |
| RequestQueueTimeMs.Mean | Type: float Unit: time,ms |
| RequestQueueTimeMs.Min | Type: float Unit: time,ms |
| RequestQueueTimeMs.StdDev | Type: float Unit: time,ms |
| ResponseQueueTimeMs.50thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.75thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.95thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.98thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.999thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.99thPercentile | Type: float Unit: time,ms |
| ResponseQueueTimeMs.Count | Type: int Unit: count |
| ResponseQueueTimeMs.Max | Type: float Unit: time,ms |
| ResponseQueueTimeMs.Mean | Type: float Unit: time,ms |
| ResponseQueueTimeMs.Min | Type: float Unit: time,ms |
| ResponseQueueTimeMs.StdDev | Type: float Unit: time,ms |
| ResponseSendTimeMs.50thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.75thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.95thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.98thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.999thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.99thPercentile | Type: float Unit: time,ms |
| ResponseSendTimeMs.Count | Type: int Unit: count |
| ResponseSendTimeMs.Max | Type: float Unit: time,ms |
| ResponseSendTimeMs.Mean | Type: float Unit: time,ms |
| ResponseSendTimeMs.Min | Type: float Unit: time,ms |
| ResponseSendTimeMs.StdDev | Type: float Unit: time,ms |
| ThrottleTimeMs.50thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.75thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.95thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.98thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.999thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.99thPercentile | Type: float Unit: time,ms |
| ThrottleTimeMs.Count | Type: int Unit: count |
| ThrottleTimeMs.Max | Type: float Unit: time,ms |
| ThrottleTimeMs.Mean | Type: float Unit: time,ms |
| ThrottleTimeMs.Min | Type: float Unit: time,ms |
| ThrottleTimeMs.StdDev | Type: float Unit: time,ms |
| TotalTimeMs.50thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.75thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.95thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.98thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.999thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.99thPercentile | Type: float Unit: time,ms |
| TotalTimeMs.Count | Type: int Unit: count |
| TotalTimeMs.Max | Type: float Unit: time,ms |
| TotalTimeMs.Mean | Type: float Unit: time,ms |
| TotalTimeMs.Min | Type: float Unit: time,ms |
| TotalTimeMs.StdDev | Type: float Unit: time,ms |
kafka_topics¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
- Metrics
| Metric | Description |
|---|---|
| BytesInPerSec.Count | Type: int Unit: count |
| BytesInPerSec.EventType | Type: string Unit: - |
| BytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesInPerSec.MeanRate | Type: float Unit: - |
| BytesInPerSec.OneMinuteRate | Type: float Unit: - |
| BytesInPerSec.RateUnit | Type: string Unit: - |
| BytesOutPerSec.Count | Type: int Unit: count |
| BytesOutPerSec.EventType | Type: string Unit: - |
| BytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesOutPerSec.MeanRate | Type: float Unit: - |
| BytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| BytesOutPerSec.RateUnit | Type: string Unit: - |
| BytesRejectedPerSec.Count | Type: int Unit: count |
| BytesRejectedPerSec.EventType | Type: string Unit: - |
| BytesRejectedPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesRejectedPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesRejectedPerSec.MeanRate | Type: float Unit: - |
| BytesRejectedPerSec.OneMinuteRate | Type: float Unit: - |
| BytesRejectedPerSec.RateUnit | Type: string Unit: - |
| FailedFetchRequestsPerSec.Count | Type: int Unit: count |
| FailedFetchRequestsPerSec.EventType | Type: string Unit: - |
| FailedFetchRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.MeanRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| FailedFetchRequestsPerSec.RateUnit | Type: string Unit: - |
| FailedProduceRequestsPerSec.Count | Type: int Unit: count |
| FailedProduceRequestsPerSec.EventType | Type: string Unit: - |
| FailedProduceRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.MeanRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| FailedProduceRequestsPerSec.RateUnit | Type: string Unit: - |
| FetchMessageConversionsPerSec.Count | Type: int Unit: count |
| FetchMessageConversionsPerSec.EventType | Type: string Unit: - |
| FetchMessageConversionsPerSec.FifteenMinuteRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.FiveMinuteRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.MeanRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.OneMinuteRate | Type: float Unit: - |
| FetchMessageConversionsPerSec.RateUnit | Type: string Unit: - |
| InvalidMagicNumberRecordsPerSec.Count | Type: int Unit: count |
| InvalidMagicNumberRecordsPerSec.EventType | Type: string Unit: - |
| InvalidMagicNumberRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.MeanRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| InvalidMagicNumberRecordsPerSec.RateUnit | Type: string Unit: - |
| InvalidMessageCrcRecordsPerSec.Count | Type: int Unit: count |
| InvalidMessageCrcRecordsPerSec.EventType | Type: string Unit: - |
| InvalidMessageCrcRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.MeanRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| InvalidMessageCrcRecordsPerSec.RateUnit | Type: string Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.Count | Type: int Unit: count |
| InvalidOffsetOrSequenceRecordsPerSec.EventType | Type: string Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.MeanRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| InvalidOffsetOrSequenceRecordsPerSec.RateUnit | Type: string Unit: - |
| MessagesInPerSec.Count | Type: int Unit: count |
| MessagesInPerSec.EventType | Type: string Unit: - |
| MessagesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| MessagesInPerSec.FiveMinuteRate | Type: float Unit: - |
| MessagesInPerSec.MeanRate | Type: float Unit: - |
| MessagesInPerSec.OneMinuteRate | Type: float Unit: - |
| MessagesInPerSec.RateUnit | Type: string Unit: - |
| NoKeyCompactedTopicRecordsPerSec.Count | Type: int Unit: count |
| NoKeyCompactedTopicRecordsPerSec.EventType | Type: string Unit: - |
| NoKeyCompactedTopicRecordsPerSec.FifteenMinuteRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.FiveMinuteRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.MeanRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.OneMinuteRate | Type: float Unit: - |
| NoKeyCompactedTopicRecordsPerSec.RateUnit | Type: string Unit: - |
| ProduceMessageConversionsPerSec.Count | Type: int Unit: count |
| ProduceMessageConversionsPerSec.EventType | Type: string Unit: - |
| ProduceMessageConversionsPerSec.FifteenMinuteRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.FiveMinuteRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.MeanRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.OneMinuteRate | Type: float Unit: - |
| ProduceMessageConversionsPerSec.RateUnit | Type: string Unit: - |
| ReassignmentBytesInPerSec.Count | Type: int Unit: count |
| ReassignmentBytesInPerSec.EventType | Type: string Unit: - |
| ReassignmentBytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.MeanRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.OneMinuteRate | Type: float Unit: - |
| ReassignmentBytesInPerSec.RateUnit | Type: string Unit: - |
| ReassignmentBytesOutPerSec.Count | Type: int Unit: count |
| ReassignmentBytesOutPerSec.EventType | Type: string Unit: - |
| ReassignmentBytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.MeanRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| ReassignmentBytesOutPerSec.RateUnit | Type: string Unit: - |
| ReplicationBytesInPerSec.Count | Type: int Unit: count |
| ReplicationBytesInPerSec.EventType | Type: string Unit: - |
| ReplicationBytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReplicationBytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| ReplicationBytesInPerSec.MeanRate | Type: float Unit: - |
| ReplicationBytesInPerSec.OneMinuteRate | Type: float Unit: - |
| ReplicationBytesInPerSec.RateUnit | Type: string Unit: - |
| ReplicationBytesOutPerSec.Count | Type: int Unit: count |
| ReplicationBytesOutPerSec.EventType | Type: string Unit: - |
| ReplicationBytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.MeanRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| ReplicationBytesOutPerSec.RateUnit | Type: string Unit: - |
| TotalFetchRequestsPerSec.Count | Type: int Unit: count |
| TotalFetchRequestsPerSec.EventType | Type: string Unit: - |
| TotalFetchRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.RateUnit | Type: string Unit: - |
| TotalProduceRequestsPerSec.Count | Type: int Unit: count |
| TotalProduceRequestsPerSec.EventType | Type: string Unit: - |
| TotalProduceRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.RateUnit | Type: string Unit: - |
kafka_topic¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
| topic | topic name |
- Metrics
| Metric | Description |
|---|---|
| BytesInPerSec.Count | Type: int Unit: count |
| BytesInPerSec.EventType | Type: string Unit: - |
| BytesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesInPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesInPerSec.MeanRate | Type: float Unit: - |
| BytesInPerSec.OneMinuteRate | Type: float Unit: - |
| BytesInPerSec.RateUnit | Type: string Unit: - |
| BytesOutPerSec.Count | Type: int Unit: count |
| BytesOutPerSec.EventType | Type: string Unit: - |
| BytesOutPerSec.FifteenMinuteRate | Type: float Unit: - |
| BytesOutPerSec.FiveMinuteRate | Type: float Unit: - |
| BytesOutPerSec.MeanRate | Type: float Unit: - |
| BytesOutPerSec.OneMinuteRate | Type: float Unit: - |
| BytesOutPerSec.RateUnit | Type: string Unit: - |
| MessagesInPerSec.Count | Type: int Unit: count |
| MessagesInPerSec.EventType | Type: string Unit: - |
| MessagesInPerSec.FifteenMinuteRate | Type: float Unit: - |
| MessagesInPerSec.FiveMinuteRate | Type: float Unit: - |
| MessagesInPerSec.MeanRate | Type: float Unit: - |
| MessagesInPerSec.OneMinuteRate | Type: float Unit: - |
| MessagesInPerSec.RateUnit | Type: string Unit: - |
| TotalFetchRequestsPerSec.Count | Type: int Unit: count |
| TotalFetchRequestsPerSec.EventType | Type: string Unit: - |
| TotalFetchRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalFetchRequestsPerSec.RateUnit | Type: string Unit: - |
| TotalProduceRequestsPerSec.Count | Type: int Unit: count |
| TotalProduceRequestsPerSec.EventType | Type: string Unit: - |
| TotalProduceRequestsPerSec.FifteenMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.FiveMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.MeanRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.OneMinuteRate | Type: float Unit: - |
| TotalProduceRequestsPerSec.RateUnit | Type: string Unit: - |
kafka_partition¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
| partition | partition number |
| topic | topic name |
- Metrics
| Metric | Description |
|---|---|
| LogEndOffset | Type: int Unit: - |
| LogStartOffset | Type: int Unit: - |
| NumLogSegments | Type: int Unit: - |
| Size | Type: int Unit: - |
| UnderReplicatedPartitions | Type: int Unit: - |
kafka_zookeeper¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
- Metrics
| Metric | Description |
|---|---|
| ZooKeeperRequestLatencyMs.50thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.75thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.95thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.98thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.999thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.99thPercentile | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.Count | Type: int Unit: count |
| ZooKeeperRequestLatencyMs.Max | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.Mean | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.Min | Type: float Unit: time,ms |
| ZooKeeperRequestLatencyMs.StdDev | Type: float Unit: time,ms |
kafka_network¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
| type | metric type |
- Metrics
| Metric | Description |
|---|---|
| ControlPlaneExpiredConnectionsKilledCount | Type: int Unit: count |
| ExpiredConnectionsKilledCount | Type: int Unit: count |
| MemoryPoolAvailable | Type: int Unit: count |
| MemoryPoolUsed | Type: int Unit: count |
| NetworkProcessorAvgIdlePercent | Type: float Unit: - |
kafka_log¶
- Tags
| Tag | Description |
|---|---|
| jolokia_agent_url | Jolokia agent url path |
| type | metric type |
- Metrics
| Metric | Description |
|---|---|
| DeadThreadCount | Type: int Unit: count |
| OfflineLogDirectoryCount | Type: int Unit: count |
| cleaner_recopy_percent | Type: float Unit: - |
| max_buffer_utilization_percent | Type: float Unit: - |
| max_clean_time_secs | Type: int Unit: time,s |
| max_compaction_delay_secs | Type: int Unit: time,s |
kafka_consumer¶
This metrics needs to be collected on the Consumer instance
- Tags
| Tag | Description |
|---|---|
| client_id | client id |
| jolokia_agent_url | Jolokia agent url path |
| type | metric type |
- Metrics
| Metric | Description |
|---|---|
| assigned_partitions | Type: int Unit: count |
| bytes_consumed_rate | Type: float Unit: - |
| bytes_consumed_total | Type: int Unit: count |
| commit_id | Type: string Unit: - |
| commit_rate | Type: float Unit: - |
| commit_total | Type: int Unit: count |
| connection_close_rate | Type: float Unit: - |
| connection_close_total | Type: int Unit: count |
| connection_count | Type: int Unit: count |
| connection_creation_rate | Type: float Unit: - |
| connection_creation_total | Type: int Unit: count |
| count | Type: int Unit: count |
| failed_authentication_rate | Type: float Unit: - |
| failed_authentication_total | Type: int Unit: count |
| failed_reauthentication_rate | Type: float Unit: - |
| failed_reauthentication_total | Type: int Unit: count |
| failed_rebalance_rate_per_hour | Type: float Unit: - |
| failed_rebalance_total | Type: float Unit: - |
| fetch_latency_avg | Type: float Unit: - |
| fetch_latency_max | Type: float Unit: - |
| fetch_rate | Type: float Unit: - |
| fetch_throttle_time_avg | Type: float Unit: - |
| fetch_throttle_time_max | Type: float Unit: - |
| fetch_total | Type: float Unit: - |
| heartbeat_rate | Type: float Unit: - |
| heartbeat_response_time_max | Type: float Unit: - |
| heartbeat_total | Type: int Unit: count |
| incoming_byte_rate | Type: float Unit: - |
| incoming_byte_total | Type: float Unit: - |
| io_ratio | Type: float Unit: - |
| io_time_ns_avg | Type: int Unit: time,ns |
| io_wait_ratio | Type: float Unit: - |
| io_wait_time_ns_avg | Type: int Unit: time,ns |
| io_waittime_total | Type: int Unit: time,ns |
| iotime_total | Type: int Unit: count |
| join_rate | Type: float Unit: - |
| join_total | Type: int Unit: count |
| last_heartbeat_seconds_ago | Type: float Unit: - |
| last_poll_seconds_ago | Type: int Unit: count |
| last_rebalance_seconds_ago | Type: float Unit: - |
| network_io_rate | Type: float Unit: - |
| network_io_total | Type: int Unit: count |
| outgoing_byte_rate | Type: float Unit: - |
| outgoing_byte_total | Type: float Unit: - |
| rebalance_latency_total | Type: int Unit: count |
| rebalance_rate_per_hour | Type: float Unit: - |
| rebalance_total | Type: int Unit: count |
| records_consumed_rate | Type: float Unit: - |
| records_consumed_total | Type: float Unit: - |
| request_rate | Type: float Unit: - |
| request_size_avg | Type: float Unit: - |
| request_size_max | Type: float Unit: - |
| request_total | Type: int Unit: count |
| response_rate | Type: float Unit: - |
| response_total | Type: int Unit: count |
| select_rate | Type: float Unit: - |
| select_total | Type: int Unit: count |
| start_time_ms | Type: int Unit: timeStamp,msec |
| successful_authentication_no_reauth_total | Type: int Unit: count |
| successful_authentication_rate | Type: float Unit: - |
| successful_authentication_total | Type: int Unit: count |
| successful_reauthentication_rate | Type: float Unit: - |
| successful_reauthentication_total | Type: int Unit: count |
| sync_rate | Type: float Unit: - |
| sync_total | Type: int Unit: - |
| version | Type: string Unit: - |
kafka_producer¶
This metrics needs to be collected on the Producer instance
- Tags
| Tag | Description |
|---|---|
| client_id | client id |
| jolokia_agent_url | Jolokia agent url path |
| type | metric type |
- Metrics
| Metric | Description |
|---|---|
| batch_split_rate | Type: float Unit: - |
| batch_split_total | Type: int Unit: - |
| buffer_available_bytes | Type: int Unit: - |
| buffer_exhausted_rate | Type: float Unit: - |
| buffer_exhausted_total | Type: float Unit: - |
| buffer_total_bytes | Type: int Unit: - |
| bufferpool_wait_ratio | Type: float Unit: - |
| bufferpool_wait_time_total | Type: int Unit: count |
| commit_id | Type: string Unit: - |
| connection_close_rate | Type: float Unit: - |
| connection_close_total | Type: int Unit: count |
| connection_count | Type: int Unit: count |
| connection_creation_rate | Type: float Unit: - |
| connection_creation_total | Type: int Unit: count |
| count | Type: int Unit: count |
| failed_authentication_rate | Type: float Unit: - |
| failed_authentication_total | Type: int Unit: count |
| failed_reauthentication_rate | Type: float Unit: - |
| failed_reauthentication_total | Type: int Unit: count |
| incoming_byte_rate | Type: float Unit: - |
| incoming_byte_total | Type: int Unit: - |
| io_ratio | Type: float Unit: - |
| io_time_ns_avg | Type: float Unit: time,ns |
| io_wait_ratio | Type: float Unit: - |
| io_wait_time_ns_avg | Type: int Unit: time,ns |
| io_waittime_total | Type: int Unit: time,ns |
| iotime_total | Type: int Unit: time,ns |
| metadata_age | Type: float Unit: - |
| network_io_rate | Type: float Unit: - |
| network_io_total | Type: int Unit: count |
| outgoing_byte_rate | Type: float Unit: - |
| outgoing_byte_total | Type: int Unit: count |
| produce_throttle_time_avg | Type: int Unit: time,ms |
| produce_throttle_time_max | Type: int Unit: time,ms |
| record_error_rate | Type: float Unit: - |
| record_error_total | Type: int Unit: - |
| record_retry_rate | Type: float Unit: - |
| record_retry_total | Type: int Unit: - |
| record_send_rate | Type: float Unit: - |
| record_send_total | Type: int Unit: count |
| request_rate | Type: float Unit: - |
| request_size_avg | Type: float Unit: - |
| request_size_max | Type: int Unit: - |
| request_total | Type: int Unit: count |
| requests_in_flight | Type: int Unit: count |
| response_rate | Type: float Unit: - |
| response_total | Type: int Unit: count |
| select_rate | Type: float Unit: - |
| select_total | Type: int Unit: count |
| start_time_ms | Type: int Unit: timeStamp,msec |
| successful_authentication_no_reauth_total | Type: int Unit: count |
| successful_authentication_rate | Type: float Unit: - |
| successful_authentication_total | Type: int Unit: count |
| successful_reauthentication_rate | Type: float Unit: - |
| successful_reauthentication_total | Type: int Unit: - |
| version | Type: string Unit: - |
| waiting_threads | Type: int Unit: count |
kafka_connect¶
This metrics needs to be collected on the Connect instance
- Tags
| Tag | Description |
|---|---|
| client_id | client id |
| connector | connector |
| jolokia_agent_url | Jolokia agent url path |
| task | task |
| type | metric type |
- Metrics
| Metric | Description |
|---|---|
| batch_size_avg | Type: int Unit: - |
| batch_size_max | Type: int Unit: - |
| commit_id | Type: string Unit: - |
| connector_class | Type: string Unit: - |
| connector_count | Type: int Unit: count |
| connector_destroyed_task_count | Type: int Unit: count |
| connector_failed_task_count | Type: int Unit: count |
| connector_paused_task_count | Type: int Unit: count |
| connector_restarting_task_count | Type: int Unit: count |
| connector_running_task_count | Type: int Unit: count |
| connector_startup_attempts_total | Type: int Unit: count |
| connector_startup_failure_percentage | Type: float Unit: - |
| connector_startup_failure_total | Type: int Unit: count |
| connector_startup_success_percentage | Type: float Unit: - |
| connector_startup_success_total | Type: int Unit: count |
| connector_total_task_count | Type: int Unit: count |
| connector_type | Type: string Unit: - |
| connector_unassigned_task_count | Type: int Unit: count |
| connector_version | Type: string Unit: - |
| count | Type: int Unit: count |
| deadletterqueue_produce_failures | Type: int Unit: count |
| deadletterqueue_produce_requests | Type: int Unit: count |
| last_error_timestamp | The epoch timestamp when this task last encountered an error in millisecond. Type: int Unit: timeStamp,msec |
| offset_commit_avg_time_ms | Type: float Unit: time,ms |
| offset_commit_completion_rate | Type: float Unit: - |
| offset_commit_completion_total | Type: int Unit: count |
| offset_commit_failure_percentage | Type: float Unit: - |
| offset_commit_max_time_ms | Type: float Unit: time,ms |
| offset_commit_seq_no | Type: int Unit: - |
| offset_commit_skip_rate | Type: float Unit: - |
| offset_commit_skip_total | Type: int Unit: count |
| offset_commit_success_percentage | Type: float Unit: - |
| partition_count | Type: int Unit: count |
| pause_ratio | Type: float Unit: - |
| put_batch_avg_time_ms | Type: float Unit: time,ms |
| put_batch_max_time_ms | Type: float Unit: time,ms |
| running_ratio | Type: float Unit: - |
| sink_record_active_count | Type: int Unit: count |
| sink_record_active_count_avg | Type: int Unit: count |
| sink_record_active_count_max | Type: int Unit: count |
| sink_record_read_rate | Type: float Unit: - |
| sink_record_read_total | Type: int Unit: count |
| sink_record_send_rate | Type: float Unit: - |
| sink_record_send_total | Type: int Unit: count |
| source_record_active_count | Type: int Unit: count |
| source_record_poll_rate | Type: float Unit: - |
| source_record_poll_total | Type: int Unit: count |
| source_record_write_rate | Type: float Unit: - |
| source_record_write_total | Type: int Unit: count |
| start_time_ms | Type: int Unit: timeStamp,msec |
| status | Type: string Unit: - |
| task_count | Type: int Unit: count |
| task_startup_attempts_total | Type: int Unit: count |
| task_startup_failure_percentage | Type: float Unit: - |
| task_startup_failure_total | Type: int Unit: count |
| task_startup_success_percentage | Type: float Unit: - |
| task_startup_success_total | Type: int Unit: count |
| total_errors_logged | Type: int Unit: count |
| total_record_errors | Type: int Unit: count |
| total_record_failures | Type: int Unit: count |
| total_records_skipped | Type: int Unit: count |
| total_retries | Type: int Unit: count |
| version | Type: string Unit: - |
Log Collection¶
To collect kafka's log, open files in kafka.conf and write to the absolute path of the kafka log file. For example:
[[inputs.kafka]]
...
[inputs.kafka.log]
files = ["/usr/local/var/log/kafka/error.log","/usr/local/var/log/kafka/kafka.log"]
When log collection is turned on, a log with a log source of kafka is generated by default.
Note: DataKit must be installed on Kafka's host to collect Kafka logs.
Example of cutting logs:
[2020-07-07 15:04:29,333] DEBUG Progress event: HTTP_REQUEST_COMPLETED_EVENT, bytes: 0 (io.confluent.connect.s3.storage.S3OutputStream:286)
The list of cut fields is as follows:
| Field Name | Field Value |
|---|---|
| msg | Progress event: HTTP_REQUEST_COMPLETED_EVENT, bytes: 0 |
| name | io.confluent.connect.s3.storage.S3OutputStream:286 |
| status | DEBUG |
| time | 1594105469333000000 |
FAQ¶
Why can't see kafka_producer/kafka_producer/kafka_connect measurements?¶
After Kafka service is started, if you need to collect Producer/Consumer/Connector indicators, you need to configure Jolokia for them respectively.
Referring to Kafka Quick Start, configure the KAFKA_OPTS environment variable for the example of Producer, as follows:
export KAFKA_OPTS="-javaagent:/usr/local/datakit/data/jolokia-jvm-agent.jar=host=127.0.0.1,port=8090"
Go into the Kafka directory and start a Producer:
Copy a Kafka.conf to open multiple Kafka collectors and configure the url:
And remove comments from the fields in the collect producer metrics section:
# The following metrics are available on producer instances.
[[inputs.kafka.metric]]
name = "kafka_producer"
mbean = "kafka.producer:type=*,client-id=*"
tag_keys = ["client-id", "type"]
Restart DataKit, which then collects metrics for the Producer instance.