Confluent Cloud

Collect Kafka Metrics data from Confluent Cloud

Configuration¶

Confluent Cloud Metrics Integration Configuration¶

1. Log in to Confluent Cloud, under ADMINISTRATION select Metrics

2. On the Metrics page click the button New integration, choose monitoring type Prometheus

3. Click the Generate Cloud API key button to generate an API Key

4. Under Resources select All Kafka clusters, generating Prometheus's scrape_configs

5. Click the Copy button on the page to copy the content

scrape_configs:
  - job_name: Confluent Cloud
    scrape_interval: 1m
    scrape_timeout: 1m
    honor_timestamps: true
    static_configs:
      - targets:
        - api.telemetry.confluent.cloud
    scheme: https
    basic_auth:
      username: H5BO.....
      password: RDCgMwguHMy.....
    metrics_path: /v2/metrics/cloud/export
    params:
      "resource.kafka.id":
        - lkc-xxxx

DataKit Collector Configuration¶

Since Confluent Cloud can directly expose a metrics url, it is possible to collect data directly using the prom collector.

Navigate to the DataKit installation directory under conf.d/prom, copy prom.conf.sample as confluent_cloud.conf.

cp prom.conf.sample confluent_cloud.conf

Adjust the contents of confluent_cloud.conf as follows:

# {"version": "1.61.0-testing_testing-graphite-metric-set", "desc": "do NOT edit this line"}

[[inputs.prom]]
  ## Exporter URLs.
  urls = ["https://api.telemetry.confluent.cloud/v2/metrics/cloud/export?resource.kafka.id=lkc-xxxx"]

  source = "confluent_cloud"

  ## Add HTTP headers for data pulling (Example basic authentication).
  [inputs.prom.http_headers]
     Authorization = "Basic QkXXXXXXXXXXXX"

  interval = "60s"

Parameter adjustment notes:

• urls: Adjust the content copied from Confluent Cloud and concatenate into a URL; if there are multiple Kafka resources, separate them with commas.

• Authorization: Convert the username and password into Basic Authorization format.

• interval: Adjust to 60s, due to Confluent Cloud API restrictions, this value cannot be less than 60s; values below this will result in data not being collected.

Restart DataKit¶

Restart DataKit

Metrics¶

Confluent Measurement Set¶