Huawei Cloud DMS Kafka

Collect Huawei Cloud DMS Kafka Metrics data

Configuration¶

Install Func¶

It is recommended to activate Guance Integration - Extension - DataFlux Func (Automata): all prerequisites will be automatically installed, please proceed with the script installation.

If you deploy Func yourself, refer to Self-deploy Func

Install Script¶

Note: Please prepare a Huawei Cloud AK that meets the requirements in advance (for simplicity, you can directly grant global read-only permission ReadOnlyAccess).

To synchronize Huawei Cloud Kafka monitoring data, we install the corresponding collection script: by accessing the web service of func and entering the 【Script Market】, 「Guance Integration (Huawei Cloud-Kafka Collection)」(ID: guance_huaweicloud_kafka)

After clicking 【Install】, enter the corresponding parameters: Huawei Cloud AK, SK, Huawei Cloud account name.

Click 【Deploy Startup Script】, the system will automatically create a Startup script set and automatically configure the corresponding startup script.

After the script is installed, find the script 「Guance Integration (Huawei Cloud-Kafka Collection)」in the "Development" section of Func, expand and modify this script. Find collector_configs and monitor_configs respectively and edit the content of region_projects. Change the region and Project ID to the actual region and Project ID, then click Save and Publish.

In addition, you can see the corresponding automatic trigger configuration in the 「Management / Automatic Trigger Configuration」. Click 【Execute】to immediately execute once without waiting for the scheduled time. After a while, you can view the execution task records and corresponding logs.

Verification¶

1. In 「Management / Automatic Trigger Configuration」, confirm whether the corresponding tasks have the corresponding automatic trigger configurations. You can also check the corresponding task records and logs to check for any abnormalities.
2. On the Guance platform, in 「Infrastructure - Resource Catalog」, check if there is any asset information.
3. On the Guance platform, in 「Metrics」, check if there are any corresponding monitoring data.

Metrics¶

Collect Huawei Cloud DMS Kafka Metrics data, you can collect more metrics through configuration Huawei Cloud DMS kafka Metrics Details

Instance Monitoring Metrics¶

Node Monitoring Metrics¶

Consumer Group Monitoring Metrics¶

Objects¶

The structure of the collected Huawei Cloud Kafka object data can be viewed in 「Infrastructure - Resource Catalog」

{
  "measurement": "huaweicloud_kafka",
  "tags": {      
    "RegionId"           : "cn-north-4",
    "charging_mode"      : "1",
    "connect_address"    : "192.168.0.161,192.168.0.126,192.168.0.31",
    "description"        : "",
    "engine"             : "kafka",
    "engine_version"     : "2.7",
    "instance_id"        : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "kafka_manager_user" : "",
    "name"               : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "port"               : "9092",
    "project_id"         : "f5f4c067d68xxxx86e173b18367bf",
    "resource_spec_code" : "",
    "service_type"       : "advanced",
    "specification"      : "kafka.2u4g.cluster.small * 3 broker",
    "status"             : "RUNNING",
    "storage_type"       : "hec",
    "user_id"            : "e4b27d49128e4bd0893b28d032a2e7c0",
    "user_name"          : "xxxx"
  },
  "fields": {
    "created_at"          : "1693203968959",
    "maintain_begin"      : "02:00:00",
    "maintain_end"        : "06:00:00",
    "storage_space"       : 186,
    "total_storage_space" : 300,
    "message"             : "{Instance JSON data}"
  }
}

Note: Fields in tags, fields may change with subsequent updates.

Tip 1: The value of tags.name is the instance ID, which serves as a unique identifier.

Tip 2: The following fields are all strings serialized in JSON.