HUAWEI CLOUD Kafka¶

Use the「Guance Synchronization」series script package in the script market to synchronize data from cloud monitoring cloud assets to the Guance.

Config¶

Install Func¶

Recommend opening 「Integrations - Extension - DataFlux Func (Automata)」: All preconditions are installed automatically, Please continue with the script installation

If you deploy Func yourself,Refer to Self-Deployment of Func

Installation script¶

Tip：Please prepare HUAWEI CLOUD AK that meets the requirements in advance（For simplicity's sake, you can directly grant the global read-only permissionReadOnlyAccess）

To synchronize the monitoring data of HUAWEI CLOUD Kafka cloud resources, we install the corresponding collection script：「Guance Integration (HUAWEI CLOUD-Kafka Collection)」(ID：guance_huaweicloud_kafka)

Click 【Install】 and enter the corresponding parameters: HUAWEI CLOUD AK, HUAWEI CLOUD account name.

tap【Deploy startup Script】,The system automatically creates Startup script sets,And automatically configure the corresponding startup script.

After this function is enabled, you can view the automatic triggering configuration in「Guance Integration (HUAWEI CLOUD-Kafka Collection)」,Click【Run】,you can immediately execute once, without waiting for a regular time.After a while, you can view task execution records and corresponding logs.

If you want to collect logs, you must enable the corresponding log collection script. If you want to collect bills, start the cloud bill collection script.

We collected some configurations by default, as described in the Metrics column Configure custom cloud object metrics

Verify¶

1. In「Management / Crontab Config」check whether the automatic triggering configuration exists for the corresponding task. In addition, you can view task records and logs to check whether exceptions exist
2. On the Guance platform, click 「Infrastructure / Custom」 to check whether asset information exists
3. On the Guance platform, press 「Metrics」 to check whether monitoring data exists

Metric¶

Configure HUAWEI CLOUD Kafka collection, the default set of metrics is as follows, you can configure the way to collect more metrics HUAWEI CLOUD cloud monitoring metrics details

Example monitoring metrics¶

Node monitoring metrics¶

Consumer Group Monitoring Metrics¶

对象¶

Collected HUAWEI CLOUD Kafka object data structure, you can see the object data from "Infrastructure - Customization".

{
  "measurement": "huaweicloud_kafka",
  "tags": {      
    "RegionId"           : "cn-north-4",
    "charging_mode"      : "1",
    "connect_address"    : "192.168.0.161,192.168.0.126,192.168.0.31",
    "description"        : "",
    "engine"             : "kafka",
    "engine_version"     : "2.7",
    "instance_id"        : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "kafka_manager_user" : "",
    "name"               : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "port"               : "9092",
    "project_id"         : "f5f4c067d68xxxx86e173b18367bf",
    "resource_spec_code" : "",
    "service_type"       : "advanced",
    "specification"      : "kafka.2u4g.cluster.small * 3 broker",
    "status"             : "RUNNING",
    "storage_type"       : "hec",
    "user_id"            : "e4b27d49128e4bd0893b28d032a2e7c0",
    "user_name"          : "xxxx"
  },
  "fields": {
    "created_at"          : "1693203968959",
    "maintain_begin"      : "02:00:00",
    "maintain_end"        : "06:00:00",
    "storage_space"       : 186,
    "total_storage_space" : 300,
    "message"             : "{Instance JSON data}"
  }
}

Note: Fields in tags, fields are subject to change with subsequent updates.

Tip 1: The tags.name value is the instance ID, which serves as a unique identifier.

Tip 2: The following fields are JSON serialized strings