Collector "Huawei Cloud - Cloud Monitoring" Configuration Manual¶

Before reading this document, please read:

Cloud Integration

Tip

Before using this collector, you must install "Integration Core" and its accompanying third-party dependencies.

Tip

This collector supports multi-threading by default (five threads are enabled by default). If you need to change the thread pool size, you can set the environment variable COLLECTOR_THREAD_POOL_SIZE.

1. Configuration Structure¶

The configuration structure of this collector is as follows:

Field	Type	Required	Description
`regions`	list	Required	List of regions to collect data from
`regions[#]`	str	Required	Region ID. For example: `'cn-north-4'` See the appendix for the complete list
`targets`	list	Required	List of cloud monitoring target configurations Multiple configurations under the same namespace are logically "AND" related
`targets[#].namespace`	str	Required	The cloud monitoring namespace to collect data from. For example: `SYS.OBS` See the appendix for the complete list
`targets[#].metrics`	list	Required	List of metric names to collect under the cloud service
`targets[#].metrics[#]`	str	Required	Metric name pattern, supports `"ALL"`, `"NOT"`, and wildcard matching Multiple patterns are logically "OR" related When `"NOT"` is included, multiple patterns are logically "AND" related See below for details

Tip

The collector will automatically obtain all IAM projects under the region, and then obtain resources based on the IAM projects.

2. Configuration Examples¶

Specify Specific Metrics¶

Collect 2 metrics named capacity_total and capacity_archive from SYS.OBS.

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['capacity_total', 'capacity_archive']
        }
    ]
}

Wildcard Matching for Metrics¶

Metric names can be matched using the * wildcard.

The following metrics will be collected in this example:

Metrics named capacity_total
Metrics starting with capacity
Metrics ending with total
Metrics containing capacity

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['capacity_total', 'capacity*', '*total', '*capacity*']
        }
    ]
}

Exclude Specific Metrics¶

Add the "NOT" marker at the beginning to exclude the following metrics.

The following metrics will [not] be collected in this example:

Metrics named capacity_total
Metrics starting with capacity
Metrics ending with total
Metrics containing capacity

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['NOT', 'capacity_total', 'capacity*', '*total', '*capacity*']
        }
    ]
}

Multi-level Filtering for Required Metrics¶

The same namespace can be specified multiple times, filtering metric names sequentially from top to bottom.

In this example, the metric names are filtered through the following steps:

Select all metrics containing capacity in their names.
From the previous result, exclude metrics named capacity_total.

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['*capacity*']
        },
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['NOT', 'capacity_total']
        }
    ]
}

Configure Filters (Optional)¶

This collector script supports custom filters, allowing users to filter target resources based on object attributes. The filter function returns True or False.

True: The target resource needs to be collected.
False: The target resource does not need to be collected.

# Example: Enable the filter to filter objects based on the instance_id and name attributes. The configuration format is as follows:
def filter_instance(instance, namespace='A'):
    '''
    Collect metrics with namespace A and id xxxx
    '''
    # return True
    instance_id = instance['tags'].get('id')
    if instance_id in ['xxx']:
        return True
    return False


###### Do not modify the following contents #####
from integration_core__runner import Runner
import integration_huaweicloud_ces__main as main


@DFF.API('HuaweiCloud-Monitor Collection', timeout=3600, fixed_crontab='*/5 * * * *')
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

Tip

When multiple filters are configured under the same namespace, all filters must be satisfied for the data to be reported.

3. Data Reporting Format¶

After data is successfully synchronized, you can view the data in the "Metrics" section of Guance.

Take the following collector configuration as an example:

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['capacity_total']
        }
    ]
}

The reported data example is as follows:

{
  "measurement": "huaweicloud_SYS.OBS",
  "tags": {
    "bucket_name": "i-xxx"
  },
  "fields": {
    "capacity_total_average" : "{...}",
    "capacity_total_max"     : "{...}",
    "capacity_total_min"     : "{...}",
    "capacity_total_sum"     : "{...}",
    "capacity_total_variance": "{...}"
  }
}

Tip

All metric values will be reported as float type.

Tip

This collector collects the capacity_total metric data under the SYS.OBS namespace (Namespace). For details, see the Data Collection Description table.

4. Integration with Custom Object Collectors¶

When other custom object collectors (e.g., OBS) are running in the same DataFlux Func, this collector will supplement fields based on the dimension information in the Data Collection Description table. For example, OBS tries to match the bucket_name field returned by cloud monitoring data with the tags.name field in the custom object.

Since it is necessary to know the custom object information in advance for integration in the cloud monitoring collector, it is generally recommended to place the cloud monitoring collector at the end of the list, such as:

# Create collectors
collectors = [
    huaweicloud_obs.DataCollector(account, common_huaweicloud_configs), # Custom object collector
    huaweicloud_ces.DataCollector(account, huaweicloud_ces_configs) # Cloud monitoring collector
]

When a successful match is made, additional fields from the custom object tags will be added to the cloud monitoring data tags, enabling effects such as filtering cloud monitoring metric data using instance names. The specific effect is as follows:

Assume the original data collected by cloud monitoring is as follows:

{
  "measurement": "huaweicloud_SYS.OBS",
  "tags": {
    "bucket_name": "i-xxx"
  },
  "fields": { 
    "key": "value"
   }
}

At the same time, the custom object data collected by the Huawei Cloud OBS collector is as follows:

{
  "measurement": "huaweicloud_cvm",
  "tags": {
    "name"           : "xxx",
    "bucket_type"    : "xxx",
    "PlatformDetails": "xxx",
    "{...}"          : "{...}"
  },
  "fields": { 
    "key": "value" 
  }
}

Then, the final reported cloud monitoring data is as follows:

{
  "measurement": "huaweicloud_SYS.OBS",
  "tags": {
    "name"            : "xxx",
    "bucket_name"     : "xxx", // Original field from cloud monitoring
    "bucket_type"     : "xxx", // Field from custom object OBS
    "PlatformDetails" : "xxx", // Field from custom object OBS
    "{...}"
  },
  "fields": { 
    "key": "value"
   }
}

5. Troubleshooting¶

When running the program, you may encounter the following error:

HTTPClientError: An HTTP Client raised an unhandled exception: SoftTimeLimitExceeded()

Reason: The task execution time is too long, causing a timeout.

Solution:

Reduce the number of collected metrics, clarify requirements, and only collect metrics that are truly needed.
Appropriately increase the timeout setting for the startup function, such as:

# Set timeout to 120 seconds
@DFF.API('Execute Collection', timeout=120)
def run():
    # Specific code omitted
    pass

7. Cloud Monitoring API Call Count Explanation¶

Tip

Huawei Cloud does not limit the number of API calls used in this script, and they are all free to use.

X. Appendix¶

Huawei Cloud Monitoring¶

Please refer to the official Huawei Cloud documentation: