Collector "Tencent Cloud - Cloud Monitor" Configuration Manual¶

Before reading this document, please read:

• Cloud Integration

1. Configuration Structure¶

The configuration structure of this collector is as follows:

2. Configuration Examples¶

Specify Specific Metrics¶

Collect 2 metrics named WanOuttraffic and WanOutpkg from QCE/CVM

tencentcloud_monitor_configs = {
    'regions': ['ap-shanghai'],
    'targets': [
        {
            'namespace': 'QCE/CVM',
            'metrics'  : ['WanOuttraffic', 'WanOutpkg'],
        }
    ]
}

Wildcard Matching Metrics¶

Metric names can use * wildcard for matching.

In this example, the following metrics will be collected:

• Metric named WanOutpkg
• Metrics starting with Wan
• Metrics ending with Outpkg
• Metrics containing Out

tencentcloud_monitor_configs = {
    'regions': ['ap-shanghai'],
    'targets': [
        {
            'namespace': 'QCE/CVM',
            'metrics'  : ['WanOutpkg', 'Wan*', '*Outpkg', '*Out*']
        }
    ]
}

Exclude Specific Metrics¶

Adding "NOT" at the beginning indicates excluding the following metrics.

In this example, the following metrics [will not] be collected:

• Metric named WanOutpkg
• Metrics starting with Wan
• Metrics ending with Outpkg
• Metrics containing Out

tencentcloud_monitor_configs = {
    'regions': ['ap-shanghai'],
    'targets': [
        {
            'namespace': 'QCE/CVM',
            'metrics'  : ['NOT', 'WanOutpkg', 'Wan*', '*Outpkg', '*Out*']
        }
    ]
}

Multiple Filters for Specified Metrics¶

The same namespace can be specified multiple times, and metrics will be filtered sequentially from top to bottom.

In this example, the following filtering steps are performed on the metric names:

1. Select all metrics containing Out in their names.

2. From the previous result, exclude the metric named WanOutpkg.

tencentcloud_monitor_configs = {
    'regions': ['ap-shanghai'],
    'targets': [
        {
            'namespace': 'QCE/CVM',
            'metrics'  : ['*Out*']
        },
        {
            'namespace': 'QCE/CVM',
            'metrics'  : ['NOT', 'WanOutpkg']
        }
    ]
}

Configure Filters (Optional)¶

This collector script supports custom filters, allowing users to filter target resources based on object attributes. The filter function returns True or False.

• True: The target resource should be collected.
• False: The target resource should not be collected.

Tencent Cloud Monitor supports filtering attributes that are consistent with cloud server (CVM), cloud database (CDB, Redis, MongoDB), load balancer (CLB), object storage (COS), and other object attribute data. For details, please refer to the Tencent Cloud Custom Object Collector documentation.

# Example: Enable filter, filter based on the object's name and RegionId attributes, configure as follows:

def filter_instance(instance,namespace='QCE/COS'):
    '''
    Collect metrics for objects with names smart-xxxxa, smart-xxxxb and RegionId ap-nanjing
    '''
    instance_name = instance['name']
    region_id = instance['RegionId']
    if instance_name in ['smart-xxxxa', 'smart-xxxxb'] and region_id in ['ap-nanjing']:
        return True
    return False

from integration_core__runner import Runner
import integration_tencentcloud_monitor__main as main

def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Collection Instructions¶

Cloud Product Configuration Information¶

Monitoring Metric Configuration Information¶

QCE/CVM¶

• Cloud Server Monitoring Metrics

QCE/CDB¶

• Cloud Database MySQL Monitoring Metrics

QCE/COS¶

• Object Storage Monitoring Metrics

QCE/LB_PRIVATE¶

• Private Load Balancer Monitoring Metrics

QCE/LB_PUBLIC¶

• Public Load Balancer Monitoring Metrics

QCE/REDIS_MEM¶

• Cloud Database Redis Memory Edition Monitoring Metrics (5 Seconds)

QCE/CMONGO¶

• Cloud Database MongoDB Monitoring Metrics

4. Data Reporting Format¶

After data is synchronized normally, you can view the data in the "Metrics" section of Guance.

Take the following collector configuration as an example:

tencentcloud_monitor_configs = {
    'regions': ['ap-shanghai'],
    'targets': [
        {
            'namespace': 'QCE/CVM',
            'metrics'  : ['WanOutpkg']
        }
    ]
}

The reported data example is as follows:

{
  "measurement": "tencentcloud_QCE/CVM",
  "tags": {
    "InstanceId": "i-xxx"
  },
  "fields": {
    "WanOutpkg_max": 0.005
  }
}

5. Interaction with Custom Object Collectors¶

When other custom object collectors (such as CVM) are running in the same DataFlux Func, this collector will supplement fields based on the dimension information in the Data Collection Instructions. For example, CVM tries to match the tags.name field in the custom object based on the InstanceId field returned by cloud monitoring data.

Since it is necessary to first obtain custom object information before interacting in the cloud monitoring collector, it is generally recommended to place the cloud monitoring collector at the end of the list, such as:

# Create collectors
collectors = [
    tencentcloud_cvm.DataCollector(account, common_tencentcloud_configs),
    tencentcloud_monitor.DataCollector(account, tencentcloud_monitor_configs) # Cloud monitoring collector is usually placed last
]

When a successful match is made, additional fields from the custom object tags will be added to the cloud monitoring data tags, thereby achieving effects such as filtering cloud monitoring metric data using instance names. The specific effect is as follows:

Assume the original data collected by cloud monitoring is as follows:

{
  "measurement": "tencentcloud_QCE/CVM",
  "tags": {
    "InstanceId": "i-xxx"
  },
  "fields": { "..." }
}

At the same time, the custom object data collected by the Tencent Cloud CVM collector is as follows:

{
  "measurement": "tencentcloud_cvm",
  "tags": {
    "name"           : "i-xxx",
    "InstanceType"   : "c6g.xxx",
    "PlatformDetails": "xxx",
    "{...}"
  },
  "fields": { "..." }
}

Then, the final reported cloud monitoring data is as follows:

{
  "measurement": "tencentcloud_QCE/CVM",
  "tags": {
    "name"            : "i-xxx",
    "InstanceId"          : "i-xxx",   // Original field from cloud monitoring
    "InstanceType"    : "c6g.xxx", // Field from custom object CVM
    "PlatformDetails" : "xxx",     // Field from custom object CVM
    "{Other fields omitted}"
  },
  "fields": { "Content omitted" }
}

6. Cloud Monitoring API Call Count Explanation¶

Tencent Cloud Cloud Monitor has a free quota limit for some API call counts (this collector uses the GetMonitorData API to request monitoring data. It belongs to the free quota API. The free request quota for each primary account is 1 million times/month. The excess part is charged at 0.25 yuan/10,000 times. In addition, after exceeding the free quota, you will not be able to continue using it. If you need to continue calling the API, you need to manually enable "API Request Pay-As-You-Go".) The following is a detailed explanation of the script set call count:

1. Users have multiple resources and need to collect multiple monitoring items. Will it exceed the free quota:

This collector uses GetMonitorData (query the latest monitoring data of specified monitoring items). One request can obtain multiple (up to 10, if exceeded, pagination is used) resources for a certain monitoring item. Example of request count:

• There are 10 cvm resources under the account that need to collect 1 monitoring item CpuUsage, 1 request is needed;
• There are 10 cvm resources under the account that need to collect 2 monitoring items CpuUsage and BaseCpuUsage, 2 requests are needed (one request per monitoring item);
• There are 11 cvm resources under the account that need to collect 1 monitoring item CpuUsage, 2 requests are needed (resources exceed 10, pagination is used);
• There are 11 cvm resources under the account that need to collect 2 monitoring items CpuUsage and BaseCpuUsage, 4 requests are needed;

2. View the real call count through the task execution log:

The collector has statistics on the API calls made for each task execution result, which can be viewed in the log, for example:

[2023-04-24 19:02:02.359] [+1156ms] The [1]th account collection is completed, it took [1155 milliseconds], and [2] API calls were made during this period.
[2023-04-24 19:02:02.360] [+0ms] Detailed calls are as follows
[2023-04-24 19:02:02.360] [+0ms] -> monitor.tencentcloudapi.com/?Action=DescribeBaseMetrics: 1 time
[2023-04-24 19:02:02.565] [+0ms] -> monitor.tencentcloudapi.com/?Action=GetMonitorData: 1 time

!!! warning "Given that cloud monitoring API calls have a free quota, it is recommended that users configure monitoring items as needed to avoid additional costs caused by wildcards"

Precautions¶

Task Error Conditions and Solutions¶

1. HTTPClientError: An HTTP Client raised an unhandled exception: SoftTimeLimitExceeded()

Reason: The task execution time is too long timeout.

Solution:

• Appropriately increase the timeout setting for the task (e.g., @DFF.API('Execute Collection', timeout=120, fixed_crontab="* * * * *"), which means setting the task timeout to 120 seconds).

• [TencentCloudSDKException] code:InvalidParameterValue message:cannot find metricName=xxx configure

Reason: Tencent Cloud does not support the collection of this metric (this situation may occur where the metric exists in the Tencent Cloud documentation but is not actually supported)

Solution:

• It is recommended to refer to the Monitoring Metric Configuration Information in this document to configure valid metric names.

• [TencentCloudSDKException] code:InvalidParameterValue message: xxxxx does not belong to the developer ....

Reason: When collecting cloud monitoring data for a certain product under a certain account, the product has been released, causing the interface to throw an error, which can be ignored.

X. Appendix¶

Tencent Cloud Cloud Monitor¶

Please refer to the official Tencent Cloud documentation:

• Region List
• Tencent Cloud Namespaces
• Monitoring Metrics
• API Billing Explanation