Collector "Alibaba Cloud - CloudMonitor" Configuration Manual¶

Before reading this document, please read:

• Cloud Integration

1. Configuration Structure¶

The configuration structure of this collector is as follows:

2. Configuration Examples¶

Specifying Specific Metrics¶

Collect 2 metrics named CPUUtilization and concurrentConnections in ECS.

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['CPUUtilization', 'concurrentConnections'],
        },
    ],
}

Wildcard Matching Metrics¶

Metric names can be matched using the * wildcard.

In this example, the following metrics will be collected:

• Metrics named CPUUtilization
• Metrics starting with CPU
• Metrics ending with Connections
• Metrics containing Conn

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['CPUUtilization', 'CPU*', '*Connections', '*Conn*'],
        },
    ],
}

Excluding Specific Metrics¶

Adding the "NOT" tag at the beginning indicates that the following metrics will be excluded.

In this example, the following metrics will [not] be collected:

• Metrics named CPUUtilization
• Metrics starting with CPU
• Metrics ending with Connections
• Metrics containing Conn

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['NOT', 'CPUUtilization', 'CPU*', '*Connections', '*Conn*'],
        },
    ],
}

Multi-level Filtering for Specific Metrics¶

The same namespace can be specified multiple times, and the metrics will be filtered step by step from top to bottom.

In this example, the following filtering steps are applied to the metric names:

1. Select all metrics containing CPU in their names.
2. From the previous result, exclude metrics named CPUUtilization.

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['*CPU*'],
        },
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['NOT', 'CPUUtilization'],
        },
    ],
}

Configuring Filters (Optional)¶

This collector script supports user-defined filters, allowing users to filter target resources through object attributes. The filter function returns True or False.

• True: The target resource should be collected.
• False: The target resource should not be collected.

Supported object attributes for filtering:

When custom object collection is enabled, more object attributes will be supported for filtering. Please refer to the corresponding product's custom object collector documentation for details (under development...).

# Example: Enable filter to filter based on the InstanceId and RegionId attributes of the object. The configuration format is as follows:

def filter_instance(instance, namespace='acs_ecs_dashboard'):
    '''
    Collect metrics for instances with InstanceId i-xxxxxa, i-xxxxxb and RegionId cn-hangzhou
    '''
    instance_id = instance['tags'].get('InstanceId')
    region_id = instance['tags'].get('RegionId')
    if instance_id in ['i-xxxxxa', 'i-xxxxxb'] and region_id in ['cn-hangzhou']:
        return True
    return False

from integration_core__runner import Runner
import integration_alibabacloud_monitor__main as main

@DFF.API('AlibabaCloud-monitor ', timeout=3600, fixed_crontab="*/5 * * * *")
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Reporting Format¶

After data is successfully synchronized, you can view the data in the "Metrics" section of Guance.

Take the following collector configuration as an example:

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['CPUUtilization'],
        },
    ],
}

The reported data example is as follows:

{
  "measurement": "aliyun_acs_ecs_dashboard",
  "tags": {
    "instanceId": "i-xxxxx",
    "userId"    : "xxxxx"
  },
  "fields": {
    "CPUUtilization_Average": 1.23,
    "CPUUtilization_Maximum": 1.23,
    "CPUUtilization_Minimum": 1.23
  }
}

4. Interaction with Custom Object Collectors¶

When other custom object collectors (such as ECS, RDS) are running in the same DataFlux Func, this collector will automatically try to match the tags.instanceId and other fields with the tags.name field in the custom objects.

Since custom object information needs to be obtained first for interaction in CloudMonitor-type collectors, it is generally recommended to place CloudMonitor collectors at the end of the list, such as:

    # Create collectors
    collectors = [
        aliyun_ecs.DataCollector(account, common_aliyun_configs),
        aliyun_rds.DataCollector(account, common_aliyun_configs),
        aliyun_slb.DataCollector(account, common_aliyun_configs),
        aliyun_oss.DataCollector(account, common_aliyun_configs),
        aliyun_monitor.DataCollector(account, monitor_collector_configs), # CloudMonitor-type collectors are usually placed at the end.
    ]

When a successful match is made, the fields in the tags of the custom object, except for name, will be added to the tags of the monitoring data, thereby achieving effects such as filtering CloudMonitor metric data using instance names. The specific effect is as follows:

Assume the original data collected by CloudMonitor is as follows:

{
  "measurement": "aliyun_acs_ecs_dashboard",
  "tags": {
    "instanceId": "i-001",
    "{...}"     : "{...}"
  },
  "fields": {
    "{...}"     : "{...}"
  }
}

At the same time, the custom object data collected by the Alibaba Cloud ECS collector is as follows:

{
  "measurement": "aliyun_ecs",
  "tags": {
    "name"      : "i-001",
    "InstanceId": "i-001",
    "RegionId"  : "cn-hangzhou",
    "{...}"     : "{...}"
  },
  "fields": {
    "{...}"     : "{...}"
  }
}

Then, the final reported CloudMonitor data is as follows:

{
"measurement": "aliyun_acs_ecs_dashboard",
  "tags": {
    "instanceId": "i-001",
    "RegionId"  : "cn-hangzhou",
    "{...}"     : "{...}"
  },
  "fields": {
    "{...}"     : "{...}"
  }
}

5. CloudMonitor API Call Count Explanation¶

Alibaba Cloud CloudMonitor has a free quota limit for some API calls (currently: Query API free quota is 1 million times/month, exceeding part is charged at 0.12 yuan/10,000 times). The DescribeMetricLast used by this collector is also within the limit. The following explains the script set call count in detail:

1. The user has multiple resources and needs to collect multiple monitoring items. Will the free quota be exceeded?

This collector uses DescribeMetricLast (query the latest monitoring data of a specified monitoring item) to obtain multiple (up to 1000, if more, paging is used) resources' monitoring items in one request. Examples of request counts:

• There are 1000 ecs resources in the account that need to collect 1 monitoring item CPUUtilization, 1 request is needed;
• There are 1000 ecs resources in the account that need to collect 2 monitoring items CPUUtilization and DiskReadBPS, 2 requests are needed (one request for each monitoring item);
• There are 1001 ecs resources in the account that need to collect 1 monitoring item CPUUtilization, 2 requests are needed (resources exceed 1000, paging is used);
• There are 1001 ecs resources in the account that need to collect 2 monitoring items CPUUtilization and DiskReadBPS, 4 requests are needed;

2. View the real call count by checking the task execution log:

The collector counts the number of API calls for each task execution result, which can be viewed in the log. For example:

[2023-04-21 15:32:13.194] [+0ms] The [1]th account collection is completed, executed in [274 milliseconds], during which [2] API calls were made.
[2023-04-21 15:32:13.194] [+0ms] Detailed calls are as follows:
[2023-04-21 15:32:13.194] [+0ms] -> metrics.aliyuncs.com/?Action=DescribeMetricMetaList: 1 call
[2023-04-21 15:32:13.194] [+0ms] -> metrics.aliyuncs.com/?Action=DescribeMetricLast: 1 call

Notes¶

Common Errors and Solutions¶

1. The number of collected instances does not match the actual number of instances

   Reason: The instance is in a shutdown state.

   Solution:

   • Start the instance.

X. Appendix¶

Please refer to the official Alibaba Cloud documentation:

• CloudMonitor / Appendix 1 Cloud Service Monitoring Items
• CloudMonitor Metric Query
• Basic CloudMonitor / Free Quota
• Basic CloudMonitor / Pay-as-you-go