Collector "AWS-CloudWatch" Configuration Manual¶

Before reading this document, please read:

• Cloud Integration

1. Configuration Structure¶

The configuration structure of this collector is as follows:

2. Configuration Examples¶

Specifying Specific Metrics¶

Collect the CPUCreditBalance and MetadataNoToken metrics from the AWS/EC2 namespace with the InstanceId dimension.

aws_cloudwatch_configs = {
    'regions': ['ap-southeast-1'],
    'targets': [
        {
            'namespace' : 'AWS/EC2',
            'dimensions': ['InstanceId'],
            'metrics'   : ['CPUCreditBalance', 'MetadataNoToken'],
        }
    ],
}

Wildcard Matching for Metrics¶

Metric names can use the * wildcard for matching.

In this example, the following metrics will be collected:

• Metrics with the InstanceId dimension

• Metrics named CPUCreditBalance

• Metrics starting with CPU

• Metrics ending with Balance

• Metrics containing Credit

aws_cloudwatch_configs = {
    'regions': ['ap-southeast-1'],
    'targets': [
        {
            'namespace' : 'AWS/EC2',
            'dimensions': ['InstanceId'],
            'metrics'   : ['CPUCreditBalance', 'CPU*', '*Balance', '*Credit*'],
        }
    ],
}

Excluding Specific Metrics¶

Adding the "NOT" marker at the beginning indicates that the following metrics should be excluded.

In this example, the following metrics will [not] be collected:

• Metrics with the InstanceId dimension

• Metrics named CPUCreditBalance

• Metrics starting with CPU

• Metrics ending with Balance

• Metrics containing Credit

aws_cloudwatch_configs = {
    'regions': ['ap-southeast-1'],
    'targets': [
        {
            'namespace' : 'AWS/EC2',
            'dimensions': ['InstanceId'],
            'metrics'   : ['NOT', 'CPUCreditBalance', 'CPU*', '*Balance', '*Credit*'],
        }
    ],
}

Multi-step Filtering for Specific Metrics¶

The same namespace can be specified multiple times, with metrics filtered step-by-step from top to bottom.

In this example, the metric names are filtered as follows:

1. All metrics containing CPU in the InstanceId dimension

2. From the previous result, exclude the metric named CPUUtilization

aws_cloudwatch_configs = {
    'regions': ['ap-southeast-1'],
    'targets': [
        {
            'namespace' : 'AWS/EC2',
            'dimensions': ['InstanceId'],
            'metrics'   : ['*CPU*'],
        },
        {
            'namespace' : 'AWS/EC2',
            'dimensions': ['InstanceId'],
            'metrics'   : ['NOT', 'CPUCreditBalance'],
        },
    ],
}

Configuring Filters (Optional)¶

This collector script supports custom filters, allowing users to filter target resources based on object attributes. The filter function returns True or False.

• True: The target resource should be collected.
• False: The target resource should not be collected.

# Example: Enabling a filter to filter based on RegionId and InstanceId attributes. Configuration format is as follows:

def filter_instance(instance, namespace='AWS/EC2'):
    '''
    Collect metrics for InstanceId i-0d7620xxxxxxxa, i-0d7620xxxxxxxb and RegionId ap-southeast-1
    '''
    region_id = instance['tags'].get('RegionId')
    instance_id = instance['tags'].get('InstanceId')

    if instance_id in ['i-0d7620xxxxxxxa', 'i-0d7620xxxxxxxb'] and region_id in ['ap-southeast-1']:
        return True
    return False

from integration_core__runner import Runner
import integration_aws_cloudwatch__main as main

@DFF.API('AWS-CloudWatch Collection', timeout=3600, fixed_crontab='*/15 * * * *')
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Collection Details¶

Cloud Product Configuration Information¶

4. Data Reporting Format¶

After data is successfully synchronized, you can view the data in the "Metrics" section of Guance.

Take the following collector configuration as an example:

aws_cloudwatch_configs = {
    'regions': ['ap-southeast-1'],
    'targets': [
        {
            'namespace' : 'AWS/EC2',
            'dimensions': ['InstanceId'],
            'metrics'   : ['CPUCreditBalance'],
        },
    ],
}

The reported data example is as follows:

{
  "measurement": "aws_AWS/EC2",
  "tags": {
    "Dimensions": "InstanceId",
    "InstanceId": "i-xxx"
  },
  "fields": {
    "CPUCreditBalance_Average"    : 576.0,
    "CPUCreditBalance_Maximum"    : 576.0,
    "CPUCreditBalance_Minimum"    : 576.0,
    "CPUCreditBalance_SampleCount": 1.0,
    "CPUCreditBalance_Sum"        : 576.0
  }
}

5. Interaction with Custom Object Collectors¶

When other custom object collectors (e.g., EC2) are running in the same DataFlux Func, this collector will collect based on the dimensions specified in Data Collection Details. For example, fields like tags.InstanceId will attempt to match the tags.name field in the custom object.

Instance-level metrics are automatically supplemented. For an explanation of instance dimensions, refer to Data Collection Details.

Since custom object information needs to be obtained first for the CloudWatch collector to interact, it is generally recommended to place the CloudWatch collector at the end of the list, e.g.:

# Create collectors
collectors = [
    aws_ec2.DataCollector(account, common_aws_configs),
    aws_cloudwatch.DataCollector(account, aws_cloudwatch_configs) # CloudWatch collector is usually placed last
]

When a match is successful, additional fields from the custom object tags will be added to the CloudWatch data tags, enabling effects like filtering CloudWatch metrics by instance name. The specific effect is as follows:

Assume the original data collected by CloudWatch is as follows:

{
  "measurement": "aws_AWS/EC2",
  "tags": {
    "Dimensions": "InstanceId",
    "InstanceId": "i-xxx"
  },
  "fields": { "Content omitted" }
},

Meanwhile, the custom object data collected by the AWS EC2 collector is as follows:

{
  "measurement": "aws_ec2",
  "tags": {
    "InstanceType"   : "c6g.xxx",
    "PlatformDetails": "xxx",
    "{key}"     : "{value}"
  },
  "fields": { "Content omitted" }
}

Then, the final CloudWatch data reported will be as follows:

{
  "measurement": "aws_AWS/EC2",
  "tags": {
    "InstanceId"          : "i-xxx",   // Original CloudWatch field
    "Dimensions"      : "InstanceId", // Dimension information field
    "InstanceType"    : "c6g.xxx", // Field from custom object EC2
    "PlatformDetails" : "xxx",     // Field from custom object EC2
    "{key}"      : "{value}"
  },
  "fields": { "Content omitted" }
},

6. Cloud Monitoring Call Costs¶

AWS CloudWatch charges for API calls to retrieve metrics. Refer to Amazon (Global) CloudWatch Pricing, Amazon (China) CloudWatch Pricing. The GetMetricData and ListMetrics APIs used by this collector are within the scope of charges.

Note that GetMetricData is not charged based on the number of API requests but on the number of metric requests. Therefore, the number of API requests is not a reference. For definitions related to metric requests, please refer to the AWS pricing documentation.

7. IAM Policy Permissions¶

This collector requires the following operation permissions:

cloudwatch:GetMetricData

cloudwatch:ListMetrics

Notes¶

Error Scenarios and Solutions for Triggered Tasks¶

1. HTTPClientError: An HTTP Client raised an unhandled exception: SoftTimeLimitExceeded()

Reason: The task execution time is too long, causing a timeout.

Solution:

• Increase the timeout setting for the task appropriately (e.g., @DFF.API('Execute Collection', timeout=120, fixed_crontab="* * * * *"), which sets the timeout to 120 seconds).

• About CloudWatch Agent Collecting Metrics from Amazon EC2 Instances and On-premises Servers

Reason: The agent collects metrics from Amazon EC2 instances and on-premises servers.

Solution: - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html

X. Appendix¶

AWS CloudWatch¶

Refer to AWS official documentation:

• Regions and Availability Zones
• S3 Using Amazon CloudWatch Monitoring Metrics
• AWS Services in CloudWatch Namespaces
• CloudWatch Viewing Available Metrics and Dimensions
• Amazon ElastiCache for Memcached Host-Level Metric Monitoring
• Amazon ElastiCache for Redis Host-Level Metric Monitoring
• Amazon (Global) CloudWatch Pricing
• Amazon (China) CloudWatch Pricing