Collector "GoogleCloud-Cloud Monitoring" Configuration Manual¶

Before reading this document, please read:

• Cloud Integration
• GoogleCloud (GCP) Client Authorization Configuration

1. Configuration Structure¶

The configuration structure of this collector is as follows:

1.1 Metric Types¶

Before understanding the collector configuration structure, let's first understand the metric types of GCP. The complete metric type consists of {namespace}/{metric}, e.g.:

Compute Engine CPU utilization: compute.googleapis.com/instance/cpu/utilization

• namespace: compute.googleapis.com

• metric: instance/cpu/utilization

Source: Google Cloud Metrics

1.2 Resource Types¶

Cloud monitoring metrics can be described as Google Cloud x service x resource types x metrics, where x resource types are the resource types (resource_type).

The collector configuration does not need to specify the resource type. Note that the measurement reported to Guance is mostly named in the format of resource type gcp_{resource_type} (some resource types may also be renamed).

2. Configuration Examples¶

Specifying Specific Metrics¶

Collect two metrics named instance/cpu/utilization and guest/disk/io_time in Compute Engine.

collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : [
              'instance/cpu/utilization',
              'guest/disk/bytes_used'
            ]
        }
    ]
}

Wildcard Matching Metrics¶

Metric names can use * wildcard for matching.

In this example, the following metrics will be collected:

• Metrics named instance/cpu/utilization
• Metrics whose names start with instance
• Metrics whose names end with cpu/utilization
• Metrics whose names contain cpu

collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : [
              'instance/cpu/utilization',
              'instance*',
              '*cpu/utilization',
              '*cpu*'
            ]
        }
    ]
}

Excluding Specific Metrics¶

Adding "NOT" at the beginning indicates excluding the following metrics.

In this example, the following metrics will [not] be collected:

• Metrics named instance/cpu/utilization
• Metrics whose names start with instance
• Metrics whose names end with cpu/utilization
• Metrics whose names contain cpu

collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : [
              'NOT',
              'instance/cpu/utilization',
              'instance*',
              '*cpu/utilization',
              '*cpu*'
            ]
        }
    ]
}

Multiple Filters to Specify Required Metrics¶

The same namespace can be specified multiple times, filtering metric names from top to bottom.

In this example, the following filtering steps are performed on the metric names:

1. Select all metrics whose names contain cpu
2. In the results of the previous step, exclude metrics named cpu/utilization

collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : ['*cpu*'],
        },
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : ['NOT', '*cpu/utilization'],
        }
    ]
}

Configuring Filters (Optional)¶

This collector script supports custom filters, allowing users to filter the data to be reported based on the tags of the reported data. The filter function returns True | False

• True: The target resource needs to be collected.

• False: The target resource does not need to be collected

# Example: Enable the filter and filter based on the InstanceId and RegionId properties of the object. The configuration format is as follows:

def filter_instance(instance, namespace='compute.googleapis.com'):
    '''
    Collect metrics with instance_id i-xxxxxa, i-xxxxxb
    '''
    # print(instance) # Consider printing the instance for the first time to view the data structure to be filtered.
    instance_id = instance['tags'].get('instance_id')
    if instance_id in ['i-xxxxxa', 'i-xxxxxb']:
        return True
    return False

from integration_core__runner import Runner
import integration_alibabacloud_monitor__main as main

@DFF.API('GoogleCloud-Monitor Collection', timeout=300, fixed_crontab="* * * * *")
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Reporting Format¶

After the data is successfully synchronized, you can view the data in the "Metrics" section of Guance.

Take the following collector configuration as an example:

collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : ['instance/cpu/utilization'],
        }
    ]
}

The reported data example is as follows:

{
  "measurement": "gcp_gce_instance",
  "tags": {
    "instance_id": "i-xxxxx",
    "userId"    : "xxxxx"
  },
  "fields": {
    "instance_cpu_utilization": 1.23
  }
}

• measurement: The definition of the field refers to the "1.2 Resource Types" section.
• tags: The dimensions corresponding to the metrics, as well as additional tags.
• fields: The metrics in the fields are the metrics configured in metrics (converted to lowercase letters and concatenated with underscores before reporting).

Since the metrics are all real-time data, the latest values, the timestamp is omitted (the timestamp of the reporting time is used by default).

4. Cloud Monitoring API Call Count Explanation¶

Google Cloud Monitoring has a free quota limit for some API calls. Before October 1, 2025: The free quota for query APIs is 1 million calls per billing account per month, and the excess is charged at $0.01 per thousand calls. The /v3/project/{project_id}/timeSeries used by this collector is also within the limit:

Viewing the Actual Call Count in Task Execution Logs¶

The collector has statistics on the number of API calls for each execution result of the task, which can be viewed in the logs, e.g.:

[2023-04-21 15:32:13.194] [+0ms] The [1] account collection is completed, executed [274 ms], and called API [2 times]
[2023-04-21 15:32:13.194] [+0ms] The detailed calls are as follows:
[2023-04-21 15:32:13.194] [+0ms] -> monitoring.googleapis.com/v3/projects/{project_id}/metricDescriptors: 1 time
[2023-04-21 15:32:13.194] [+0ms] -> monitoring.googleapis.com/v3/project/{project_id}/timeSeries: 1 time

!!! warning "Given that cloud monitoring API calls have a free quota, it is recommended that users configure monitoring items as needed to avoid additional costs caused by wildcards."

X. Appendix¶

Refer to the official documentation:

• Google Cloud Observability Pricing
• Google Cloud Metrics
• Get Google Cloud Metrics API Method:projects.timeSeries.list