Skip to content

APM Metrics Detection


Positioning of This Document

This document is the second step in the configuration process of detection rules. After completing the configuration, please return to the main document to continue with the third step: Event Notification.

Data Range: Traces (T), used to monitor key APM Metrics data in the workspace. The system counts the number of traces that meet the conditions within a specified time period, and triggers an anomaly event when the custom threshold is exceeded.

Detection Configuration

Detection Frequency

Set the time period for executing detections.

  • Preset options: 1 minute, 5 minutes, 10 minutes, 15 minutes, 30 minutes, 1 hour

  • Crontab mode: Click "Switch to Crontab mode" to configure custom periods, supporting scheduled task execution based on seconds, minutes, hours, days, months, weeks, etc.

Detection Interval

Set the data time range for each detection query (❗️The detection interval should be greater than or equal to the detection frequency, and must match the actual data reporting cycle to avoid missed detections or false alarms).

Detection Frequency Detection Interval (Dropdown options)
30s 1m/5m/15m/30m/1h/3h
1m 1m/5m/15m/30m/1h/3h
5m 5m/15m/30m/1h/3h
15m 15m/30m/1h/3h/6h
30m 30m/1h/3h/6h
1h 1h/3h/6h/12h/24h
6h 6h/12h/24h
12h 12h/24h
24h 24h
  • Custom format: Manually enter the detection interval, e.g., 20m (last 20 minutes), 2h (last 2 hours), 1d (last 1 day).

Detection Metrics

Set the metrics for detection data, supporting two detection modes:

  • Service Metrics

  • Trace Statistics

Note

Avoid selecting high-cardinality fields as detection dimensions. Improper configuration with overly loose trigger conditions may cause frequent alerts. The maximum number of records returned by a single query is 100,000.

Service Metrics

Monitor APM services in the current workspace.

Configuration Item Description
Service Select an APM service in the current workspace, supports selecting "All" or a specific service
Metric Specific detection metrics, including: Request Count, Error Request Count, Request Error Rate, Average Requests per Second, Average Response Time, P50 Response Time, P75 Response Time, P90 Response Time, P99 Response Time
Filter Conditions Filter detection data based on metric tags to limit the detection scope. Supports adding one or more tag filters, as well as fuzzy match and fuzzy not match conditions
Detection Dimensions Any string type (keyword) field in the data can be selected as a detection dimension. Currently supports up to three fields. A specific detection object can be determined by combining multiple detection dimension fields (e.g., {service: svc1, host: host1})
Additional Information Select additional fields to display in the event details

Trace Statistics

Count the number of spans that meet the conditions within a specified time period, and trigger an anomaly event when the custom threshold is exceeded. This can be used for notifications of service trace anomaly errors.

Configuration Item Description
Source Select the trace data source (service) to be counted, supports keyword filtering
Filter Conditions Filter trace spans by tags to limit the detection data range. Supports adding one or more tag filter conditions
Aggregation Algorithm Default is "*", with the corresponding aggregation function count (counts the number of spans). If another field is selected, the aggregation function automatically changes to count distinct (counts the number of data points where the keyword appears, i.e., deduplication count)
Detection Dimensions Any string type (keyword) field in the data can be selected as a detection dimension. Currently supports up to three fields. A specific detection object can be determined by combining multiple detection dimension fields

Trigger Conditions

Configure trigger conditions and severity levels. When the query result has multiple values, an event is generated if any value meets the trigger condition.

Supports configuring four levels of thresholds: Critical, Error, Warning, Info, and OK recovery conditions.

Level Configuration Description
Critical When Result >= [value] Highest level alert, requires immediate action
Error When Result >= [value] High level alert, requires priority action
Warning When Result >= [value] Medium level alert, requires attention
Info When Result >= [value] Low level alert, requires awareness
OK No events generated for [N] consecutive detections If the detection metric triggers a "Critical", "Error", "Warning", or "Info" anomaly event, and the next consecutive N detections are normal, an "OK" event is generated. Used to determine whether the anomaly event has returned to normal. It is recommended to configure.

For more details, refer to Event Level Description.

Advanced Options

Continuous Trigger Judgment

When enabled, events are generated only when the trigger condition is continuously met, avoiding false alarms due to transient fluctuations (❗️The maximum configuration limit is 10 times).

High Volume Alert Protection

Enabled by default.

When the number of alerts generated by a single detection exceeds the preset threshold, the system automatically switches to a summary strategy by status: Instead of processing each alert object individually, it generates a small number of summary alerts based on event status and pushes them.

This ensures timely notifications while significantly reducing alert noise and avoiding timeout risks due to processing too many alerts.

When this switch is enabled, subsequent event details generated by the monitor after detecting anomalies will not display historical records or related events.

Recovery Conditions

Configure recovery conditions and severity levels. When the query result has multiple values, a recovery event is generated if any value meets the trigger condition.

Set independent recovery thresholds for different levels to achieve degraded recovery. For example, a critical alert recovers when the value drops below 70, while a warning alert recovers when below 80.

Default Recovery Logic

When the graded recovery condition configuration is not enabled, the default behavior is to automatically recover when the detection result no longer meets the trigger condition.

Data Gap

Handling strategy when the detection metric query result is empty within the detection interval:

Option Description
Do not trigger events (default) No alert is generated when there is no data, suitable for scenarios where data loss is acceptable
Treat query result as 0 Treat empty data as a value of 0 for threshold judgment
Trigger data gap event Treat no data as an anomaly and trigger a data gap event
Trigger critical event Trigger a critical-level event when no data is present
Trigger error event Trigger an error-level event when no data is present
Trigger warning event Trigger a warning-level event when no data is present
Trigger info event Trigger an info-level event when no data is present
Trigger recovery event Trigger a recovery event when no data is present

When trigger conditions, data gap, and information generation are configured simultaneously, the priority is: Data Gap > Trigger Conditions > Information Event Generation.

That is: first determine if there is a data gap, then determine if the threshold is triggered, and finally determine whether to generate an information event.

Information Generation

When this option is enabled, you need to configure Information Generation Conditions. The system writes an "Info" event only when the detection result does not trigger any of the "Critical", "Error", "Warning", or "Info" thresholds and meets the information generation conditions.

It is suitable for scenarios where normal state changes or low-priority information needs to be recorded.

Subsequent Configuration

After completing the above detection configuration, continue to configure:

  1. Event Notification: Define event title, content, notification members, data gap handling, and related incidents;
  2. Alert Configuration: Select alert strategies, set notification targets, and mute periods;
  3. Association: Link dashboards for quick navigation to view data;
  4. Permissions: Set operation permissions to control who can edit/delete this monitor.

Feedback

Is this page helpful? ×