APM Metrics Detection¶
Positioning of This Document
This document is the second step in the configuration process of detection rules. After completing the configuration, please return to the main document to continue with the third step: Event Notification.
Data Range: Traces (T), used to monitor key APM Metrics data in the workspace. The system counts the number of traces that meet the conditions within a specified time period, and triggers an anomaly event when the custom threshold is exceeded.
Detection Configuration¶
Detection Frequency¶
Set the time period for executing detections.
-
Preset options: 1 minute, 5 minutes, 10 minutes, 15 minutes, 30 minutes, 1 hour
-
Crontab mode: Click "Switch to Crontab mode" to configure custom periods, supporting scheduled task execution based on seconds, minutes, hours, days, months, weeks, etc.
Detection Interval¶
Set the data time range for each detection query (❗️The detection interval should be greater than or equal to the detection frequency, and must match the actual data reporting cycle to avoid missed detections or false alarms).
| Detection Frequency | Detection Interval (Dropdown options) |
|---|---|
| 30s | 1m/5m/15m/30m/1h/3h |
| 1m | 1m/5m/15m/30m/1h/3h |
| 5m | 5m/15m/30m/1h/3h |
| 15m | 15m/30m/1h/3h/6h |
| 30m | 30m/1h/3h/6h |
| 1h | 1h/3h/6h/12h/24h |
| 6h | 6h/12h/24h |
| 12h | 12h/24h |
| 24h | 24h |
- Custom format: Manually enter the detection interval, e.g.,
20m(last 20 minutes),2h(last 2 hours),1d(last 1 day).
Detection Metrics¶
Set the metrics for detection data, supporting two detection modes:
-
Service Metrics
-
Trace Statistics
Note
Avoid selecting high-cardinality fields as detection dimensions. Improper configuration with overly loose trigger conditions may cause frequent alerts. The maximum number of records returned by a single query is 100,000.
Service Metrics¶
Monitor APM services in the current workspace.
| Configuration Item | Description |
|---|---|
| Service | Select an APM service in the current workspace, supports selecting "All" or a specific service |
| Metric | Specific detection metrics, including: Request Count, Error Request Count, Request Error Rate, Average Requests per Second, Average Response Time, P50 Response Time, P75 Response Time, P90 Response Time, P99 Response Time |
| Filter Conditions | Filter detection data based on metric tags to limit the detection scope. Supports adding one or more tag filters, as well as fuzzy match and fuzzy not match conditions |
| Detection Dimensions | Any string type (keyword) field in the data can be selected as a detection dimension. Currently supports up to three fields. A specific detection object can be determined by combining multiple detection dimension fields (e.g., {service: svc1, host: host1}) |
| Additional Information | Select additional fields to display in the event details |
Trace Statistics¶
Count the number of spans that meet the conditions within a specified time period, and trigger an anomaly event when the custom threshold is exceeded. This can be used for notifications of service trace anomaly errors.
| Configuration Item | Description |
|---|---|
| Source | Select the trace data source (service) to be counted, supports keyword filtering |
| Filter Conditions | Filter trace spans by tags to limit the detection data range. Supports adding one or more tag filter conditions |
| Aggregation Algorithm | Default is "*", with the corresponding aggregation function count (counts the number of spans). If another field is selected, the aggregation function automatically changes to count distinct (counts the number of data points where the keyword appears, i.e., deduplication count) |
| Detection Dimensions | Any string type (keyword) field in the data can be selected as a detection dimension. Currently supports up to three fields. A specific detection object can be determined by combining multiple detection dimension fields |
Trigger Conditions¶
Configure trigger conditions and severity levels. When the query result has multiple values, an event is generated if any value meets the trigger condition.
Supports configuring four levels of thresholds: Critical, Error, Warning, Info, and OK recovery conditions.
| Level | Configuration | Description |
|---|---|---|
| Critical | When Result >= [value] |
Highest level alert, requires immediate action |
| Error | When Result >= [value] |
High level alert, requires priority action |
| Warning | When Result >= [value] |
Medium level alert, requires attention |
| Info | When Result >= [value] |
Low level alert, requires awareness |
| OK | No events generated for [N] consecutive detections |
If the detection metric triggers a "Critical", "Error", "Warning", or "Info" anomaly event, and the next consecutive N detections are normal, an "OK" event is generated. Used to determine whether the anomaly event has returned to normal. It is recommended to configure. |
For more details, refer to Event Level Description.
Advanced Options¶
Continuous Trigger Judgment¶
When enabled, events are generated only when the trigger condition is continuously met, avoiding false alarms due to transient fluctuations (❗️The maximum configuration limit is 10 times).
High Volume Alert Protection¶
Enabled by default.
When the number of alerts generated by a single detection exceeds the preset threshold, the system automatically switches to a summary strategy by status: Instead of processing each alert object individually, it generates a small number of summary alerts based on event status and pushes them.
This ensures timely notifications while significantly reducing alert noise and avoiding timeout risks due to processing too many alerts.
When this switch is enabled, subsequent event details generated by the monitor after detecting anomalies will not display historical records or related events.
Recovery Conditions¶
Configure recovery conditions and severity levels. When the query result has multiple values, a recovery event is generated if any value meets the trigger condition.
Set independent recovery thresholds for different levels to achieve degraded recovery. For example, a critical alert recovers when the value drops below 70, while a warning alert recovers when below 80.
Default Recovery Logic
When the graded recovery condition configuration is not enabled, the default behavior is to automatically recover when the detection result no longer meets the trigger condition.
Data Gap¶
Handling strategy when the detection metric query result is empty within the detection interval:
| Option | Description |
|---|---|
| Do not trigger events (default) | No alert is generated when there is no data, suitable for scenarios where data loss is acceptable |
| Treat query result as 0 | Treat empty data as a value of 0 for threshold judgment |
| Trigger data gap event | Treat no data as an anomaly and trigger a data gap event |
| Trigger critical event | Trigger a critical-level event when no data is present |
| Trigger error event | Trigger an error-level event when no data is present |
| Trigger warning event | Trigger a warning-level event when no data is present |
| Trigger info event | Trigger an info-level event when no data is present |
| Trigger recovery event | Trigger a recovery event when no data is present |
When trigger conditions, data gap, and information generation are configured simultaneously, the priority is: Data Gap > Trigger Conditions > Information Event Generation.
That is: first determine if there is a data gap, then determine if the threshold is triggered, and finally determine whether to generate an information event.
Information Generation¶
When this option is enabled, you need to configure Information Generation Conditions. The system writes an "Info" event only when the detection result does not trigger any of the "Critical", "Error", "Warning", or "Info" thresholds and meets the information generation conditions.
It is suitable for scenarios where normal state changes or low-priority information needs to be recorded.
Subsequent Configuration¶
After completing the above detection configuration, continue to configure:
- Event Notification: Define event title, content, notification members, data gap handling, and related incidents;
- Alert Configuration: Select alert strategies, set notification targets, and mute periods;
- Association: Link dashboards for quick navigation to view data;
- Permissions: Set operation permissions to control who can edit/delete this monitor.