Skip to content

Threshold Detection


Threshold detection is used to monitor anomalies in metrics, logs, links and other data. Users can set threshold ranges and receive alerts and notifications when the metrics reach the thresholds. It supports detecting multiple metric data simultaneously and setting alert levels.

Use Case

Support detection of metrics, logs, infrastructure, custom, events, APM, RUM, security check, and network data anomalies. For example, you can monitor issues such as high host memory usage.

Setup

Step 1: Detection Configuration

Detection Frequency: The execution frequency of detection rules, including 1m/5m/15/30m/1h/6h (5m is selected by default).

Detection Interval: The time range of detection index query when each task is executed. The optional detection interval will be different due to the influence of detection frequency.

Detection Frequency Detection Interval (Drop-down Option)
1m 1m/5m/15m/30m/1h/3h
5m 5m/15m/30m/1h/3h
15m 15m/30m/1h/3h/6h
30m 30m/1h/3h/6h
1h 1h/3h/6h/12h/24h
6h 6h/12h/24h

Detection Metrics: Set the detection data.

Field Description
Data Type The current detected data type supports detecting data types such as Metrics, Logs, Infrastructure, Custom, Events, APM, RUM, Security Check and Network.
Measurements The measurement where the current detection metric is located. (take the "metric" data type as an example.)
Metrics The metric for which the current test is aimed. (take the "metric" data type as an example.)
Aggregation Algorithm Contains Avg by (average), Min by (minimum), Max by (maximum), Sum by (sum), Last (Last), First by (first), Count by (data points), Count_distinct by (non-duplicate data points), p50 (median), p75 (75%), p90 (90%), p99 (99%).
Detection Dimension The corresponding string type (keyword) fields in the configuration data can be selected as detection dimensions. At present, the detection dimensions support selecting up to three fields. Through the combination of fields of multiple detection dimensions, a certain detection object can be determined, and the guance will judge whether the statistical index corresponding to a detection object meets the threshold of trigger conditions, and if it meets the conditions, an event will be generated. (For example, if the detection dimensions host and host_ip are selected, the detection object can be {host: host1, host_ip: 127.0.0.1}) When the detection object is "log", the default detection dimensions are status, host, service, source and filename.
Filtering Metirc-based labels filter the data of detection metrics, limit the scope of detection data, and support adding one or more labels for screening. Non-metric data support fuzzy matching and fuzzy mismatching screening conditions.
Alias Custom metrics name
Query Mode Support simple query and expression query, refer to query.

Trigger Condition: Set the trigger condition of alert level; You can configure any of the following trigger conditions: Critical, Error, Warning, No Data, or Information.

Configure the trigger condition and severity. When the query result is multiple values, an event will be generated if any value meets the trigger condition.

See Event Levels.

I. Alert levels: Critical (red), Important (orange), Warning (yellow): Based on the configured conditions using operators.

For likeTrue and likeFalse details, see Truth Table Description.

II. Alert levels: OK (green), Information (blue): Based on the configured number of detections, as explained below:

  • One test is performed for each test task, if "test frequency = 5 minutes", then one test = 5 minutes
  • You can customize the number of tests, such as "Test frequency = 5 minutes", then 3 tests = 15 minutes
Level Description
OK After the detection rule takes effect, if the result of an urgent, important, or warning abnormal event returns to normal within the configured number of custom detections, a recovery alert event is generated.
⚠ Recovery alert events are not affected by Mute Alerting. If no detection count is set for recovery alert events, the alert event will not recover and will always appear in the Events > Unrecovered Events List.
Information Events are generated even for normal detection results.

III. Alert level: No Data (gray): The no data state supports three configuration strategies: Trigger No-Data Event, Trigger Recovery Event, and Untrigger Event.

  • The Trigger No-Data Event, Trigger Recovery Event and Untrigger Event are three configuration strategies supported for the no data staus.
  • When configuring the data monitor for basic objects and custom objects, the Trigger Recovery Event strategy is selected by default.

Step 2: Event Notification

Event Title: Set the event name of the alert trigger condition; support the use of preset template variables.

Note: In the latest version, the Monitor Name will be automatically generated based on the Event Title input. In older monitors, there may be inconsistencies between the Monitor Name and the Event Title. To enjoy a better user experience, please synchronize to the latest version as soon as possible. One-click replacement with event title is supported.

Event Content: The content of the event notification sent when the trigger conditions are met. Support inputting text in Markdown format, previewing effects, the use of preset associated links and the use of preset template variables.

Note: Different alert notification objects support different Markdown syntax. For example, WeCom does not support unordered lists.

  • Relevant Links: Automatically generate jump links based on detection metrics, supporting adjustment of filtering conditions and time range after inserting links. Usually, it is a fixed link address prefix, which includes the current domain name and workspace ID; you can also customize jump links.

Among them, if you need to insert a link to the dashboard, based on the above logic, you also need to supplement the ID and name of the dashboard, supporting adjustment of view variables and time range.

No Data Notification Configuration: Support customizing the content of the no data notification. If not configured, the official default notification template will be automatically used.

Synchronously create Issue: If abnormal events occur under this monitor, an issue for anomaly tracking will be created synchronously and delivered to the channel for anomaly tracking. You can go to Incident > Your selected Channel to view it.

Step 3: Alert Configuration

Alert Strategy: After the monitoring meets the trigger conditions, immediately send an alert message to the specified notification targets. The Alert Strategy includes the event level that needs to be notified, the notification targets and the mute alerting period.

Step 4: Association

Associate Dashboard: Every monitor supports associating with a dashboard for quick navigation and viewing.

Example

Monitor host memory Swap usage is too high.

Feedback

Is this page helpful? ×