Detection Rules¶

The system provides rich built-in detection rules that precisely match the monitoring needs of various data types, effectively preventing false alerts and missed alerts.

Configuration Process¶

To create a monitor, complete the configuration in the following order:

1. Select Rule Type: Determines the data scope and algorithm logic of the detection configuration;
2. Detection Configuration: Different types correspond to different configuration items;
3. Event Notification: Defines the event title, content, related information, and data gap handling;
4. Alert Configuration: Sets the notification policy and mute period;
5. Link to Dashboard: Configures dashboard association;
6. Permissions: Sets operation permissions.

Except for Detection Configuration, which varies by rule type, the configuration logic for Event Notification, Alert Configuration, Link to Dashboard, and Permissions is consistent across all rule types.

Rule Types¶

You can select the appropriate detection logic based on your monitoring targets:

After selecting a rule type, the available parameters in the Detection Configuration module change accordingly. Event Notification and subsequent configurations remain consistent across all types.

Detection Configuration¶

Depending on the rule type, you can configure the corresponding detection frequency, detection range, and detection metrics.

Detection configurations vary significantly across rule types. Refer to the detailed configuration documentation for the corresponding type.

Data Waiting Window¶

The data waiting window applies to all monitor types. After the scheduled execution time is reached, the monitor waits for the specified duration before performing the monitoring evaluation, reducing the chance of false alerts caused by data latency.

• Default: No wait;
• Optional wait durations are 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, 15 minutes, and 30 minutes;
• The wait duration must be shorter than the interval between two adjacent scheduled execution times. Otherwise, the monitor cannot be saved and the message "The data waiting window must be shorter than the monitoring execution interval" is displayed. Crontab mode is validated in the same way based on adjacent scheduled execution times.

For example, if a monitor is scheduled to run at 10:00 and the data waiting window is set to 5 minutes, the task starts around 10:05. The wait only delays the execution time and does not change the query time range. If the detection range is the last 1 hour, data from 09:00 ~ 10:00 is still queried.

For detailed execution logic, see Monitor Internals.

Event Notification¶

Defines the event title, content, notified members, and related handling when a monitor is triggered.

Event Title¶

Defines the event name for the alert trigger condition. Predefined template variables can be used.

Event Content¶

Write the event notification content. When the trigger condition is met, the system sends this content externally. It generally includes the following:

• Markdown-formatted body;
• Insertable related links and template variables;
• Related logs or error information can be added via Advanced Settings;
• Target notification members who receive the event content.

Event content supports syntax validation and template preview, allowing you to check the rendering of variables and conditional branches before saving.

Related Links¶

Click + Link. The system automatically generates jump links based on the current detection metric. The link address includes the current domain, workspace ID, detection time range （{{df_check_range_start}} ~ {{df_check_range_end}}）, and dynamic filter conditions.

Link format examples:

Log Explorer: [View Related Logs](<{{STUDIO_CONSOLE_BASE_URL}}/logIndi/log/all?time={{df_check_range_start}},{{df_check_range_end}}&w={{df_workspace_uuid}}>)

Trace Explorer: [View Related Traces](<{{STUDIO_CONSOLE_BASE_URL}}/tracing/link/all?time={{df_check_range_start}},{{df_check_range_end}}&w={{df_workspace_uuid}}>)

Template Variables¶

Click + Variable to insert predefined template variables. When an event is triggered, variables are dynamically replaced with actual values:

Click to view all supported template variables.

Advanced Settings¶

With Advanced Settings, you can use DQL to embed related data context into events.

1. Add Related Logs

Click to auto-generate the template:

{% set dql_data = DQL("L::RE(`.*`):(`message`) { `index` = 'default' } LIMIT 1") %}
{{ dql_data.message | limit_lines(10) }}

Configuration notes:

• Replace {index= 'default' } with the actual index name
• RE(``.*``) supports regex matching, e.g., RE(``error\|exception``)
• limit_lines(10) limits the number of output lines to avoid overly long notifications

2. Add Related Error Stack

Click to auto-generate the template:

{% set dql_data = DQL("T::re(`.*`):(`error_message`,`error_stack`){ (`source` NOT IN ['service_map', 'tracing_stat', 'service_list_1m', 'service_list_1d', 'service_list_1h', 'profile']) AND (`error_stack` = exists()) } LIMIT 1") %}
{{ dql_data.error_message | limit_lines(10) }}
{{ dql_data.error_stack | limit_lines(10) }}

Configuration notes:

• source NOT IN [...] excludes statistical aggregated data and keeps only raw traces
• (error_stack= exists()) ensures that only errors containing stack information are returned

Notify Members (@)¶

Click to select workspace members.

How it works:

1. Only when Incident Association is enabled, the @ members configuration takes effect and sends the event content to the specified members.
2. This configuration is independent of the notification targets in Alert Configuration and does not affect the alert notification scope.

Custom Notification Content¶

By default, the system uses the event content as the alert notification content. If you need to customize the notification actually sent externally, enable the toggle here and enter the notification information.

• Expand an independent editor to separately define the notification content sent externally;
• The original event content remains in the platform for event details display;
• The independent editor also supports Markdown, template variables, related links, and advanced settings.

Data Gap Events¶

Customize the notification content for data gaps (no data reported). You can also configure the title, content, and other information for these events when they are sent externally.

If not customized, the system sends data gap alerts using the official default template.

Custom data gap event content also supports syntax validation and template preview. The preview uses the current data gap title and content you entered.

Incident Association¶

When association is enabled, if an abnormal event is generated by this monitor, an Incident is created accordingly.

Configuration Items¶

1. Add tags to automatically created incidents to facilitate classification and filtering in the Incident Center.
2. Configure the mapping between event levels and incident levels. Multiple rules are supported.
   • When events at the levels Fatal/Critical/Important/Warning/Data Gap are generated, incidents at the new levels P0/P1/P2/P3/... are created accordingly.

Incidents generated here can be viewed in the Incident Center. (❗️ Such incidents include tag filter conditions.)

Linkage Mechanism¶

1. When an event is triggered, an incident record is automatically created in the Incident Center, and the incident description is automatically synchronized with the event content.
2. A notification about the new incident is sent based on the @ members list in the event content.
3. You can view incident details in the Incident Center. The system automatically correlates and displays all related data for the incident, including performance metrics, error logs, call traces, and infrastructure topology.

Alert Configuration¶

When a monitor meets the trigger condition, an alert message is immediately sent to the specified notification targets.

Alert Policy¶

Select one or more existing alert policies. Click a policy name to expand and view details. Click Edit Alert Policy to modify the configuration:

Link to Dashboard¶

Select an existing dashboard to establish an association between the monitor and the dashboard, allowing quick jumps and visual inspection of monitoring data.

Permissions¶

Set operation permissions for the monitor to ensure that different users can perform operations according to their roles and permission levels. The Owner of the current workspace is not affected by these operation permission settings.

• When disabled: follows the default permissions of Monitor Configuration Management;
• When enabled and custom permission subjects are selected: only the creator and the assigned subjects can enable/disable, edit, or delete the rules configured for this monitor;
• When enabled but no custom permission subjects are selected: only the creator has permissions to enable/disable, edit, and delete this monitor.

In addition, the Workspace Owner can grant monitor operation permissions to admins, roles, or members through Unified Permission Configuration. Authorized subjects can operate all monitors with custom operation permissions enabled without being added to the custom permission subjects of each monitor individually.

Trigger Detection Now¶

After the rule configuration is complete, click Trigger Detection Now to manually run a test and verify the overall effect of the current rule configuration. The test execution does not generate actual alert notifications.

Alert Cache Protection Mechanism¶

After configuration is complete, the monitor executes the following protection strategies during operation. The system prevents system pressure caused by excessive detection objects from high-cardinality aggregation through the following mechanisms:

The alert cache limit is 100,000 entries, and the warning threshold is 80%.

Pause Recovery Mechanism¶

After a monitor is paused by the system due to alert cache overflow, performing either of the following operations automatically recovers it:

• Modify the query conditions and save the monitor again

• Directly save the monitor again

The system automatically clears the alert cache marker and restores the monitor to normal operation. No additional action is required.

Further Reading¶

After a monitor rule is successfully created, you may need to: