Monitor Internal Principles¶

Due to factors such as network conditions and system load, the evaluation execution of monitors involves some special internal processing mechanisms.

Evaluation Trigger Time¶

The evaluation frequency configured by the user is internally converted to a Crontab expression. Monitors are triggered strictly according to this expression, rather than simply executing every N minutes after creation or saving.

For example, if the execution frequency configured for "Monitor A" is "5 minutes", the corresponding Crontab expression is */5 * * * *. The specific trigger timing is as follows:

Data Waiting Window¶

The data waiting window is used to actively delay the monitor judgment after the scheduled execution time, allowing more time for late-arriving data to be ingested.

Assume the monitor is scheduled to execute at 10:00, the evaluation window is the last 1 hour, and the data waiting window is 5 minutes:

The data waiting window only delays the actual execution time; the query range is still based on the scheduled execution time and does not shift overall. During the waiting period, the execution record status shows "Waiting for data". If the actual execution time is later than the "scheduled execution time + waiting duration" due to task queuing, the task will still be executed, and the execution record retains the scheduled execution time, waiting configuration, and actual start time.

Note:

• By default, there is no waiting; existing monitors also execute without waiting.
• The waiting duration must be less than the interval between two adjacent scheduled execution times.
• If the monitor is disabled or deleted during the waiting period, any pending waiting tasks will be canceled.

Evaluation Range Calibration¶

Because the platform must handle thousands of monitors configured by all users, evaluation tasks triggered at the same time cannot all execute simultaneously. Most tasks enter a queue and wait.

Therefore, most evaluation tasks experience a situation where they are scheduled to trigger at time T, but actually execute at time T + Δt.

If the actual execution time were used directly as the end time of the query, the evaluation time ranges would overlap or have gaps. For example:

Assume the evaluation time range is 5 minutes:

In this case:

• The evaluation ranges of "Action 1" and "Action 2" overlap between 00:05:05 and 00:05:10.
• The evaluation ranges of "Action 2" and "Action 3" have a gap between 00:10:05 and 00:10:30, causing data in this period to be uncovered.

Current Solution¶

To avoid fluctuations in the evaluation range caused by task queuing, the data query range of a monitor is calibrated based on its scheduled trigger time, not the actual execution time.

Assume the evaluation time range is 5 minutes:

Thus, regardless of how long an evaluation task is queued, its data query range is always based on the scheduled trigger time, ensuring continuity and stability of the time window.

Evaluation Range Drift¶

Due to factors such as network latency and data processing, it usually takes seconds to tens of seconds for reported data to be fully ingested (i.e., queryable via DQL). During this period, monitor evaluations cannot query these "in-flight" data.

This can easily lead to data loss when evaluating with a fixed time range. For example:

Assume the evaluation time range is 5 minutes:

Although Data A was reported before the second evaluation, it was missed by the first evaluation because it had not yet been ingested. After it is ingested, its timestamp is too early to fall within the subsequent evaluation range, so it remains undetected.

Current Solution¶

To address this issue, all monitors automatically drift the data query range backward by 1 minute during evaluation, to avoid the window during which data is still being ingested.

Applying this solution, the above example becomes:

Assume the evaluation time range is 5 minutes:

Although Data A was missed by the evaluation at 00:10:00, its timestamp 00:09:59 falls within the range of the evaluation at 00:15:00 (00:09:00 ~ 00:14:00), so it is successfully captured during the second evaluation.

Data Gap Judgment Logic¶

Guance, as a time-series data platform, does not have the concept of an "asset inventory" like traditional asset management software. The system can only determine what "exists" based on the data it has queried, but it cannot know what "should exist but is currently absent".

For example:

A box is known to contain a pencil and an eraser. We can clearly say "the box contains a pencil and an eraser", but we cannot assert that "the box does not contain a pen", because we do not know what the box "should" contain.

Therefore, the "data gap detection" in monitors actually uses an edge-triggered mechanism, detecting changes by comparing the results of two consecutive evaluations.

The core logic is: "If the previous round of evaluation found that object X existed, and the current round of evaluation finds that object X has disappeared, then X is considered to have a data gap."

Assume the evaluation time range is 5 minutes:

Data Gap / Data Recovery Events¶

When a monitor determines that a "data gap" or "data recovery" has occurred, it decides whether to generate a "data gap event" or "data recovery event" based on the user's configuration.

To avoid generating duplicate or meaningless alerts, the system refers to the state of existing events before generating a new event:

Therefore, "data gap events" and "data recovery events" always alternate. There will never be consecutive data gap events or consecutive recovery events.

Frequently Asked Questions¶

The time indicated in the event does not match the time the event was generated

The time seen in the event details or alert notification (e.g., 00:15:00) is the monitor's scheduled trigger time (i.e., the normalized time based on the Crontab expression), not the actual time the event was generated in the system.

Exception: If you manually click "Execute" in the monitor list, the event time will be the actual time of the click.

The time indicated in the event does not match the actual time of the failure

The time indicated in the event is the monitor's scheduled trigger time. Due to the "evaluation range drift" mechanism, the actual data range evaluated is from scheduled trigger time - evaluation range - drift time to scheduled trigger time - drift time.

Therefore, the actual time of the failure data point is likely not within the intuitive range of scheduled trigger time - evaluation range to scheduled trigger time. This is normal.

When querying directly in the platform, suspected failure data is found, but the monitor did not generate an alert

This issue is usually caused by one of the following reasons:

1. High data ingestion latency: The failure data was not yet queryable when the evaluation was executed due to latency.
2. DQL query execution failure: The evaluation was interrupted due to a query failure.

Such situations are usually caused by issues in the data pipeline or query engine, which are beyond the control of the monitor itself.