Monitor Internals¶

Due to factors such as network conditions and system load, the execution of monitor evaluations involves some special internal handling mechanisms.

Evaluation Trigger Time¶

The evaluation frequency configured by the user is converted to a Crontab expression internally. The monitor starts strictly according to this expression, rather than simply executing once every N minutes after creation or save.

For example, if the user configures "Monitor A" with an evaluation frequency of "5 minutes", the corresponding Crontab expression is */5 * * * *. The specific trigger times are:

Data Waiting Window¶

The Data Waiting Window is used to proactively delay the monitor evaluation after the scheduled execution time arrives, giving late-reported data more time to complete ingestion.

Assume the monitor is scheduled to execute at 10:00, the evaluation range is the last 1 hour, and the data waiting window is 5 minutes:

The data waiting window only delays the actual execution time; the query range remains based on the scheduled execution time and does not shift forward. During the waiting period, the execution record status shows "Waiting for data"; if the actual execution time is later than "scheduled execution time + waiting duration" due to task queueing, the task still executes, and the execution record retains the scheduled execution time, waiting configuration, and actual start time.

Note:

• No waiting is applied by default; existing monitors also execute without waiting;
• The waiting duration must be shorter than the interval between two adjacent scheduled execution times;
• If the monitor is disabled or deleted during the waiting period, waiting tasks that have not yet executed are canceled.

Evaluation Range Calibration¶

Because the platform needs to handle thousands of monitors configured by all users, evaluation tasks triggered at the same moment cannot execute simultaneously; most tasks enter a queue and wait.

Therefore, most evaluation tasks encounter a situation where they are scheduled to trigger at time T but actually execute at time T + Δt.

If the actual execution time is directly used as the end time of the query, the evaluation time ranges will overlap or have gaps. For example:

Assume the evaluation time range is 5 minutes:

In this case:

• The evaluation ranges of "Action 1" and "Action 2" overlap between 00:05:05 ~ 00:05:10;
• There is a gap between "Action 2" and "Action 3" from 00:10:05 ~ 00:10:30, so data in this period is not covered.

Current Solution¶

To avoid evaluation range fluctuations caused by task queueing, the monitor's data query range is calibrated based on its scheduled trigger time, rather than the actual execution time.

Assume the evaluation time range is 5 minutes:

It can be seen that regardless of how long an evaluation task waits in the queue, its data query range is always based on the scheduled trigger time, ensuring the continuity and stability of the time window.

Evaluation Range Drift¶

Due to factors such as network latency and data processing, after data is reported it usually takes several seconds to tens of seconds to complete persistence (i.e., be queryable via DQL). During this period, monitor evaluations cannot query this "in-flight" data.

This easily leads to missing data when evaluating with a fixed time range. For example:

Assume the evaluation time range is 5 minutes:

Although data A was reported before the second evaluation executed, it was missed because it was not persisted during the first evaluation. After it is persisted, because its timestamp is earlier, it is no longer within the time ranges of subsequent evaluations and therefore continues to go undetected.

Current Solution¶

To solve this problem, all monitors automatically drift the data query range 1 minute toward the past when executing evaluations, to avoid the blind spot during data persistence.

After applying this solution, the example above becomes:

Assume the evaluation time range is 5 minutes:

Although data A was missed in the evaluation at 00:10:00, its timestamp 00:09:59 falls within the range of the evaluation at 00:15:00 (00:09:00 ~ 00:14:00), so it can be successfully captured in the second evaluation.

Data Gap Detection Logic¶

Guance is a time-series data platform and does not have the concept of an "asset master table" found in traditional asset management software. The system can only determine what "exists" based on the data that has been queried, and cannot know which objects "should exist but currently do not."

For example:

It is known that a box contains pencils and erasers. We can clearly state that "pencils and erasers exist in the box," but we cannot assert that "there are no pens in the box," because we do not know what "should" be in the box.

Therefore, "data gap detection" in monitors actually uses an "edge-triggered" mechanism to make determinations, that is, detecting changes by comparing the results of two consecutive evaluations.

Its core logic is: "If the previous evaluation found object X present, but object X disappears in this evaluation, then X is determined to have a data gap."

Assume the evaluation time range is 5 minutes:

Data Gap / Data Recovery Events¶

When the monitor determines that a "data gap" or "data resumption" has occurred, it decides whether to generate a "data gap event" or a "data gap recovery event" based on user configuration.

To avoid duplicate or meaningless alerts, the system references existing event states before generating events to make a decision:

Therefore, "data gap events" and "data gap recovery events" always alternate; consecutive data gap events or consecutive recovery events never occur.

FAQ¶

The time indicated in the event does not match the event generation time

The time seen in the event details or alert notification (e.g., 00:15:00) is the monitor's scheduled trigger time (i.e., the regularized time based on the Crontab expression), not the actual time when the event was generated in the system.

Exception: If you manually click "Execute" in the monitor list, the event time will be the actual time of the manual execution.

The time indicated in the event does not match the actual failure occurrence time

The time indicated in the event is the monitor's scheduled trigger time. Due to the "evaluation range drift" mechanism, the actual data range evaluated is from scheduled trigger time - evaluation range - drift time to scheduled trigger time - drift time.

Therefore, the timestamp of the data point where the failure actually occurred is likely not within the intuitive range from scheduled trigger time - evaluation range to scheduled trigger time. This is normal.

Suspected failure data is found when querying directly on the platform, but the monitor did not generate an alert

This issue is usually caused by:

1. Excessive data persistence latency: When the evaluation executes, the failure data is not yet queryable due to latency;
2. DQL query execution failure: The evaluation process is interrupted because the query failed.

Such cases usually stem from issues in the data pipeline or query engine, which are outside the monitor's own control.