On-call¶

The On-call feature helps teams establish a 7x24 fault response mechanism, ensuring each fault has a clear responsible person and automatically escalates when not handled within the timeout period, achieving "guaranteed alert delivery".

Core Concepts¶

On-call Rules¶

On-call Rules define who is responsible for what type of fault at what time. Each rule includes the following elements:

• On-call personnel: Members, teams, or notification targets.
• Working hours: The effective time period for the on-call duty (supports timezone settings).
• Effective scope: The validity period of the scheduling rule.
• Matching tags/dimensions: Determine which faults are routed to this rule.
• Escalation policy: Notification escalation rules when not handled within the timeout.

Escalation Policy¶

An escalation policy is a multi-level notification mechanism attached to an on-call rule. When a fault is not claimed or resolved within a specified time, the system will gradually expand the notification scope according to the preset levels, ensuring no fault is missed.

The Level 0 rule of the escalation policy supports selecting "Current on-call personnel", which points to the on-call personnel currently in effect when the on-call strategy rotation is enabled.

Matching Tag Logic¶

Faults automatically match on-call rules based on the tags they carry. Matching rules support:

• AND: Multiple tags must be satisfied simultaneously (full match).
• OR: Any one of multiple tags being satisfied is sufficient (partial match).
• Wildcard: key:value* supports prefix matching.

Example:

Fault tags: {service:payment, env:prod, team:backend}

• On-call Rule A: Tags service:payment AND env:prod → Matches ✓
• On-call Rule B: Tag team:frontend → Does not match ✗
• On-call Rule C: No tags (global) → Matches ✓ (fallback)

On-call Overview¶

After entering the On-call page, on-call rules are displayed in card view by default. The top of the page shows your current on-call status, including the name of the rule you are currently on duty for, the end time, and the remaining duration.

You can quickly switch the view scope via "All/Mine" to display all on-call rules or only those you participate in. The number of items in the current list is displayed on the right side of the filter results. You can also search by on-call name using the search box. If you are currently on duty, the corresponding rule will be pinned to the top and marked with an "On-call" label.

Card View¶

Displays on-call rules in card form. Each card contains the on-call name, current on-call personnel, next on-call personnel, matching dimensions, and the bound escalation policy. Click any on-call card to enter the detailed configuration page for that on-call duty.

Calendar View¶

Click "Calendar" to switch to the calendar view. The current filter and search conditions will be retained.

• The left side displays the list of on-call duties within the current filter scope. You can check the on-call rules you want to display in the calendar.
• The right side is the monthly calendar view, supporting month switching, returning to today, and timezone switching.

The calendar displays the on-call schedule by day, with each on-call personnel occupying a separate row, showing the personnel name and start time. Clicking expands the entire row to show complete information. Hovering the mouse over the on-call personnel name allows you to view detailed information about that duty.

The system-built "Default On-call" is always displayed in the on-call list and cannot be deleted or hidden. Clicking the on-call personnel name on the calendar allows you to view detailed information about that duty, including the associated on-call rule, escalation policy, and specific on-call time periods. The top left corner supports switching timezone and date to view historical or future schedules.

On-call Management¶

The "On-call Management" page centrally displays all on-call rules in a list format. Each rule lists key information such as on-call timezone, execution cycle, on-call personnel, matching tags, and escalation policy. The list includes both the system default on-call and custom on-call rules. Click any entry to enter its detail page for in-depth configuration.

To ensure fault notifications can be accurately delivered and responsibility is closed-loop, the core of configuring an on-call strategy lies in establishing the following two-layer guarantee mechanism:

1. Clarify "who is responsible at what time": By setting on-call personnel, effective time periods, and enabling notification rotation (supporting automatic handover by day, week, etc.), the system achieves clear scheduling and automatic rotation of responsibilities, ensuring there is always a clear "first responder" at any time.

2. Preset escalation path ("how to report if no response"): By configuring escalation policies, a "T+N minutes" progressive notification timeline is constructed. When a fault is not handled by anyone within the set time, the system will automatically escalate the alert to other levels of members or broader teams according to this rule, ensuring critical faults are guaranteed to be delivered.

Create On-call Rule¶

Creating an on-call rule requires completing the following configuration steps.

Basic Information¶

1. Enter the on-call name.
2. Select the timezone on which the on-call duty is based.
3. Configure working hours: Define during which periods the on-call duty is effective.
   • All day: Covers the entire day.
   • Specified time: Configure multiple time periods by day of the week and start/end times. The final effective time is the union of these periods.
4. Configure effective scope: Define the validity period of the scheduling rule.
   • Long-term effective: No end time limit.
   • Specified date: Configure start date and end date. The start date cannot be earlier than today.

Matching Tags/Dimensions (Optional)¶

This section determines which faults will be handled by this rule. If no tags/dimensions are added, this rule is globally matched.

1. Matching tags:
   • Select existing tags from the dropdown list.
   • Supports directly entering new tags and quickly creating them, or directly going to "Global Tags" for management.
2. Matching dimensions:
   • You can select detection dimensions (such as service, host) and set specific matching values.
   • Supports logical relationships: AND (full match, all conditions must be met) or OR (partial match, any one condition being met is sufficient). Default is AND.
   • Values support wildcards, format is key:value*, e.g., service:auth* can match auth-api, auth-service, etc.

On-call Personnel Settings¶

1. Select on-call personnel: Can be one or multiple members, or an entire team.
2. Enable rotation: If rotation is needed, enable the rotation function. Set the rotation cycle (e.g., daily, weekly, monthly). The system will automatically schedule rotations according to the order of the member list and intuitively display the scheduling effect in the calendar on the right.
   • Custom rotation cycle: Supports entering 1–31 days or 1–4 weeks. The system will automatically calculate the actual rotation effect based on working hours. For example, if working hours are set from Monday to Friday, a 4-day rotation will have the actual effect: A is on duty from Monday to Thursday, B is on duty from Friday and next Monday to next Wednesday.
3. Automatic claim: When enabled, if a fault matches only one on-call personnel, that member will be automatically assigned as the fault handler, and the fault status will be updated to "Working". Level 0 notifications of the escalation policy will still be sent normally.

Rotation example:

• Before enabling rotation:

• After enabling rotation:

Temporary Substitute¶

Used to temporarily adjust on-call personnel within a specific time period.

1. Select the substitute and the person being substituted (both are single-select), and specify the start and end time of the substitution.
2. The calendar on the right previews the substitution effect in real-time. The top shows the substitute, the person being substituted, and the substitution start/end time and duration.
3. After the substitution is created, the person being substituted will no longer receive fault alert notifications during that period; the substitute will receive them instead.
4. In the calendar, already effective substitutions are directly displayed on the corresponding dates, with the original on-call personnel shown with a strikethrough. Clicking the substitution module opens a detail overlay containing both parties of the substitution, start/end time, and supports "Adjust substitution" or deletion.
5. After the substitution takes effect, the system will send an email notification to the substitute; after deleting a substitution, the system will send an email notification to the person being substituted.

Configure Escalation Policy¶

The escalation policy ensures that when a fault is not handled within the timeout, the notification scope is automatically expanded to more people or higher levels (❗️The escalation policy is the core of the on-call rule, strongly recommended to configure).

Timeline Mechanism (T+N)¶

All time point calculations are based on the moment the fault is generated (denoted as T=0). The system triggers notifications at each level sequentially according to preset time intervals:

Level Configuration Description¶

1. Level 0 (Initial Notification) (Required)

• Trigger timing: Immediate notification when the fault is generated (T=0).
• Notification targets: By default, the current on-call personnel (i.e., the on-call personnel currently in effect in the on-call rule) are filled in. Supports additionally adding other personnel or teams.
• Notification method: Individually check for each notification target (email, SMS, phone call, multiple selections allowed).

2. Level 1~10 (Escalation Levels) (Optional)

• Trigger conditions: This level will only be triggered if all the following conditions are met simultaneously:
  • The fault duration has reached the set wait time (e.g., T+20 minutes).
  • The fault severity is within the specified range (e.g., only for P0, P1).
  • The fault status is a specified value (e.g., Open or Working).
• Notification targets: Only notify the personnel or teams configured in this level, not those configured in Level 0.
• Notification method: Set the notification method individually for newly added personnel.

Repeat Notification Mechanism¶

Within each level, you can choose whether to enable repeat notifications:

• Disable repeat notification: This level only sends one notification, then waits to enter the next level.
• Enable repeat notification: Periodically send notifications at the set frequency (e.g., every 5 minutes) until the fault status changes or it enters the next level.

Example:

• Level 1 wait time: 30 minutes.
• Repeat interval: 5 minutes.
• Final effect: Send a notification at T+5, T+10, T+15, T+20, T+25, T+30 minutes.

Handling Cross On-call Handover¶

If the fault duration spans an on-call handover time, subsequent escalation notifications will be transferred to the new on-call personnel and executed according to the new on-call personnel's escalation policy.

Example:

• Fault generated at 23:55, current on-call personnel is A.
• The wait time for Level 1 in the escalation policy is 15 minutes, configured to repeat every 5 minutes.
• The first repeat notification triggers 5 minutes after fault generation (i.e., 0:00). At this time, the on-call personnel has switched to B, so this notification will be sent to B, and subsequent escalation notifications (including remaining repeats and the next level) will be executed according to B's escalation policy.

After crossing days, the system will continue processing the fault based on the new on-call personnel B's escalation rules.

Multi-escalation Policy Deduplication¶

When the same fault matches multiple on-call rules (thus matching multiple escalation policies) simultaneously, the system will automatically deduplicate notifications to ensure the same user does not receive duplicate notifications. Deduplication logic is based on user, fault, and notification content.

Escalation Policy Configuration Example¶

Scenario: Escalation Policy for Core Service P0 Fault

In this example:

• When the fault is generated, immediately notify the current on-call personnel A.
• If the fault is not handled after 5 minutes, additionally notify the on-call team lead B (notification targets are now A + B).
• If still not handled after 15 minutes, additionally notify the department manager C (notification targets are now A + B + C).
• If still not handled after 30 minutes, additionally notify CTO D, and Level 3 has repeat notification enabled (e.g., every 10 minutes) until someone responds.

Notification Method Description¶

The system supports three notification channels. You can individually check for each notification target:

Default On-call¶

The system has a built-in "Default On-call", which is a simplified version of an on-call rule suitable for simple scenarios. Its characteristics are as follows:

• Only on-call personnel, on-call personnel rotation, and escalation policy can be configured.
• Non-configurable items: timezone (fixed as empty, follows system timezone), matching tags/dimensions (not supported for setting, defaults to global match).
• The default on-call will always be displayed in the on-call list and cannot be deleted.

Rule Limitations¶

1. An on-call rule can set up to 10 escalation levels (Level 0 + Level 1~10).
2. The maximum single wait time is 360 minutes (6 hours), exceeding which cannot be saved.
3. The fault severity and status range of higher levels must be a subset of the range already selected in lower levels.

Next Steps¶

After configuring the on-call rules, you can see the on-call information automatically associated with faults in the Incident List. When a fault is generated, the system will automatically notify the corresponding personnel according to your set rules and execute the escalation policy after timeout, ensuring every fault receives timely response.