Log Index¶

The system automatically archives log data into corresponding indexes based on the configured filter conditions. By creating multiple log indexes, you can:

• Isolate log data by business line, environment, or project
• Configure differentiated storage policies for different indexes
• Optimize query performance by reducing scanning of irrelevant data

Create¶

1. Go to the Create > Log Index page.
2. Enter a custom index name.
3. Add a description as needed.
4. Add filter conditions.

5. Configure data storage policy: select the log retention period, after which data is automatically deleted.

   • For Deployment Plan users, custom retention periods are supported, ranging from 1d to 1,800d.

   • Optionally configure a clone index.

   • Clone configuration includes name, retention period, and clone conditions.

   • Optionally configure advanced options.
   • Confirm.

Filter Conditions¶

Filter conditions define the rules for determining which logs are routed to the current index. Two configuration modes are supported: Standard Matching and Advanced Matching.

Standard Matching¶

Standard Matching uses the original configuration method and is suitable for simple filtering scenarios.

1. Select the condition relationship:

   • Match all conditions: The log must satisfy all filter conditions to be routed to the current index.
   • Match any condition: The log can satisfy any one of the filter conditions to be routed to the current index.

2. Add filter conditions:

   • Field name: Select a log field, such as source, service, host, etc.
   • Operator: Select the matching method.
   • Value: Enter the field value. Multiple values should be separated by commas.

Standard Matching is suitable for quickly configuring index routing based on fixed fields.

Advanced Matching¶

Advanced Matching is suitable for scenarios requiring more flexible combination of matching rules. When Advanced Matching is enabled, log data will be processed through a Pipeline for matching, and corresponding charges will apply.

In Advanced Matching, you can configure multiple filter conditions, each containing:

• Field name: Enter the log field to be evaluated.
• Match method: Supports =, !=, in, not in, match, not match, wildcard, not wildcard.
• Value: Enter the corresponding value for the field.

Click Add filter to add more conditions. Click Clear all conditions to remove all configured conditions.

The Expression preview at the bottom of the page shows the final matching expression generated from the current filter conditions. If the conditions are incomplete or the expression is invalid, the page will prompt "Please complete or correct the filter conditions." In this case, you must fill in the field name, match method, or value before saving.

Advanced Options¶

Advanced options are expanded by default. Here you can select full-text index fields.

Full-Text Index Fields¶

1. Index only the message field (default): Full-text search matches only the original message field. The system builds a full-text index for message.
2. Full-row index: Full-text search matches all business fields in the log, excluding system fields. The system writes all business fields into variant and builds a full-text index only for variant.

When a log contains message, the system does not delete it; instead, it is written into variant as a regular business field. The system does not build two separate full-text indexes for both message and variant to avoid additional storage costs from duplicate indexing.

Without message:
Business fields → variant → full-text index

With message:
message + other business fields → variant → full-text index

For instructions on configuring DataKit JSON field extraction or deleting the original message after Pipeline field extraction, see Full-Row Index.

The original "Match Service" capability has been migrated to Log Index Mapping.

Clone Index¶

A clone index is used to further filter and copy a subset of log data from a regular log index. For logs that are frequently queried and have a well-defined data scope, you can copy them into a dedicated clone index to reduce the amount of data scanned during queries, improving log retrieval efficiency.

Logs are first matched against the regular index order and written to the first matching regular index. After the write is complete, the system asynchronously evaluates whether the clone conditions configured for that index are met and decides whether to copy the log to the clone index.

Log reporting
   ↓
Match regular indexes in order
   ↓
Write to the first matching regular index
   ↓
Check if clone rules are configured
   ↓
Clone conditions met ──→ Asynchronously copy to clone index
Clone conditions not met ──→ No clone performed

Clone tasks are executed asynchronously. Even if a clone fails, the log write to the source index is not affected.

Use Cases¶

Clone indexes are suitable for the following scenarios:

• High-frequency queries only focus on a subset of logs within a regular index.
• Need to narrow query scope by service, environment, status, business type, etc.
• The regular index contains a large amount of data, and you want to reduce the scan volume for common queries.
• You want to assign a shorter retention period to high-frequency query data.
• You want to retain complete logs in the source index while also creating a simplified dataset for daily troubleshooting.

For example, the regular index application stores all application logs, but daily troubleshooting mainly focuses on error logs in the production environment. You can configure a clone index application_error for application with the following clone conditions:

env = production AND status = error

New logs that meet the conditions will be stored in both:

• Source index: application
• Clone index: application_error

When querying production error logs, you can directly select application_error, reducing the scan of the full data in application.

Configuring a Clone Index¶

When creating or editing a regular log index, you can optionally configure a clone index:

1. Go to Logs > Index.
2. Create a new log index or edit an existing regular log index.
3. Find the Clone Index configuration.
4. Enable clone index.
5. Enter the clone index name.
6. Set the data retention period for the clone index.
7. Add clone conditions.
8. Save the index configuration.

Each regular index can have at most one clone index.

Clone Index Name¶

The clone index name is used to identify the index in the index list and the Log Explorer.

When configuring the name, note:

• The clone index name must be unique within the current workspace.
• The clone index cannot have the same name as any existing regular index, native direct-write index, external index, or other clone index.
• After creation, the clone index is displayed as a sub-row of its source index.
• A clone index cannot serve as the source for another clone index.

Retention Period¶

A clone index can have its own independent data retention period, but it cannot exceed the retention period of the source index.

If you modify the retention period of the source index, ensure the clone index's retention period still complies with the limit.

Logs in the clone index are automatically deleted when they reach the retention period, and this does not affect the original logs in the source index.

Clone Conditions¶

Clone conditions determine whether a log in the source index should be copied to the clone index.

You can set filter rules based on log fields, for example:

• source
• service
• env
• status
• host
• Other log fields

Only logs that meet all the following conditions are routed to the clone index:

1. The log has been successfully written to the source regular index.
2. The log satisfies the configured clone conditions.
3. The clone rule was already in effect before the log was written.

Logs that do not meet the clone conditions are still stored normally in the source index, but are not copied to the clone index.

Data Content¶

The clone index retains all fields and tags of the source log; it does not only retain the fields used in the clone conditions.

The copied log content includes:

• Original log business fields.
• Original log tags.
• message if present in the original log.
• Information required for querying and displaying the log.

The clone operation does not modify or delete logs in the source index.

Scope of Effect¶

Clone rules only apply to logs written to the source index after the rule takes effect.

After the following operations, the system processes subsequent logs from the time the operation takes effect:

• Creating a new clone rule.
• Enabling a clone rule.
• Modifying clone conditions.
• Modifying clone index configuration.

The system does not automatically scan the source index and backfill historical data. Therefore, logs written to the source index before the rule takes effect are not automatically copied to the clone index.

For example, enabling a clone rule at 2026-08-19 10:00:00:

• Logs written after 10:00:00 that meet the conditions can enter the clone index.
• Historical logs that existed before 10:00:00 are not automatically backfilled.

Viewing Clone Indexes¶

Clone indexes are displayed as sub-rows of their source index in the index list.

application
└── application_error

The list display follows these rules:

• Regular indexes are displayed as first-level rows.
• Clone indexes are located below their source index.
• Clone indexes do not participate in regular index sorting.
• Clone indexes do not count toward the regular index pagination count.
• When adjusting the order of regular indexes, clone indexes move with their source index.

In the Logs > Explorer, you can explicitly select a clone index in the index list to query its logs.

Editing, Enabling, and Disabling¶

When editing a regular index, you can modify the corresponding clone rules.

After modifying or re-enabling clone rules:

• New rules only apply to logs written after the rule takes effect.
• Historical logs already written to the clone index are not reprocessed according to the new rules.
• Historical logs already written to the source index are not automatically backfilled.
• Modifying the rules does not alter the original log data in the source index.

When a clone rule is disabled, subsequent logs are no longer copied to the clone index. Historical logs within the clone index that are still within their retention period can still be queried.

Limitations¶

Index Limitations¶

Further Reading¶