Create Forwarding Rules¶

Prerequisites¶

Only available in the Commercial Plan.

Create¶

Go to the Data Forwarding > Forwarding Rules > Create page.

After the data forwarding rule is created, the system will perform rule validation every 5 minutes.

Step 1: Enter Rule Name¶

The name of the current data forwarding rule.

Step 2: Define Forwarding Rules¶

1. Data Source

Includes Logs, APM, RUM, LLM, Events, and Audit Events.

1. Filter Conditions

Supports custom logic between conditions; multiple conditions can be added.

All Conditions: Only log data that matches all filter conditions will be saved to data forwarding.
Any Conditions: Log data that matches any one filter condition will be saved to data forwarding.

Condition Operators:

Condition Operator	Match Type
in, not in	Exact match, supports multiple values (separated by commas)

If no filter conditions are added here, it means all data will be saved.

Pipeline Processing for Forwarded Data

The central Pipeline script will affect the forwarding condition filtering and final content.

When creating the script, you can check "Enable Pipeline to process forwarded data": if checked, data will be processed by the script before being filtered and forwarded; if not checked, the original data will be forwarded directly.

1. Advanced Settings

When Logs is selected as the data source, you can further configure:

Include Extended Fields: By default, only the message field content of matching logs is forwarded. If "Include Extended Fields" is checked, the entire log data that matches the condition will be forwarded.
- When creating multiple data forwarding rules, rules with "Include Extended Fields" checked will be matched first. If different rules match the same data, the rule with "Include Extended Fields" will take precedence to display the full log data.
Restrict Index: Select from the dropdown Native Write Index. After enabling, only log data from the selected index will be forwarded. (❗️ Enabling Restrict Index will significantly reduce the performance consumption of this data forwarding rule)

Step 3: Select Archive Type¶

To provide a more comprehensive data forwarding storage method, the system supports the following storage paths.

Guance: Matched log data will be saved to Guance's OSS, S3, OBS object storage.

Note

The site support for the above archive types is as follows:
- Full site support: AWS S3, Alibaba Cloud OSS, Huawei Cloud OBS, Volcengine TOS, Kafka
- Partial site support: GCP GCS (only supported in Hong Kong China and Global sites)
When the data forwarding storage target is Guance, the minimum log data retention is 180 days, and once the rule is created, it cannot be canceled. Storage fees will be charged daily. You can go to Management > Workspace Settings > Change Data Storage Policy to modify it.

Storage Format¶

Select the data storage format as needed:

JSON: Text format (❗️ When forwarding data to Guance, only JSON storage format is supported).
Parquet: Columnar storage format.

Feature	JSON	Parquet
Functional Positioning	Standard format for immediate consumption and integration	Optimized format for low-cost archiving and offline analysis
Data Status	Hot data that can be viewed. Forwarded data will be stored and displayed normally in the Guance platform (e.g., Log Explorer), maintaining consistent visibility on both ends	Cold data for external processing. After forwarding, the original logs will still be stored and displayed normally in the Guance platform, but the copy forwarded in Parquet format, due to format limitations, cannot be reloaded or viewed in the Guance interface
Core Scenarios	While maintaining data observability in Guance, provide a copy that can be directly parsed by external systems (e.g., SIEM, self-built log libraries) for real-time consumption	Generate a dedicated copy for efficient batch analysis by external big data systems, with better storage cost, for qualifying data
Business Use	Downstream business systems can obtain readable logs consistent with the Guance interface in near real-time via API, object storage files, or message queues, used for: Integrating with security audit (SIEM) platforms. Secondary processing in self-built log analysis tools	Downstream big data systems can periodically (e.g., hourly/daily) read Parquet files from object storage in batches, used for: Long-term historical log compliance archiving. Cross-month/year offline aggregation analysis and report generation
Key Impact	Dual write, dual-end visibility. Incurs additional storage costs but ensures data consistency and immediate availability within and outside the Guance ecosystem	Format-specific, with viewing limitations. The generated Parquet copy is optimized for external analysis and is not suitable for backflow viewing, so this copy cannot be viewed in Guance. However, the original logs' observability in Guance is unaffected

Encrypted Storage¶

After enabling encrypted storage, the system will perform symmetric encryption on the forwarded data. Later, if you need to query or view this data, the system can decrypt the encrypted data back to the original content for display.

What is Symmetric Encryption?

Symmetric encryption is a method that uses the same key for both encryption and decryption, just like a single key can both lock and unlock the same lock.

Step 4: Define Data View Permissions¶

Set viewing permissions for the forwarded data to enhance data security.

Unrestricted: All members of the workspace can view the forwarded data.
Custom: Specify which member roles can view the forwarded data.