Create Data Forwarding Rules¶

Prerequisites¶

Available only for Commercial Plan.

Create¶

Go to the Data Forwarding > Forwarding Rules > Create page.

After a data forwarding rule is created, the system performs rule validation every 5 minutes.

1. Enter Rule Name¶

This is the name of the current data forwarding rule.

2. Define Forwarding Rules¶

1. Data Source

Includes Logs, APM, RUM, Events, Audit Events.

2. Filter Conditions

Supports custom logic operations between conditions. Multiple conditions can be added.

All Conditions: Only log data matching all filter conditions will be saved for forwarding.
Any Condition: Data matching any single filter condition will be saved for forwarding.

Condition operators are shown in the table below:

Condition Operator	Match Type
in, not in	Exact match, supports multiple values (separated by commas)
match, not match	Fuzzy match, supports regular expression syntax

If no filter conditions are added here, it means all data will be saved.

Pipeline Processing for Forwarded Data

The central Pipeline script affects the forwarding condition filtering and final content.

When creating a script, you can check "Enable Pipeline processing for forwarded data": If checked, data is processed by the script first and then filtered for storage/forwarding; if not checked, the original data is forwarded directly.

3. Advanced Settings

When Logs is selected as the data source, further configuration is available:

Include Extended Fields: By default, only the message field content of logs matching the conditions is forwarded. If "Include Extended Fields" is checked, the entire log data matching the conditions will be forwarded.
- When creating multiple data forwarding rules, rules with "Include Extended Fields" checked are prioritized. If different rules hit the same piece of data, the logic for including extended fields takes precedence, showing the entire log data.
Limit Index: Dropdown to select a Native Write Index. Once enabled, only log data from the selected index will be forwarded. (❗️Enabling Limit Index will significantly reduce the performance consumption of this data forwarding rule.)

3. Select Storage Type¶

To provide more comprehensive data forwarding storage methods, the system supports the following storage paths.

Guance: Matched log data will be saved to Guance's OSS, S3, or OBS object storage.

Note

All the above storage types are available site-wide.
When selecting Guance as the data forwarding storage object, the minimum storage period for log data defaults to 180 days. Once created, the rule cannot be canceled, and fees will be charged daily during the storage period. You can modify this by going to Manage > Workspace Settings > Change Data Storage Policy.

Storage Format¶

Select the data storage format as needed:

JSON: Text format (❗️When forwarding data to Guance, only the JSON storage format is supported).
Parquet: Columnar storage format.

Characteristic	JSON	Parquet
Functional Positioning	Standard format for immediate consumption and integration	Optimized format for low-cost archiving and offline analysis
Data State	Hot data for review. Forwarded data is normally stored and displayed within the Guance platform (e.g., Log Explorer), maintaining visibility and consistency across both ends.	Cold data for external processing. After data is forwarded, the original logs are still normally stored and displayed within the Guance platform. However, the copy forwarded in Parquet format, due to its format limitations, cannot be reloaded or reviewed within the Guance interface.
Core Scenarios	Provides a copy for external systems (e.g., SIEM, self-built log libraries) to consume in real-time, directly parsable, while maintaining data observability within Guance.	Generates a dedicated copy for external big data systems to perform efficient batch analysis, with optimized storage costs, for data meeting the conditions.
Business Use	Downstream business systems can obtain readable logs, identical to those in the Guance interface, quasi-real-time via API, object storage files, or message queues, for: Integrating with security audit (SIEM) platforms. Feeding into self-built log analysis tools for secondary processing.	Downstream big data systems can periodically (e.g., hourly/daily) read Parquet files from object storage in batches, for: Long-term historical log compliance archiving. Cross-month/year offline aggregation analysis and report generation.
Key Impact	Data is written twice and visible at both ends. Incurs additional storage costs but ensures data consistency and immediate availability both inside and outside the Guance ecosystem.	Format is specialized, with review limitations. The generated Parquet copy is optimized for external analysis and is not suitable for backflow viewing. Therefore, this copy itself cannot be reviewed within Guance. However, the observability of the original logs within Guance remains completely unaffected.

Encrypted Storage¶

After enabling encrypted storage, the system performs symmetric encryption on the forwarded data. If you need to query or view this data later, the system can decrypt the encrypted data back to its original content for display.

What is symmetric encryption?

Symmetric encryption is an encryption method that uses the same key to encrypt and decrypt data, much like a single key that can both lock and unlock the same lock.