Create Data Forwarding Rules¶

Prerequisites¶

Available only for Commercial Plan.

Create¶

Navigate to Data Forwarding > Forwarding Rules > Create.

After a data forwarding rule is created, the system will perform rule validation every 5 minutes.

Step 1: Enter Rule Name¶

The name of the current data forwarding rule.

Step 2: Define Forwarding Rules¶

1. Data Source

Includes LOG, APM, RUM, LLM, Event, Audit Event.

2. Filter Conditions

Supports custom logic operations between conditions. Multiple conditions can be added.

All Conditions: Only log data that meets all filter conditions will be saved to data forwarding.
Any Condition: Data will be saved to data forwarding if any one filter condition is met.

Condition operators are listed below:

Condition Operator	Match Type
in, not in	Exact match, supports multiple values (separated by commas)

If no filter conditions are added here, it means saving all data.

Pipeline Processing for Forwarded Data

The central Pipeline script affects the forwarding condition filtering and final content.

When creating a script, you can check "Enable Pipeline processing for forwarded data": If checked, data is processed by the script first before being filtered and forwarded; if not checked, the original data is forwarded directly.

3. Advanced Settings

When LOG is selected as the data source, further configuration is available:

Include Extended Fields: By default, only the message field content of logs meeting the conditions is forwarded. If "Include Extended Fields" is checked, the entire log data meeting the conditions will be forwarded.
- When creating multiple data forwarding rules, rules with "Include Extended Fields" checked are matched first. If different rules hit the same piece of data, the logic of including extended fields takes priority to display the entire log data.
Limit Index: Dropdown to select a native direct-write index. When enabled, only log data from the selected index is forwarded. (❗️After enabling Limit Index, the performance consumption of this data forwarding rule will be significantly reduced.)

Step 3: Select Storage Type¶

To provide more comprehensive data forwarding storage methods, the system supports the following storage paths.

Guance: Matched log data will be saved to Guance-side OSS, S3, OBS object storage.

Note

The site support for the above storage types is as follows:
- Open to all sites: AWS S3, Alibaba Cloud OSS, Huawei Cloud OBS, Volcengine TOS, Kafka.
- Open to some sites: Google Cloud GCS (only supported in Hong Kong, China and global region sites).
When selecting Guance as the data forwarding storage object, the minimum log data storage period defaults to 180 days, and the rule cannot be canceled once created; fees will be charged daily during the storage period. You can go to Manage > Workspace Settings > Change Data Storage Policy to modify it.

Storage Format¶

Select the data storage format as needed:

JSON: Text format (❗️When forwarding data to Guance, only the JSON storage format can be selected).
Parquet: Columnar storage format.

Feature	JSON	Parquet
Functional Positioning	Standard format for immediate consumption and integration	Optimized format for low-cost archiving and offline analysis
Data State	Hot data that can be reviewed. After forwarding, the data will be normally stored and displayed within the Guance platform (e.g., in the log explorer), maintaining consistency and visibility on both ends.	Cold data for external processing. After data forwarding, the original logs will still be normally stored and displayed within the Guance platform. However, the copy forwarded in Parquet format, due to format limitations, cannot be reloaded or reviewed within the Guance interface.
Core Scenarios	While maintaining data observability within Guance, provides a copy for external systems (e.g., SIEM, self-built log libraries) for real-time consumption and direct parsing.	Generates a dedicated copy for external big data systems to perform efficient batch analysis with better storage cost optimization for eligible data.
Business Usage	Downstream business systems can obtain readable logs, completely consistent with those within the Guance interface, in near real-time via API, object storage files, or message queues, for: Integrating with security audit (SIEM) platforms. Flowing into self-built log analysis tools for secondary processing.	Downstream big data systems can periodically (e.g., hourly/daily) read Parquet files in object storage in batches, for: Compliance archiving of long-term historical logs. Offline aggregation analysis and report generation across months/years.
Key Impact	Data is written twice and visible on both ends. Incurs additional storage costs but ensures data consistency and immediate availability within and outside the Guance ecosystem.	Format-specific, with review limitations. The generated Parquet copy is optimized for external analysis and is not suitable for backflow viewing; therefore, this copy itself cannot be reviewed within Guance. However, the observability of the original logs within Guance is not affected in any way.

Encrypted Storage¶

After enabling encrypted storage, the system will perform symmetric encryption on the forwarded data. Later, if you need to query or view this data, the system can restore the encrypted data to its original content for display.

What is symmetric encryption?

Symmetric encryption is an encryption method that uses the same key to encrypt and decrypt data, much like a key that can both lock and unlock the same lock.

Step 4: Define Data Viewing Permissions¶

Set viewing permissions for the forwarded data to enhance data security.

No Restrictions: All members of the workspace can view the forwarded data.
Custom: Specify member roles that can view the forwarded data.