Data Forwarding¶

For data that requires long-term storage but has a low update frequency (such as logs), you can use the Data Forwarding feature to automatically save it to object storage or forward it in real time to external systems like Kafka. This feature filters data based on rules to achieve low-cost long-term archiving while supporting subsequent secondary processing.

Once rules take effect, you can quickly retrieve stored data by setting a query time range and rule on the Data Forwarding page.

How It Works¶

When data is forwarded to object storage, the workflow is as follows: data reported by users that matches the rules is first written line by line into a temporary file on the local disk of the server. When the temporary file reaches a preset size (e.g., 256 MB) or the continuous write time exceeds a set duration (e.g., 1 hour), the system automatically closes the current file and creates a new temporary file to continue receiving data.

Meanwhile, a background service continuously scans these closed temporary files, compresses them using gzip to reduce size, and then uploads them to the specified object storage location according to predefined path rules. When you need to search this stored data, the system locates the relevant files in object storage using the same path rules, downloads them, decompresses them, and matches each line against your search criteria.

File Format Description¶

The files ultimately stored in object storage are compressed with gzip. After decompression, the file content is multi-line text, with each line corresponding to an original data record, and the data is saved in JSON format. Any empty lines in the file are automatically ignored by the system.

A typical log-type data forwarding file looks like the following:

date is a required field, indicating the timestamp of the data record in milliseconds (Unix timestamp), and the message field contains the specific log content.

{"__docid":"L_1750649205520_d1cciupkac7k1683bhq0","__namespace":"backup_log","date":1750649205520,"date_ns":168000,"df_metering_size":-9223372036854775808,"filepath":"/var/log/datakit/gin.log","host":"X.local","log_read_lines":2,"message":"[GIN] 2025/06/23 - 11:26:43 | 200 | 1.012923708s |       127.0.0.1 | GET     \"/metrics\"","message_length":87,"service":"default","source":"default","status":"unknown"}

{"__docid":"L_1750649205516_d1cciupkac7k1683bhqg","__namespace":"backup_log","date":1750649205516,"date_ns":897000,"df_metering_size":-9223372036854775808,"filepath":"/var/log/datakit/gin.log","host":"X.local","log_read_lines":1,"message":"[GIN] 2025/06/23 - 11:26:38 | 200 | 1.012696542s |       127.0.0.1 | GET     \"/metrics\"","message_length":87,"service":"default","source":"default","status":"unknown"}

{"__docid":"L_1750649206520_d1cciupkac7k1683bhr0","__namespace":"backup_log","date":1750649206520,"date_ns":948000,"df_metering_size":-9223372036854775808,"filepath":"/var/log/datakit/log","host":"X.local","log_read_lines":150,"message":"2025-06-23T11:26:46.520+0800\tWARN\thost_processes\tprocess/input.go:332\tprocess: {\"pid\":411}, proc.PageFaults(): not implemented yet","message_length":130,"service":"default","source":"default","status":"unknown"}

{"__docid":"L_1750649205520_d1cciupkac7k1683bhrg","__namespace":"backup_log","date":1750649205520,"date_ns":419000,"df_metering_size":-9223372036854775808,"filepath":"/var/log/datakit/log","host":"X.local","log_read_lines":9,"message":"2025-06-23T11:26:43.876+0800\tWARN\tcontainer\tcontainer/impl.go:254\tendpoint unix:///var/run/crio/crio.sock does not exist, maybe it is not running, skip","message_length":151,"service":"default","source":"default","status":"unknown"}

{"__docid":"L_1750649205517_d1cciupkac7k1683bhs0","__namespace":"backup_log","date":1750649205517,"date_ns":79000,"df_metering_size":-9223372036854775808,"filepath":"/var/log/datakit/log","host":"X.local","log_read_lines":1,"message":"2025-06-23T11:26:38.365+0800\tWARN\thttp\thttpapi/http.go:494\tlistener.Close failed: close tcp [::]:9529: use of closed network connection","message_length":135,"service":"default","source":"default","status":"unknown"}
{"__docid":"L_1750649205517_d1cciupkac7k1683bhsg","__namespace":"backup_log","date":1750649205517,"date_ns":80000,"df_metering_size":-9223372036854775808,"filepath":"/var/log/datakit/log","host":"X.local","log_read_lines":2,"message":"2025-06-23T11:26:38.365+0800\tWARN\thttp\thttpapi/http.go:494\tlistener.Close failed: close tcp [::]:9529: use of closed network connection","message_length":135,"service":"default","source":"default","status":"unknown"}

File Naming and Storage Path¶

[{$path_prefix}/]{$workspace_uuid}/[{$data_type}/]

{$rule_name}/{$year}/{$month}/{$day}/{$hour}/{$time}-{$hostname}.gz

Parts enclosed in [] are optional. See the table below for details:

Path examples:

wksp_d9a1851859e040469d290409bc17cceb/backup_logging_for_test/2025/05/06/17/175950000-c6a92aafa992599c.gz

path/to/backup/wksp_d9a1851859e040469d290409bc17cceb/tracing/test-minio/2025/05/06/17/175950000-c6a92aafa992599c.gz

You can upload external files to object storage following the format and path rules generated by the data forwarding rules, and the console will search and display them in the same way.

Getting Started¶

Create a Forwarding Rule¶

Based on different types of forwarding destinations, create data forwarding rules that meet your business needs:

Manage Forwarding Rules¶

On the data forwarding rules list, you can perform a series of operations to manage them.