Managing Pipelines¶

Options¶

Single Operation¶

Click the button under the operations on the right side of the Pipeline file to edit, delete, enable, or disable that data entry.

Batch Operations¶

You can batch-enable, disable, export, or delete multiple Pipeline files.

Note: This feature is only visible to workspace owners, administrators, and standard members; read-only members do not see this option.

Import¶

You can create a Pipeline by importing a JSON file.

Note: The imported JSON file must be a configuration JSON file from Guance.

Official Pipeline Library¶

Guance provides a standard official library for log data, helping you quickly structure your log data.

In the Guance workspace Logs > Pipelines, click Official Pipeline Library to view the built-in standard Pipeline files, including nginx, apache, redis, elasticsearch, mysql, etc.

Select and open any Pipeline file, such as apache.p, to see the built-in parsing rules. If you need to customize modifications, click the Clone button in the top-right corner.

The Pipeline file name is automatically generated based on the selected log source. Click Confirm to create a custom Pipeline file.

Note: DataKit will automatically retrieve the official library Pipeline file. If the cloned custom Pipeline file has the same name as the official Pipeline, DataKit will prioritize the newly created custom Pipeline file configuration. If the cloned custom Pipeline file has a different name from the official Pipeline, you need to modify the corresponding Pipeline file name in the relevant collector's Pipeline.

Precautions¶

Pipelines can perform the following operations on data collected by DataKit:

• Add, delete, or modify the values or data types of field and tag;

• Convert field to tag;

• Modify the name of the Measurement;

• Discard current data (drop());

• Terminate the execution of the Pipeline script (exit()).

When using Pipelines to process different data types, the original data structure may be affected. It is recommended to confirm that the data processing results meet expectations through debugging before use.