Skip to content

Full-Line Index


What Is Full-Line Index

Full-Line Index is a log full-text search mode. After it is enabled, the system includes all business fields in logs into the full-text index. When querying, you do not need to specify field names in advance; entering the value of any business field retrieves logs containing that value.

For example, a log contains the following fields:

{
  "service": "order-api",
  "trace_id": "8d4f2a",
  "order_id": "O-1001",
  "amount": 128.5
}

After Full-Line Index is enabled, searching directly for 8d4f2a, O-1001, or 128.5 retrieves the log.

Full-Line Index is suitable for the following scenarios:

  • Logs have been extracted into multiple structured fields and no longer rely on an entire message;
  • You need to quickly search across different business fields such as trace_id, order IDs, and user IDs;
  • A log index contains multiple log structures, making it impossible to predefine a unified search field for all logs;
  • You want to keep field-based query capability while supporting full-text search across business fields.

Differences from Message-Only Indexing

Item message-only index Full-Line Index
Full-text search scope Only the message field All business fields, including the existing message
Whether raw logs need to be retained message must be retained Not required; can be retained or deleted after field extraction
Suitable data Plain-text logs, unstructured logs JSON logs, structured logs extracted by Pipeline
Full-text indexed field message variant

Full-Line Index only changes the full-text search scope; it does not replace field filtering. For extracted fields, you can still use field conditions such as service:order-api for precise filtering, aggregation, or analysis.

How It Works

The system uses variant to uniformly carry the business fields in the Full-Line Index and creates a full-text index only for variant.

No message:
business fields → variant → full-text index

With message:
message + other business fields → variant → full-text index

The specific rules are as follows:

  • Full-Line Index includes the business fields in logs, not system fields;
  • When a log has no message, other business fields can still participate in full-text search;
  • When a log has message, the system does not delete the field but writes it into variant as an ordinary business field;
  • The system does not create two full-text indexes for both message and variant;
  • The same log index can store logs with and without message at the same time.

Whether message exists is determined by the original log content and the actual processing results of DataKit and Pipeline. Enabling Full-Line Index does not require deleting message.

Using or Switching to Full-Line Index

When creating a new log index, the system uses Full-Line Index by default, with no additional setup required.

If the original index has query acceleration configured for JSON Path in the message field, record the related configuration before switching, and reconfigure the corresponding JSON Path query acceleration after switching. For details, see JSON Path Embedded Field Query Acceleration.

For existing log indexes that still use the message-only index field, perform the following operations:

  1. Go to Logs > Index;
  2. Edit the target log index;
  3. Expand Advanced Options;
  4. Under Full-Text Index Field, select Full-Line Index;
  5. Confirm the impact scope and save the configuration.
Note

Switching from a message-only index to Full-Line Index is a one-way operation. After saving, you cannot switch back to a message-only index. Confirm related queries and data usage before performing this operation.

Prerequisites

The current workspace must support Full-Line Index. If this option is not shown on the page, check the workspace version and related feature permissions.

Querying with Full-Line Index

After configuration is complete and logs are written, go to Logs > Explorer and select the corresponding log index.

The Explorer supports methods such as text search, field filtering, combined search, JSON search, and DQL queries. For the complete search syntax and usage instructions, see Explorer Search.

Enter the value of a business field in the search box to search across all business fields without specifying a field name.

Using the order log above as an example:

  • Entering O-1001 matches order_id;
  • Entering 8d4f2a matches trace_id;
  • Entering order-api matches service;
  • When the log retains message, you can also search for content in message.

Text search tokenizes the input content. To match complete, continuous content, use ASCII double quotation marks to wrap the search content. For more information, see Text Search.

Field Filtering

If you already know the field name, you can still use field conditions to narrow the query scope. For example:

service:order-api

Full-text search suits scenarios where it is unclear which field the content is in; field filtering suits scenarios where the field name is known and precise filtering or aggregation analysis is needed. The two can be combined. For the complete field filtering syntax, see Filtering.

Query Notes

  • Full-Line full-text search matches only business fields, not system fields;
  • When a log has no message, the Explorer combines the current log's business fields to display the log content;
  • System fields are not part of the Full-Line Index log content;
  • If business fields have not yet been extracted from the raw log, Full-Line Index can only search the fields that currently exist. Therefore, for JSON or plain-text logs, extract the required fields first using the methods below.

Extracting Log Fields

Full-Line Index indexes business fields that already exist in logs, but it does not automatically parse and split the content in message. If the raw log is still an entire block of JSON or plain text, you can use DataKit or Pipeline to extract the content into structured fields.

After field extraction, whether to retain the original message depends on actual requirements:

  • When you need to view the complete original text or maintain compatibility with existing usage habits, you can retain message;
  • When sufficient structured fields already exist and the original text no longer needs to be retained, you can delete message;
  • Regardless of whether message is retained, extracted business fields can participate in Full-Line full-text indexing.

Extracting JSON Fields with DataKit

Suitable for logs where each line is a standard JSON object and you want to directly extract all top-level fields. DataKit 2.9.0 and later can enable json_as_fields.

After it is enabled, DataKit converts the top-level properties of the JSON root object into log fields after character decoding, ANSI cleaning, and multi-line merging, and then executes the Pipeline.

Host Log Collection

Edit logging.conf:

[[inputs.logging]]
  logfiles = ["/var/log/order/*.json"]
  source = "order"
  service = "order-api"
  json_as_fields = true

After saving the configuration, restart DataKit.

Kubernetes Container Log Collection

You can enable JSON field mode for a specific container via Pod Annotation:

metadata:
  annotations:
    datakit/order-api.logs: >-
      [{"source":"order","service":"order-api","json_as_fields":true}]

Here, order-api is the container name. You can also use the same JSON configuration in the container environment variable DATAKIT_LOGS_CONFIG.

Container Configuration Limitations

json_as_fields currently supports only JSON log configuration in container environment variables and Pod Annotations/Labels. The ClusterLoggingConfig CRD is not supported yet.

Log Streaming

Enable it in logstreaming.conf:

[inputs.logstreaming]
  json_as_fields = true

json_as_fields is a collector configuration, not an HTTP URL parameter. This configuration does not apply to the influxdb, firelens, or firehose types.

Extraction Results

Raw log:

{"timestamp":"2026-08-18T10:00:00+08:00","level":"INFO","service":"order-api","trace_id":"8d4f2a","order_id":"O-1001","amount":128.5,"labels":{"channel":"web"}}

After successful conversion:

  • timestamp, level, service, trace_id, order_id, and amount become independent fields;
  • The labels object is saved as a compact JSON string;
  • DataKit no longer separately generates a message field that stores the entire raw JSON;
  • If the raw JSON itself contains message, this field is retained normally.

The main field conversion rules are as follows:

  • Top-level strings, booleans, integers, and decimals keep their types; objects and arrays are saved as compact JSON strings; null is ignored;
  • When the JSON is invalid, the root node is not an object, or there are no valid fields, it falls back to the original message;
  • . in field names is converted to _, newline characters are converted to spaces, and field names are up to 256 bytes;
  • Each log retains up to 1024 fields, excluding tags;
  • JSON fields overwrite same-named tags or fields in the collector;
  • time, source, date, and storage_index in JSON are renamed to json_time, json_source, json_date, and json_storage_index, respectively.

For the complete rules, see JSON Field Mode.

Extracting Fields with Pipeline

Suitable for the following situations:

  • Only some fields need to be extracted;
  • Fields need to be renamed or converted;
  • Logs are not standard JSON;
  • An existing Pipeline processing flow needs to be reused.

Extracting All Top-Level Scalar Fields

DataKit 2.2.0 and later can use json_all():

# Raw data:
# {"service":"order-api","status":"info","trace_id":"8d4f2a","order_id":"O-1001","amount":128.5}

json_all(_, key_patterns=["*"])

After processing, independent fields such as service, status, trace_id, order_id, and amount are obtained.

json_all() extracts only top-level strings, numbers, and booleans. It does not recursively expand objects or arrays, and it does not retain null. When neither include_keys nor key_patterns is configured, no fields are extracted. To extract all top-level scalar fields, you must explicitly set key_patterns=["*"].

Extracting Only Specified Fields

json(_, service)
json(_, level, status)
json(_, trace_id)
json(_, order_id)
json(_, amount)

This approach controls the field scope included in the Full-Line Index and can also unify fields from different logs under the same names.

Objects or arrays are not extracted by json_all(). To retain such content, use json() with specified fields for extraction; the extraction result is saved as a JSON string.

Deleting Message After Field Extraction

When extracting fields with Pipeline, the raw log is still saved in message by default. If structured fields already meet the display and query requirements, you can call drop_origin_data() after field extraction to stop saving the original content.

json_all(_, key_patterns=["*"])
drop_origin_data()

You can also delete it after extracting specified fields:

json(_, service)
json(_, level, status)
json(_, trace_id)
json(_, order_id)
json(_, amount)
drop_origin_data()

If the JSON itself contains message and this field is also extracted, you can retain or delete it as needed. When it is explicitly not needed, use:

json_all(_, key_patterns=["*"])
drop_origin_data()
drop_key(message)

The three functions serve different purposes:

  • drop_origin_data(): Stops outputting the original text saved during initialization; the entire log is still uploaded;
  • drop_key(message): Deletes the extracted message field;
  • drop(): Discards the entire log. The log is not uploaded, and this function cannot be used to delete message.
Do not use drop() to delete message

drop() marks the current log as pending discard. After the Pipeline finishes execution, the entire log is not uploaded.

For more syntax, see json(), json_all(), drop_origin_data(), and drop_key().

Validating Pipeline Locally

After saving the Pipeline script, you can run the following on the host where DataKit is installed:

datakit pipeline -P full_line_json.p \
  -T '{"service":"order-api","status":"info","trace_id":"8d4f2a","order_id":"O-1001","amount":128.5}'

Check that the output meets the following expectations:

  • All fields that need to be searched are extracted;
  • If drop_origin_data() is configured, the original message is deleted;
  • The log is not marked as drop: true;
  • Fields such as status and log time meet business expectations.

FAQ

Does Full-Line Index require deleting message?

No. When message exists, the system writes it into variant as an ordinary business field and does not create a separate full-text index for message. Whether to delete message should be decided based on raw content retention and display requirements.

After enabling Full-Line Index, why can I still not search JSON fields in message?

If the entire JSON is still just the string content of message, its attributes are not independent business fields. Although you can search text in message, you cannot directly use these attributes for field filtering or aggregation. It is recommended to use DataKit or Pipeline to extract the required fields first.

How does the log Explorer display content when there is no message?

When a log has no message, the log Explorer combines the current log's business fields to display the log content. System fields are not part of the log content.

Can the same index contain logs with and without message at the same time?

Yes. Both types of logs write business fields into variant according to unified rules, and variant participates in full-text indexing.

Can Full-Line Index search system fields?

No. Full-Line Index includes only business fields, not system fields. System fields can still be queried through field filtering and other methods.

Can it be restored to a message index after switching to Full-Line Index?

No. Switching from a message-only index to Full-Line Index is a one-way operation and cannot be restored after saving. Existing message-only indexes that have not been switched can continue to be used in the original way.

Feedback

Is this page helpful?