Comprehensive Guide to DataKit Log Collection¶

Introduction¶

Log collection is one of the core capabilities of Guance DataKit. It processes log data collected actively or received passively and then uploads it to the Guance backend.

By data source, logs can be divided into two categories:

• Log file data:

  • Local disk files: fixed files stored on disk and collected by the logging collector
  • Container standard output: the container runtime persists stdout / stderr to disk, and the container collector discovers the corresponding log file path from the runtime and collects it
  • Log files inside containers: files are mounted out through Volume / VolumeMount so that DataKit can access them

• Network stream data:

  • TCP / UDP data: DataKit listens on the corresponding TCP / UDP ports and receives log data
  • Logstreaming: data is received over HTTP

These data types differ substantially in both usage scenarios and configuration methods. The following sections explain them in detail.

Usage Scenarios and Configuration Methods¶

Collecting Local Disk Files¶

This section only describes the host-deployed DataKit scenario. If DataKit is deployed in Kubernetes, collecting local files is not recommended.

First, enable the logging collector. Go to the conf.d/log directory under the DataKit installation path, copy logging.conf.sample, and rename it to logging.conf, for example:

  [[inputs.logging]]
   # List of log files. Absolute paths are supported, and glob rules can be used
   # to specify files in batches.
   # Absolute paths with explicit file suffixes are recommended.
   # Keep the scope as narrow as possible to avoid collecting archives or binary files.
   logfiles = [
    "/var/log/*.log",           # all .log files under this path
    "/var/log/*.txt",           # all .txt files under this path
    "/var/log/sys*",            # all files prefixed with sys under this path
    "/var/log/syslog",          # Unix-style file path
    "c:/path/with spaces/some.txt", # Windows-style file path
   ]

   ## Socket currently supports two protocols: TCP and UDP.
   ## It is recommended to expose only intranet ports to avoid security risks.
   ## File and socket collection can run together in one collector instance.
   sockets = [
    "tcp://0.0.0.0:9540",
    "udp://0.0.0.0:9541",
   ]

  # File path filters using glob rules. A file will not be collected if it matches
  # any filter rule.
   ignore = [""]

   # Data source. If empty, the default value is 'default'.
   source = ""

   # Service tag to add. If empty, $source is used by default.
   service = ""

   # Pipeline script path. If empty, $source.p is used. If $source.p does not exist,
   # no pipeline is applied.
   pipeline = ""

   # Character encoding. Use the default empty value unless explicit conversion is needed.
   # `utf-8`,`gbk`
   character_encoding = ""

   ## Regular expression used to identify the first line of a log entry.
   ## Example: ^\d{4}-\d{2}-\d{2} matches lines starting with yyyy-mm-dd.
   ## Lines that match are treated as valid new log entries; otherwise they are
   ## appended to the previous valid entry.
   ## Use triple single quotes '''this-regexp''' to avoid excessive escaping.
   ## Regular expression syntax: https://golang.org/pkg/regexp/syntax/#hdr-syntax
   # multiline_match = '''^\s'''

   ## Whether to enable multiline processing.
   enable_multiline = true
   ## Deprecated: use enable_multiline. Existing configurations remain compatible.
   # auto_multiline_detection = true
   ## Additional automatic multiline patterns. These patterns are checked only after
   ## the built-in automatic rules do not match. If empty, only built-in rules are used.
   auto_multiline_extra_patterns = []

   ## Remove ANSI escape sequences such as terminal color codes.
   remove_ansi_escape_codes = false

   ## Ignore inactive files. For example, if the last modification time was 20 minutes
   ## ago and exceeds the configured 10m threshold, the file will be ignored.
   ## Supported time units: "ms", "s", "m", "h".
   ignore_dead_log = "1h"

   ## Whether to read from the beginning of the file.
   from_beginning = false

   # Custom tags
   [inputs.logging.tags]
   # some_tag = "some_value"
   # more_tag = "some_other_value"

This is a basic logging.conf. The functions of fields such as multiline_match and pipeline are described later in this document.

Collecting Container Standard Output¶

Collecting the standard output of containerized applications is the most common approach. You can view such logs with commands like docker logs or crictl logs.

The container runtime persists stdout / stderr to disk, and DataKit automatically obtains the corresponding logpath for collection.

If you want to customize collection behavior, you can do so through container environment variables or Kubernetes Pod annotations.

• The configuration key can take the following forms:
  • For container environment variables, the key is always datakit_logs_config
  • Pod annotations support two formats:
    • datakit/<container_name>.logs, where <container_name> must be replaced with the container name in the current Pod, which is useful in multi-container scenarios
    • datakit/logs, which applies to all containers in the Pod

• Example configuration value:

[
 {
  "disable" : false,
  "source" : "<your-source>",
  "service" : "<your-service>",
  "pipeline": "<your-pipeline.p>",
  "remove_ansi_escape_codes": false,
  "from_beginning"     : false,
  "tags" : {
   "<some-key>" : "<some_other_value>"
  }
 }
]

Field descriptions:

Complete examples:

  $ cat dockerfile
  from pubrepo.guance.com/base/ubuntu:18.04 as base
  run mkdir -p /opt
  run echo 'i=0; \n\
  while true; \n\
  do \n\
    echo "$(date +"%y-%m-%d %h:%m:%s") [$i] bash for loop examples. hello, world! testing output."; \n\
    i=$((i+1)); \n\
    sleep 1; \n\
  done \n'\
  >> /opt/s.sh
  cmd ["/bin/bash", "/opt/s.sh"]

  ## Build the image
  $ docker build -t testing/log-output:v1 .

  ## Start the container and add the datakit_logs_config environment variable
  $ docker run --name log-output -env datakit_logs_config='[{"disable":false,"source":"log-source","service":"log-service"}]' -d testing/log-output:v1

apiversion: apps/v1
kind: deployment
metadata:
 name: log-demo-deployment
 labels:
  app: log-demo
spec:
 replicas: 1
 selector:
  matchlabels:
   app: log-demo
 template:
  metadata:
   labels:
    app: log-demo
   annotations:
    ## Add the configuration and bind it to the log-output container
    datakit/log-output.logs: |
     [{
       "disable": false,
       "source": "log-output-source",
       "service": "log-output-service",
       "tags" : {
        "some_tag": "some_value"
       }
     }]
  spec:
   containers:
   - name: log-output
    image: pubrepo.guance.com/base/ubuntu:18.04
    args:
    - /bin/sh
    - -c
    - >
     i=0;
     while true;
     do
      echo "$(date +'%f %h:%m:%s') [$i] bash for loop examples. hello, world! testing output.";
      i=$((i+1));
      sleep 1;
     done

$ kubectl apply -f log-output.yaml
#...

The value of multiline_match is double-escaped. Four backslashes represent one actual backslash. For example, \"multiline_match\":\"^\\\\d{4}\" is equivalent to "multiline_match":"^\d{4}". Example:

$ kubectl annotate pods my-pod datakit/logs="[{\"disable\":false,\"source\":\"log-source\",\"service\":\"log-service\",\"pipeline\":\"test.p\",\"multiline_match\":\"^\\\\d{4}-\\\\d{2}\"}]"
#...

If a Pod or container log is already being collected, adding configuration later with kubectl annotate does not take effect.

Collecting Log Files Inside Containers¶

For log files inside a container, the main difference from standard output collection is that you must specify the file path. The other configuration items are mostly the same.

You still use container environment variables or Kubernetes Pod annotations. The key and value formats are almost identical to those described earlier.

Complete examples:

  $ cat dockerfile
  from pubrepo.guance.com/base/ubuntu:18.04 as base
  run mkdir -p /opt
  run echo 'i=0; \n\
  while true; \n\
  do \n\
    echo "$(date +"%y-%m-%d %h:%m:%s") [$i] bash for loop examples. hello, world! testing output." >> /tmp/opt/1.log; \n\
    i=$((i+1)); \n\
    sleep 1; \n\
  done \n'\
  >> /opt/s.sh
  cmd ["/bin/bash", "/opt/s.sh"]

  ## Build the image
  $ docker build -t testing/log-to-file:v1 .

  ## Start the container and add the datakit_logs_config environment variable.
  ## Note the required escaping.
  ## When collecting a non-stdout path, "type" and "path" are required fields,
  ## and a Volume must be created for the collection path.
  ## For example, to collect `/tmp/opt/1.log`, add an anonymous Volume for `/tmp/opt`.
  $ docker run --env datakit_logs_config="[{\"disable\":false,\"type\":\"file\",\"path\":\"/tmp/opt/1.log\",\"source\":\"log-source\",\"service\":\"log-service\"}]" -v /tmp/opt -d testing/log-to-file:v1

apiversion: apps/v1
kind: deployment
metadata:
 name: log-demo-deployment
 labels:
  app: log-demo
spec:
 replicas: 1
 selector:
  matchlabels:
   app: log-demo
 template:
  metadata:
   labels:
    app: log-demo
   annotations:
    ## Add the configuration and bind it to the logging-demo container
    ## Both file and stdout collection are configured here.
    ## To collect "/tmp/opt/1.log", add an emptyDir Volume for "/tmp/opt" first.
    datakit/logging-demo.logs: |
     [
      {
       "disable": false,
       "type": "file",
       "path":"/tmp/opt/1.log",
       "source": "logging-file",
       "tags" : {
        "some_tag": "some_value"
       }
      },
      {
       "disable": false,
       "source": "logging-output"
      }
     ]
  spec:
   containers:
   - name: logging-demo
    image: pubrepo.guance.com/base/ubuntu:18.04
    args:
    - /bin/sh
    - -c
    - >
     i=0;
     while true;
     do
      echo "$(date +'%f %h:%m:%s') [$i] bash for loop examples. hello, world! testing output.";
      echo "$(date +'%f %h:%m:%s') [$i] bash for loop examples. hello, world! testing output." >> /tmp/opt/1.log;
      i=$((i+1));
      sleep 1;
     done
    volumemounts:
    - mountpath: /tmp/opt
     name: datakit-vol-opt
   volumes:
   - name: datakit-vol-opt
    emptydir: {}

$ kubectl apply -f logging.yaml
#...

Receiving TCP / UDP Data¶

Configure sockets in logging.conf. Existing logfiles can remain enabled if file logs should also be collected. For example:

[[inputs.logging]]
 sockets = [
  "tcp://127.0.0.1:9540",
  "udp://127.0.0.1:9541",
 ]

 source = "socket-testing"

With this configuration, DataKit listens on TCP port 9540 and UDP port 9541 at the same time. Log data received on both ports uses socket-testing as the unified source.

Use the Linux nc command to test it:

# Send TCP data. Press ctrl-c to quit.
$ nc 127.0.0.1 9540
This is a tcp message-1
This is a tcp message-2
This is a tcp message-3

# Send one UDP message
$ echo "This is a udp message" | nc -w 3 -v -u 127.0.0.1 9541
Connection to 127.0.0.1 9541 port [udp/*] succeeded!

Note: Because UDP data has no context, multiline matching does not apply.

Receiving HTTP Data¶

Start an HTTP server to receive log text data and report it to Guance. The fixed HTTP URL is /v1/write/logstreaming, that is, http://datakit_ip:port/v1/write/logstreaming.

Note: If DataKit is deployed in Kubernetes as a DaemonSet, you can access it through a Service at http://datakit-service.datakit:9529.

[inputs.logstreaming]
  ignore_url_tags = false

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.logstreaming.threads]
  #   buffer = 100
  #   threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.logstreaming.storage]
  #   path = "./log_storage"
  #   capacity = 5120

Supported Logstreaming Parameters¶

Logstreaming supports flexible processing of log data through query parameters in the HTTP URL. The supported parameters are as follows:

• type: data format. Currently only influxdb and firelens are supported.
  • When type=influxdb (/v1/write/logstreaming?type=influxdb), the data itself is line protocol format (the default precision is s). DataKit only adds built-in tags and performs no further processing.
  • When type=firelens (/v1/write/logstreaming?type=firelens), the data format should be JSON containing multiple log entries.
  • When this value is empty, DataKit performs line splitting, Pipeline processing, and related operations.
• source: identifies the data source, that is, the measurement in line protocol. For example, nginx or redis (/v1/write/logstreaming?source=nginx).
  • This value is ignored when type=influxdb.
  • The default value is default.
• service: adds the service tag, for example /v1/write/logstreaming?service=nginx_service.
  • By default, it uses the value of source.
• pipeline: specifies the Pipeline name to use, for example nginx.p (/v1/write/logstreaming?pipeline=nginx.p).
• tags: adds custom tags separated by commas ,, for example key1=value1,key2=value2 (/v1/write/logstreaming?tags=key1=value1,key2=value2).

Logstreaming Examples¶

• Fluentd with influxdb output documentation
• Fluentd with HTTP output documentation
• Logstash with influxdb output documentation
• Logstash with HTTP output documentation

You only need to configure the output host as the logstreaming URL (http://datakit_ip:port/v1/write/logstreaming) and append the required parameters.

Log Collection Details¶

Discovering Log Files¶

Configuring File Paths with Glob¶

Using glob rules makes it easier to specify log files, automatically discover new files, and filter files.

In addition to the standard glob rules above, the collector also supports ** for recursive traversal, as shown in the examples below. For more on glob syntax, see this article.

Discovering Local Log Files¶

Ordinary log files are the most common form. A process writes readable log records directly to disk files. Examples include well-known frameworks such as log4j, or a simple command like echo "this is log" >> /tmp/log.

In most cases, the file path of this kind of log is fixed. For example, on Linux the MySQL log path is usually /var/log/mysql/mysql.log. If you run the DataKit MySQL collector, it looks for that path by default. However, because log storage paths are configurable, DataKit cannot cover every case, so manual file-path configuration is necessary.

DataKit uses glob expressions to configure file paths. Wildcards are used to locate file names, though you can also specify an exact path.

For example, assume the following files exist:

$ tree /tmp

/tmp
├── datakit
│   ├── datakit-01.log
│   ├── datakit-02.log
│   └── datakit-03.log
└── mysql.d
  └── mysql
    └── mysql.log

3 directories, 4 files

In the DataKit logging collector, you can configure logfiles to specify which files to collect:

• Collect all files in the datakit directory with /tmp/datakit/*
• Collect all files whose name contains datakit with /tmp/datakit/datakit-*log
• Collect mysql.log, where there are two intermediate directories, mysql.d and mysql:
  • Exact path: /tmp/mysql.d/mysql/mysql.log
  • Single-star pattern: /tmp/*/*/mysql.log, which is rarely used
  • Double star: /tmp/**/mysql.log, which replaces the intermediate multi-level directory structure and is the more concise, common approach

After you specify file paths with glob rules, DataKit periodically scans the disk for matching files. If a file is found and is not already in the collection list, it is added and collected.

Locating Container Standard Output Logs¶

There are two ways for logs to be produced inside a container:

• They are written directly to a mounted disk directory. From the host's perspective, this is the same as an ordinary log file stored at a fixed disk path.
• They are written to stdout / stderr, and then persisted and managed by the container runtime. This is the more common approach. The persisted path can be obtained through the runtime API.

DataKit connects to the Docker or containerd socket and calls the runtime API to retrieve the container logpath, similar to running docker inspect --format='{{.logpath}}' <container_id>:

$ docker inspect --format='{{.logpath}}' <container_id>

/var/lib/docker/containers/<container_id>/<container_id>-json.log

After DataKit gets the container logpath, it creates a log collection task with that path and the corresponding configuration.

Locating File Paths Inside Containers¶

To collect files inside containers, the target directory must first be mounted out. This section only describes how to locate such paths in Kubernetes.

Using the earlier "Collecting Log Files Inside Containers" example, suppose you want to collect /tmp/opt/1.log inside the container. First create an emptyDir Volume for /tmp/opt. Then run the following command on the host (using containerd as the runtime in this example):

$ crictl inspect <container_id>
#...

You will see a mount definition like the following:

  "mounts": [
    {
      "containerpath": "/tmp/opt",
      "hostpath": "/var/lib/kubelet/pods/<pod_id>/volumes/kubernetes.io~empty-dir/datakit-vol-opt",
      "propagation": "propagation_private",
      "selinuxrelabel": false
    }
  ]

Here, containerpath is the mounted path inside the container, while hostpath is the corresponding directory on the host. Under that host directory there is a file named 1.log, which is the target log file.

DataKit gets this hostpath from the container runtime and appends the configured target file name 1.log, resulting in /var/lib/kubelet/pods/<pod_id>/volumes/kubernetes.io~empty-dir/datakit-vol-opt/1.log.

Because DataKit mounts the host root directory into the container at /rootfs, the final path actually collected by DataKit becomes /rootfs/var/lib/kubelet/pods/<pod_id>/volumes/kubernetes.io~empty-dir/datakit-vol-opt/1.log.

When DataKit is deployed as a Kubernetes DaemonSet, it creates a VolumeMount that mounts the host root directory / into the DataKit container at /rootfs, allowing the DataKit process to access host files in read-only mode.

File Filtering¶

When you use glob rules to configure file paths, many files may be discovered. Some of them may remain unchanged for a long time, and you may not want to open or collect them. This is what ignore_dead_log is for.

ignore_dead_log is a duration setting. For example, 1h means 1 hour and 20m means 20 minutes.

When DataKit discovers a file, it gets the file's last modification time and calculates how long it has been since that file was last updated. If the interval exceeds the configured threshold, such as 1 hour or 20 minutes, the file is ignored.

When the file is discovered again later, the same judgment is performed again. Only when the file changes will DataKit create a collection task for it.

For files that are already open and being collected, if they remain unchanged beyond the configured duration, collection exits and the file handle is closed to avoid long-term resource occupation.

For both container standard output and files inside containers, ignore_dead_log is fixed at 1h.

Collection Start Position¶

The start position for file collection is important. If you always read from the head, duplicate collection may occur; if you always read from the tail, some data may be missed.

DataKit chooses the starting position according to the following priority:

1. Continue from the previous collection position DataKit calculates a hash from file metadata and records the current collection position of the file in /usr/local/datakit/cache/logtail.history. If the position can be found accurately and is less than or equal to the current file size, which means the file has not been truncated, collection resumes from that position.

2. Force reading from the head with from_beginning If from_beginning=true and no historical position record is found for the file, DataKit starts from the head.

3. Decide based on from_beginning_threshold_size from_beginning_threshold_size is an integer. When DataKit discovers a new file, it starts from the head if the file size is smaller than from_beginning_threshold_size; otherwise it starts from the tail. The default is 1000*1000*20, that is, 20 MB.

4. Start from the tail By default, DataKit reads from the tail.

When should you use from_beginning?

• When you need to collect an entire log file, for example when backing up historical data. Note that historical files often remain unchanged for a long time, so they may be filtered out by ignore_dead_log.
• On non-Linux systems, when log volume is very large, for example more than 20 MB is written within 10 seconds, which exceeds from_beginning_threshold_size, you may need to enable from_beginning.

Regarding the second point:

DataKit scans for new log files every 10 seconds. If a file happens to fall on the boundary of this interval and is already larger than 20 MB when discovered 10 seconds later, the default policy would start from the tail. In that case, from_beginning should be forced on so that collection starts from the head. On Linux, because inotify event notification exists, DataKit can theoretically detect the file as soon as it appears and start collection immediately, so the interval is too short to exceed 20 MB. In that case, enabling from_beginning is usually unnecessary.

Character Encoding Conversion¶

DataKit supports collecting files encoded in utf-8 and gbk.

Multiline Data Assembly¶

By identifying the characteristics of the first line of a multiline log, DataKit can determine whether a line starts a new log entry. If a line does not match that characteristic, it is treated as an append to the previous multiline log.

For example, most logs are left-aligned, but some logs are not, such as stack traces produced when a program crashes. This kind of text is a typical multiline log.

In DataKit, multiline log boundaries are identified with regular expressions. A line matching the regex is treated as the start of a new log entry. All subsequent lines that do not match are appended to that new entry until another line matches the regex and starts the next entry.

In logging.conf, configure it like this:

  enable_multiline = true
  multiline_match = '''put the specific regular expression here''' # triple ASCII single quotes are recommended

The regex syntax used by the log collector is documented here.

Assume the raw data is:

2020-10-23 06:41:56,688 info demo.py 1.0
2020-10-23 06:54:20,164 error /usr/local/lib/python3.6/dist-packages/flask/app.py exception on /0 [get]
traceback (most recent call last):
 file "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
  response = self.full_dispatch_request()
zerodivisionerror: division by zero
2020-10-23 06:41:56,688 info demo.py 5.0

When multiline_match is configured as ^\\d{4}-\\d{2}-\\d{2}.*, meaning it matches lines starting with values such as 2020-10-23,

the collector produces the following three line-protocol points, corresponding to original lines 1, 2, and 8. As you can see, the traceback ... block on lines 3 to 6 does not become a separate log. Instead, it is appended to the message field of the previous log that starts on line 2.

testing,filename=/tmp/094318188 message="2020-10-23 06:41:56,688 info demo.py 1.0" 1611746438938808642
testing,filename=/tmp/094318188 message="2020-10-23 06:54:20,164 error /usr/local/lib/python3.6/dist-packages/flask/app.py exception on /0 [get]
traceback (most recent call last):
 file \"/usr/local/lib/python3.6/dist-packages/flask/app.py\", line 2447, in wsgi_app
  response = self.full_dispatch_request()
zerodivisionerror: division by zero
" 1611746441941718584
testing,filename=/tmp/094318188 message="2020-10-23 06:41:56,688 info demo.py 5.0" 1611746443938917265

In addition to handwritten multiline rules, DataKit provides automatic multiline detection. When enable_multiline is enabled and multiline_match is empty, DataKit first checks the line against its built-in rules. If none of those rules match, it checks the additional rules in auto_multiline_extra_patterns. The old auto_multiline_detection option is deprecated but remains compatible: setting it to true also enables multiline processing.

During automatic detection, built-in rules are divided into numeric, alphabetic, and symbolic groups based on the first character of the line, and only the corresponding group is checked. Whitespace characters such as spaces and tabs usually mark stack-trace continuation lines and therefore do not enter a built-in rule group. To recognize a new log entry that begins with whitespace, add an explicit rule to auto_multiline_extra_patterns. These customer-defined rules are supplementary rather than mixed into the built-in rule priority: DataKit checks them only when no built-in rule matches, or when the line begins with a space or tab. If no rule matches after the full process, the line is treated as a mismatch.

The follow-up behavior is the same as normal multiline collection. On a successful match, the currently accumulated multiline entry is flushed out and the current line starts a new entry. On a mismatch, the current line is appended to the end of the existing multiline buffer.

Because multiple multiline configurations may exist, the priority order is:

1. If enable_multiline is disabled, multiline processing is not used. The legacy setting auto_multiline_detection = true enables it through the compatibility path.
2. If multiline processing is enabled and multiline_match is not empty, only that rule is used.
3. If multiline processing is enabled and multiline_match is empty, DataKit checks the built-in automatic rules first and then auto_multiline_extra_patterns if no built-in rule matches.
4. For container logs, a matching source rule from logging_source_multiline_map is applied last and overrides any explicit or automatic multiline rule selected above.

The built-in automatic multiline rules mainly cover common first-line formats, including:

• RFC3339/ISO8601, yyyy-mm-dd HH:MM:SS, slash-separated dates, dot-separated dates, and compact dates;
• timestamps or log levels enclosed in [] or ();
• Syslog, Apache/Nginx access and error logs, Kubernetes glog/klog, Android logcat, and kernel ring-buffer formats;
• logfmt/key-value styles such as time=..., ts=..., and level=...;
• lines beginning with levels such as INFO, ERROR, and WARN;
• common component formats including Redis, MongoDB, RabbitMQ/Erlang, Ruby logger, Gin, and MySQL/TiDB slow logs;
• JSON objects and arrays, and explicit Go stack entry lines such as panic: and fatal error:.

The default rules identify the start of a new log entry; they are not complete field-parsing rules. Common Python and Java lines such as Traceback, Exception, and Caused by are usually continuations of the preceding error entry, so they are not treated as default first-line patterns.

Filtering Special Escape Sequences¶

Logs may contain unreadable special characters, such as terminal color codes. You can set remove_ansi_escape_codes=true to strip them out.

Here, "special characters" specifically refers to color escape sequences embedded in the text. For example, the following command prints a red word red in the terminal:

$ red='\033[0;31m' && nc='\033[0m' && print "${red}red${nc}"
#...

If these characters are not handled, the collected log will contain \033[0;31m, which is not only unsightly and wasteful in storage, but may also negatively affect downstream data processing. For this reason, DataKit provides filtering for these color escape sequences.

The benchmark results for this processing are shown below for reference only:

goos: linux
goarch: arm64
pkg: ansi
benchmarkstrip
benchmarkstrip-2     653751       1775 ns/op       272 b/op     3 allocs/op
benchmarkstrip-4     673238       1801 ns/op       272 b/op     3 allocs/op
pass
ok   ansi   2.422s

The processing time added per text record is roughly 1700 ns. If this feature is disabled, there is no additional overhead.

Pipeline Field Extraction¶

Pipeline is mainly used to parse unstructured text data or to extract specific information from structured text such as JSON.

For log data, the main extracted fields are:

• time: the timestamp when the log was generated. If time is not extracted or parsing fails, the current system time is used.
• status: the log level. If status is not extracted, status defaults to unknown.

Valid status values are listed below, case-insensitive:

Example input:

12115:m 08 jan 17:45:41.572 # server started, redis version 3.0.6

Pipeline script:

add_pattern("date2", "%{monthday} %{month} %{year}?%{time}")
grok(_, "%{int:pid}:%{word:role} %{date2:time} %{notspace:severity} %{greedydata:msg}")
group_in(severity, ["#"], "warning", status)
cast(pid, "int")
default_time(time)

Result:

{
  "message": "12115:m 08 jan 17:45:41.572 # server started, redis version 3.0.6",
  "msg": "server started, redis version 3.0.6",
  "pid": 12115,
  "role": "m",
  "severity": "#",
  "status": "warning",
  "time": 1610127941572000000
}

Notes about Pipeline:

• If pipeline is empty in logging.conf, DataKit uses <source-name>.p by default. For example, if source is nginx, the default script is nginx.p.
• If <source-name>.p does not exist, Pipeline is not enabled.
• All Pipeline script files are stored under the pipeline directory of the DataKit installation path.

Retaining Specific Fields with a Whitelist¶

Log collection has the following built-in fields:

In some scenarios, many of these built-in fields are unnecessary. A whitelist feature is therefore provided so that only specified fields are retained.

Field whitelisting is supported only through environment variables, for example ENV_LOGGING_FIELD_WHITE_LIST='["host", "service", "filepath", "container_name"]'. The behavior is:

• If the whitelist is empty, all built-in fields are added
• If the whitelist is not empty and contains valid fields, for example ["filepath", "container_name"], only those fields are retained
• If the whitelist is not empty and every field is invalid, for example ["no-exist"] or ["no-exist-key1", "no-exist-key2"], the log is discarded

For tags coming from other sources, note the following:

• The whitelist does not affect DataKit global tags
• Debug fields enabled by ENV_ENABLE_DEBUG_FIELDS="true" are unaffected, including log_read_offset and log_file_inode from log collection, as well as Pipeline debug fields

Maximum Number of Collected Files¶

You can set the maximum number of simultaneously collected files through ENV_LOGGING_MAX_OPEN_FILES. For example, ENV_LOGGING_MAX_OPEN_FILES="1000" means up to 1000 files can be collected at the same time.

This helps avoid excessive DataKit resource consumption when glob rules match too many files.

ENV_LOGGING_MAX_OPEN_FILES is a global configuration that applies to both the logging collector and container log collection. The default value is 500.

More Collection Details¶

Source Setting for Container Log Collection¶

In container environments, the log source (source) is an important configuration item because it directly affects how logs are displayed on the page. Manually assigning a source to every container log would be impractical. If you do not manually configure the source, DataKit infers it automatically according to the following rules, in descending priority:

"Not manually specifying the container log source" means it is not specified in Pod annotations and is not specified in container.conf either. At present, container.conf does not provide a dedicated configuration item for container log source.

• The Kubernetes-assigned container name, taken from the container label io.kubernetes.container.name
• The container's own name, as visible through docker ps or crictl ps
• default, the fallback source

Limits for Extra-Large Multiline Logs¶

DataKit supports up to 512 KiB for a single log line. If the line exceeds 512 KiB, it is truncated and the next byte starts a new log entry.

For multiline logs, the limit is ENV_DATAWAY_MAX_RAW_BODY_SIZE * 0.8. By default this is 819 KiB. If the log exceeds this limit, it is also truncated and a new log entry is created.

How Disk Files Are Discovered¶

DataKit discovers log files through two mechanisms: periodic scans based on glob rules every 10 seconds, and inotify event notifications. inotify is supported only when DataKit runs on Linux.

Recommended File Rotation Strategies¶

Log rotation is a common behavior and is supported by most logging frameworks.

In general, rotation falls into two categories:

• File names change over time or sequence number, and new files are continuously created

For example, suppose there is a log file /opt/log/access-20241203-10.log. When it reaches a condition such as exceeding a configured size, it stops being written to, and a new file /opt/log/access-20241203-11.log is created for subsequent writes.

For this pattern, it is recommended to keep the total file count relatively small, ideally around 5. Too many files reduce file discovery performance.

The recommended collection path is /opt/log/*.log. DataKit will watch the directory or periodically search for matching log files. For old files that are no longer updated after collection ends, DataKit periodically cleans them up and releases the associated resources.

• The file name stays fixed and old files are produced by renaming

For example, suppose there is a file /opt/log/access-0.log. Once it is full, it is renamed to /opt/log/access-1.log as a backup, and a new /opt/log/access-0.log is created for continued writing.

This is how Docker file-json rotation works. When starting a Docker container, you can control the file size and number of rotated files with --log-opt max-size=100m and --log-opt max-file=3.

For this pattern, the collection path must not be *.log, otherwise backup files will also be collected and duplicates will occur.

The collection path should be /opt/log/access-0.log so that only the active file is collected.

There is also another rotation method similar to running truncate -s 0 on Linux, which truncates the file size to 0 bytes and clears the content. This is not recommended. It clears the file completely, and if DataKit has not yet collected the tail, that tail data is lost permanently.

FAQ¶

Remote File Collection Solution¶

On Linux, you can mount the log directory from the source host onto the DataKit host through NFS, and then configure the logging collector to read the mounted path.

operation not permitted on macOS¶

On macOS, because of system security policies, the DataKit log collector may fail to open files and report operation not permitted. See the Apple developer documentation for the solution.

How to Estimate Total Log Volume¶

Because log billing is based on the number of log entries, and because in most cases logs are written to disk, what you typically see first is disk usage, such as 100 GB of logs per day.

A practical estimate can be made with the following shell commands:

# Count how many lines are in 1 GB of logs
head -c 1g path/to/your/log.txt | wc -l

Sometimes you also need to estimate how much network traffic log collection may produce:

# Count the compressed size in bytes of 1 GB of logs
head -c 1g path/to/your/log.txt | gzip | wc -c

The value obtained here is the compressed byte count. Converting bytes to network bits by multiplying by 8 gives an approximate bandwidth estimate:

bytes * 2 * 8 /1024/1024 = xxx mbit

In practice, DataKit's compression ratio is not this high because it does not send 1 GB in one shot. Data is sent in batches, and a more realistic compression ratio is about 85%, meaning 100 MB is compressed to 15 MB. A rough estimate is therefore:

1gb * 2 * 8 * 0.15/1024/1024 = xxx mbit

Symbolic Links in Log Directories¶

Under normal circumstances, DataKit gets the log file path from the container runtime and then collects that file.

In some special environments, the directory containing the log file is itself a symbolic link. DataKit cannot know the final target in advance, so it cannot mount that directory automatically, which means the log file cannot be found or collected.

For example, suppose a container log file is found at /var/log/pods/default_log-demo_f2617302-9d3a-48b5-b4e0-b0d59f1f0cd9/log-output/0.log, but in the current environment /var/log/pods is a symbolic link pointing to /mnt/container_logs:

$ ls /var/log -lh
total 284k
lrwxrwxrwx 1 root root  20 oct 8 10:06 pods -> /mnt/container_logs/

In this case, DataKit must mount the host path /mnt/container_logs in order to collect logs correctly. For example, add the following to datakit.yaml:

  # omitted
  spec:
   containers:
   - name: datakit
    image: pubrepo.guance.com/datakit/datakit:1.16.0
    volumemounts:
    - mountpath: /mnt/container_logs
     name: container-logs
   # omitted
   volumes:
   - hostpath:
     path: /mnt/container_logs
    name: container-logs

This situation is uncommon. Usually you only need to do this when you already know the path is symlinked, or when you inspect DataKit logs and see collection errors.

Adjusting Log Collection by Container Image¶

By default, DataKit collects the standard output logs of all containers on the host, which can produce a large amount of data. You can filter containers by image or namespace.

  ## Use image as an example
  ## If the container image matches `datakit`, the container's logs will be collected
  container_include_log = ["image:datakit"]

  ## Ignore all kodo containers
  container_exclude_log = ["image:kodo"]

  ## This causes all containers to be filtered out
  ## For example, a container named `datakit` matches include and also matches exclude,
  ## so it is filtered and its logs are not collected.
  ## A container named `nginx` does not match include in the first place,
  ## so it is also filtered out.
  container_include_log = ["image_name:datakit"]
  container_exclude_log = ["image_name:*"]

  ## If the container matches either `image_name` or `namespace`,
  ## its logs will not be collected.
  container_include_log = []
  container_exclude_log = ["image_name:datakit", "namespace:datakit-ns"]

  - env:
   - name: ENV_INPUT_CONTAINER_CONTAINER_INCLUDE_LOG
    value: image:hello* # specify the image name or a wildcard pattern

  - env:
   - name: ENV_INPUT_CONTAINER_CONTAINER_EXCLUDE_LOG
    value: namespace:foo # do not collect container logs from this namespace

$ docker inspect --format '{{.config.image}}' <container_id>
#...

$ kubectl get pod -o=jsonpath="{.spec.containers[0].image}" <pod_name>
#...

[{
  "disable": false,
  "source": "logging-output"
}]