Several Approaches to Log Collection in Kubernetes Clusters¶

Introduction¶

Logs are critical for enterprise application systems, especially in Kubernetes environments where log collection becomes more complex. DataKit provides robust support for log collection across multiple environments and technology stacks. The following sections detail the usage of DataKit log collection.

Prerequisites¶

Log in to the Guance, navigate to Integration → Datakit → Kubernetes, and follow the instructions to install DataKit in the Kubernetes cluster. The datakit.yaml file used for deployment will be referenced in subsequent operations.

DataKit Advanced Configuration¶

1 Setting the Log Level¶

The default log level of DataKit is Info. To change the log level to Debug, add an environment variable in datakit.yaml.

        - name: ENV_LOG_LEVEL
          value: debug

2 Setting the Log Output Method¶

By default, DataKit outputs logs to /var/log/datakit/gin.log and /var/log/datakit/log. To avoid generating log files inside the container, add the following environment variables in datakit.yaml.

    - name: ENV_LOG
      value: stdout
    - name: ENV_GIN_LOG
      value: stdout

You can view the logs generated by DataKit using the kubectl logs command with the POD name.

kubectl logs datakit-2fnrz -n datakit #

Note: When ENV_LOG_LEVEL is set to debug, a large amount of logs will be generated. In this case, it is not recommended to set ENV_LOG to stdout.

Log Collection¶

1 stdout Collection¶

1.1 Collecting All stdout Logs¶

DataKit can collect container logs output to stdout. After deploying DataKit with datakit.yaml, the container collector is enabled by default.

        - name: ENV_DEFAULT_ENABLED_INPUTS
          value: cpu,disk,diskio,mem,swap,system,hostobject,net,host_processes,container

This will generate the configuration file /usr/local/datakit/conf.d/container/container.conf in the DataKit container. The default configuration collects all stdout logs except those from images starting with pubrepo.guance.com/datakit/logfwd.

  container_include_log = []  # equivalent to image:*
  container_exclude_log = ["image:pubrepo.guance.com/datakit/logfwd*"]

1.2 Customizing stdout Log Collection¶

To better distinguish log sources, add tags, and specify a custom log parsing pipeline file, use the annotations approach in the deployment YAML file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-demo-service
  labels:
    app: log-demo-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: log-demo-service
  template:
    metadata:
      labels:
        app: log-demo-service
      annotations:
        # Add the following section
        datakit/logs: |
          [
            {
              "source": "pod-logging-testing-demo",
              "service": "pod-logging-testing-demo",
              "pipeline": "pod-logging-demo.p",
              "multiline_match": "^\\d{4}-\\d{2}-\\d{2}"
            }
          ]

Annotations Parameter Description

source: Data source
service: Tag label
pipeline: Pipeline script name
ignore_status:
multiline_match: Regular expression to match a line of log. For example, a line starting with a date (e.g., 2021-11-26) is considered a single log line; if the following line does not start with such a date, it is considered part of the previous log line.
remove_ansi_escape_codes: Whether to remove ANSI escape codes, such as text color for standard output.

1.3 Excluding Container stdout Logs from Collection¶

When the container collector is enabled, it automatically collects logs output to stdout. To exclude specific logs, use the following methods.

1.3.1 Disabling stdout Log Collection for a POD¶

Add an annotation in the application deployment YAML file and set disable to true.

apiVersion: apps/v1
kind: Deployment
metadata:

...

spec:
  ...
  template:
    metadata:      
      annotations:
        ## Add the following content
        datakit/logs: |
          [
            {
              "disable": true  
            }
          ]

1.3.2 Redirecting Standard Output¶

If stdout log collection is enabled and container logs are also written to stdout, and you do not want to modify either, you can modify the startup command to redirect standard output.

java ${JAVA_OPTS}   -jar ${jar} ${PARAMS}  2>&1 > /dev/null

1.3.3 Filtering with the container Collector¶

For more convenient control over stdout log collection, it is recommended to override the container.conf file by using a ConfigMap to define container.conf, modifying the values of container_include_log and container_exclude_log, and then mounting it into DataKit. Modify datakit.yaml as follows:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
    #### container
    container.conf: |-  
      [inputs.container]
        docker_endpoint = "unix:///var/run/docker.sock"
        containerd_address = "/var/run/containerd/containerd.sock"

        enable_container_metric = true
        enable_k8s_metric = true
        enable_pod_metric = true

        ## Containers logs to include and exclude, default collect all containers. Globs accepted.
        container_include_log = []
        container_exclude_log = ["image:pubrepo.guance.com/datakit/logfwd*", "image:pubrepo.guance.com/datakit/datakit*"]

        exclude_pause_container = true

        ## Removes ANSI escape codes from text strings
        logging_remove_ansi_escape_codes = false

        kubernetes_url = "https://kubernetes.default:443"

        ## Authorization level:
        ##   bearer_token -> bearer_token_string -> TLS
        ## Use bearer token for authorization. ('bearer_token' takes priority)
        ## linux at:   /run/secrets/kubernetes.io/serviceaccount/token
        ## windows at: C:\var\run\secrets\kubernetes.io\serviceaccount\token
        bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
        # bearer_token_string = "<your-token-string>"

        [inputs.container.tags]
          # some_tag = "some_value"
          # more_tag = "some_other_value"

        volumeMounts:
        - mountPath: /usr/local/datakit/conf.d/container/container.conf
          name: datakit-conf
          subPath: container.conf

container_include and container_exclude must start with image, in the format "image:<glob规则>", indicating that the glob rule applies to the container image.
Glob rules are lightweight regular expressions supporting basic matching units such as * and ?.

For example, to collect only logs from images whose name contains log-order but not log-pay, configure as follows:

        container_include_log = ["image:*log-order*"]
        container_exclude_log = ["image:*log-pay*"]

Note: If a POD has stdout log collection enabled, do not also use logfwd or socket log collection on the same POD, otherwise logs will be duplicated.

2 logfwd Collection¶

This is a log collection method using the Sidecar pattern. It leverages shared storage among containers within the same POD. logfwd reads the business container's log files in Sidecar mode and sends them to DataKit. For more details, refer to Pod Log Collection Best Practices, Solution 2.

3 Socket Collection¶

DataKit opens a socket port, such as 9542, to which logs can be pushed. Java log4j and logback support log pushing. The following example uses Spring Boot with Logback to implement socket log collection.

3.1 Adding an Appender¶

Add a socket Appender in the logback-spring.xml file.

<?xml version="1.0" encoding="UTF-8"?>

<configuration scan="true" scanPeriod="60 seconds" debug="false">
    <springProperty scope="context" name="dkSocketHost" source="datakit.socket.host" />
    <springProperty scope="context" name="dkSocketPort" source="datakit.socket.port" />
    <contextName>logback</contextName>

    <!-- 日志根目录 -->
    <property name="log.path" value="./logs"/>
    <!-- 日志输出格式 -->
    <property name="log.pattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{20} - [%method,%line] -  - %msg%n" />

    <!-- 打印日志到控制台 -->
    <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${log.pattern}</pattern>
        </encoder>
    </appender>
    ... 
    <!--下面是增加的 Socket appender--> 
    <appender name="socket" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <!-- datakit host: logsocket_port -->
        <destination>${dkSocketHost}:${dkSocketPort}</destination>
        <!-- 日志输出编码 -->
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>UTC+8</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        "severity": "%level",
                        "appName": "${logName:-}",
                        "trace": "%X{dd.trace_id:-}",
                        "span": "%X{dd.span_id:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "msg": "%message\n%exception"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="Console"/>
        <appender-ref ref="file_info"/>
        <appender-ref ref="socket" />
    </root>
</configuration>

3.2 Adding Configuration¶

Add the configuration to the application.yml file of the Spring Boot project.

datakit:
  socket:
    host: 120.26.218.200  # 
    port: 9542

3.3 Adding Dependencies¶

Add the dependency to the pom.xml file of the Spring Boot project.

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>4.9</version>
</dependency>

3.4 Adding the logging-socket.conf File in DataKit¶

In the datakit.yaml file of DataKit:

        volumeMounts:  # Add the following three lines in this section
        - mountPath: /usr/local/datakit/conf.d/log/logging-socket.conf
          name: datakit-conf
          subPath: logging-socket.conf

---           
apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:       
    logging-socket.conf: |-
      [[inputs.logging]]
        # only two protocols are supported:TCP and UDP
        sockets = [
          "tcp://0.0.0.0:9542",
        #"udp://0.0.0.0:9531",                  
        ]
        ignore = [""]
        source = "demo-socket-service"
        service = ""
        pipeline = ""
        ignore_status = []
        character_encoding = ""
        # multiline_match = '''^\S'''
        remove_ansi_escape_codes = false

        [inputs.logging.tags]
        # some_tag = "some_value"
        # more_tag = "some_other_value"

For more information about socket log collection, refer to logback Socket Log Collection Best Practices.

4 Log File Collection¶

When DataKit is installed on a Linux host, the way to collect logs from that host is to copy the logging.conf file and then modify the logfiles value in logging.conf to the absolute path of the log file.

cd /usr/local/datakit/conf.d/log
cp logging.conf.sample  logging.conf

In a Kubernetes environment, first mount the Pod's log directory /data/app/logs/demo-system to the host machine at /var/log/k8s/demo-system. Then deploy DataKit with DaemonSet and mount the /var/log/k8s/demo-system directory, so that DataKit can collect the log file /rootfs/var/log/k8s/demo-system/info.log on the host.

        volumeMounts:
        - name: app-log
          mountPath: /data/app/logs/demo-system

      ...

      volumes:   
      - name: app-log
        hostPath:
          path: /var/log/k8s/demo-system

        volumeMounts:  # Add the following three lines in this section
        - mountPath: /usr/local/datakit/conf.d/log/logging.conf
          name: datakit-conf
          subPath: logging.conf

---           
apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:           
    #### logging
    logging.conf: |-
        [[inputs.logging]]
          ## required
          logfiles = [
            "/rootfs/var/log/k8s/demo-system/info.log",
          ]

          ## glob filteer
          ignore = [""]

          ## your logging source, if it's empty, use 'default'
          source = "k8s-demo-system-log"

          ## add service tag, if it's empty, use $source.
          #service = "k8s-demo-system-log"

          ## grok pipeline script path
          pipeline = ""
          ## optional status:
          ##   "emerg","alert","critical","error","warning","info","debug","OK"
          ignore_status = []

          ## optional encodings:
          ##    "utf-8", "utf-16le", "utf-16le", "gbk", "gb18030" or ""
          character_encoding = ""

          ## The pattern should be a regexp. Note the use of '''this regexp'''
          ## regexp link: https://golang.org/pkg/regexp/syntax/#hdr-Syntax
          multiline_match = '''^\d{4}-\d{2}-\d{2}'''

          [inputs.logging.tags]
          # some_tag = "some_value"
          # more_tag = "some_other_value"

Note: Since logs are already persisted using Guance, there is no need to write logs to the host disk again. Therefore, this collection method is not recommended in Kubernetes environments.

Pipeline¶

Pipeline is mainly used to parse unstructured text data or extract partial information from structured text (such as JSON). For logs, it is primarily used to extract fields like log generation time, log level, etc. Special note: logs collected via socket are in JSON format and must be parsed before they can be searched by keyword in the search bar. For details on using Pipeline, see the articles below.

Anomaly Detection¶

When log anomalies have a significant impact on the application, use the Guance log anomaly detection feature and configure alerts to promptly notify the observed targets. Guance alerts support notification methods such as email, DingTalk, SMS, WeCom, and Lark. The following uses email as an example to introduce alerting.

1 Creating a Notification Target¶

Log in to Guance, go to Manage → Notification Targets → Create Notification Target, select Email Group, and enter a name and email address.

2 Creating a Monitor¶

Click Monitor → Create Monitor → Log Monitor.

Enter the rule name. The detection metric log_fwd_demo is the source configured during log collection. The following error is the content contained in the log. host_ip is a log tag, and in the event content you can use {{host_ip}} to output the specific tag value. Set the trigger condition to 1. The title and content will be sent via email. After filling in the fields, click Save.

3 Configuring Alerts¶

On the Monitor page, click the monitor you just created, then click Alert Configuration.

Select the email group created in the first step as the alert notification target, select the alert silence duration, and click OK.

4 Triggering an Alert¶

When the application triggers an error log, a notification email will be received.