Several Playbooks for Log Collection in Kubernetes Clusters¶

Introduction¶

For enterprise application systems, logs are of great importance, especially in a Kubernetes environment where log collection becomes more complex. Therefore, DataKit provides very robust support for log collection, supporting multiple environments and various technology stacks. The following will provide a detailed explanation of the usage of DataKit's log collection.

Prerequisites¶

Log in to Guance, 【Integration】->【Datakit】-> 【Kubernetes】, please follow the instructions to install DataKit in your Kubernetes cluster. The datakit.yaml file used during deployment will be referenced in subsequent operations.

Advanced Configuration for DataKit¶

1 Set Log Level¶

The default log level for DataKit is Info. If you need to adjust the log level to Debug, add an environment variable in the datakit.yaml file.

        - name: ENV_LOG_LEVEL
          value: debug

2 Set Log Output Method¶

By default, DataKit outputs logs to /var/log/datakit/gin.log and /var/log/datakit/log. If you do not want to generate log files inside the container, add an environment variable in the datakit.yaml file.

    - name: ENV_LOG
      value: stdout
    - name: ENV_GIN_LOG
      value: stdout

Logs generated by DataKit can be viewed using the kubectl command with the POD name.

kubectl logs datakit-2fnrz -n datakit #

Note: After setting ENV_LOG_LEVEL to debug, a large amount of logs will be generated. It is not recommended to set ENV_LOG to stdout at this time.

Log Collection¶

1 Stdout Collection¶

1.1 Full Collection of Stdout Logs¶

DataKit can collect container logs output to stdout. After deploying DataKit using the datakit.yaml file, the container collector is enabled by default.

        - name: ENV_DEFAULT_ENABLED_INPUTS
          value: cpu,disk,diskio,mem,swap,system,hostobject,net,host_processes,container

At this point, a configuration file named /usr/local/datakit/conf.d/container/container.conf will be generated within the DataKit container. By default, it collects all stdout logs except those starting with the image pubrepo.guance.com/datakit/logfwd.

  container_include_log = []  # Equivalent to image:*
  container_exclude_log = ["image:pubrepo.guance.com/datakit/logfwd*"]

1.2 Custom Stdout Log Collection¶

To better distinguish the source of logs, add tags and specify log segmentation pipeline files, which requires using custom methods. That is, add annotations in the deployed yaml file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-demo-service
  labels:
    app: log-demo-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: log-demo-service
  template:
    metadata:
      labels:
        app: log-demo-service
      annotations:
        # Add the following section
        datakit/logs: |
          [
            {
              "source": "pod-logging-testing-demo",
              "service": "pod-logging-testing-demo",
              "pipeline": "pod-logging-demo.p",
              "multiline_match": "^\\d{4}-\\d{2}-\\d{2}"
            }
          ]

Annotations Parameter Explanation

source: Data source
service: Tag marker
pipeline: Pipeline script name
ignore_status:
multiline_match: Regular expression matching one line of logs; for example, in the sample, lines starting with dates (such as 2021-11-26) are considered individual log entries, while lines that do not start with such dates are treated as part of the previous log entry.
remove_ansi_escape_codes: Whether to remove ANSI escape codes, such as text colors in standard output.

1.3 Not Collecting Container Stdout Logs¶

When the container collector is enabled, it automatically collects logs output by containers to stdout. For logs you do not wish to collect, there are several methods.

1.3.1 Disable Stdout Log Collection for PODs¶

Add annotations in the yaml file when deploying applications, and set disable to true.

apiVersion: apps/v1
kind: Deployment
metadata:

...

spec:
  ...
  template:
    metadata:      
      annotations:
        ## Add the following content
        datakit/logs: |
          [
            {
              "disable": true  
            }
          ]

1.3.2 Redirect Standard Output¶

If stdout log collection is enabled and the container logs also output to stdout, and neither side wishes to make changes, modify the startup command to redirect standard output.

java ${JAVA_OPTS}   -jar ${jar} ${PARAMS}  2>&1 > /dev/null

1.3.3 Filtering Functionality of the Container Collector¶

If you want more convenient control over stdout log collection, it is recommended to rewrite the container.conf file, i.e., define container.conf using ConfigMap, modify the values of container_include_log and container_exclude_log, then mount them into DataKit. Modify datakit.yaml as follows:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
    #### container
    container.conf: |-  
      [inputs.container]
        docker_endpoint = "unix:///var/run/docker.sock"
        containerd_address = "/var/run/containerd/containerd.sock"

        enable_container_metric = true
        enable_k8s_metric = true
        enable_pod_metric = true

        ## Containers logs to include and exclude, default collect all containers. Globs accepted.
        container_include_log = []
        container_exclude_log = ["image:pubrepo.guance.com/datakit/logfwd*", "image:pubrepo.guance.com/datakit/datakit*"]

        exclude_pause_container = true

        ## Removes ANSI escape codes from text strings
        logging_remove_ansi_escape_codes = false

        kubernetes_url = "https://kubernetes.default:443"

        ## Authorization level:
        ##   bearer_token -> bearer_token_string -> TLS
        ## Use bearer token for authorization. ('bearer_token' takes priority)
        ## linux at:   /run/secrets/kubernetes.io/serviceaccount/token
        ## windows at: C:\var\run\secrets\kubernetes.io\serviceaccount\token
        bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
        # bearer_token_string = "<your-token-string>"

        [inputs.container.tags]
          # some_tag = "some_value"
          # more_tag = "some_other_value"

        volumeMounts:
        - mountPath: /usr/local/datakit/conf.d/container/container.conf
          name: datakit-conf
          subPath: container.conf

container_include and container_exclude must start with image, in the format "image:<glob rule>", indicating that the glob rule applies to the container image.
Glob Rule is a lightweight regular expression that supports * and ? basic matching units.

For instance, if you only want to collect logs from images containing log-order but not containing log-pay, you can configure as follows.

        container_include_log = ["image:*log-order*"]
        container_exclude_log = ["image:*log-pay*"]

Note: If a certain POD has enabled stdout log collection, avoid using logfwd or socket log collection, otherwise, logs may be collected redundantly.

2 Logfwd Collection¶

This is a Sidecar mode log collection method, utilizing shared storage within the same POD so that logfwd reads the log files of the business container in Sidecar mode and sends them to DataKit. For specific usage, refer to the Best Practices for Pod Log Collection Solution Two.

3 Socket Collection¶

DataKit opens a Socket port like 9542, and logs are pushed to this port. Java’s log4j and logback support log pushing. Below is an example of achieving socket log collection by integrating SpringBoot with Logback.

3.1 Add Appender¶

Add a socket Appender in the logback-spring.xml file.

<?xml version="1.0" encoding="UTF-8"?>

<configuration scan="true" scanPeriod="60 seconds" debug="false">
    <springProperty scope="context" name="dkSocketHost" source="datakit.socket.host" />
    <springProperty scope="context" name="dkSocketPort" source="datakit.socket.port" />
    <contextName>logback</contextName>

    <!-- Root log directory -->
    <property name="log.path" value="./logs"/>
    <!-- Log output format -->
    <property name="log.pattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{20} - [%method,%line] -  - %msg%n" />

    <!-- Print logs to console -->
    <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${log.pattern}</pattern>
        </encoder>
    </appender>
    ... 
    <!-- Below is the added Socket appender --> 
    <appender name="socket" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <!-- datakit host: logsocket_port -->
        <destination>${dkSocketHost}:${dkSocketPort}</destination>
        <!-- Log output encoding -->
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>UTC+8</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        "severity": "%level",
                        "appName": "${logName:-}",
                        "trace": "%X{dd.trace_id:-}",
                        "span": "%X{dd.span_id:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "msg": "%message\n%exception"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="Console"/>
        <appender-ref ref="file_info"/>
        <appender-ref ref="socket" />
    </root>
</configuration>

3.2 Add Configuration¶

Add configurations in the application.yml file of the SpringBoot project.

datakit:
  socket:
    host: 120.26.218.200  # 
    port: 9542

3.3 Add Dependencies¶

Add dependencies in the pom.xml file of the SpringBoot project.

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>4.9</version>
</dependency>

3.4 Add logging-socket.conf File to DataKit¶

In the datakit.yaml file of DataKit

        volumeMounts:  # Add the following three lines at this location
        - mountPath: /usr/local/datakit/conf.d/log/logging-socket.conf
          name: datakit-conf
          subPath: logging-socket.conf

---           
apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:       
    logging-socket.conf: |-
      [[inputs.logging]]
        # Only two protocols are supported: TCP and UDP
        sockets = [
          "tcp://0.0.0.0:9542",
        #"udp://0.0.0.0:9531",                  
        ]
        ignore = [""]
        source = "demo-socket-service"
        service = ""
        pipeline = ""
        ignore_status = []
        character_encoding = ""
        # multiline_match = '''^\S'''
        remove_ansi_escape_codes = false

        [inputs.logging.tags]
        # some_tag = "some_value"
        # more_tag = "some_other_value"

For more content about Socket log collection, refer to Best Practices for Logback Socket Log Collection.

4 Log File Collection¶

The way DataKit installed on Linux hosts collects logs from the host is by copying the logging.conf file and then modifying the value of logfiles in the logging.conf file to the absolute path of the logs.

cd /usr/local/datakit/conf.d/log
cp logging.conf.sample  logging.conf

In a Kubernetes environment, you need to first mount the log directory /data/app/logs/demo-system of the Pod to the host machine at /var/log/k8s/demo-system, then deploy DataKit using DaemonSet, mounting the /var/log/k8s/demo-system directory so that DataKit can collect the /rootfs/var/log/k8s/demo-system/info.log log file on the host machine.

        volumeMounts:
        - name: app-log
          mountPath: /data/app/logs/demo-system

      ...

      volumes:   
      - name: app-log
        hostPath:
          path: /var/log/k8s/demo-system

    ---ap ki me

Note: Since Guance is used to collect logs and the logs have been persisted, name="__codelineno-17-1"> volumeMounts: # Add the following three lines at this location - mountPath: /usr/local/datakit/conf.d/log/logging.conf name: datakit-conf subPath: logging.conf class="w"> iVersion: v1 nd: ConfigMap tadata: name: datakit-conf namespace: datakit >data: #### logging logging.conf: |- [[inputs.logging]] ## required logfiles = [ "/rootfs/var/log/k8s/demo-system/info.log", ] ## glob filter ignore = [""] ## your logging source, if it's empty, use 'default' source = "k8s-demo-system-log" ## add service tag, if it's empty, use $source. #service = "k8s-demo-system-log" ## grok pipeline script path pipeline = "" ## optional status: ## "emerg","alert","critical","error","warning","info","debug","OK" ignore_status = [] ## optional encodings: ## "utf-8", "utf-16le", "utf-16le", "gbk", "gb18030" or "" character_encoding = "" ## The pattern should be a regexp. Note the use of '''this regexp''' ## regexp link: https://golang.org/pkg/regexp/syntax/#hdr-Syntax multiline_match = '''^\d{4}-\d{2}-\d{2}''' [inputs.logging.tags] # some_tag = "some_value" # more_tag = "some_other_value" there is no need to write logs to disk on the host machine, so this collection method is not recommended in a Kubernetes environment.

Pipeline¶

Pipeline is mainly used to split unstructured text data or extract partial information from structured text (such as JSON). For logs, it primarily extracts log generation times, log levels, etc. Here we particularly note that logs collected via Socket are in JSON format and need to be split before they can be searched by keyword in the search box. For details on Pipeline usage, see the articles below.

Anomaly Detection¶

When logs exhibit anomalies that significantly impact applications, using the log anomaly detection feature of Guance and configuring alerts can promptly notify monitoring objects. Guance alert notifications support email, DingTalk, SMS, WeCom, Lark, and other notification methods. Below, we'll introduce alerts using email as an example.

1 Create Notification Target¶

Log in to Guance, 【Manage】->【Notification Targets】-> 【Create Notification Target】, choose Email Group, input the name and email address.

2 Create Monitor¶

Click 【Monitor】->【Create Monitor】-> 【Log Monitoring】.

Input the rule name, the monitored metric log_fwd_demo is the source configured during log collection, the subsequent error refers to the content included in the logs, host_ip is the log label, and {host_ip} can be used in the event content to output the specific value of the label. Fill in the trigger condition as 1, and both title and content will be sent via email. After filling out, click 【Save】.

3 Configure Alerts¶

On the 【Monitors】interface, click on the monitor created earlier, then click 【Alert Configuration】.

Select the email group created in step one as the alert notification target, choose the alert mute time, and click 【Confirm】.

4 Trigger Alert¶

When the application triggers error logs, a notification email will be received.