Skip to content

Best Practices for Pod Log Collection

Introduction

When deploying microservices using containers, the microservices run inside containers. A Pod is the smallest scheduling unit in Kubernetes, consisting of one or a group of tightly coupled containers.

For logs in Pods, this article describes three approaches to collecting logs via DataKit.

Approach 1

DataKit enables the Logfwd input, and Logfwd collects business container logs in Sidecar mode.

1 Enable the Logfwd Input

If Kubernetes is not yet integrated with DataKit, log in to Guance, go to Integrations → Datakit → Kubernetes, and integrate DataKit using the datakit.yaml file.

image

Next, modify the datakit.yaml file to mount the logfwdserver.conf file into the DataKit directory /usr/local/datakit/conf.d/log/.

Add the following configuration in datakit.yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
  #### logfwdserver
  logfwdserver.conf: |-
    [inputs.logfwdserver]
      ## Logfwd receiver listening address and port
      address = "0.0.0.0:9531"

      [inputs.logfwdserver.tags]
      # some_tag = "some_value"
      # more_tag = "some_other_value"

In the DaemonSet resource, add:

- mountPath: /usr/local/datakit/conf.d/log/logfwdserver.conf
  name: datakit-conf
  subPath: logfwdserver.conf

2 Mount Pipeline

Modify the datakit.yaml file to mount the pod-logging-demo.p file into the DataKit directory /usr/local/datakit/pipeline/.

In the ConfigMap resource, add:

    pod-logging-demo.p: |-
        # Log format
        #2021-12-01 10:41:06.015 [http-nio-8090-exec-2] INFO  c.s.d.c.HealthController - [getPing,19] -  - 调用 ping接口
        grok(_, "%{TIMESTAMP_ISO8601:time} %{NOTSPACE:thread_name} %{LOGLEVEL:status}%{SPACE}%{NOTSPACE:class_name} - \\[%{NOTSPACE:method_name},%{NUMBER:line}\\] -  - %{GREEDYDATA:msg}")


        default_time(time,"Asia/Shanghai")

In the DaemonSet resource, add:

- mountPath: /usr/local/datakit/pipeline/pod-logging-demo.p
  name: datakit-conf
  subPath: pod-logging-demo.p

Note: If you do not need to use Pipeline for log parsing, this step can be skipped.

3 Restart DataKit

kubectl delete -f datakit.yaml
kubectl apply -f datakit.yaml

4 Collect Logs with Logfwd Sidecar

Deploy the Logfwd image and the business image in the same Pod. The following example uses log-demo-service:v1 as the business image, which generates the log file /data/app/logs/log.log. Logfwd reads the log file via a shared volume and forwards the logs to DataKit. The pod-logging-demo.p pipeline is used to parse the logs, and date-based multiline matching is applied.

Example Configuration File 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-fwd-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: log-fwd-pod
  template:
    metadata:
      labels:
        app: log-fwd-pod
      annotations:
    spec:
      nodeName: k8s-node2
      containers:
        - name: log-fwd-container
          image: 172.16.0.238/df-demo/log-demo-service:v2
          ports:
            - containerPort: 8090
              protocol: TCP
          volumeMounts:
            - mountPath: /data/app/logs
              name: varlog
        - name: logfwd
          image: pubrepo.guance.com/datakit/logfwd:1.2.12
          env:
            - name: LOGFWD_DATAKIT_HOST
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: LOGFWD_DATAKIT_PORT
              value: "9531"
            - name: LOGFWD_ANNOTATION_DATAKIT_LOGS
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.annotations['datakit/logs']
            - name: LOGFWD_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: LOGFWD_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
          volumeMounts:
            - mountPath: /var/log
              name: varlog
            - mountPath: /opt/logfwd/config
              name: logfwd-config
              subPath: config
      restartPolicy: Always
      volumes:
        - name: varlog
          emptyDir: {}
        - configMap:
            name: logfwd-conf
          name: logfwd-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logfwd-conf
data:
  config: |
    [
        {            
            "loggings": [
                {
                    "logfiles": ["/var/log/log.log"],
                    "source": "log_fwd_demo",                    
                    "pipeline": "pod-logging-demo.p",
                    "multiline_match": "^\\d{4}-\\d{2}-\\d{2}",
                    "tags": {
                        "flag": "tag1"
                    }
                }
            ]
        }
    ]

logfwd-conf Parameter Description

  • logfiles: List of log files.
  • ignore: File path filter using glob rules. If any filter condition matches, the file is not collected.
  • source: Data source.
  • service: Additional tag. If empty, defaults to $source.
  • pipeline: Pipeline script path.
  • character_encoding: Encoding selection.
  • multiline_match: Multiline matching pattern.
  • remove_ansi_escape_codes: Whether to remove ANSI escape codes, e.g., text color from standard output. Values: true or false.
  • tags: Tags defined as key-value pairs. Optional.

Environment Variable Description

  • LOGFWD_DATAKIT_HOST: DataKit address.
  • LOGFWD_DATAKIT_PORT: Logfwd port.
kubectl apply -f log-fwd-deployment.yaml

5 View Logs

Log in to Guance, go to Logs, and search for the data source log_fwd_demo.

image

image

Approach 2

DataKit by default collects logs output to Stdout from Pods. To apply special processing to log formats, you typically add Annotations to the Deployment controller YAML file of the Pod.

Below is an example of log collection for a Spring Boot microservice project. The JAR file is log-springboot-demo-1.0-SNAPSHOT.jar, and the logging uses Logback. The steps are as follows:

1 Write logback-spring.xml

logback-spring.xml 
<?xml version="1.0" encoding="UTF-8"?>

<configuration scan="true" scanPeriod="60 seconds" debug="false">
    <contextName>logback</contextName>

    <!-- Log root directory -->
    <property name="log.root.dir" value="./logs"/>
    <!-- Log output format -->
    <property name="log.pattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{20} - [%method,%line] -  - %msg%n" />

    <!-- Print logs to console -->
    <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${log.pattern}</pattern>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="Console"/>
    </root>
</configuration>

2 Build the Image

Dockerfile:

FROM openjdk:8u292

RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' >/etc/timezone
ENV jar log-springboot-demo-1.0-SNAPSHOT.jar

ENV workdir /data/app/
RUN mkdir -p ${workdir}
WORKDIR ${workdir}
ENTRYPOINT ["sh", "-ec", "exec java ${JAVA_OPTS} -jar ${jar} "]

Build the image and push it to the Harbor registry:

 docker build -t <your-harbor>/log-demo-service:v1  .
 docker push <your-harbor>/log-demo-service:v1

3 Write pod-log-service.yaml

pod-log-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: log-demo-service
  labels:
    app: log-demo-service
spec:
  selector:
    app: log-demo-service
  ports:
    - protocol: TCP
      port: 8090
      nodePort: 30053
      targetPort: 8090
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-demo-service
  labels:
    app: log-demo-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: log-demo-service
  template:
    metadata:
      labels:
        app: log-demo-service
      annotations:
        datakit/logs: |
          [
            {
              "source": "pod-logging-testing-demo",
              "service": "pod-logging-testing-demo",
              "pipeline": "pod-logging-demo.p",
              "multiline_match": "^\\d{4}-\\d{2}-\\d{2}"
            }
          ]

    spec:
      containers:
        - env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          name: log-service
          image: <your-harbor>/log-demo-service:v1
          ports:
            - containerPort: 8090
              protocol: TCP

      restartPolicy: Always
      volumes:
        - name: ddagent
          emptyDir: {}

Annotations Parameter Description

  • source: Data source.
  • service: Tag label.
  • pipeline: Pipeline script path.
  • ignore_status: (Not used in this example).
  • multiline_match: Regular expression to match the start of a log line. For example, lines starting with a date (e.g., 2021-11-26) are considered a single log line. If the next line does not start with this pattern, it is treated as part of the previous line.
  • remove_ansi_escape_codes: Whether to remove ANSI escape codes, e.g., text color from standard output.

4 Configure Pipeline

Add the pod-logging-demo.p section to the ConfigMap resource in the datakit-default.yaml file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
  pod-logging-demo.p: |-
    # Log format
    #2021-12-01 10:41:06.015 [http-nio-8090-exec-2] INFO  c.s.d.c.HealthController - [getPing,19] -  - 调用 ping接口
    grok(_, "%{TIMESTAMP_ISO8601:time} %{NOTSPACE:thread_name} %{LOGLEVEL:status}%{SPACE}%{NOTSPACE:class_name} - \\[%{NOTSPACE:method_name},%{NUMBER:line}\\] -  - %{GREEDYDATA:msg}")

    default_time(time)

Mount pod-logging-demo.p into DataKit:

- mountPath: /usr/local/datakit/pipeline/pod-logging-demo.p
  name: datakit-conf
  subPath: pod-logging-demo.p

5 View Logs

Deploy the Pod by running the following command:

kubectl apply -f pod-log-service.yaml

Access the microservice:

curl localhost:30053/ping

Log in to Guance, go to the Logs module, and search for log-demo-service. The logs will be visible.

image

image

Approach 3

Mount a Volume in the Pod using the hostPath volume type, bind the log file to the host node, and then deploy DataKit as a DaemonSet, also mounting a hostPath volume. This allows DataKit to collect the log files from the Pod.

Feedback

Is this page helpful?