Best Practices for Kubernetes StdOut Log Whitelisting¶

Environment Preparation¶

An existing Kubernetes environment (shortened as K8s). This practice is based on a self-built Kubernetes v1.23.1, Guance Datakit version 1.2.13, and Nginx 1.17.

Datakit has already been deployed, with the Datakit configuration file container.conf managed via ConfigMap.

Note: The configuration principles for Alibaba Cloud Container Service for Kubernetes or other cloud providers' Kubernetes services are similar.

Prerequisites¶

Nginx logs in the K8s environment are output via StdOut rather than file-based. Guance Datakit, when deployed as a DaemonSet, by default collects all StdOut log outputs within K8s, including those from cluster internal components such as CoreDNS (if logging is enabled). All logs mentioned in this article are Stdout outputs.

Note: StdOut is the console output method chosen by developers while writing code, such as:

<appender name="console" class="ch.qos.logback.core.ConsoleAppender">

Whitelist Requirements¶

After deploying Datakit, logs from specified business Pods and K8s cluster components are collected as needed. Logs from newly added, unspecified business Pods will not be collected. Additionally, for multiple containers within the same Pod, only one or more may be collected.

This article demonstrates how to achieve this through different log filtering methods of the Guance collector Datakit, using Annotations added to logs (including filtering out logs generated by other containers within a Pod) and combining it with container_include_log = [] in container.conf.

For more detailed log processing principles, refer to <Datakit Log Processing Overview>

Implementation Methods¶

Method One: Using container_include_log = []¶

Collect only the logs from cluster components coredns and nginx. The container_include_log uses regular expression syntax to specify the image names.

Refer to <Configure Metrics and Log Collection Based on Container Image>

      [inputs.container]
        docker_endpoint = "unix:///var/run/docker.sock"
        containerd_address = "/var/run/containerd/containerd.sock"

        enable_container_metric = true
        enable_k8s_metric = true
        enable_pod_metric = true

        ## Containers logs to include and exclude, default collect all containers. Globs accepted.
        container_include_log = ["image:*coredns*","image:*nginx*"]
        container_exclude_log = ["image:pubrepo.guance.com/datakit/logfwd*", "image:pubrepo.guance.com/datakit/datakit*"]

        exclude_pause_container = true

        ## Removes ANSI escape codes from text strings
        logging_remove_ansi_escape_codes = false

        kubernetes_url = "https://kubernetes.default:443"

        ## Authorization level:
        ##   bearer_token -> bearer_token_string -> TLS
        ## Use bearer token for authorization. ('bearer_token' takes priority)
        ## linux at:   /run/secrets/kubernetes.io/serviceaccount/token
        ## windows at: C:\var\run\secrets\kubernetes.io\serviceaccount\token
        bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
        # bearer_token_string = "<your-token-string>"

        [inputs.container.tags]
          # some_tag = "some_value"
          # more_tag = "some_other_value"

Implementation Results¶

This collects logs from specified image-named Pods as shown in the following figure:

Method Two: Combining container_include_log = [] and Annotation Marking¶

Collect only the logs from cluster components coredns and nginx. Meanwhile, mark nginx using Annotations. Images not included in the container_include_log whitelist, such as busybox, can also be marked via Annotations and then collected. This is because the priority of marking via Annotations is higher.

For more detailed log processing principles, refer to <Datakit Log Processing Overview>

Annotation marking for Nginx:

      labels:
         app: nginx-pod
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "nginx-source",
              "service": "nginx-source",
              "pipeline": "",
              "multiline_match": ""
            }
          ]
    spec:

[inputs.container]
  docker_endpoint = "unix:///var/run/docker.sock"
  containerd_address = "/var/run/containerd/containerd.sock"

  ## Containers metrics to include and exclude, default not collect. Globs accepted.
  container_include_metric = []
  container_exclude_metric = ["image:*"]

  ## Containers logs to include and exclude, default collect all containers. Globs accepted.
  container_include_log = ["image:*coredns*","image:*nginx*"]
  container_exclude_log = []

  exclude_pause_container = true

  ## Removes ANSI escape codes from text strings
  logging_remove_ansi_escape_codes = false
  ## Maximum length of logging, default 32766 bytes.
  max_logging_length = 32766

  kubernetes_url = "https://kubernetes.default:443"

  ## Authorization level:
  ##   bearer_token -> bearer_token_string -> TLS
  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## linux at:   /run/secrets/kubernetes.io/serviceaccount/token
  ## windows at: C:\var\run\secrets\kubernetes.io\serviceaccount\token
  bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  # bearer_token_string = "<your-token-string>"

  [inputs.container.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

Implementation Results¶

Method Three: Filtering Logs from Specific Containers in a Pod¶

Collect only the logs from cluster components coredns and nginx. Meanwhile, use the "only_images" field in the nginx annotation to specify that only the container's image should be included, which means there is also a whitelist strategy within the Pod.

Before Enabling the Whitelist Inside the Pod¶

As shown in the following figure, both nginx and busybox logs are collected.

After Enabling the Whitelist Inside the Pod¶

      labels:
         app: nginx-pod
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "nginx-source",
              "service": "nginx-source",
              "pipeline": "",
              "only_images": ["image:*nginx*"],
              "multiline_match": ""
            }
          ]
    spec:

Implementation Results¶

Only retain Nginx logs within the Pod.

Summary¶

It is not recommended to enable the whitelist strategy. Whitelists may cause many issues and are difficult to debug. Unexpected effects of whitelists may occur, such as developers not seeing certain logs due to missing Tags. To filter log sources, the worst-case scenario with blacklists is that data is still collected, whereas blacklist filtering (e.g., in the Datakit collector container.conf)

container_exclude_log = ["image:pubrepo.guance.com/datakit/logfwd*"]

Method one does not use Annotation marking but instead relies on the built-in filtering mechanism in the collector's container.conf, achieving a more fundamental implementation. However, this method is less flexible compared to method two, since marking allows better tagging of log sources, making future problem analysis and filtering easier. Additionally, marking can be applied directly to business Pods, enabling fine-grained log filtering control for the same batch of business images.
Method three, combined with specific business scenarios, filters out unnecessary Sidecar logs, achieving noise reduction in log collection.