Kubernetes

In the case of a pure Docker or Containerd environment, the DataKit can only be installed on the host machine.

Go to the conf.d/samples directory under the DataKit installation directory, copy container.conf.sample and name it container.conf. Examples are as follows:

# Container input plugin configuration
[inputs.container]

  # Container runtime endpoints to connect to
  endpoints = [
    "unix:///var/run/docker.sock",
    "unix:///var/run/containerd/containerd.sock",
    "unix:///var/run/crio/crio.sock",
  ]

  # Collection intervals (uncomment to customize)
  # metric_collect_interval = "60s"    # Default: 60s
  # object_collect_interval = "5m"     # Default: 5m
  # logging_search_interval = "60s"    # Default: 60s

  # GKE Autopilot Cloud API mode. Authentication uses Workload Identity through
  # the GKE metadata server; project, cluster, and location are auto-discovered.
  # gcp_cloud_api_enabled        = false
  # Optional overrides for cross-project or non-standard collection:
  # gcp_project_id               = "my-project"
  # gcp_cluster_name             = "my-autopilot-cluster"
  # gcp_cluster_location         = "us-central1"
  # Advanced feature overrides (both enabled by default):
  # enable_gcp_cloud_monitoring  = true
  # enable_gcp_cloud_logging     = true
  # gcp_cloud_logging_lookback   = "2m"
  # gcp_cloud_logging_overlap    = "2m"

  # Feature toggles
  enable_container_metric = true  # Collect container metrics
  enable_k8s_metric       = true  # Collect Kubernetes metrics
  enable_pod_metric       = false # Collect pod-specific metrics
  enable_k8s_event        = true  # Collect Kubernetes events
  enable_k8s_node_local   = true  # Enable node-local collection
  enable_collect_k8s_job  = true  # Collect Kubernetes jobs

  # Kubernetes label extraction as tags
  # Specify label keys to extract as tags (containers use pod labels)
  # Example: ["app", "name", "version"]
  # extract_k8s_label_as_tags_v2            = []
  # extract_k8s_label_as_tags_v2_for_metric = []

  # Container log filtering (supports glob patterns)
  container_include_log = []                                    # Include specific containers (empty = all)
  container_exclude_log = ["image:*logfwd*", "image:*datakit*"] # Exclude containers by image pattern

  # Pod metric filtering (supports glob patterns)
  pod_include_metric = [] # Include specific pods (empty = all)
  pod_exclude_metric = [] # Exclude specific pods

  # Logging configuration
  logging_enable_multiline              = true  # Enable multiline log detection
  logging_auto_multiline_extra_patterns = []    # Additional multiline patterns

  # Log field filtering
  logging_field_white_list = [] # Only retain specified fields (empty = all fields)

  # Log processing options
  logging_remove_ansi_escape_codes = false # Remove ANSI escape codes from logs
  logging_file_from_beginning      = false # Start reading from beginning of log files

  # Performance tuning
  # logging_max_open_files = 500 # Maximum open files (-1 = unlimited)

  # Source mapping for log collection
  # Maps regex patterns to new source names
  [inputs.container.logging_extra_source_map]
    # ".*nginx.*" = "nginx_logs"
    # ".*apache.*" = "apache_logs"

  # Multiline configuration per source
  # Define multiline patterns for specific log sources
  [inputs.container.logging_source_multiline_map]
    # "java_logs" = '''^\d{4}-\d{2}-\d{2}'''
    # "python_logs" = '''^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'''

  # Global tags for all collected data
  [inputs.container.tags]
    # environment = "production"
    # cluster = "main"
    # region = "us-west-1"

Can be turned on by ConfigMap Injection Collector Configuration or Config ENV_DATAKIT_INPUTS .

Can also be turned on by environment variables, (needs to be added as the default collector in ENV_DEFAULT_ENABLED_INPUTS):

• ENV_INPUT_CONTAINER_ENDPOINTS

  Append to container endpoints

  Type: List

  input.conf: endpoints

  Example: "unix:///var/run/docker.sock,unix:///var/run/containerd/containerd.sock,unix:///var/run/crio/crio.sock"

• ENV_INPUT_CONTAINER_METRIC_COLLEC_INTERVAL

  The metric collection interval

  Type: Duration

  input.conf: metric_collec_interval

  Default: 60s

• ENV_INPUT_CONTAINER_OBJECT_COLLEC_INTERVAL

  The object collection interval

  Type: Duration

  input.conf: object_collec_interval

  Default: 5m

• ENV_INPUT_CONTAINER_LOGGING_SEARCH_INTERVAL

  The time interval of log discovery, that is, how often logs are retrieved. If the interval is too long, some logs with short survival will be ignored

  Type: Duration

  input.conf: logging_search_interval

  Default: 60s

• ENV_INPUT_CONTAINER_G_C_P_CLOUD_A_P_I_ENABLED

  Collect GKE container metrics and stdout/stderr logs through Google Cloud APIs

  Type: Boolean

  input.conf: g_c_p_cloud_a_p_i_enabled

  Default: false

• ENV_INPUT_CONTAINER_G_C_P_PROJECT_ID

  Optional Google Cloud project ID override; discovered from the GKE metadata server by default

  Type: String

  input.conf: g_c_p_project_id

• ENV_INPUT_CONTAINER_G_C_P_CLUSTER_NAME

  Optional GKE cluster name override; discovered from the GKE metadata server by default

  Type: String

  input.conf: g_c_p_cluster_name

• ENV_INPUT_CONTAINER_G_C_P_CLUSTER_LOCATION

  Optional GKE cluster region or zone override; discovered from the GKE metadata server by default

  Type: String

  input.conf: g_c_p_cluster_location

• ENV_INPUT_CONTAINER_ENABLE_G_C_P_CLOUD_MONITORING

  Advanced override to collect container metrics from Cloud Monitoring

  Type: Boolean

  input.conf: enable_g_c_p_cloud_monitoring

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_G_C_P_CLOUD_LOGGING

  Advanced override to collect container stdout/stderr from Cloud Logging

  Type: Boolean

  input.conf: enable_g_c_p_cloud_logging

  Default: true

• ENV_INPUT_CONTAINER_G_C_P_CLOUD_LOGGING_LOOKBACK

  Initial Cloud Logging lookback window

  Type: Duration

  input.conf: g_c_p_cloud_logging_lookback

  Default: 2m

• ENV_INPUT_CONTAINER_G_C_P_CLOUD_LOGGING_OVERLAP

  Cloud Logging overlap window used to tolerate delayed entries

  Type: Duration

  input.conf: g_c_p_cloud_logging_overlap

  Default: 2m

• ENV_INPUT_CONTAINER_G_C_P_CLOUD_LOGGING_STATE_FILE

  File used to persist the Cloud Logging watermark and deduplication state

  Type: String

  input.conf: g_c_p_cloud_logging_state_file

• ENV_INPUT_CONTAINER_ENABLE_CONTAINER_METRIC

  Start container index collection

  Type: Boolean

  input.conf: enable_container_metric

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_K8S_METRIC

  Start k8s index collection

  Type: Boolean

  input.conf: enable_k8s_metric

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_POD_METRIC

  Turn on Pod index collection

  Type: Boolean

  input.conf: enable_pod_metric

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_K8S_EVENT

  Enable event collection mode

  Type: Boolean

  input.conf: enable_k8s_event

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_K8S_NODE_LOCAL

  Enable sub-Node collection mode, where the DataKit deployed on each Node independently collects the resources of the current Node. Version-1.5.7 Need new RABC link

  Type: Boolean

  input.conf: enable_k8s_node_local

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_COLLECT_KUBE_JOB

  Turn off collection of Kubernetes Job resources (including metrics data and object data)

  Type: Boolean

  input.conf: enable_collect_kube_job

  Default: true

• ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2

  Append the labels of the resource to the tag of the non-metric (like object, logging and change) data. Label keys should be specified, if there is only one key and it is an empty string (e.g. [""]), all labels will be added to the tag. The container will inherit the Pod labels. If the key of the label has the dot character, it will be changed to a horizontal line

  Type: JSON

  input.conf: extract_k8s_label_as_tags_v2

  Example: '["app","name"]'

• ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2_FOR_METRIC

  Append the labels of the resource to the tag of the metric data. Label keys should be specified, if there is only one key and it is an empty string (e.g. [""]), all labels will be added to the tag. The container will inherit the Pod labels. If the key of the label has the dot character, it will be changed to a horizontal line

  Type: JSON

  input.conf: extract_k8s_label_as_tags_v2_for_metric

  Example: '["app","name"]'

• ENV_INPUT_CONTAINER_CONTAINER_MAX_CONCURRENT

  Maximum number of concurrency when collecting container data, recommended to be turned on only when the collection delay is large

  Type: Int

  input.conf: container_max_concurrent

  Default: cpu cores + 1

• ENV_INPUT_CONTAINER_CONTAINER_INCLUDE_LOG

  Include condition of container log, filtering with image

  Type: List

  input.conf: container_include_log

  Example: "image:pubrepo.jiagouyun.com/datakit/logfwd*"

• ENV_INPUT_CONTAINER_CONTAINER_EXCLUDE_LOG

  Exclude condition of container log, filtering with image

  Type: List

  input.conf: container_exclude_log

  Example: "image:pubrepo.jiagouyun.com/datakit/logfwd*"

• ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC

  Include condition of pod metrics, filtering with namespace

  Type: List

  input.conf: pod_include_metric

  Example: "namespace:datakit*"

• ENV_INPUT_CONTAINER_POD_EXCLUDE_METRIC

  Exclude condition of pod metrics, filtering with namespace

  Type: List

  input.conf: pod_exclude_metric

  Example: "namespace:kube-system"

• ENV_INPUT_CONTAINER_LOGGING_EXTRA_SOURCE_MAP

  Log collection configures additional source matching, and the regular source will be renamed

  Type: Map

  input.conf: logging_extra_source_map

  Example: source_regex*=new_source,regex*=new_source2

• ENV_INPUT_CONTAINER_LOGGING_SOURCE_MULTILINE_MAP_JSON

  Log collection with multiline configuration as specified by the source

  Type: JSON

  input.conf: logging_source_multiline_map

  Example: '{"source_nginx":"^\d{4}", "source_redis":"^[A-Za-z_]"}'

• ENV_INPUT_CONTAINER_LOGGING_AUTO_MULTILINE_EXTRA_PATTERNS_JSON

  Supplementary automatic multiline patterns for log collection, checked only after built-in rules do not match

  Type: JSON

  input.conf: logging_auto_multiline_extra_patterns

  Example: '["^\d{4}-\d{2}", "^[A-Za-z_]"]'

  Default: For more default rules, see doc

• ENV_INPUT_CONTAINER_LOGGING_REMOVE_ANSI_ESCAPE_CODES

  Remove ansi escape codes and color characters, referred to ansi-decode doc

  Type: Boolean

  input.conf: logging_remove_ansi_escape_codes

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_FILE_FROM_BEGINNING_THRESHOLD_SIZE

  Decide whether or not to from_beginning based on the file size, if the file size is smaller than this value when the file is found, start the collection from the begin

  Type: Int

  input.conf: logging_file_from_beginning_threshold_size

  Default: 20,000,000

• ENV_INPUT_CONTAINER_LOGGING_FILE_FROM_BEGINNING

  Whether to collect logs from the begin of the file

  Type: Boolean

  input.conf: logging_file_from_beginning

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_MAX_OPEN_FILES

  The maximum allowed number of open files. If it is set to -1, it means there is no limit.

  Type: Int

  input.conf: logging_max_open_files

  Default: 500

• ENV_INPUT_CONTAINER_LOGGING_FIELD_WHITE_LIST

  "Only retain the fields specified in the whitelist."

  Type: List

  input.conf: logging_field_white_list

  Example: '["service","container_id"]'

• ENV_INPUT_CONTAINER_TAGS

  Customize tags. If there is a tag with the same name in the configuration file, it will be overwritten

  Type: Map

  input.conf: tags

  Example: tag1=value1,tag2=value2

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_POD_ANNOTATIONS

  Deprecated. Whether to turn on Prometheus Pod Annotations and collect metrics automatically

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_pod_annotations

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_SERVICE_ANNOTATIONS

  Deprecated. Whether to turn on Prometheus Service Annotations and collect metrics automatically

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_service_annotations

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_POD_MONITORS

  Deprecated. Whether to turn on automatic discovery of Prometheus PodMonitor CRD and collection of metrics, see [Prometheus-Operator CRD doc](kubernetes-prometheus-operator-crd

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_pod_monitors

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_SERVICE_MONITORS

  Deprecated. Whether to turn on automatic discovery of Prometheus ServiceMonitor CRD and collection of metrics, see [Prometheus-Operator CRD doc](kubernetes-prometheus-operator-crd

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_service_monitors

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_AUTO_MULTILINE_DETECTION

  Deprecated. Whether the automatic multi-line mode is turned on for log collection; the applicable multi-line rules will be matched in the patterns list after it is turned on

  Type: Boolean

  input.conf: logging_auto_multiline_detection

  Default: false

Additional description of environment variables:

• ENV_INPUT_CONTAINER_TAGS: If there is a tag with the same name in the configuration file (container.conf), it will be overwritten by the configuration here.

• ENV_INPUT_CONTAINER_LOGGING_EXTRA_SOURCE_MAP: Specifying the replacement source with the argument format regular expression=new_source, which is replaced by new_source when a source matches the regular expression. If the replacement is successful, the source（ Version-1.4.7）configured in annotations/labels is no longer used. If you want to make an exact match, you need to use ^ and $ to enclose the content. For example, if a regular expression is written as datakit, it can not only match the word datakit , but also match datakit123; Written as ^datakit$ , you can only match datakit.

• ENV_INPUT_CONTAINER_LOGGING_SOURCE_MULTILINE_MAP_JSON: Used to specify the mapping of source to multi-row configuration. If a log is not configured with multiline_match, the corresponding multiline_match is found and used here based on its source. Because the multiline_match value is a regular expression, it is more complex, so the value format is a JSON string that can be coded and compressed into a single line using json.cn.

Modify the containerd_address configuration entry of container.conf to set it to the corresponding sock path.

Change datakit.yaml, mount the directory that contains the socket into DataKit, and configure ENV_INPUT_CONTAINER_ENDPOINTS:

# add envs
- env:
  - name: ENV_INPUT_CONTAINER_ENDPOINTS
    value: '["unix:///path/to/new/containerd/containerd.sock"]'

# mount the directory that contains the socket
  - mountPath: /path/to/new/containerd
    name: containerd-run
    readOnly: true

# modify volumes
volumes:
- hostPath:
    path: /path/to/new/containerd
    type: Directory
  name: containerd-run

Prefer mounting the directory that contains the socket. Avoid subPath or mounting only the socket file. A container runtime may delete and recreate its socket during restart. A directory mount exposes the new file to DataKit, while a single-file mount may continue to reference the stale file.

  ## Collect metrics for Pods whose namespace matches `datakit`
  pod_include_metric = ["namespace:datakit"]

  ## Exclude all Pods with namespace `kodo`
  pod_exclude_metric = ["namespace:kodo"]

• Both include and exclude settings must start with a field name, formatted as a glob pattern: "<field name>:<glob rule>".
• Currently, only the namespace field is supported for filtering, e.g., namespace:datakit-ns.

If both include and exclude are configured, the Pod must meet the following conditions:

• It must satisfy the include rule
• It must not satisfy the exclude rule

For example, the following configuration will filter out all Pods:

  ## Collect only Pods in `namespace:datakit` and exclude all namespaces
  pod_include_metric = ["namespace:datakit"]
  pod_exclude_metric = ["namespace:*"]

In a Kubernetes environment, you can configure the following environment variables:

• ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC
• ENV_INPUT_CONTAINER_POD_EXCLUDE_METRIC

For example, if you want to collect metrics only for Pods in the kube-system namespace, you can set the ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC environment variable as shown below:

  - env:
      - name: ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC
        value: namespace:kube-system  # Specify the namespace to collect metrics from

By using this method, you can flexibly control the range of Pods from which DataKit collects metrics, reducing the collection of unnecessary data and optimizing system performance and resource usage.