Kubernetes

In the case of a pure Docker or Containerd environment, the DataKit can only be installed on the host machine.

Go to the conf.d/container directory under the DataKit installation directory, copy container.conf.sample and name it container.conf. Examples are as follows:

[inputs.container]
  endpoints = [
    "unix:///var/run/docker.sock",
    "unix:///var/run/containerd/containerd.sock",
    "unix:///var/run/crio/crio.sock",
  ]

  enable_container_metric = true
  enable_k8s_metric       = true
  enable_pod_metric       = false
  enable_k8s_event        = true
  enable_k8s_node_local   = true
  enable_collect_k8s_job  = true

  ## Add resource Label as Tags (container use Pod Label), need to specify Label keys.
  ## e.g. ["app", "name"]
  # extract_k8s_label_as_tags_v2            = []
  # extract_k8s_label_as_tags_v2_for_metric = []

  ## Containers logs to include and exclude, default collect all containers. Globs accepted.
  container_include_log = []
  container_exclude_log = ["image:*logfwd*", "image:*datakit*"]

  ## Pods metric to include and exclude, default collect all pods. Globs accepted.
  pod_include_metric = []
  pod_exclude_metric = []

  logging_enable_multiline              = true
  logging_auto_multiline_detection      = true
  logging_auto_multiline_extra_patterns = []

  ## Only retain the fields specified in the whitelist.
  logging_field_white_list = []

  ## Removes ANSI escape codes from text strings.
  logging_remove_ansi_escape_codes = false

  ## Whether to collect logs from the begin of the file.
  logging_file_from_beginning = false

  ## The maximum allowed number of open files, default is 500. If it is -1, it means no limit.
  # logging_max_open_files = 500

  ## Search logging interval, default "60s".
  #logging_search_interval = ""

  ## Log collection configures additional source matching, and the regular source will be renamed.
  [inputs.container.logging_extra_source_map]
    # source_regexp = "new_source"

  ## Log collection with multiline configuration as specified by the source.
  [inputs.container.logging_source_multiline_map]
    # source = '''^\d{4}'''

  [inputs.container.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

Can be turned on by ConfigMap Injection Collector Configuration or Config ENV_DATAKIT_INPUTS .

Can also be turned on by environment variables, (needs to be added as the default collector in ENV_DEFAULT_ENABLED_INPUTS):

• ENV_INPUT_CONTAINER_ENDPOINTS

  Append to container endpoints

  Type: List

  input.conf: endpoints

  Example: "unix:///var/run/docker.sock,unix:///var/run/containerd/containerd.sock,unix:///var/run/crio/crio.sock"

• ENV_INPUT_CONTAINER_ENABLE_CONTAINER_METRIC

  Start container index collection

  Type: Boolean

  input.conf: enable_container_metric

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_K8S_METRIC

  Start k8s index collection

  Type: Boolean

  input.conf: enable_k8s_metric

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_POD_METRIC

  Turn on Pod index collection

  Type: Boolean

  input.conf: enable_pod_metric

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_K8S_EVENT

  Enable event collection mode

  Type: Boolean

  input.conf: enable_k8s_event

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_K8S_NODE_LOCAL

  Enable sub-Node collection mode, where the DataKit deployed on each Node independently collects the resources of the current Node. Version-1.5.7 Need new RABC link

  Type: Boolean

  input.conf: enable_k8s_node_local

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_COLLECT_KUBE_JOB

  Turn off collection of Kubernetes Job resources (including metrics data and object data)

  Type: Boolean

  input.conf: enable_collect_kube_job

  Default: true

• ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2

  Append the labels of the resource to the tag of the non-metric (like object and logging) data. Label keys should be specified, if there is only one key and it is an empty string (e.g. [""]), all labels will be added to the tag. The container will inherit the Pod labels. If the key of the label has the dot character, it will be changed to a horizontal line

  Type: JSON

  input.conf: extract_k8s_label_as_tags_v2

  Example: ["app","name"]

• ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2_FOR_METRIC

  Append the labels of the resource to the tag of the metric data. Label keys should be specified, if there is only one key and it is an empty string (e.g. [""]), all labels will be added to the tag. The container will inherit the Pod labels. If the key of the label has the dot character, it will be changed to a horizontal line

  Type: JSON

  input.conf: extract_k8s_label_as_tags_v2_for_metric

  Example: ["app","name"]

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_POD_ANNOTATIONS

  Deprecated. Whether to turn on Prometheus Pod Annotations and collect metrics automatically

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_pod_annotations

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_SERVICE_ANNOTATIONS

  Deprecated. Whether to turn on Prometheus Service Annotations and collect metrics automatically

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_service_annotations

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_POD_MONITORS

  Deprecated. Whether to turn on automatic discovery of Prometheus PodMonitor CRD and collection of metrics, see [Prometheus-Operator CRD doc](kubernetes-prometheus-operator-crd

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_pod_monitors

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_SERVICE_MONITORS

  Deprecated. Whether to turn on automatic discovery of Prometheus ServiceMonitor CRD and collection of metrics, see [Prometheus-Operator CRD doc](kubernetes-prometheus-operator-crd

  Type: Boolean

  input.conf: enable_auto_discovery_of_prometheus_service_monitors

  Default: false

• ENV_INPUT_CONTAINER_CONTAINER_MAX_CONCURRENT

  Maximum number of concurrency when collecting container data, recommended to be turned on only when the collection delay is large

  Type: Int

  input.conf: container_max_concurrent

  Default: cpu cores + 1

• ENV_INPUT_CONTAINER_CONTAINER_INCLUDE_LOG

  Include condition of container log, filtering with image

  Type: List

  input.conf: container_include_log

  Example: "image:pubrepo.jiagouyun.com/datakit/logfwd*"

• ENV_INPUT_CONTAINER_CONTAINER_EXCLUDE_LOG

  Exclude condition of container log, filtering with image

  Type: List

  input.conf: container_exclude_log

  Example: "image:pubrepo.jiagouyun.com/datakit/logfwd*"

• ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC

  Include condition of pod metrics, filtering with namespace

  Type: List

  input.conf: pod_include_metric

  Example: "namespace:datakit*"

• ENV_INPUT_CONTAINER_POD_EXCLUDE_METRIC

  Exclude condition of pod metrics, filtering with namespace

  Type: List

  input.conf: pod_exclude_metric

  Example: "namespace:kube-system"

• ENV_INPUT_CONTAINER_LOGGING_SEARCH_INTERVAL

  The time interval of log discovery, that is, how often logs are retrieved. If the interval is too long, some logs with short survival will be ignored

  Type: Duration

  input.conf: logging_search_interval

  Default: 60s

• ENV_INPUT_CONTAINER_LOGGING_EXTRA_SOURCE_MAP

  Log collection configures additional source matching, and the regular source will be renamed

  Type: Map

  input.conf: logging_extra_source_map

  Example: source_regex*=new_source,regex*=new_source2

• ENV_INPUT_CONTAINER_LOGGING_SOURCE_MULTILINE_MAP_JSON

  Log collection with multiline configuration as specified by the source

  Type: JSON

  input.conf: logging_source_multiline_map

  Example: {"source_nginx":"^\d{4}", "source_redis":"^[A-Za-z_]"}

• ENV_INPUT_CONTAINER_LOGGING_AUTO_MULTILINE_DETECTION

  Whether the automatic multi-line mode is turned on for log collection; the applicable multi-line rules will be matched in the patterns list after it is turned on

  Type: Boolean

  input.conf: logging_auto_multiline_detection

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_AUTO_MULTILINE_EXTRA_PATTERNS_JSON

  Automatic multi-line pattern pattens list for log collection, supporting manual configuration of multiple multi-line rules

  Type: JSON

  input.conf: logging_auto_multiline_extra_patterns

  Example: ["^\d{4}-\d{2}", "^[A-Za-z_]"]

  Default: For more default rules, see doc

• ENV_INPUT_CONTAINER_LOGGING_REMOVE_ANSI_ESCAPE_CODES

  Remove ansi escape codes and color characters, referred to ansi-decode doc

  Type: Boolean

  input.conf: logging_remove_ansi_escape_codes

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_FILE_FROM_BEGINNING_THRESHOLD_SIZE

  Decide whether or not to from_beginning based on the file size, if the file size is smaller than this value when the file is found, start the collection from the begin

  Type: Int

  input.conf: logging_file_from_beginning_threshold_size

  Default: 20,000,000

• ENV_INPUT_CONTAINER_LOGGING_FILE_FROM_BEGINNING

  Whether to collect logs from the begin of the file

  Type: Boolean

  input.conf: logging_file_from_beginning

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_MAX_OPEN_FILES

  The maximum allowed number of open files. If it is set to -1, it means there is no limit.

  Type: Int

  input.conf: logging_max_open_files

  Default: 500

• ENV_INPUT_CONTAINER_LOGGING_FIELD_WHITE_LIST

  "Only retain the fields specified in the whitelist."

  Type: List

  input.conf: logging_field_white_list

  Example: '["service","container_id"]'

• ENV_INPUT_CONTAINER_TAGS

  Customize tags. If there is a tag with the same name in the configuration file, it will be overwritten

  Type: Map

  input.conf: tags

  Example: tag1=value1,tag2=value2

Additional description of environment variables:

• ENV_INPUT_CONTAINER_TAGS: If there is a tag with the same name in the configuration file (container.conf), it will be overwritten by the configuration here.

• ENV_INPUT_CONTAINER_LOGGING_EXTRA_SOURCE_MAP: Specifying the replacement source with the argument format regular expression=new_source, which is replaced by new_source when a source matches the regular expression. If the replacement is successful, the source（ Version-1.4.7）configured in annotations/labels is no longer used. If you want to make an exact match, you need to use ^ and $ to enclose the content. For example, if a regular expression is written as datakit, it can not only match the word datakit , but also match datakit123; Written as ^datakit$ , you can only match datakit.

• ENV_INPUT_CONTAINER_LOGGING_SOURCE_MULTILINE_MAP_JSON: Used to specify the mapping of source to multi-row configuration. If a log is not configured with multiline_match, the corresponding multiline_match is found and used here based on its source. Because the multiline_match value is a regular expression, it is more complex, so the value format is a JSON string that can be coded and compressed into a single line using json.cn.

Modify the containerd_address configuration entry of container.conf to set it to the corresponding sock path.

Change the volumes containerd-socket of DataKit.yaml, mount the new path into the DataKit, and configure the environment variablesENV_INPUT_CONTAINER_ENDPOINTS：

# add envs
- env:
  - name: ENV_INPUT_CONTAINER_ENDPOINTS
    value: ["unix:///path/to/new/containerd/containerd.sock"]

# modify mountPath
  - mountPath: /path/to/new/containerd/containerd.sock
    name: containerd-socket
    readOnly: true

# modify volumes
volumes:
- hostPath:
    path: /path/to/new/containerd/containerd.sock
  name: containerd-socket

  ## Collect metrics for Pods whose namespace matches `datakit`
  pod_include_metric = ["namespace:datakit"]

  ## Exclude all Pods with namespace `kodo`
  pod_exclude_metric = ["namespace:kodo"]

• Both include and exclude settings must start with a field name, formatted as a glob pattern: "<field name>:<glob rule>".
• Currently, only the namespace field is supported for filtering, e.g., namespace:datakit-ns.

If both include and exclude are configured, the Pod must meet the following conditions:

• It must satisfy the include rule
• It must not satisfy the exclude rule

For example, the following configuration will filter out all Pods:

  ## Collect only Pods in `namespace:datakit` and exclude all namespaces
  pod_include_metric = ["namespace:datakit"]
  pod_exclude_metric = ["namespace:*"]

In a Kubernetes environment, you can configure the following environment variables:

• ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC
• ENV_INPUT_CONTAINER_POD_EXCLUDE_METRIC

For example, if you want to collect metrics only for Pods in the kube-system namespace, you can set the ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC environment variable as shown below:

  - env:
      - name: ENV_INPUT_CONTAINER_POD_INCLUDE_METRIC
        value: namespace:kube-system  # Specify the namespace to collect metrics from

By using this method, you can flexibly control the range of Pods from which DataKit collects metrics, reducing the collection of unnecessary data and optimizing system performance and resource usage.