Container Data Collection¶

In the case of a pure Docker or Containerd environment, the DataKit can only be installed on the host machine.

Go to the conf.d/container directory under the DataKit installation directory, copy container.conf.sample and name it container.conf. Examples are as follows:

[inputs.container]
  endpoints = [
    "unix:///var/run/docker.sock",
    "unix:///var/run/containerd/containerd.sock",
    "unix:///var/run/crio/crio.sock",
  ]

  enable_container_metric = true
  enable_k8s_metric = true
  enable_pod_metric = false
  enable_k8s_event = true
  enable_k8s_node_local = true

  ## Add resource Label as Tags (container use Pod Label), need to specify Label keys.
  ## e.g. ["app", "name"]
  # extract_k8s_label_as_tags_v2 = []
  # extract_k8s_label_as_tags_v2_for_metric = []

  ## Auto-Discovery of PrometheusMonitoring Annotations/CRDs
  enable_auto_discovery_of_prometheus_pod_annotations = false
  enable_auto_discovery_of_prometheus_service_annotations = false
  enable_auto_discovery_of_prometheus_pod_monitors = false
  enable_auto_discovery_of_prometheus_service_monitors = false

  ## Containers logs to include and exclude, default collect all containers. Globs accepted.
  container_include_log = []
  container_exclude_log = ["image:*logfwd*", "image:*datakit*"]

  kubernetes_url = "https://kubernetes.default:443"

  ## Authorization level:
  ##   bearer_token -> bearer_token_string -> TLS
  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## linux at:   /run/secrets/kubernetes.io/serviceaccount/token
  bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  # bearer_token_string = "<your-token-string>"

  ## Set true to enable election for k8s metric collection
  election = true

  logging_auto_multiline_detection = true
  logging_auto_multiline_extra_patterns = []

  ## Removes ANSI escape codes from text strings.
  logging_remove_ansi_escape_codes = false

  ## Search logging interval, default "60s"
  #logging_search_interval = ""

  [inputs.container.logging_extra_source_map]
    # source_regexp = "new_source"

  [inputs.container.logging_source_multiline_map]
    # source = '''^\d{4}'''

  [inputs.container.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

Can be turned on by ConfigMap Injection Collector Configuration or Config ENV_DATAKIT_INPUTS .

Can also be turned on by environment variables, (needs to be added as the default collector in ENV_DEFAULT_ENABLED_INPUTS):

• ENV_INPUT_CONTAINER_ENDPOINTS

  Append to container endpoints

  Type: List

  ConfField: endpoints

  Example: "unix:///var/run/docker.sock,unix:///var/run/containerd/containerd.sock,unix:///var/run/crio/crio.sock"

• ENV_INPUT_CONTAINER_DOCKER_ENDPOINT

  Deprecated. Specify the endpoint of Docker Engine

  Type: String

  ConfField: docker_endpoint

  Example: unix:///var/run/docker.sock

• ENV_INPUT_CONTAINER_CONTAINERD_ADDRESS

  Deprecated. Specify the endpoint of Containerd

  Type: String

  ConfField: containerd_address

  Example: /var/run/containerd/containerd.sock

• ENV_INPUT_CONTAINER_ENABLE_CONTAINER_METRIC

  Start container index collection

  Type: Boolean

  ConfField: enable_container_metric

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_K8S_METRIC

  Start k8s index collection

  Type: Boolean

  ConfField: enable_k8s_metric

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_POD_METRIC

  Turn on Pod index collection

  Type: Boolean

  ConfField: enable_pod_metric

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_K8S_EVENT

  Enable event collection mode

  Type: Boolean

  ConfField: enable_k8s_event

  Default: true

• ENV_INPUT_CONTAINER_ENABLE_K8S_NODE_LOCAL

  Enable sub-Node collection mode, where the Datakit deployed on each Node independently collects the resources of the current Node. Version-1.5.7 Need new RABC link

  Type: Boolean

  ConfField: enable_k8s_node_local

  Default: true

• ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS

  Should the labels of the resources be appended to the tags collected? Only Pod metrics, objects, and Node objects will be added, and the labels of container logs belonging to the Pod will also be added. If the key of a label contains a dot character, it will be replaced with a hyphen

  Type: Boolean

  ConfField: extract_k8s_label_as_tags

  Default: false

• ENV_INPUT_CONTAINER_ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2

  Append the labels of the resource to the tag of the non-metric (like object and logging) data. Label keys should be specified, if there is only one key and it is an empty string (e.g. [""]), all labels will be added to the tag. The container will inherit the Pod labels. If the key of the label has the dot character, it will be changed to a horizontal line

  Type: JSON

  ConfField: env_input_container_extract_k8s_label_as_tags_v2

  Example: ["app","name"]

• ENV_INPUT_CONTAINER_ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2_FOR_METRIC

  Append the labels of the resource to the tag of the metric data. Label keys should be specified, if there is only one key and it is an empty string (e.g. [""]), all labels will be added to the tag. The container will inherit the Pod labels. If the key of the label has the dot character, it will be changed to a horizontal line

  Type: JSON

  ConfField: env_input_container_extract_k8s_label_as_tags_v2_for_metric

  Example: ["app","name"]

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_POD_ANNOTATIONS

  Whether to turn on Prometheus Pod Annotations and collect metrics automatically

  Type: Boolean

  ConfField: enable_auto_discovery_of_prometheus_pod_annotations

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_SERVICE_ANNOTATIONS

  Whether to turn on Prometheus Service Annotations and collect metrics automatically

  Type: Boolean

  ConfField: enable_auto_discovery_of_prometheus_service_annotations

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_POD_MONITORS

  Whether to turn on automatic discovery of Prometheus PodMonitor CRD and collection of metrics, see [Prometheus-Operator CRD doc](kubernetes-prometheus-operator-crd

  Type: Boolean

  ConfField: enable_auto_discovery_of_prometheus_pod_monitors

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_AUTO_DISCOVERY_OF_PROMETHEUS_SERVICE_MONITORS

  Whether to turn on automatic discovery of Prometheus ServiceMonitor CRD and collection of metrics, see [Prometheus-Operator CRD doc](kubernetes-prometheus-operator-crd

  Type: Boolean

  ConfField: enable_auto_discovery_of_prometheus_service_monitors

  Default: false

• ENV_INPUT_CONTAINER_CONTAINER_INCLUDE_LOG

  Include condition of container log, filtering with image

  Type: List

  ConfField: container_include_log

  Example: "image:pubrepo.jiagouyun.com/datakit/logfwd*"

• ENV_INPUT_CONTAINER_CONTAINER_EXCLUDE_LOG

  Exclude condition of container log, filtering with image

  Type: List

  ConfField: container_exclude_log

  Example: "image:pubrepo.jiagouyun.com/datakit/logfwd*"

• ENV_INPUT_CONTAINER_KUBERNETES_URL

  k8s api-server access address

  Type: String

  ConfField: kubernetes_url

  Example: https://kubernetes.default:443

• ENV_INPUT_CONTAINER_BEARER_TOKEN

  The path to the token file required to access k8s api-server

  Type: String

  ConfField: bearer_token

  Example: /run/secrets/kubernetes.io/serviceaccount/token

• ENV_INPUT_CONTAINER_BEARER_TOKEN_STRING

  Token string required to access k8s api-server

  Type: String

  ConfField: bearer_token_string

  Example: your-token-string

• ENV_INPUT_CONTAINER_LOGGING_SEARCH_INTERVAL

  The time interval of log discovery, that is, how often logs are retrieved. If the interval is too long, some logs with short survival will be ignored

  Type: TimeDuration

  ConfField: logging_search_interval

  Default: 60s

• ENV_INPUT_CONTAINER_LOGGING_EXTRA_SOURCE_MAP

  Log collection configures additional source matching, and the regular source will be renamed

  Type: Map

  ConfField: logging_extra_source_map

  Example: source_regex=new_source,regex=new_source2

• ENV_INPUT_CONTAINER_LOGGING_SOURCE_MULTILINE_MAP_JSON

  Log collection configures additional source matching, and the regular source will be renamed

  Type: JSON

  ConfField: logging_source_multiline_map

  Example: {"source_nginx":"^\d{4}", "source_redis":"^[A-Za-z_]"}

• ENV_INPUT_CONTAINER_LOGGING_AUTO_MULTILINE_DETECTION

  Whether the automatic multi-line mode is turned on for log collection; the applicable multi-line rules will be matched in the patterns list after it is turned on

  Type: Boolean

  ConfField: logging_auto_multiline_detection

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_AUTO_MULTILINE_EXTRA_PATTERNS_JSON

  Automatic multi-line pattern pattens list for log collection, supporting manual configuration of multiple multi-line rules

  Type: JSON

  ConfField: logging_auto_multiline_extra_patterns

  Example: ["^\d{4}-\d{2}", "^[A-Za-z_]"]

  Default: For more default rules, see doc

• ENV_INPUT_CONTAINER_LOGGING_MAX_MULTILINE_LIFE_DURATION

  Maximum single multi-row life cycle of log collection. At the end of this cycle, existing multi-row data will be emptied and uploaded to avoid accumulation

  Type: TimeDuration

  ConfField: logging_max_multiline_life_duration

  Default: 3s

• ENV_INPUT_CONTAINER_LOGGING_REMOVE_ANSI_ESCAPE_CODES

  Remove ansi escape codes and color characters, referred to ansi-decode doc

  Type: Boolean

  ConfField: logging_remove_ansi_escape_codes

  Default: false

• ENV_INPUT_CONTAINER_LOGGING_FORCE_FLUSH_LIMIT

  If there are consecutive N empty collections, the existing data will be uploaded to prevent memory occupation caused by accumulated

  Type: Int

  ConfField: logging_force_flush_limit

  Default: 5

• ENV_INPUT_CONTAINER_CONTAINER_MAX_CONCURRENT

  Maximum number of concurrency when collecting container data, recommended to be turned on only when the collection delay is large

  Type: Int

  ConfField: container_max_concurrent

  Default: cpu cores + 1

• ENV_INPUT_CONTAINER_DISABLE_COLLECT_KUBE_JOB

  Turn off collection of Kubernetes Job resources (including metrics data and object data)

  Type: Boolean

  ConfField: disable_collect_kube_job

  Default: false

• ENV_INPUT_CONTAINER_DISABLE_COLLECT_KUBE_JOB

  Turn off collection of Kubernetes Job resources (including metrics data and object data)

  Type: Boolean

  ConfField: disable_collect_kube_job

  Default: false

• ENV_INPUT_CONTAINER_ENABLE_K8S_SELF_METRIC_BY_PROM

  Enable collection of Kubernetes Prometheus data, including APIServer, Scheduler, Etcd, etc.(Experimental)

  Type: Boolean

  ConfField: enable_k8s_self_metric_by_prom

  Default: false

• ENV_INPUT_CONTAINER_TAGS

  Customize tags. If there is a tag with the same name in the configuration file, it will be overwritten

  Type: Map

  ConfField: tags

  Example: tag1=value1,tag2=value2

Additional description of environment variables:

• ENV_INPUT_CONTAINER_TAGS: If there is a tag with the same name in the configuration file (container.conf), it will be overwritten by the configuration here.

• ENV_INPUT_CONTAINER_LOGGING_EXTRA_SOURCE_MAP: Specifying the replacement source with the argument format regular expression=new_source, which is replaced by new_source when a source matches the regular expression. If the replacement is successful, the source（ Version-1.4.7）configured in annotations/labels is no longer used. If you want to make an exact match, you need to use ^ and $ to enclose the content. For example, if a regular expression is written as datakit, it can not only match the word datakit , but also match datakit123; Written as ^datakit$ , you can only match datakit.

• ENV_INPUT_CONTAINER_LOGGING_SOURCE_MULTILINE_MAP_JSON: Used to specify the mapping of source to multi-row configuration. If a log is not configured with multiline_match, the corresponding multiline_match is found and used here based on its source. Because the multiline_match value is a regular expression, it is more complex, so the value format is a JSON string that can be coded and compressed into a single line using json.cn.

Modify the containerd_address configuration entry of container.conf to set it to the corresponding sock path.

Change the volumes containerd-socket of DataKit.yaml, mount the new path into the DataKit, and configure the environment variablesENV_INPUT_CONTAINER_ENDPOINTS：

# add envs
- env:
  - name: ENV_INPUT_CONTAINER_ENDPOINTS
    value: ["unix:///path/to/new/containerd/containerd.sock"]

# modify mountPath
  - mountPath: /path/to/new/containerd/containerd.sock
    name: containerd-socket
    readOnly: true

# modify volumes
volumes:
- hostPath:
    path: /path/to/new/containerd/containerd.sock
  name: containerd-socket