Datakit Operator User Guide¶

Overview and Installation¶

Datakit Operator is a collaborative project between Datakit and Kubernetes orchestration. It aims to assist the deployment of Datakit as well as other functions such as verification and injection.

Currently, Datakit Operator provides the following functions:

• Injection DDTrace SDK(Java/Python/Node.js) and related environments. See documentation.
• Injection Sidecar logfwd to collect Pod logging. See documentation.
• Support task distribution for Datakit plugins. See documentation.

Prerequisites:

• Recommended Kubernetes version 1.24.1 or above and internet access (to download yaml file and pull images).
• Ensure MutatingAdmissionWebhook and ValidatingAdmissionWebhook controllers are enabled.
• Ensure admissionregistration.k8s.io/v1 API is enabled.

Installation Steps¶

$ kubectl create namespace datakit
$ wget https://static.guance.com/datakit-operator/datakit-operator.yaml
$ kubectl apply -f datakit-operator.yaml
$ kubectl get pod -n datakit

NAME                               READY   STATUS    RESTARTS   AGE
datakit-operator-f948897fb-5w5nm   1/1     Running   0          15s

$ helm install datakit-operator datakit-operator \
     --repo  https://pubrepo.guance.com/chartrepo/datakit-operator \
     -n datakit --create-namespace

$ helm -n datakit list

$ helm -n datakit get values datakit-operator -a -o yaml > values.yaml
$ helm upgrade datakit-operator datakit-operator \
    --repo https://pubrepo.guance.com/chartrepo/datakit-operator \
    -n datakit \
    -f values.yaml

$ helm uninstall datakit-operator -n datakit

Configuration Explanation¶

Version-1.4.2

The Datakit Operator configuration is in JSON format and is stored in Kubernetes as a separate ConfigMap. It is loaded into the container as environment variables.

The default configuration is as follows:

{
    "server_listen": "0.0.0.0:9543",
    "log_level":     "info",
    "admission_inject": {
        "ddtrace": {
            "enabled_namespaces":     [],
            "enabled_labelselectors": [],
            "images": {
                "java_agent_image":   "pubrepo.guance.com/datakit-operator/dd-lib-java-init:v1.30.1-guance",
            },
            "envs": {
                "DD_AGENT_HOST":           "datakit-service.datakit.svc",
                "DD_TRACE_AGENT_PORT":     "9529",
                "DD_JMXFETCH_STATSD_HOST": "datakit-service.datakit.svc",
                "DD_JMXFETCH_STATSD_PORT": "8125",
                "DD_SERVICE":              "{fieldRef:metadata.labels['service']}",
                "POD_NAME":                "{fieldRef:metadata.name}",
                "POD_NAMESPACE":           "{fieldRef:metadata.namespace}",
                "NODE_NAME":               "{fieldRef:spec.nodeName}",
                "DD_TAGS":                 "pod_name:$(POD_NAME),pod_namespace:$(POD_NAMESPACE),host:$(NODE_NAME)"
            }
        },
        "logfwd": {
            "options": {
                "reuse_exist_volume": "false"
            },
            "images": {
                "logfwd_image": "pubrepo.guance.com/datakit/logfwd:1.28.1"
            }
        },
        "profiler": {
            "images": {
                "java_profiler_image":   "pubrepo.guance.com/datakit-operator/async-profiler:0.1.0",
                "python_profiler_image": "pubrepo.guance.com/datakit-operator/py-spy:0.1.0",
                "golang_profiler_image": "pubrepo.guance.com/datakit-operator/go-pprof:0.1.0"
            },
            "envs": {
                "DK_AGENT_HOST":  "datakit-service.datakit.svc",
                "DK_AGENT_PORT":  "9529",
                "DK_PROFILE_VERSION": "1.2.333",
                "DK_PROFILE_ENV": "prod",
                "DK_PROFILE_DURATION": "240",
                "DK_PROFILE_SCHEDULE": "0 * * * *"
            }
        }
    }
}

The main configuration items are ddtrace, logfwd, and profiler, which specify the injected images and environment variables. In addition, ddtrace also supports batch injection based on enabled_namespaces and enabled_selectors, as detailed in the section below.

Configuration of Images¶

The primary function of the Datakit Operator is to inject images and environment variables, using the images configuration to specify the image addresses. The images configuration consists of multiple Key/Value pairs, where the Key is fixed, and the Value is modified to specify the image address.

Under normal circumstances, images are stored in pubrepo.guance.com/datakit-operator. However, for some special environments where accessing this image repository is not convenient, you can use the following method (taking the dd-lib-java-init image as an example):

1. In an environment where pubrepo.guance.com is accessible, pull the image pubrepo.guance.com/datakit-operator/dd-lib-java-init:v1.30.1-guance, and then re-store it in your own image repository, for example, inside.image.hub/datakit-operator/dd-lib-java-init:v1.30.1-guance.
2. Modify the JSON configuration, changing admission_inject->ddtrace->images->java_agent_image to inside.image.hub/datakit-operator/dd-lib-java-init:v1.30.1-guance, and apply this YAML file.
3. After this, the Datakit Operator will use the new Java Agent image path.

Adding Environment Variables¶

All environment variables that need to be injected must be specified in the configuration file, as the Datakit Operator does not add any environment variables by default.

The environment variable configuration is called envs, envs consists of multiple Key/Value pairs: the Key is a fixed value, and the Value can either be a fixed value or a placeholder, depending on the actual situation.

For example, to add an environment variable testing-env in envs:

    "admission_inject": {
        "ddtrace": {
            # other..
            "envs": {
                "DD_AGENT_HOST":           "datakit-service.datakit.svc",
                "DD_TRACE_AGENT_PORT":     "9529",
                "FAKE_ENV":                "ok"
            }
        }
    }

All containers that have ddtrace agent injected into them will have five environment variables added to their envs.

In Datakit Operator v1.4.2 and later versions, envs envs support for the Kubernetes Downward API environment variable fetch field. The following are now supported:

• metadata.name: The pod's name.
• metadata.namespace: The pod's namespace.
• metadata.uid: The pod's unique ID.
• metadata.annotations['<KEY>']: The value of the pod's annotation named <KEY> (for example, metadata.annotations['myannotation']).
• metadata.labels['<KEY>']: The text value of the pod's label named <KEY> (for example, metadata.labels['mylabel']).
• spec.serviceAccountName: The name of the pod's service account.
• spec.nodeName: The name of the node where the Pod is executing.
• status.hostIP: The primary IP address of the node to which the Pod is assigned.
• status.hostIPs: The IP addresses is a dual-stack version of status.hostIP, the first is always the same as status.hostIP. The field is available if you enable the PodHostIPs feature gate.
• status.podIP: The pod's primary IP address (usually, its IPv4 address).
• status.podIPs: The IP addresses is a dual-stack version of status.podIP, the first is always the same as status.podIP.

For example, if there is a Pod with the name nginx-123 and the namespace middleware, and you want to inject the environment variables POD_NAME and POD_NAMESPACE, refer to the following configuration:

{
    "admission_inject": {
        "ddtrace": {
            "envs": {
                "POD_NAME":      "{fieldRef:metadata.name}",
                "POD_NAMESPACE": "{fieldRef:metadata.namespace}"
            }
        }
    }
}

Eventually, the environment variables can be seen in the Pod:

$ env | grep POD
POD_NAME=nginx-123
POD_NAMESPACE=middleware

Injection Methods¶

Datakit-Operator supports two methods for resource injection global configuration namespaces and selectors, and adding specific annotations to target Pods. The differences between them are as follows:

• Global Configuration: Namespace and Selector: By modifying the Datakit-Operator configuration, you specify the target Pod's Namespace and Selector. If a Pod matches the criteria, the resource injection will occur.

  • Advantages: No need to add annotations to the target Pod (but the target Pod must be restarted).
  • Disadvantages: The scope is not precise enough, which may lead to unnecessary injections.

• Adding Annotations to Target Pods: Add annotations to the target Pod, and Datakit-Operator will check the Pod's annotations to decide whether to perform the injection based on the conditions.

  • Advantages: Precise scope, preventing unnecessary injections.
  • Disadvantages: You must manually add annotations to the target Pod, and the Pod needs to be restarted.

Global Configuration: Namespaces and Selectors¶

The enabled_namespaces and enabled_labelselectors fields are specific to ddtrace. They are object arrays that require the specification of namespace and language. The relationships between the arrays are "OR" (i.e., any match in the array will trigger injection). The configuration is written as follows (refer to the configuration details later):

{
    "server_listen": "0.0.0.0:9543",
    "log_level":     "info",
    "admission_inject": {
        "ddtrace": {
            "enabled_namespaces": [
                {
                    "namespace": "testns",  # Specify the namespace
                    "language": "java"      # Specify the language for the agent to inject
                }
            ],
            "enabled_labelselectors": [
                {
                    "labelselector": "app=log-output",  # Specify the label selector
                    "language": "java"                  # Specify the language for the agent to inject
                }
            ]
            # other..
        }
    }
}

If a Pod satisfies both the enabled_namespaces rule and the enabled_labelselectors rule, the configuration in enabled_labelselectors will take precedence (usually applied when the language value is used).

For guidelines on how to write labelselector, please refer to the official documentation.

Adding Annotation Configuration for Injection¶

To inject the ddtrace file into a Pod, add the specified annotation to the Deployment. Make sure to add the annotation to the template section.

The annotation format is as follows:

• The key is admission.datakit/%s-lib.version, where %s should be replaced with the desired language. Currently, it supports java.
• The value is the specified version number. By default, it uses the version specified by the Datakit-Operator configuration in the java_agent_image setting. For more details, see the configuration explanation below.

For example, to add an annotation:

      annotations:
        admission.datakit/java-lib.version: "v1.36.2-guance"

This indicates that the image version to be injected for this Pod is v1.36.2-guance. The image address is taken from the configuration admission_inject -> ddtrace -> images -> java_agent_image, where the image version is replaced with "v1.36.2-guance", similar to pubrepo.guance.com/datakit-operator/dd-lib-java-init:v1.36.2-guance.

Using Datakit-Operator to Inject Files and Programs¶

In large Kubernetes clusters, it can be quite difficult to make bulk configuration changes. Datakit-Operator will determine whether or not to modify or inject data based on Annotation configuration.

The following functions are currently supported:

• Injection of ddtrace agent and environment
• Mounting of logfwd sidecar and enabling log collection
• Inject async-profiler for JVM profiling Experimental
• Inject py-spy for Python profiling Experimental

DDTrace Agent¶

Usage¶

1. On the target Kubernetes cluster, download and install Datakit-Operator.
2. Add a Annotation admission.datakit/java-lib.version: "" in deployment.

Example¶

The following is an example of Deployment that injects dd-java-lib into all Pods created by Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        admission.datakit/java-lib.version: ""
    spec:
      containers:
      - name: nginx
        image: nginx:1.22
        ports:
        - containerPort: 80

Create a resource using yaml file:

kubectl apply -f nginx.yaml

Verify as follows:

$ kubectl get pod
NAME                                   READY   STATUS    RESTARTS      AGE
nginx-deployment-7bd8dd85f-fzmt2       1/1     Running   0             4s

$ kubectl get pod nginx-deployment-7bd8dd85f-fzmt2 -o=jsonpath={.spec.initContainers\[\*\].name}
datakit-lib-init

logfwd¶

Prerequisites¶

logfwd is a proprietary log collection application for Datakit. To use it, you need to first deploy Datakit in the same Kubernetes cluster and satisfy the following two conditions:

1. The Datakit logfwdserver collector is enabled, for example, listening on port 9533.
2. The Datakit service needs to open port 9533 to allow other Pods to access datakit-service.datakit.svc:9533.

Instructions¶

1. On the target Kubernetes cluster, download and install Datakit-Operator.
2. In the deployment, add the specified Annotation to indicate that a logfwd sidecar needs to be mounted. Note that the Annotation should be added in the template.
   • The key is uniformly admission.datakit/logfwd.instances.
   • The value is a JSON string of specific logfwd configuration, as shown below:

[
    {
        "datakit_addr": "datakit-service.datakit.svc:9533",
        "loggings": [
            {
                "logfiles": ["<your-logfile-path>"],
                "ignore": [],
                "source": "<your-source>",
                "service": "<your-service>",
                "pipeline": "<your-pipeline.p>",
                "character_encoding": "",
                "multiline_match": "<your-match>",
                "tags": {}
            },
            {
                "logfiles": ["<your-logfile-path-2>"],
                "source": "<your-source-2>"
            }
        ]
    }
]

Parameter explanation can refer to logfwd configuration:

• datakit_addr is the Datakit logfwdserver address.
• loggings is the main configuration and is an array that can refer to Datakit logging collector.
  • logfiles is a list of log files, which can specify absolute paths and support batch specification using glob rules. Absolute paths are recommended.
  • ignore filters file paths using glob rules. If it meets any filtering condition, the file will not be collected.
  • source is the data source. If it is empty, 'default' will be used by default.
  • service adds a new tag. If it is empty, $source will be used by default.
  • pipeline is the Pipeline script path. If it is empty, $source.p will be used. If $source.p does not exist, the Pipeline will not be used. (This script file exists on the DataKit side.)
  • character_encoding selects an encoding. If the encoding is incorrect, the data cannot be viewed. It is recommended to leave it blank. Supported encodings include utf-8, utf-16le, utf-16le, gbk, gb18030, or "".
  • multiline_match is for multiline matching, as described in Datakit Log Multiline Configuration. Note that since it is in the JSON format, it does not support the "unescaped writing method" of three single quotes. The regex ^\d{4} needs to be written as ^\\d{4} with an escape character.
  • tags adds additional tags in JSON map format, such as { "key1":"value1", "key2":"value2" }.

When injecting logfwd, it is allowed to reuse a volume with the same path to avoid injection errors caused by an existing volume at the same path. To enable this, set the configuration admission_inject->logfwd->options->reuse_exist_volume to true.

For example, if the target Pod has a volume mounted at the path /var/log, and /var/log is the directory to be collected:

spec:
  container:
    # other...
    volumeMounts:
    - name: volume-log
      mountPath: /var/log
  volumes:
  - name: volume-log
    emptyDir: {}

When reuse_exist_volume is enabled, no new volume or volumeMount will be added, and the existing volume-log will be reused.

Example¶

Here is an example Deployment that continuously writes data to a file using shell and configures the collection of that file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logging-deployment
  labels:
    app: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logging
  template:
    metadata:
      labels:
        app: logging
      annotations:
        admission.datakit/logfwd.instances: '[{"datakit_addr":"datakit-service.datakit.svc:9533","loggings":[{"logfiles":["/var/log/log-test/*.log"],"source":"deployment-logging","tags":{"key01":"value01"}}]}]'
    spec:
      containers:
      - name: log-container
        image: busybox
        args: [/bin/sh, -c, 'mkdir -p /var/log/log-test; i=0; while true; do printf "$(date "+%F %H:%M:%S") [%-8d] Bash For Loop Examples.\\n" $i >> /var/log/log-test/1.log; i=$((i+1)); sleep 1; done']

Creating Resources Using yaml File:

$ kubectl apply -f logging.yaml
...

Verify as follows:

$ kubectl get pod
NAME                                   READY   STATUS    RESTARTS      AGE
logging-deployment-5d48bf9995-vt6bb       1/1     Running   0             4s

$ kubectl get pod logging-deployment-5d48bf9995-vt6bb -o=jsonpath={.spec.containers\[\*\].name}
log-container datakit-logfwd

Finally, you can check whether the logs have been collected on the Guance Cloud Log Platform.

FAQ¶

• How to specify that a certain Pod should not be injected? Add the annotation "admission.datakit/enabled": "false" to the Pod. This will prevent any actions from being performed on it, with the highest priority.

• Datakit-Operator utilizes Kubernetes Admission Controller functionality for resource injection. For detailed mechanisms, please refer to the official documentation