DataKit Operator¶

DataKit Operator is a project integrated with DataKit in Kubernetes orchestration, aimed at facilitating easier deployment of DataKit, as well as providing other functions such as verification and injection.

Overview¶

DataKit Operator provides automated injection capabilities for Kubernetes clusters through the Kubernetes Admission Controller mechanism, helping users integrate observability more easily. Key features include:

• DDTrace Injection: Automatically inject APM tracing agents for Java applications
• Log Collection: Automatically collect container logs via logfwd sidecar
• Performance Profiling: Inject Flameshot or Profiler components for application performance monitoring
• Configuration Management: Support both global configuration and declarative configuration injection methods

Core Advantages:

• Automated Deployment: No need to manually modify application YAML, reducing configuration errors
• Batch Management: Implement batch injection through namespace and label selectors
• Flexible Configuration: Support JSON configuration and fine-grained control via Annotations
• Version Compatibility: Maintain backward compatibility and support smooth upgrades

Prerequisites¶

• Kubernetes v1.24.1 or higher is recommended, with internet access (to download yaml files and pull corresponding images)
• Ensure MutatingAdmissionWebhook and ValidatingAdmissionWebhook controllers are enabled
• Ensure admissionregistration.k8s.io/v1 API is enabled

Installation¶

$ kubectl create namespace datakit
$ wget https://static.guance.com/datakit-operator/datakit-operator.yaml
$ kubectl apply -f datakit-operator.yaml
$ kubectl get pod -n datakit

NAME                               READY   STATUS    RESTARTS   AGE
datakit-operator-f948897fb-5w5nm   1/1     Running   0          15s

$ helm install datakit-operator datakit-operator \
    --repo https://pubrepo.guance.com/chartrepo/datakit-operator \
    -n datakit --create-namespace

$ helm -n datakit list

$ helm -n datakit get values datakit-operator -a -o yaml > values.yaml
$ helm upgrade datakit-operator datakit-operator \
    --repo https://pubrepo.guance.com/chartrepo/datakit-operator \
    -n datakit \
    -f values.yaml

$ helm uninstall datakit-operator -n datakit

Configuration Explanation¶

DataKit Operator configuration is in JSON format, stored separately as a ConfigMap in Kubernetes, and loaded into the container as environment variables.

{
    "server_listen": "0.0.0.0:9543", // Operator service listening address
    "log_level": "info",             // Operator log level
    "admission_inject_v2": {         // Injection configuration v2
        "ddtraces": [...],           // DDTrace configuration array
        "logfwds": [...],            // Log forwarding configuration array
        "flameshots": [...]          // Profiling configuration array
    },
    "admission_mutate": {            // Configuration mutation
        "loggings": [...]            // Log configuration mutation
    }
}

{
    "server_listen": "0.0.0.0:9543",
    "log_level":     "info",
    "admission_inject": {
        "ddtrace": {...},
        "profiler": {...},
        "logfwd": {...}
    },
    "admission_mutate": {
        "loggings": [...]
    }
}

Injection Methods¶

DataKit Operator supports two resource input methods:

1. Selector Configuration Injection (Imperative)

   Specify the Namespace and Selector of the target Pod by modifying the DataKit-Operator config. If a Pod meets the conditions, injection is performed.

   Advantages: No need to add Annotations to the target Pod (but the target Pod needs to be restarted)

   Disadvantages: Scope is not precise enough, potentially leading to invalid injections

2. Annotation Configuration Injection (Declarative)

   Add Annotations to the target Pod to enable its own injection.

   Advantages: Injection rejection can be precisely controlled via Annotation

   Disadvantages: Injection cannot be triggered solely by Annotation; matching rules still need to be configured, meaning besides enabling injection in the target Pod annotation, other fields in the Operator configuration are also required.

Selector Configuration Injection¶

Batch injection can be achieved by configuring namespace_selectors and label_selectors.

In the admission_inject_v2 configuration, namespace_selectors and label_selectors are configured directly in the array item. Taking DDTrace injection as an example:

{
    "admission_inject_v2": {
        "ddtraces": [
            {
                "namespace_selectors": ["testns"],
                "label_selectors":     ["app=log-output"],
                ...
            }
        ]
    }
}

• namespace_selectors: Namespace selector array, supports regular expression matching. For exact matching, surround the pattern with ^ and $, e.g., ^testns$
• label_selectors: Label selector array, uses Kubernetes Label Selector syntax

If both selectors are configured, the target Pod must satisfy both conditions simultaneously. For guidelines on writing label selectors, refer to this official documentation.

Annotation Configuration Injection¶

Adding specific Annotations to Deployment can control whether injection is allowed. Note that Annotations should be added in the template.

Supported Annotations are as follows:

Example:

    annotations:
    admission.datakit/ddtrace.enabled: "true"
    admission.datakit/logfwd.enabled: "true"

check_annotation Configuration Item Explanation¶

check_annotation is a configuration field used to control how DataKit Operator handles version annotations on Pods. The values and behaviors of this field are as follows:

Annotation Type Explanation¶

DataKit Operator supports two types of annotations with different behaviors:

1. Enable/Disable Annotations (unaffected by check_annotation) These annotations are used to control whether to enable or disable specific functions, unaffected by check_annotation configuration:

2. Version Annotations (affected by check_annotation) These annotations are used to specify component versions, controlled by check_annotation configuration:

Injection Logic Explanation¶

Core Rules:

• admission.datakit/enabled:"false" rejects all injections (highest priority)
• Function-specific enable annotations (e.g., admission.datakit/ddtrace.enabled: "false") reject that function's injection
• When check_annotation: true, corresponding version annotations must exist to inject
• When check_annotation: false, version annotation checks are ignored

Injection Condition Comparison:

Use Case Examples¶

1. Batch Injection (check_annotation: false): Suitable for scenarios requiring automatic injection for large numbers of Pods, without adding version annotations to each Pod.

2. Precise Control (check_annotation: true): Suitable for scenarios requiring strict version control, injecting only for explicitly marked Pods.

Supported Injection Function List¶

Downward API¶

In DataKit Operator v1.4.2 and later versions, envs supports Kubernetes Downward API environment variable value fields. The following are currently supported:

For example, if there is a Pod named nginx-123 in the middleware namespace, and you want to inject the environment variables POD_NAME and POD_NAMESPACE, refer to the following:

{
    "admission_inject_v2": {
        "ddtraces": [
            {
                "namespace_selectors": ["middleware"],
                "language":            "java",
                "image":               "dd-lib-java-init:latest",
                "envs": {
                    "POD_NAME":      "{fieldRef:metadata.name}",
                    "POD_NAMESPACE": "{fieldRef:metadata.namespace}"
                }
            }
        }
    }
}

Ultimately, in that Pod you can see:

$ env | grep POD
POD_NAME=nginx-123
POD_NAMESPACE=middleware

Important Notes on {resourceFieldRef:*}¶

{resourceFieldRef:*} placeholders are used to reference resource limits (limits) and requests (requests) of the first container in a Pod. Note the following when using them:

1. Resource Check: If the Pod's first container does not have the corresponding resource limit or request configured, environment variables using this placeholder will not be injected. For example:
2. If the container does not have limits.cpu set, the {resourceFieldRef:limits.cpu} environment variable will be ignored

3. If the container does not have requests.memory set, the {resourceFieldRef:requests.memory} environment variable will be ignored

4. Unit Explanation:

5. CPU unit is millicores (m), e.g., 500 means 500m (i.e., 0.5 CPU)

6. Memory unit is MiB, e.g., 1024 means 1024Mi (i.e., 1GiB)

7. Only Supports First Container: {resourceFieldRef:*} can only reference resources of the first container in the Pod, not resources of other containers.

8. Usage Example:

{
    "envs": {
        "APP_CPU_LIMIT": "{resourceFieldRef:limits.cpu}",
        "APP_MEMORY_REQUEST": "{resourceFieldRef:requests.memory}"
    }
}

1. Verification Method: You can confirm whether resource placeholders are correctly parsed by checking the injected Pod's environment variables:

kubectl exec <pod-name> -- env | grep APP_
APP_CPU_LIMIT=500
APP_MEMORY_REQUEST=512

FAQ¶

How to disable injection for a specific Pod?¶

Add Annotation "admission.datakit/enabled": "false" to that Pod, and no operations will be performed for it. This has the highest priority.

How does it work?¶

DataKit-Operator uses Kubernetes Admission Controller function for resource injection. For detailed mechanisms, please check the official documentation

What to note in AWS EKS environment?¶

Deploying in an AWS EKS environment may cause DataKit-Operator not to take effect; you need to open port 9543 in the security group.

Troubleshooting Guide¶