DataKit Operator¶

DataKit Operator uses a Kubernetes Admission Webhook to inject tracing, log collection, and profiling components into newly created Pods. It also provides an in-cluster Pod query API.

Overview¶

DataKit Operator provides the following features:

Injection occurs during the Pod CREATE phase and does not modify running Pods. After changing the Operator configuration, an injection image, or workload Annotations, recreate the Pod for the change to take effect.

Admission is fail-open: if injection fails, the Operator logs the error but does not block creation of the application Pod. The webhook in the deployment manifest also uses failurePolicy: Ignore.

Prerequisites¶

• Kubernetes must support admissionregistration.k8s.io/v1; Kubernetes v1.24 or later is recommended.
• The MutatingAdmissionWebhook Admission Controller must be enabled in the cluster.
• Cluster nodes must be able to pull the configured images. In offline environments, synchronize the images to a private registry in advance.

Installation¶

kubectl create namespace datakit
wget https://static.guance.com/datakit-operator/datakit-operator.yaml
kubectl apply -f datakit-operator.yaml
kubectl get pod -n datakit

NAME                                READY   STATUS    RESTARTS   AGE
datakit-operator-f948897fb-5w5nm    1/1     Running   0          15s

helm install datakit-operator datakit-operator \
    --repo https://pubrepo.guance.com/chartrepo/datakit-operator \
    -n datakit --create-namespace

helm -n datakit list

helm -n datakit get values datakit-operator -a -o yaml > values.yaml
helm upgrade datakit-operator datakit-operator \
    --repo https://pubrepo.guance.com/chartrepo/datakit-operator \
    -n datakit \
    -f values.yaml

helm uninstall datakit-operator -n datakit

Configuration¶

The Operator uses JSON configuration. Deployment manifests generally store the configuration in a ConfigMap and load it through the ENV_JSON_CONFIG environment variable.

admission_inject_v2 is used starting with v1.8.0:

{
    "server_listen": "0.0.0.0:9543",
    "log_level": "info",
    "admission_inject_v2": {
        "ddtraces": [],
        "otels": [],
        "logfwds": [],
        "flameshots": [],
        "profilers": []
    },
    "admission_mutate": {
        "loggings": []
    }
}

The preceding example shows only the configuration structure. The distributed deployment templates retain one Java DDTrace rule that matches Pods in the default Namespace; otels is empty, so OpenTelemetry injection is not enabled automatically. This default is a runtime policy that preserves compatibility with existing deployments and does not mean that the Operator supports only Java.

The Operator does not detect the language in application containers automatically. DDTrace also supports Python, PHP, and Node.js, while OpenTelemetry supports Java, Python, and Node.js. To enable these capabilities, follow the DDTrace automatic injection and OpenTelemetry automatic injection documentation and add rules that use mutually exclusive language labels.

The legacy admission_inject configuration remains supported. A valid ddtrace, logfwd, or profiler entry in the legacy configuration overrides the corresponding v2 rules. Do not maintain two valid configurations for the same feature during an upgrade.

Cluster API¶

DataKit Operator v1.8.1 and later provide the Cluster API. This API uses the Operator's Pod informer cache to proxy Pod data queries, reducing direct access to the API Server from each DataKit instance.

The Cluster API is enabled by default. At startup, the Operator checks whether its ServiceAccount has permission to read Pods. If permission is insufficient or cache synchronization fails, the related routes are not registered, but other features can continue to run. The minimum required permissions are:

- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

The API uses the Operator Service. Its default address is https://datakit-operator.datakit.svc:443:

Starting with v1.8.9, use view=ebpf-v1 to obtain a reduced Pod representation for eBPF:

curl -k "https://datakit-operator.datakit.svc:443/v1/cluster/api/v1/pods?view=ebpf-v1"

Injection Rules¶

Every injection configuration must first match a Pod through selectors. An Annotation can reject injection or, when check_annotation is enabled, further restrict injection. It cannot trigger injection independently of selectors.

Selector Configuration¶

namespace_selectors and label_selectors are both arrays:

• Multiple selectors in the same array are matched with OR.
• When both arrays are configured, both the Namespace and Label dimensions must match.
• When an injection rule configures only one dimension, only that dimension is used. If neither dimension is configured, the current rule is disabled and evaluation continues with subsequent rules.
• Logging mutation rules require both dimensions. If either dimension is missing, the current rule is disabled.

namespace_selectors uses Go regular expressions. The string "*" is shorthand for matching all Namespaces; use ^ and $ for exact matches. label_selectors uses Kubernetes Label Selector syntax and extends =, ==, and != with glob matching.

Empty strings, whitespace-only values, invalid Namespace regular expressions, and invalid or empty-constraint Label selectors are invalid entries. At startup, the Operator logs a warning and ignores each invalid entry. If a configured dimension has no valid entries left, the current rule is disabled and evaluation continues with subsequent rules. Use "*" to explicitly match all Namespaces.

The following rule matches only Pods in the production Namespace that have the admission.datakit/ddtrace-language=java label:

{
    "namespace_selectors": ["^production$"],
    "label_selectors": ["admission.datakit/ddtrace-language=java"]
}

The same Pod may match multiple DDTrace or OTel rules. The Operator selects the first rule in configuration order that satisfies the Annotation conditions. Once selected, it does not fall back to subsequent rules because of an invalid language, image, or other configuration error. Use mutually exclusive labels for rules targeting different languages.

When DDTrace and OTel both match, DDTrace takes precedence. If DDTrace injection fails, the Operator does not fall back to OTel.

Annotation Configuration¶

Add Annotations to the Pod or to .spec.template.metadata.annotations in a controller such as a Deployment.

These switch Annotations use permissive semantics: only a value that can be parsed as false disables the corresponding feature. A missing or unparseable Annotation is treated as true. Even when explicitly set to true, the Pod must still match the corresponding configuration rule.

metadata:
  annotations:
    admission.datakit/ddtrace.enabled: "false"
    admission.datakit/otel.enabled: "true"

check_annotation¶

DDTrace, OTel, logfwd, and legacy Profiler rules support check_annotation:

The mappings are:

When check_annotation: true and the Pod provides the corresponding version Annotation, DDTrace, OTel, and Profiler replace the tag in the rule's image without changing the image registry or name. Feature switch Annotations always apply, regardless of check_annotation.

Supported Injection Features¶

Environment Variable Value References¶

The envs field in an injection rule supports literal values and placeholders that are converted to native Kubernetes fieldRef, resourceFieldRef, and secretKeyRef references.

Environment variables preserve configuration order. Later values can reference earlier variables using Kubernetes $(VAR) syntax:

{
    "envs": {
        "POD_NAME": "{fieldRef:metadata.name}",
        "POD_NAMESPACE": "{fieldRef:metadata.namespace}",
        "RESOURCE_TAGS": "pod_name=$(POD_NAME),pod_namespace=$(POD_NAMESPACE)"
    }
}

Unrecognized placeholders are injected as ordinary strings. resourceFieldRef references only the first application container. If that container does not declare the corresponding request or limit, the environment variable is not injected.

{secretKeyRef:*}¶

The Secret reference format is:

{secretKeyRef:<secret-name>.<key>}

For example:

{
    "envs": {
        "ACCESS_KEY_ID": "{secretKeyRef:flameshot-oss.access_key_id}",
        "ACCESS_KEY_SECRET": "{secretKeyRef:flameshot-oss.access_key_secret}"
    }
}

The Operator does not read the Secret; Kubernetes resolves the reference when the container starts. The Secret must be in the same Namespace as the application Pod. If the Secret or key does not exist, the Pod enters CreateContainerConfigError. Because . separates the Secret name from the key, the Secret name in this format cannot contain ..

FAQ¶

Disable Pod Changes¶

Add the following to the Pod template:

admission.datakit/enabled: "false"

Why Did Injection Not Take Effect?¶

Check the following in order:

1. Whether the Pod was recreated after the configuration update.
2. Whether its Namespace and Label match the same rule.
3. Whether admission.datakit/enabled or a feature switch Annotation is false.
4. When check_annotation: true, whether the correct version or configuration Annotation is present.
5. Whether the Operator logs contain warnings about selectors, images, environment variables, or security context conflicts.

What Should I Consider on AWS EKS?¶

The EKS control plane must be able to access port 9543 on the Operator webhook. If injection does not take effect, check whether the cluster security groups and network policies allow traffic in that direction.