Skip to content

OpenTelemetry


OpenTelemetry (OTEL) is an observability project in CNCF. This document explains how to collect OTEL data on DataKit and what you should configure for Java and Go scenarios.

Configuration

Go to conf.d/opentelemetry in the DataKit installation directory, copy opentelemetry.conf.sample to opentelemetry.conf, and edit it.

[[inputs.opentelemetry]]
  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to center is "project_name"
  # customer_tags = ["sink_project", "custom.otel.tag", "reg:key_*"]

  ## If set to true, all Attributes will be extracted and message.Attributes will be empty.
  # customer_tags_all = false

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
  # compatible_ddtrace=false

  ## split service.name form xx.system.
  ## see: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md
  split_service_name = true

  ## delete trace message
  # del_message = true

  ## logging message data max length,default is 500kb
  log_max = 500

  ## JSON marshaler: set JSON marshaler. available marshaler are:
  ##   gojson/jsoniter/protojson
  ##
  ## For better performance, gojson and jsoniter is better than protojson,
  ## for compatible reason we still use protojson as default.
  jmarshaler = "protojson"

  ## cleaned the top-level fields in message. Default true
  clean_message = true

  ## tracing_metric_enable: trace_hits trace_hits_by_http_status trace_latency trace_errors trace_errors_by_http_status trace_apdex.
  ## Extract the above metrics from the collection traces.
  # tracing_metric_enable = true

  ## If true, global host tags will not be added to OTEL metrics.
  # tracing_metric_disable_global_host_tags = false

  ## Blacklist of metric tags: There are many labels in the metric: "tracing_metrics".
  ## If you want to remove certain tag, you can use the blacklist to remove them.
  ## By default, it includes: source,span_name,env,service,status,version,resource,http_status_code,http_status_class
  ## and "customer_tags", k8s related tags, and others service.
  # tracing_metric_tag_blacklist = ["resource", "operation", "tag_a", "tag_b"]

  ## White list of metric tags: There are many labels in the metric: "tracing_metrics".
  # tracing_metric_tag_whitelist = []

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.opentelemetry.close_resource]
  # service1 = ["resource1", "resource2", ...]
  # service2 = ["resource1", "resource2", ...]
  # "*" = ["close_resource_under_all_services"]
  # ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.opentelemetry.sampler]
  # sampling_rate = 1.0

  # [inputs.opentelemetry.tags]
  # key1 = "value1"
  # key2 = "value2"
  # ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.opentelemetry.threads]
  # buffer = 100
  # threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.opentelemetry.storage]
  # path = "./otel_storage"
  # capacity = 5120

  ## OTEL agent HTTP config for trace and metrics
  ## If enable set to be true, trace and metrics will be received on path respectively, by default is:
  ## trace : /otel/v1/traces
  ## metric: /otel/v1/metrics
  ## and the client side should be configured properly with Datakit listening port(default: 9529)
  ## or custom HTTP request path.
  ## for example http://127.0.0.1:9529/otel/v1/traces
  ## The acceptable http_status_ok values will be 200 or 202.
  [inputs.opentelemetry.http]
    http_status_ok = 200
    trace_api = "/otel/v1/traces"
    metric_api = "/otel/v1/metrics"
    logs_api = "/otel/v1/logs"

  ## OTEL agent GRPC config for trace and metrics.
  ## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
  ## add is the listening on address for GRPC server.
  [inputs.opentelemetry.grpc]
    addr = "127.0.0.1:4317"
    max_payload = 16777216 # default 16MiB

  ## If 'expected_headers' is well configed, then the obligation of sending certain wanted HTTP headers is on the client side,
  ## otherwise HTTP status code 400(bad request) will be provoked.
  ## Note: expected_headers will be effected on both trace and metrics if setted up.
  # [inputs.opentelemetry.expected_headers]
  # ex_version = "1.2.3"
  # ex_name = "env_resource_name"
  # ...

Restart DataKit after configuration: restart DataKit

Configure the collector with ConfigMap or ENV_DATAKIT_INPUTS: ConfigMap injection or ENV configuration.

Or set environment variables. Add the collector to ENV_DEFAULT_ENABLED_INPUTS first:

  • ENV_INPUT_OTEL_CUSTOMER_TAGS

    Whitelist to tags

    Type: JSON

    input.conf: customer_tags

    Example: '["project_id", "custom.tag"]'

  • ENV_INPUT_OTEL_CUSTOMER_TAGS_ALL

    extracted all attributes to tags

    Type: Boolean

    input.conf: customer_tags_all

    Default: false

  • ENV_INPUT_OTEL_METRIC_ENABLE_AGGREGATE

    Enable metric aggregate

    Type: Boolean

    input.conf: metric_enable_aggregate

    Default: false

  • ENV_INPUT_OTEL_KEEP_RARE_RESOURCE

    Keep rare tracing resources list switch

    Type: Boolean

    input.conf: keep_rare_resource

    Default: false

  • ENV_INPUT_OTEL_COMPATIBLE_DD_TRACE

    Convert trace_id to decimal, compatible with DDTrace

    Type: Boolean

    input.conf: compatible_dd_trace

    Default: false

  • ENV_INPUT_OTEL_SPLIT_SERVICE_NAME

    Get xx.system from span.Attributes to replace service name

    Type: Boolean

    input.conf: split_service_name

    Default: false

  • ENV_INPUT_OTEL_TRACING_METRIC_ENABLE

    These metrics capture request counts, error counts, and latency measures.

    Type: Boolean

    input.conf: tracing_metric_enable

    Default: false

  • ENV_INPUT_OTEL_TRACING_METRIC_DISABLE_GLOBAL_HOST_TAGS

    Disable add global host tags to OTEL metrics

    Type: Boolean

    input.conf: tracing_metric_disable_global_host_tags

    Default: false

  • ENV_INPUT_OTEL_TRACING_METRIC_TAG_BLACKLIST

    Blacklist of tags in the metric: tracing_metrics

    Type: JSON

    input.conf: tracing_metric_tag_blacklist

    Example: '["tag_a", "tag_b"]'

  • ENV_INPUT_OTEL_TRACING_METRIC_TAG_WHITELIST

    Whitelist of tags in the metric: tracing_metrics

    Type: JSON

    input.conf: tracing_metric_tag_whitelist

    Example: '["tag_a", "tag_b"]'

  • ENV_INPUT_OTEL_DEL_MESSAGE

    Delete trace message

    Type: Boolean

    input.conf: del_message

    Default: false

  • ENV_INPUT_OTEL_OMIT_ERR_STATUS

    Whitelist to error status

    Type: JSON

    input.conf: omit_err_status

    Example: '["404", "403", "400"]'

  • ENV_INPUT_OTEL_CLOSE_RESOURCE

    Ignore tracing resources that service (regular)

    Type: JSON

    input.conf: close_resource

    Example: '{"service1":["resource1","other"],"service2":["resource2","other"]}'

  • ENV_INPUT_OTEL_SAMPLER

    Global sampling rate

    Type: Float

    input.conf: sampler

    Example: 0.3

  • ENV_INPUT_OTEL_THREADS

    Total number of threads and buffer

    Type: JSON

    input.conf: threads

    Example: '{"buffer":1000, "threads":100}'

  • ENV_INPUT_OTEL_STORAGE

    Local cache file path and size (MB)

    Type: JSON

    input.conf: storage

    Example: '{"storage":"./otel_storage", "capacity": 5120}'

  • ENV_INPUT_OTEL_HTTP

    HTTP agent config

    Type: JSON

    input.conf: http

    Example: '{"enable":true, "http_status_ok": 200, "trace_api": "/otel/v1/traces", "metric_api": "/otel/v1/metrics"}'

  • ENV_INPUT_OTEL_GRPC

    GRPC agent config

    Type: JSON

    input.conf: grpc

    Example: '{"addr": "127.0.0.1:4317", "max_payload": 16777216 }'

  • ENV_INPUT_OTEL_EXPECTED_HEADERS

    If expected_headers is well config, then the obligation of sending certain wanted HTTP headers is on the client side

    Type: JSON

    input.conf: expected_headers

    Example: '{"ex_version": "1.2.3", "ex_name": "env_resource_name"}'

  • ENV_INPUT_OTEL_CLEAN_MESSAGE

    Clean message generate smaller message field

    Type: Boolean

    input.conf: clean_message

    Example: true/false

  • ENV_INPUT_OTEL_TAGS

    Customize tags. If there is a tag with the same name in the configuration file, it will be overwritten

    Type: JSON

    input.conf: tags

    Example: '{"k1":"v1", "k2":"v2", "k3":"v3"}'

The customer_tags field supports regular-expression matching. Use a fixed prefix reg: for regex values, for example reg:key_*.

Notes

  1. We recommend gRPC for OTEL ingestion by default because it has better compression and lower overhead.
  2. Since DataKit 1.10.0, OTEL HTTP routes are configurable. Defaults:
  3. traces: /otel/v1/traces
  4. metrics: /otel/v1/metrics
  5. logs: /otel/v1/logs
  6. For float/double values, precision is kept to two decimal places.
  7. Both HTTP and gRPC support gzip compression. Enable it with exporter-side config, for example OTEL_EXPORTER_OTLP_COMPRESSION=gzip.
  8. HTTP requests support both JSON and Protobuf in clients, but DataKit HTTP handlers parse Protobuf only.
Warning
  • DDTrace service names are based on DDTrace service tags or referenced libraries.
  • OTEL service names are determined by otel.service.name.
  • To split service names by resource type (for example db.system, rpc.system, messaging.system), enable:

split_service_name = true - The default priority is to derive service name from db.system, then rpc.system, then messaging.system when the split mode is enabled.

If you are using OTEL HTTP exporter, configure the endpoint paths explicitly in DataKit: traces /otel/v1/traces, metrics /otel/v1/metrics, logs /otel/v1/logs (default port is 9529).

Java Agent V2 protocol behavior

In OTEL Java Agent V2, default OTLP protocol is http/protobuf. To keep compatibility, you can still switch back to gRPC:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
  -Dotel.service.name=app \
  -jar app.jar

For HTTP mode (DataKit default path), configure each exporter endpoint:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.logs.endpoint=http://localhost:9529/otel/v1/logs \
  -Dotel.exporter.otlp.traces.endpoint=http://localhost:9529/otel/v1/traces \
  -Dotel.exporter.otlp.metrics.endpoint=http://localhost:9529/otel/v1/metrics \
  -Dotel.service.name=app \
  -jar app.jar

Disable OTEL logs when not needed:

-Dotel.logs.exporter=none

For V2 release notes, see: GitHub-v2.0.0

Common commands

The table below is a practical subset of configuration items used for DataKit ingestion.

ENV (System Property) Description
OTEL_SDK_DISABLED(otel.sdk.disabled) Disable entire SDK (false by default).
OTEL_RESOURCE_ATTRIBUTES(otel.resource.attributes) Add global resource tags, e.g. service.name=app,project=app-a.
OTEL_SERVICE_NAME(otel.service.name) Override service name, priority higher than resource tags.
OTEL_LOG_LEVEL(otel.log.level) SDK log level (info by default).
OTEL_PROPAGATORS(otel.propagators) Propagation format (tracecontext,baggage by default).
OTEL_TRACES_SAMPLER(otel.traces.sampler) Sampling strategy.
OTEL_TRACES_SAMPLER_ARG(otel.traces.sampler.arg) Sampler arguments, default 1.0 (0~1.0).
OTEL_EXPORTER_OTLP_PROTOCOL(otel.exporter.otlp.protocol) Transport protocol, default grpc; supported grpc and http/protobuf.
OTEL_EXPORTER_OTLP_ENDPOINT(otel.exporter.otlp.endpoint) General OTLP endpoint, e.g. http://datakit-host:4317 (gRPC) or http://datakit-host:9529 (HTTP base).
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT(otel.exporter.otlp.traces.endpoint) Trace endpoint for HTTP mode, e.g. http://datakit-host:9529/otel/v1/traces.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT(otel.exporter.otlp.metrics.endpoint) Metrics endpoint for HTTP mode, e.g. http://datakit-host:9529/otel/v1/metrics.
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT(otel.exporter.otlp.logs.endpoint) Logs endpoint for HTTP mode, e.g. http://datakit-host:9529/otel/v1/logs.
OTEL_TRACES_EXPORTER(otel.traces.exporter) Trace exporter type; otlp by default.
OTEL_LOGS_EXPORTER(otel.logs.exporter) Log exporter type; set to otlp explicitly when logs are needed.
OTEL_METRICS_EXPORTER(otel.metrics.exporter) Metric exporter type; set to otlp explicitly for OTEL metric ingestion.

Starting from DataKit 1.85.0, http/json is removed from supported DataKit HTTP content types. Use http/protobuf only.

Pass otel.javaagent.debug=true to print Java Agent debug logs (use with caution in production).

Trace sampling

Choose head-based or tail-based sampling.

Tag extraction

From DataKit 1.22.0, fixed tags are extracted by default (blacklist mode was removed).

Attributes Tags Description
http.url http_url Full request URL
http.hostname http_hostname Request hostname
http.route http_route HTTP route
http.status_code http_status_code HTTP status code
http.request.method http_request_method HTTP request method
http.method http_method Same as above
http.client_ip http_client_ip Client IP
http.scheme http_scheme Request protocol
url.full url_full Full request URL
url.scheme url_scheme URL scheme
url.path url_path Request path
url.query url_query Query string
span_kind span_kind Span role
db.system db_system DB system
db.operation db_operation DB operation
db.name db_name Database name
db.statement db_statement SQL statement
server.address server_address Service host
net.host.name net_host_name Host name
server.port server_port Service port
net.host.port net_host_port Host port
network.peer.address network_peer_address Peer host
network.peer.port network_peer_port Peer port
network.transport network_transport Network protocol
messaging.system messaging_system Message queue type
messaging.operation messaging_operation Message operation
messaging.message messaging_message Message details
messaging.destination messaging_destination Message destination
rpc.service rpc_service RPC service name
rpc.system rpc_system RPC system
error error Whether error
error.message error_message Error message
error.stack error_stack Error stack
error.type error_type Error type
project project Project tag
version version Version
env env Environment
host host Host tag
pod_name pod_name Pod name
pod_namespace pod_namespace Pod namespace
telemetry.sdk.language sdk_language SDK language
telemetry.sdk.name sdk_name SDK name
telemetry.sdk.version sdk_version SDK version

To add custom resource tags:

-Dotel.resource.attributes=service.name=app,version=1.1.0,env=prod
Span kind
  • unspecified: not set
  • internal: internal span
  • server: server side
  • client: client side
  • producer: message producer
  • consumer: message consumer

Metrics

Java Instrumentation can report JVM and JMX metrics through its built-in metrics Pipeline.

  • Control JMX collection with otel.jmx.enabled=true/false (default is enabled).
  • Adjust scan interval by otel.jmx.discovery.delay (milliseconds).

For built-in JMX extension details, see: GitHub OTEL JMX Metrics

Histogram conversion

OTEL histogram metrics are converted to Prometheus style metrics:

Input buckets:

[0, 10), [10, 50), [50, 100)

Converted metrics:

my_histogram_bucket{le="10"} 100
my_histogram_bucket{le="50"} 200
my_histogram_bucket{le="100"} 250

And DataKit additionally generates:

my_histogram_count 250
my_histogram_max 100
my_histogram_min 50
my_histogram_sum 12345.67

Metrics ending with _bucket correspond to histogram buckets and should also include _count, _sum, _min, _max.

Log collection

Version-1.33.0

DataKit receives OTEL logs through OTLP. Default is disabled in OTEL V1 and needs to be enabled explicitly.

# environment variable form
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<DataKit Addr>:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
java -jar app.jar

# command line form
java -javaagent:/path/to/agent.jar \
  -Dotel.logs.exporter=otlp \
  -Dotel.exporter.otlp.endpoint=http://<DataKit Addr>:4317 \
  -Dotel.exporter.otlp.protocol=grpc \
  -jar app.jar

For HTTP protocol in OTEL V2, use -Dotel.exporter.otlp.protocol=http/protobuf and set -Dotel.exporter.otlp.logs.endpoint=http://<DataKit Addr>:9529/otel/v1/logs.

Default maximum message field size is 500KB. Can not parts above the limit are truncated.

Feedback

Is this page helpful? ×