OpenTelemetry
OpenTelemetry (OTEL) is an observability project in CNCF. This document explains how to collect OTEL data on DataKit and what you should configure for Java and Go scenarios.
Configuration¶
Go to conf.d/opentelemetry in the DataKit installation directory, copy opentelemetry.conf.sample to opentelemetry.conf, and edit it.
[[inputs.opentelemetry]]
## customer_tags will work as a whitelist to prevent tags send to data center.
## All . will replace to _ ,like this :
## "project.name" to send to center is "project_name"
# customer_tags = ["sink_project", "custom.otel.tag", "reg:key_*"]
## If set to true, all Attributes will be extracted and message.Attributes will be empty.
# customer_tags_all = false
## Keep rare tracing resources list switch.
## If some resources are rare enough(not presend in 1 hour), those resource will always send
## to data center and do not consider samplers and filters.
# keep_rare_resource = false
## By default every error presents in span will be send to data center and omit any filters or
## sampler. If you want to get rid of some error status, you can set the error status list here.
# omit_err_status = ["404"]
## compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
# compatible_ddtrace=false
## split service.name form xx.system.
## see: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md
split_service_name = true
## delete trace message
# del_message = true
## logging message data max length,default is 500kb
log_max = 500
## JSON marshaler: set JSON marshaler. available marshaler are:
## gojson/jsoniter/protojson
##
## For better performance, gojson and jsoniter is better than protojson,
## for compatible reason we still use protojson as default.
jmarshaler = "protojson"
## cleaned the top-level fields in message. Default true
clean_message = true
## tracing_metric_enable: trace_hits trace_hits_by_http_status trace_latency trace_errors trace_errors_by_http_status trace_apdex.
## Extract the above metrics from the collection traces.
# tracing_metric_enable = true
## If true, global host tags will not be added to OTEL metrics.
# tracing_metric_disable_global_host_tags = false
## Blacklist of metric tags: There are many labels in the metric: "tracing_metrics".
## If you want to remove certain tag, you can use the blacklist to remove them.
## By default, it includes: source,span_name,env,service,status,version,resource,http_status_code,http_status_class
## and "customer_tags", k8s related tags, and others service.
# tracing_metric_tag_blacklist = ["resource", "operation", "tag_a", "tag_b"]
## White list of metric tags: There are many labels in the metric: "tracing_metrics".
# tracing_metric_tag_whitelist = []
## Ignore tracing resources map like service:[resources...].
## The service name is the full service name in current application.
## The resource list is regular expressions uses to block resource names.
## If you want to block some resources universally under all services, you can set the
## service name as "*". Note: double quotes "" cannot be omitted.
# [inputs.opentelemetry.close_resource]
# service1 = ["resource1", "resource2", ...]
# service2 = ["resource1", "resource2", ...]
# "*" = ["close_resource_under_all_services"]
# ...
## Sampler config uses to set global sampling strategy.
## sampling_rate used to set global sampling rate.
# [inputs.opentelemetry.sampler]
# sampling_rate = 1.0
# [inputs.opentelemetry.tags]
# key1 = "value1"
# key2 = "value2"
# ...
## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
## buffer is the size of jobs' buffering of worker channel.
## threads is the total number fo goroutines at running time.
# [inputs.opentelemetry.threads]
# buffer = 100
# threads = 8
## Storage config a local storage space in hard dirver to cache trace data.
## path is the local file path used to cache data.
## capacity is total space size(MB) used to store data.
# [inputs.opentelemetry.storage]
# path = "./otel_storage"
# capacity = 5120
## OTEL agent HTTP config for trace and metrics
## If enable set to be true, trace and metrics will be received on path respectively, by default is:
## trace : /otel/v1/traces
## metric: /otel/v1/metrics
## and the client side should be configured properly with Datakit listening port(default: 9529)
## or custom HTTP request path.
## for example http://127.0.0.1:9529/otel/v1/traces
## The acceptable http_status_ok values will be 200 or 202.
[inputs.opentelemetry.http]
http_status_ok = 200
trace_api = "/otel/v1/traces"
metric_api = "/otel/v1/metrics"
logs_api = "/otel/v1/logs"
## OTEL agent GRPC config for trace and metrics.
## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
## add is the listening on address for GRPC server.
[inputs.opentelemetry.grpc]
addr = "127.0.0.1:4317"
max_payload = 16777216 # default 16MiB
## If 'expected_headers' is well configed, then the obligation of sending certain wanted HTTP headers is on the client side,
## otherwise HTTP status code 400(bad request) will be provoked.
## Note: expected_headers will be effected on both trace and metrics if setted up.
# [inputs.opentelemetry.expected_headers]
# ex_version = "1.2.3"
# ex_name = "env_resource_name"
# ...
Restart DataKit after configuration: restart DataKit
Configure the collector with ConfigMap or ENV_DATAKIT_INPUTS:
ConfigMap injection
or
ENV configuration.
Or set environment variables. Add the collector to ENV_DEFAULT_ENABLED_INPUTS first:
-
ENV_INPUT_OTEL_CUSTOMER_TAGS
Whitelist to tags
Type: JSON
input.conf:
customer_tagsExample:
'["project_id", "custom.tag"]' -
ENV_INPUT_OTEL_CUSTOMER_TAGS_ALL
extracted all attributes to tags
Type: Boolean
input.conf:
customer_tags_allDefault: false
-
ENV_INPUT_OTEL_METRIC_ENABLE_AGGREGATE
Enable metric aggregate
Type: Boolean
input.conf:
metric_enable_aggregateDefault: false
-
ENV_INPUT_OTEL_KEEP_RARE_RESOURCE
Keep rare tracing resources list switch
Type: Boolean
input.conf:
keep_rare_resourceDefault: false
-
ENV_INPUT_OTEL_COMPATIBLE_DD_TRACE
Convert trace_id to decimal, compatible with DDTrace
Type: Boolean
input.conf:
compatible_dd_traceDefault: false
-
ENV_INPUT_OTEL_SPLIT_SERVICE_NAME
Get xx.system from span.Attributes to replace service name
Type: Boolean
input.conf:
split_service_nameDefault: false
-
ENV_INPUT_OTEL_TRACING_METRIC_ENABLE
These metrics capture request counts, error counts, and latency measures.
Type: Boolean
input.conf:
tracing_metric_enableDefault: false
-
ENV_INPUT_OTEL_TRACING_METRIC_DISABLE_GLOBAL_HOST_TAGS
Disable add global host tags to OTEL metrics
Type: Boolean
input.conf:
tracing_metric_disable_global_host_tagsDefault: false
-
ENV_INPUT_OTEL_TRACING_METRIC_TAG_BLACKLIST
Blacklist of tags in the metric:
tracing_metricsType: JSON
input.conf:
tracing_metric_tag_blacklistExample:
'["tag_a", "tag_b"]' -
ENV_INPUT_OTEL_TRACING_METRIC_TAG_WHITELIST
Whitelist of tags in the metric:
tracing_metricsType: JSON
input.conf:
tracing_metric_tag_whitelistExample:
'["tag_a", "tag_b"]' -
ENV_INPUT_OTEL_DEL_MESSAGE
Delete trace message
Type: Boolean
input.conf:
del_messageDefault: false
-
ENV_INPUT_OTEL_OMIT_ERR_STATUS
Whitelist to error status
Type: JSON
input.conf:
omit_err_statusExample:
'["404", "403", "400"]' -
ENV_INPUT_OTEL_CLOSE_RESOURCE
Ignore tracing resources that service (regular)
Type: JSON
input.conf:
close_resourceExample:
'{"service1":["resource1","other"],"service2":["resource2","other"]}' -
ENV_INPUT_OTEL_SAMPLER
Global sampling rate
Type: Float
input.conf:
samplerExample: 0.3
-
ENV_INPUT_OTEL_THREADS
Total number of threads and buffer
Type: JSON
input.conf:
threadsExample:
'{"buffer":1000, "threads":100}' -
ENV_INPUT_OTEL_STORAGE
Local cache file path and size (MB)
Type: JSON
input.conf:
storageExample:
'{"storage":"./otel_storage", "capacity": 5120}' -
ENV_INPUT_OTEL_HTTP
HTTP agent config
Type: JSON
input.conf:
httpExample:
'{"enable":true, "http_status_ok": 200, "trace_api": "/otel/v1/traces", "metric_api": "/otel/v1/metrics"}' -
ENV_INPUT_OTEL_GRPC
GRPC agent config
Type: JSON
input.conf:
grpcExample:
'{"addr": "127.0.0.1:4317", "max_payload": 16777216 }' -
ENV_INPUT_OTEL_EXPECTED_HEADERS
If
expected_headersis well config, then the obligation of sending certain wanted HTTP headers is on the client sideType: JSON
input.conf:
expected_headersExample:
'{"ex_version": "1.2.3", "ex_name": "env_resource_name"}' -
ENV_INPUT_OTEL_CLEAN_MESSAGE
Clean message generate smaller
messagefieldType: Boolean
input.conf:
clean_messageExample:
true/false -
ENV_INPUT_OTEL_TAGS
Customize tags. If there is a tag with the same name in the configuration file, it will be overwritten
Type: JSON
input.conf:
tagsExample:
'{"k1":"v1", "k2":"v2", "k3":"v3"}'
The customer_tags field supports regular-expression matching. Use a fixed prefix reg: for regex values, for example reg:key_*.
Notes¶
- We recommend gRPC for OTEL ingestion by default because it has better compression and lower overhead.
- Since DataKit 1.10.0, OTEL HTTP routes are configurable. Defaults:
- traces:
/otel/v1/traces - metrics:
/otel/v1/metrics - logs:
/otel/v1/logs - For
float/doublevalues, precision is kept to two decimal places. - Both HTTP and gRPC support gzip compression. Enable it with exporter-side config, for example
OTEL_EXPORTER_OTLP_COMPRESSION=gzip. - HTTP requests support both JSON and Protobuf in clients, but DataKit HTTP handlers parse Protobuf only.
Warning
- DDTrace service names are based on DDTrace service tags or referenced libraries.
- OTEL service names are determined by
otel.service.name. - To split service names by resource type (for example
db.system,rpc.system,messaging.system), enable:
split_service_name = true
- The default priority is to derive service name from db.system, then rpc.system, then messaging.system when the split mode is enabled.
If you are using OTEL HTTP exporter, configure the endpoint paths explicitly in DataKit:
traces /otel/v1/traces, metrics /otel/v1/metrics, logs /otel/v1/logs (default port is 9529).
Java Agent V2 protocol behavior¶
In OTEL Java Agent V2, default OTLP protocol is http/protobuf.
To keep compatibility, you can still switch back to gRPC:
java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.exporter.otlp.endpoint=http://localhost:4317 \
-Dotel.service.name=app \
-jar app.jar
For HTTP mode (DataKit default path), configure each exporter endpoint:
java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/protobuf \
-Dotel.exporter.otlp.logs.endpoint=http://localhost:9529/otel/v1/logs \
-Dotel.exporter.otlp.traces.endpoint=http://localhost:9529/otel/v1/traces \
-Dotel.exporter.otlp.metrics.endpoint=http://localhost:9529/otel/v1/metrics \
-Dotel.service.name=app \
-jar app.jar
Disable OTEL logs when not needed:
-Dotel.logs.exporter=none
For V2 release notes, see: GitHub-v2.0.0
Common commands¶
The table below is a practical subset of configuration items used for DataKit ingestion.
| ENV (System Property) | Description |
|---|---|
OTEL_SDK_DISABLED(otel.sdk.disabled) |
Disable entire SDK (false by default). |
OTEL_RESOURCE_ATTRIBUTES(otel.resource.attributes) |
Add global resource tags, e.g. service.name=app,project=app-a. |
OTEL_SERVICE_NAME(otel.service.name) |
Override service name, priority higher than resource tags. |
OTEL_LOG_LEVEL(otel.log.level) |
SDK log level (info by default). |
OTEL_PROPAGATORS(otel.propagators) |
Propagation format (tracecontext,baggage by default). |
OTEL_TRACES_SAMPLER(otel.traces.sampler) |
Sampling strategy. |
OTEL_TRACES_SAMPLER_ARG(otel.traces.sampler.arg) |
Sampler arguments, default 1.0 (0~1.0). |
OTEL_EXPORTER_OTLP_PROTOCOL(otel.exporter.otlp.protocol) |
Transport protocol, default grpc; supported grpc and http/protobuf. |
OTEL_EXPORTER_OTLP_ENDPOINT(otel.exporter.otlp.endpoint) |
General OTLP endpoint, e.g. http://datakit-host:4317 (gRPC) or http://datakit-host:9529 (HTTP base). |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT(otel.exporter.otlp.traces.endpoint) |
Trace endpoint for HTTP mode, e.g. http://datakit-host:9529/otel/v1/traces. |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT(otel.exporter.otlp.metrics.endpoint) |
Metrics endpoint for HTTP mode, e.g. http://datakit-host:9529/otel/v1/metrics. |
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT(otel.exporter.otlp.logs.endpoint) |
Logs endpoint for HTTP mode, e.g. http://datakit-host:9529/otel/v1/logs. |
OTEL_TRACES_EXPORTER(otel.traces.exporter) |
Trace exporter type; otlp by default. |
OTEL_LOGS_EXPORTER(otel.logs.exporter) |
Log exporter type; set to otlp explicitly when logs are needed. |
OTEL_METRICS_EXPORTER(otel.metrics.exporter) |
Metric exporter type; set to otlp explicitly for OTEL metric ingestion. |
Starting from DataKit 1.85.0, http/json is removed from supported DataKit HTTP content types. Use http/protobuf only.
Pass otel.javaagent.debug=true to print Java Agent debug logs (use with caution in production).
Trace sampling¶
Choose head-based or tail-based sampling.
- Tail-based sampling (collector side): OpenTelemetry Sampling Best Practices
- Head-based sampling (agent side): OpenTelemetry Java Agent Sampling Strategy
Tag extraction¶
From DataKit 1.22.0, fixed tags are extracted by default (blacklist mode was removed).
| Attributes | Tags | Description |
|---|---|---|
http.url |
http_url |
Full request URL |
http.hostname |
http_hostname |
Request hostname |
http.route |
http_route |
HTTP route |
http.status_code |
http_status_code |
HTTP status code |
http.request.method |
http_request_method |
HTTP request method |
http.method |
http_method |
Same as above |
http.client_ip |
http_client_ip |
Client IP |
http.scheme |
http_scheme |
Request protocol |
url.full |
url_full |
Full request URL |
url.scheme |
url_scheme |
URL scheme |
url.path |
url_path |
Request path |
url.query |
url_query |
Query string |
span_kind |
span_kind |
Span role |
db.system |
db_system |
DB system |
db.operation |
db_operation |
DB operation |
db.name |
db_name |
Database name |
db.statement |
db_statement |
SQL statement |
server.address |
server_address |
Service host |
net.host.name |
net_host_name |
Host name |
server.port |
server_port |
Service port |
net.host.port |
net_host_port |
Host port |
network.peer.address |
network_peer_address |
Peer host |
network.peer.port |
network_peer_port |
Peer port |
network.transport |
network_transport |
Network protocol |
messaging.system |
messaging_system |
Message queue type |
messaging.operation |
messaging_operation |
Message operation |
messaging.message |
messaging_message |
Message details |
messaging.destination |
messaging_destination |
Message destination |
rpc.service |
rpc_service |
RPC service name |
rpc.system |
rpc_system |
RPC system |
error |
error |
Whether error |
error.message |
error_message |
Error message |
error.stack |
error_stack |
Error stack |
error.type |
error_type |
Error type |
project |
project |
Project tag |
version |
version |
Version |
env |
env |
Environment |
host |
host |
Host tag |
pod_name |
pod_name |
Pod name |
pod_namespace |
pod_namespace |
Pod namespace |
telemetry.sdk.language |
sdk_language |
SDK language |
telemetry.sdk.name |
sdk_name |
SDK name |
telemetry.sdk.version |
sdk_version |
SDK version |
To add custom resource tags:
Span kind¶
unspecified: not setinternal: internal spanserver: server sideclient: client sideproducer: message producerconsumer: message consumer
Metrics¶
Java Instrumentation can report JVM and JMX metrics through its built-in metrics Pipeline.
- Control JMX collection with
otel.jmx.enabled=true/false(default is enabled). - Adjust scan interval by
otel.jmx.discovery.delay(milliseconds).
For built-in JMX extension details, see: GitHub OTEL JMX Metrics
Histogram conversion¶
OTEL histogram metrics are converted to Prometheus style metrics:
Input buckets:
Converted metrics:
And DataKit additionally generates:
Metrics ending with _bucket correspond to histogram buckets and should also include
_count, _sum, _min, _max.
Log collection¶
DataKit receives OTEL logs through OTLP. Default is disabled in OTEL V1 and needs to be enabled explicitly.
# environment variable form
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<DataKit Addr>:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
java -jar app.jar
# command line form
java -javaagent:/path/to/agent.jar \
-Dotel.logs.exporter=otlp \
-Dotel.exporter.otlp.endpoint=http://<DataKit Addr>:4317 \
-Dotel.exporter.otlp.protocol=grpc \
-jar app.jar
For HTTP protocol in OTEL V2, use -Dotel.exporter.otlp.protocol=http/protobuf and set
-Dotel.exporter.otlp.logs.endpoint=http://<DataKit Addr>:9529/otel/v1/logs.
Default maximum message field size is 500KB. Can not parts above the limit are truncated.