OpenTelemetry
OpenTelemetry(简称 OTEL)是 CNCF 的可观测性标准体系。DataKit 的 opentelemetry 输入用于接收 OTEL 的 traces、metrics、logs。
配置¶
进入 DataKit 安装目录的 conf.d/opentelemetry,拷贝 opentelemetry.conf.sample 为 opentelemetry.conf 并修改。
[[inputs.opentelemetry]]
## customer_tags will work as a whitelist to prevent tags send to data center.
## All . will replace to _ ,like this :
## "project.name" to send to center is "project_name"
# customer_tags = ["sink_project", "custom.otel.tag", "reg:key_*"]
## If set to true, all Attributes will be extracted and message.Attributes will be empty.
# customer_tags_all = false
## Keep rare tracing resources list switch.
## If some resources are rare enough(not presend in 1 hour), those resource will always send
## to data center and do not consider samplers and filters.
# keep_rare_resource = false
## By default every error presents in span will be send to data center and omit any filters or
## sampler. If you want to get rid of some error status, you can set the error status list here.
# omit_err_status = ["404"]
## compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
# compatible_ddtrace=false
## split service.name form xx.system.
## see: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md
split_service_name = true
## delete trace message
# del_message = true
## logging message data max length,default is 500kb
log_max = 500
## JSON marshaler: set JSON marshaler. available marshaler are:
## gojson/jsoniter/protojson
##
## For better performance, gojson and jsoniter is better than protojson,
## for compatible reason we still use protojson as default.
jmarshaler = "protojson"
## cleaned the top-level fields in message. Default true
clean_message = true
## tracing_metric_enable: trace_hits trace_hits_by_http_status trace_latency trace_errors trace_errors_by_http_status trace_apdex.
## Extract the above metrics from the collection traces.
# tracing_metric_enable = true
## If true, global host tags will not be added to OTEL metrics.
# tracing_metric_disable_global_host_tags = false
## Blacklist of metric tags: There are many labels in the metric: "tracing_metrics".
## If you want to remove certain tag, you can use the blacklist to remove them.
## By default, it includes: source,span_name,env,service,status,version,resource,http_status_code,http_status_class
## and "customer_tags", k8s related tags, and others service.
# tracing_metric_tag_blacklist = ["resource", "operation", "tag_a", "tag_b"]
## White list of metric tags: There are many labels in the metric: "tracing_metrics".
# tracing_metric_tag_whitelist = []
## Ignore tracing resources map like service:[resources...].
## The service name is the full service name in current application.
## The resource list is regular expressions uses to block resource names.
## If you want to block some resources universally under all services, you can set the
## service name as "*". Note: double quotes "" cannot be omitted.
# [inputs.opentelemetry.close_resource]
# service1 = ["resource1", "resource2", ...]
# service2 = ["resource1", "resource2", ...]
# "*" = ["close_resource_under_all_services"]
# ...
## Sampler config uses to set global sampling strategy.
## sampling_rate used to set global sampling rate.
# [inputs.opentelemetry.sampler]
# sampling_rate = 1.0
# [inputs.opentelemetry.tags]
# key1 = "value1"
# key2 = "value2"
# ...
## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
## buffer is the size of jobs' buffering of worker channel.
## threads is the total number fo goroutines at running time.
# [inputs.opentelemetry.threads]
# buffer = 100
# threads = 8
## Storage config a local storage space in hard dirver to cache trace data.
## path is the local file path used to cache data.
## capacity is total space size(MB) used to store data.
# [inputs.opentelemetry.storage]
# path = "./otel_storage"
# capacity = 5120
## OTEL agent HTTP config for trace and metrics
## If enable set to be true, trace and metrics will be received on path respectively, by default is:
## trace : /otel/v1/traces
## metric: /otel/v1/metrics
## and the client side should be configured properly with Datakit listening port(default: 9529)
## or custom HTTP request path.
## for example http://127.0.0.1:9529/otel/v1/traces
## The acceptable http_status_ok values will be 200 or 202.
[inputs.opentelemetry.http]
http_status_ok = 200
trace_api = "/otel/v1/traces"
metric_api = "/otel/v1/metrics"
logs_api = "/otel/v1/logs"
## OTEL agent GRPC config for trace and metrics.
## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
## add is the listening on address for GRPC server.
[inputs.opentelemetry.grpc]
addr = "127.0.0.1:4317"
max_payload = 16777216 # default 16MiB
## If 'expected_headers' is well configed, then the obligation of sending certain wanted HTTP headers is on the client side,
## otherwise HTTP status code 400(bad request) will be provoked.
## Note: expected_headers will be effected on both trace and metrics if setted up.
# [inputs.opentelemetry.expected_headers]
# ex_version = "1.2.3"
# ex_name = "env_resource_name"
# ...
配置完成后重启 DataKit 生效: 重启 DataKit
通过 ConfigMap 或环境变量方式开启采集器: ConfigMap 注入方式 或 ENV_DATAKIT_INPUTS 方式。
也可以使用环境变量直接配置(需将该采集器加入 ENV_DEFAULT_ENABLED_INPUTS):
-
ENV_INPUT_OTEL_CUSTOMER_TAGS
标签白名单
字段类型: JSON
采集器配置字段:
customer_tags示例:
'["project_id", "custom.tag"]' -
ENV_INPUT_OTEL_CUSTOMER_TAGS_ALL
提取所有标签
字段类型: Boolean
采集器配置字段:
customer_tags_all默认值: false
-
ENV_INPUT_OTEL_METRIC_ENABLE_AGGREGATE
开启 OpenTelemetry 上报的指标聚合
字段类型: Boolean
采集器配置字段:
metric_enable_aggregate默认值: false
-
ENV_INPUT_OTEL_KEEP_RARE_RESOURCE
保持稀有跟踪资源列表
字段类型: Boolean
采集器配置字段:
keep_rare_resource默认值: false
-
ENV_INPUT_OTEL_COMPATIBLE_DD_TRACE
将 trace_id 转成 10 进制,兼容 DDTrace
字段类型: Boolean
采集器配置字段:
compatible_dd_trace默认值: false
-
ENV_INPUT_OTEL_SPLIT_SERVICE_NAME
从 span.Attributes 中获取 xx.system 去替换服务名
字段类型: Boolean
采集器配置字段:
split_service_name默认值: false
-
ENV_INPUT_OTEL_TRACING_METRIC_ENABLE
开启请求计数,错误计数和延迟指标的采集
字段类型: Boolean
采集器配置字段:
tracing_metric_enable默认值: false
-
ENV_INPUT_OTEL_TRACING_METRIC_DISABLE_GLOBAL_HOST_TAGS
禁止追加全局主机 tag 到 OTEL 指标数据上
字段类型: Boolean
采集器配置字段:
tracing_metric_disable_global_host_tags默认值: false
-
ENV_INPUT_OTEL_TRACING_METRIC_TAG_BLACKLIST
指标集
tracing_metrics中标签的黑名单字段类型: JSON
采集器配置字段:
tracing_metric_tag_blacklist示例:
'["tag_a", "tag_b"]' -
ENV_INPUT_OTEL_TRACING_METRIC_TAG_WHITELIST
指标集
tracing_metrics中标签的白名单字段类型: JSON
采集器配置字段:
tracing_metric_tag_whitelist示例:
'["tag_a", "tag_b"]' -
ENV_INPUT_OTEL_DEL_MESSAGE
删除 trace 消息
字段类型: Boolean
采集器配置字段:
del_message默认值: false
-
ENV_INPUT_OTEL_OMIT_ERR_STATUS
错误状态白名单
字段类型: JSON
采集器配置字段:
omit_err_status示例:
'["404", "403", "400"]' -
ENV_INPUT_OTEL_CLOSE_RESOURCE
忽略指定服务器的 tracing(正则匹配)
字段类型: JSON
采集器配置字段:
close_resource示例:
'{"service1":["resource1","other"],"service2":["resource2","other"]}' -
ENV_INPUT_OTEL_SAMPLER
全局采样率
字段类型: Float
采集器配置字段:
sampler示例: 0.3
-
ENV_INPUT_OTEL_THREADS
线程和缓存的数量
字段类型: JSON
采集器配置字段:
threads示例:
'{"buffer":1000, "threads":100}' -
ENV_INPUT_OTEL_STORAGE
本地缓存路径和大小(MB)
字段类型: JSON
采集器配置字段:
storage示例:
'{"storage":"./otel_storage", "capacity": 5120}' -
ENV_INPUT_OTEL_HTTP
代理 HTTP 配置
字段类型: JSON
采集器配置字段:
http示例:
'{"enable":true, "http_status_ok": 200, "trace_api": "/otel/v1/traces", "metric_api": "/otel/v1/metrics"}' -
ENV_INPUT_OTEL_GRPC
代理 GRPC 配置
字段类型: JSON
采集器配置字段:
grpc示例:
'{"addr": "127.0.0.1:4317", "max_payload": 16777216 }' -
ENV_INPUT_OTEL_EXPECTED_HEADERS
配置使用客户端的 HTTP 头
字段类型: JSON
采集器配置字段:
expected_headers示例:
'{"ex_version": "1.2.3", "ex_name": "env_resource_name"}' -
ENV_INPUT_OTEL_CLEAN_MESSAGE
精简
message字段大小字段类型: Boolean
采集器配置字段:
clean_message示例:
true/false -
ENV_INPUT_OTEL_TAGS
自定义标签。如果配置文件有同名标签,将会覆盖它
字段类型: JSON
采集器配置字段:
tags示例:
'{"k1":"v1", "k2":"v2", "k3":"v3"}'
customer_tags 支持正则匹配,使用时必须以 reg: 为前缀,比如 reg:key_*。
注意事项¶
- 推荐使用 gRPC,压缩率高、序列化快,资源开销相对更低。
- 自 DataKit 1.10.0 起,HTTP 路由可配置。默认值:
- traces:
/otel/v1/traces - metrics:
/otel/v1/metrics - logs:
/otel/v1/logs float/double类型在 DataKit 侧会保留最多两位小数。- HTTP 与 gRPC 都支持 gzip。可通过 exporter 配置开启,例如
OTEL_EXPORTER_OTLP_COMPRESSION=gzip。 - HTTP 支持 JSON 与 Protobuf 两种序列化提交格式;但 DataKit HTTP 采集仅支持
application/x-protobuf。
Warning
- DDTrace 链路的服务名通常来自 DDTrace 或框架库的
service.name。 - OTEL 链路的服务名由
otel.service.name决定。 - 如果你希望按
db.system、rpc.system、messaging.system将服务名拆分展示,可开启:
split_service_name = true
- 开启后优先级为 db.system,其次 rpc.system,最后 messaging.system。
使用 OTEL HTTP exporter 时,请按 DataKit 实际地址分别配置 OTEL 路由:
traces /otel/v1/traces、metrics /otel/v1/metrics、logs /otel/v1/logs(默认监听端口 9529)。
Java Agent V2 协议行为¶
OTEL Java Agent V2 默认使用 http/protobuf 作为 OTLP 协议,若需切回 gRPC:
java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.exporter.otlp.endpoint=http://localhost:4317 \
-Dotel.service.name=app \
-jar app.jar
如使用 HTTP 方式,请为每类数据显式配置 endpoint:
java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/protobuf \
-Dotel.exporter.otlp.logs.endpoint=http://localhost:9529/otel/v1/logs \
-Dotel.exporter.otlp.traces.endpoint=http://localhost:9529/otel/v1/traces \
-Dotel.exporter.otlp.metrics.endpoint=http://localhost:9529/otel/v1/metrics \
-Dotel.service.name=app \
-jar app.jar
需要关闭日志采集时设置:
-Dotel.logs.exporter=none
更多 V2 变更请参见: GitHub-v2.0.0
常用配置¶
以下为接入 DataKit 常用的 OTEL 配置项(节选):
| 配置项(环境变量/系统属性) | 说明 |
|---|---|
OTEL_SDK_DISABLED(otel.sdk.disabled) |
是否关闭 SDK,默认 false。 |
OTEL_RESOURCE_ATTRIBUTES(otel.resource.attributes) |
全局资源标签,例如 service.name=app,project=app-a。 |
OTEL_SERVICE_NAME(otel.service.name) |
服务名,优先级高于资源标签。 |
OTEL_LOG_LEVEL(otel.log.level) |
SDK 日志级别,默认 info。 |
OTEL_PROPAGATORS(otel.propagators) |
透传协议,默认 tracecontext,baggage。 |
OTEL_TRACES_SAMPLER(otel.traces.sampler) |
采样器类型。 |
OTEL_TRACES_SAMPLER_ARG(otel.traces.sampler.arg) |
与采样器配合的参数,范围 0~1.0,默认 1.0。 |
OTEL_EXPORTER_OTLP_PROTOCOL(otel.exporter.otlp.protocol) |
传输协议,默认 grpc,支持 grpc、http/protobuf。 |
OTEL_EXPORTER_OTLP_ENDPOINT(otel.exporter.otlp.endpoint) |
统一 OTLP 地址,例如 gRPC 模式使用 http://datakit-host:4317,HTTP 模式使用服务地址 http://datakit-host:9529。 |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT(otel.exporter.otlp.traces.endpoint) |
HTTP traces 端点,例如 http://datakit-host:9529/otel/v1/traces。 |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT(otel.exporter.otlp.metrics.endpoint) |
HTTP metrics 端点,例如 http://datakit-host:9529/otel/v1/metrics。 |
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT(otel.exporter.otlp.logs.endpoint) |
HTTP logs 端点,例如 http://datakit-host:9529/otel/v1/logs。 |
OTEL_TRACES_EXPORTER(otel.traces.exporter) |
链路导出器,默认 otlp。 |
OTEL_LOGS_EXPORTER(otel.logs.exporter) |
日志导出器,启用日志时需设置为 otlp。 |
OTEL_METRICS_EXPORTER(otel.metrics.exporter) |
指标导出器,启用指标时需设置为 otlp。 |
从 DataKit 1.85.0 起,HTTP http/json 已不再支持;使用 http/protobuf。
可设置 otel.javaagent.debug=true 打开 Java Agent 调试日志,请勿在生产持续开启。
链路采样¶
可选 head-based 或 tail-based 方案:
- 尾部采样(collector):OpenTelemetry 采样最佳实践
- 头部采样(Agent):OpenTelemetry Java Agent 采样策略
Tag 提取¶
从 DataKit 1.22.0 起,tags 提取从黑名单改为白名单,以下为固定映射清单:
| Attributes | Tags | 说明 |
|---|---|---|
http.url |
http_url |
请求完整 URL |
http.hostname |
http_hostname |
Hostname |
http.route |
http_route |
路由 |
http.status_code |
http_status_code |
状态码 |
http.request.method |
http_request_method |
请求方法 |
http.method |
http_method |
同上 |
http.client_ip |
http_client_ip |
客户端 IP |
http.scheme |
http_scheme |
请求协议 |
url.full |
url_full |
完整请求 URL |
url.scheme |
url_scheme |
URL 协议 |
url.path |
url_path |
请求路径 |
url.query |
url_query |
请求参数 |
span_kind |
span_kind |
Span 类型 |
db.system |
db_system |
数据库系统 |
db.operation |
db_operation |
DB 操作 |
db.name |
db_name |
数据库名 |
db.statement |
db_statement |
SQL 文本 |
server.address |
server_address |
服务地址 |
net.host.name |
net_host_name |
Host 名 |
server.port |
server_port |
服务端口 |
net.host.port |
net_host_port |
主机端口 |
network.peer.address |
network_peer_address |
对端地址 |
network.peer.port |
network_peer_port |
对端端口 |
network.transport |
network_transport |
网络协议 |
messaging.system |
messaging_system |
消息系统 |
messaging.operation |
messaging_operation |
消息动作 |
messaging.message |
messaging_message |
消息 |
messaging.destination |
messaging_destination |
消息目标 |
rpc.service |
rpc_service |
RPC 服务名 |
rpc.system |
rpc_system |
RPC 系统 |
error |
error |
是否错误 |
error.message |
error_message |
错误信息 |
error.stack |
error_stack |
堆栈 |
error.type |
error_type |
错误类型 |
project |
project |
项目 |
version |
version |
版本 |
env |
env |
环境 |
host |
host |
主机 |
pod_name |
pod_name |
Pod 名 |
pod_namespace |
pod_namespace |
Pod 命名空间 |
telemetry.sdk.language |
sdk_language |
SDK 语言 |
telemetry.sdk.name |
sdk_name |
SDK 名 |
telemetry.sdk.version |
sdk_version |
SDK 版本 |
添加自定义资源标签:
Span Kind¶
unspecified:未指定internal:内部 spanserver:服务端 spanclient:客户端 spanproducer:消息生产者consumer:消息消费者
指标采集¶
Java Agent 通过内置 JMX 支持上报 JVM 和相关中间件指标。可在应用中通过:
otel.jmx.enabled=true/false(默认开启)控制是否上报 JMX;otel.jmx.discovery.delay调整探测间隔(单位:毫秒)。
更多 JMX 扩展支持见: GitHub OTEL JMX Metric
直方图指标转换¶
OTEL 直方图会转换为 Prometheus 直方图语义:
输入区间:
转换后的指标:
并补齐:
凡 _bucket 结尾的指标应对应同一维度的 _count、_sum、_min、_max。
日志采集¶
OTEL logs 通过 OTLP 上报到 DataKit。OTEL V1 默认不采集日志,需显式开启:
# 环境变量
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<DataKit Addr>:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
java -jar app.jar
# 命令行
java -javaagent:/path/to/agent.jar \
-Dotel.logs.exporter=otlp \
-Dotel.exporter.otlp.endpoint=http://<DataKit Addr>:4317 \
-Dotel.exporter.otlp.protocol=grpc \
-jar app.jar
如果你使用 V2 的 HTTP/Protobuf,需要设置:
-Dotel.exporter.otlp.protocol=http/protobuf 和
-Dotel.exporter.otlp.logs.endpoint=http://<DataKit Addr>:9529/otel/v1/logs。
默认 message 字段最大 500KB,超出部分会截断,tag 最大 32KB。