跳转至

OpenTelemetry


OpenTelemetry(简称 OTEL)是 CNCF 的可观测性标准体系。DataKit 的 opentelemetry 输入用于接收 OTEL 的 traces、metrics、logs。

配置

进入 DataKit 安装目录的 conf.d/opentelemetry,拷贝 opentelemetry.conf.sampleopentelemetry.conf 并修改。

[[inputs.opentelemetry]]
  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to center is "project_name"
  # customer_tags = ["sink_project", "custom.otel.tag", "reg:key_*"]

  ## If set to true, all Attributes will be extracted and message.Attributes will be empty.
  # customer_tags_all = false

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
  # compatible_ddtrace=false

  ## split service.name form xx.system.
  ## see: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md
  split_service_name = true

  ## delete trace message
  # del_message = true

  ## logging message data max length,default is 500kb
  log_max = 500

  ## JSON marshaler: set JSON marshaler. available marshaler are:
  ##   gojson/jsoniter/protojson
  ##
  ## For better performance, gojson and jsoniter is better than protojson,
  ## for compatible reason we still use protojson as default.
  jmarshaler = "protojson"

  ## cleaned the top-level fields in message. Default true
  clean_message = true

  ## tracing_metric_enable: trace_hits trace_hits_by_http_status trace_latency trace_errors trace_errors_by_http_status trace_apdex.
  ## Extract the above metrics from the collection traces.
  # tracing_metric_enable = true

  ## If true, global host tags will not be added to OTEL metrics.
  # tracing_metric_disable_global_host_tags = false

  ## Blacklist of metric tags: There are many labels in the metric: "tracing_metrics".
  ## If you want to remove certain tag, you can use the blacklist to remove them.
  ## By default, it includes: source,span_name,env,service,status,version,resource,http_status_code,http_status_class
  ## and "customer_tags", k8s related tags, and others service.
  # tracing_metric_tag_blacklist = ["resource", "operation", "tag_a", "tag_b"]

  ## White list of metric tags: There are many labels in the metric: "tracing_metrics".
  # tracing_metric_tag_whitelist = []

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.opentelemetry.close_resource]
  # service1 = ["resource1", "resource2", ...]
  # service2 = ["resource1", "resource2", ...]
  # "*" = ["close_resource_under_all_services"]
  # ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.opentelemetry.sampler]
  # sampling_rate = 1.0

  # [inputs.opentelemetry.tags]
  # key1 = "value1"
  # key2 = "value2"
  # ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.opentelemetry.threads]
  # buffer = 100
  # threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.opentelemetry.storage]
  # path = "./otel_storage"
  # capacity = 5120

  ## OTEL agent HTTP config for trace and metrics
  ## If enable set to be true, trace and metrics will be received on path respectively, by default is:
  ## trace : /otel/v1/traces
  ## metric: /otel/v1/metrics
  ## and the client side should be configured properly with Datakit listening port(default: 9529)
  ## or custom HTTP request path.
  ## for example http://127.0.0.1:9529/otel/v1/traces
  ## The acceptable http_status_ok values will be 200 or 202.
  [inputs.opentelemetry.http]
    http_status_ok = 200
    trace_api = "/otel/v1/traces"
    metric_api = "/otel/v1/metrics"
    logs_api = "/otel/v1/logs"

  ## OTEL agent GRPC config for trace and metrics.
  ## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
  ## add is the listening on address for GRPC server.
  [inputs.opentelemetry.grpc]
    addr = "127.0.0.1:4317"
    max_payload = 16777216 # default 16MiB

  ## If 'expected_headers' is well configed, then the obligation of sending certain wanted HTTP headers is on the client side,
  ## otherwise HTTP status code 400(bad request) will be provoked.
  ## Note: expected_headers will be effected on both trace and metrics if setted up.
  # [inputs.opentelemetry.expected_headers]
  # ex_version = "1.2.3"
  # ex_name = "env_resource_name"
  # ...

配置完成后重启 DataKit 生效: 重启 DataKit

通过 ConfigMap 或环境变量方式开启采集器: ConfigMap 注入方式ENV_DATAKIT_INPUTS 方式

也可以使用环境变量直接配置(需将该采集器加入 ENV_DEFAULT_ENABLED_INPUTS):

  • ENV_INPUT_OTEL_CUSTOMER_TAGS

    标签白名单

    字段类型: JSON

    采集器配置字段: customer_tags

    示例: '["project_id", "custom.tag"]'

  • ENV_INPUT_OTEL_CUSTOMER_TAGS_ALL

    提取所有标签

    字段类型: Boolean

    采集器配置字段: customer_tags_all

    默认值: false

  • ENV_INPUT_OTEL_METRIC_ENABLE_AGGREGATE

    开启 OpenTelemetry 上报的指标聚合

    字段类型: Boolean

    采集器配置字段: metric_enable_aggregate

    默认值: false

  • ENV_INPUT_OTEL_KEEP_RARE_RESOURCE

    保持稀有跟踪资源列表

    字段类型: Boolean

    采集器配置字段: keep_rare_resource

    默认值: false

  • ENV_INPUT_OTEL_COMPATIBLE_DD_TRACE

    将 trace_id 转成 10 进制,兼容 DDTrace

    字段类型: Boolean

    采集器配置字段: compatible_dd_trace

    默认值: false

  • ENV_INPUT_OTEL_SPLIT_SERVICE_NAME

    从 span.Attributes 中获取 xx.system 去替换服务名

    字段类型: Boolean

    采集器配置字段: split_service_name

    默认值: false

  • ENV_INPUT_OTEL_TRACING_METRIC_ENABLE

    开启请求计数,错误计数和延迟指标的采集

    字段类型: Boolean

    采集器配置字段: tracing_metric_enable

    默认值: false

  • ENV_INPUT_OTEL_TRACING_METRIC_DISABLE_GLOBAL_HOST_TAGS

    禁止追加全局主机 tag 到 OTEL 指标数据上

    字段类型: Boolean

    采集器配置字段: tracing_metric_disable_global_host_tags

    默认值: false

  • ENV_INPUT_OTEL_TRACING_METRIC_TAG_BLACKLIST

    指标集 tracing_metrics 中标签的黑名单

    字段类型: JSON

    采集器配置字段: tracing_metric_tag_blacklist

    示例: '["tag_a", "tag_b"]'

  • ENV_INPUT_OTEL_TRACING_METRIC_TAG_WHITELIST

    指标集 tracing_metrics 中标签的白名单

    字段类型: JSON

    采集器配置字段: tracing_metric_tag_whitelist

    示例: '["tag_a", "tag_b"]'

  • ENV_INPUT_OTEL_DEL_MESSAGE

    删除 trace 消息

    字段类型: Boolean

    采集器配置字段: del_message

    默认值: false

  • ENV_INPUT_OTEL_OMIT_ERR_STATUS

    错误状态白名单

    字段类型: JSON

    采集器配置字段: omit_err_status

    示例: '["404", "403", "400"]'

  • ENV_INPUT_OTEL_CLOSE_RESOURCE

    忽略指定服务器的 tracing(正则匹配)

    字段类型: JSON

    采集器配置字段: close_resource

    示例: '{"service1":["resource1","other"],"service2":["resource2","other"]}'

  • ENV_INPUT_OTEL_SAMPLER

    全局采样率

    字段类型: Float

    采集器配置字段: sampler

    示例: 0.3

  • ENV_INPUT_OTEL_THREADS

    线程和缓存的数量

    字段类型: JSON

    采集器配置字段: threads

    示例: '{"buffer":1000, "threads":100}'

  • ENV_INPUT_OTEL_STORAGE

    本地缓存路径和大小(MB)

    字段类型: JSON

    采集器配置字段: storage

    示例: '{"storage":"./otel_storage", "capacity": 5120}'

  • ENV_INPUT_OTEL_HTTP

    代理 HTTP 配置

    字段类型: JSON

    采集器配置字段: http

    示例: '{"enable":true, "http_status_ok": 200, "trace_api": "/otel/v1/traces", "metric_api": "/otel/v1/metrics"}'

  • ENV_INPUT_OTEL_GRPC

    代理 GRPC 配置

    字段类型: JSON

    采集器配置字段: grpc

    示例: '{"addr": "127.0.0.1:4317", "max_payload": 16777216 }'

  • ENV_INPUT_OTEL_EXPECTED_HEADERS

    配置使用客户端的 HTTP 头

    字段类型: JSON

    采集器配置字段: expected_headers

    示例: '{"ex_version": "1.2.3", "ex_name": "env_resource_name"}'

  • ENV_INPUT_OTEL_CLEAN_MESSAGE

    精简 message 字段大小

    字段类型: Boolean

    采集器配置字段: clean_message

    示例: true/false

  • ENV_INPUT_OTEL_TAGS

    自定义标签。如果配置文件有同名标签,将会覆盖它

    字段类型: JSON

    采集器配置字段: tags

    示例: '{"k1":"v1", "k2":"v2", "k3":"v3"}'

customer_tags 支持正则匹配,使用时必须以 reg: 为前缀,比如 reg:key_*

注意事项

  1. 推荐使用 gRPC,压缩率高、序列化快,资源开销相对更低。
  2. 自 DataKit 1.10.0 起,HTTP 路由可配置。默认值:
  3. traces: /otel/v1/traces
  4. metrics: /otel/v1/metrics
  5. logs: /otel/v1/logs
  6. float/double 类型在 DataKit 侧会保留最多两位小数。
  7. HTTP 与 gRPC 都支持 gzip。可通过 exporter 配置开启,例如 OTEL_EXPORTER_OTLP_COMPRESSION=gzip
  8. HTTP 支持 JSON 与 Protobuf 两种序列化提交格式;但 DataKit HTTP 采集仅支持 application/x-protobuf
Warning
  • DDTrace 链路的服务名通常来自 DDTrace 或框架库的 service.name
  • OTEL 链路的服务名由 otel.service.name 决定。
  • 如果你希望按 db.systemrpc.systemmessaging.system 将服务名拆分展示,可开启:

split_service_name = true - 开启后优先级为 db.system,其次 rpc.system,最后 messaging.system

使用 OTEL HTTP exporter 时,请按 DataKit 实际地址分别配置 OTEL 路由: traces /otel/v1/traces、metrics /otel/v1/metrics、logs /otel/v1/logs(默认监听端口 9529)。

Java Agent V2 协议行为

OTEL Java Agent V2 默认使用 http/protobuf 作为 OTLP 协议,若需切回 gRPC:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
  -Dotel.service.name=app \
  -jar app.jar

如使用 HTTP 方式,请为每类数据显式配置 endpoint:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.logs.endpoint=http://localhost:9529/otel/v1/logs \
  -Dotel.exporter.otlp.traces.endpoint=http://localhost:9529/otel/v1/traces \
  -Dotel.exporter.otlp.metrics.endpoint=http://localhost:9529/otel/v1/metrics \
  -Dotel.service.name=app \
  -jar app.jar

需要关闭日志采集时设置:

-Dotel.logs.exporter=none

更多 V2 变更请参见: GitHub-v2.0.0

常用配置

以下为接入 DataKit 常用的 OTEL 配置项(节选):

配置项(环境变量/系统属性) 说明
OTEL_SDK_DISABLED(otel.sdk.disabled) 是否关闭 SDK,默认 false
OTEL_RESOURCE_ATTRIBUTES(otel.resource.attributes) 全局资源标签,例如 service.name=app,project=app-a
OTEL_SERVICE_NAME(otel.service.name) 服务名,优先级高于资源标签。
OTEL_LOG_LEVEL(otel.log.level) SDK 日志级别,默认 info
OTEL_PROPAGATORS(otel.propagators) 透传协议,默认 tracecontext,baggage
OTEL_TRACES_SAMPLER(otel.traces.sampler) 采样器类型。
OTEL_TRACES_SAMPLER_ARG(otel.traces.sampler.arg) 与采样器配合的参数,范围 0~1.0,默认 1.0
OTEL_EXPORTER_OTLP_PROTOCOL(otel.exporter.otlp.protocol) 传输协议,默认 grpc,支持 grpchttp/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT(otel.exporter.otlp.endpoint) 统一 OTLP 地址,例如 gRPC 模式使用 http://datakit-host:4317,HTTP 模式使用服务地址 http://datakit-host:9529
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT(otel.exporter.otlp.traces.endpoint) HTTP traces 端点,例如 http://datakit-host:9529/otel/v1/traces
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT(otel.exporter.otlp.metrics.endpoint) HTTP metrics 端点,例如 http://datakit-host:9529/otel/v1/metrics
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT(otel.exporter.otlp.logs.endpoint) HTTP logs 端点,例如 http://datakit-host:9529/otel/v1/logs
OTEL_TRACES_EXPORTER(otel.traces.exporter) 链路导出器,默认 otlp
OTEL_LOGS_EXPORTER(otel.logs.exporter) 日志导出器,启用日志时需设置为 otlp
OTEL_METRICS_EXPORTER(otel.metrics.exporter) 指标导出器,启用指标时需设置为 otlp

从 DataKit 1.85.0 起,HTTP http/json 已不再支持;使用 http/protobuf

可设置 otel.javaagent.debug=true 打开 Java Agent 调试日志,请勿在生产持续开启。

链路采样

可选 head-based 或 tail-based 方案:

Tag 提取

从 DataKit 1.22.0 起,tags 提取从黑名单改为白名单,以下为固定映射清单:

Attributes Tags 说明
http.url http_url 请求完整 URL
http.hostname http_hostname Hostname
http.route http_route 路由
http.status_code http_status_code 状态码
http.request.method http_request_method 请求方法
http.method http_method 同上
http.client_ip http_client_ip 客户端 IP
http.scheme http_scheme 请求协议
url.full url_full 完整请求 URL
url.scheme url_scheme URL 协议
url.path url_path 请求路径
url.query url_query 请求参数
span_kind span_kind Span 类型
db.system db_system 数据库系统
db.operation db_operation DB 操作
db.name db_name 数据库名
db.statement db_statement SQL 文本
server.address server_address 服务地址
net.host.name net_host_name Host 名
server.port server_port 服务端口
net.host.port net_host_port 主机端口
network.peer.address network_peer_address 对端地址
network.peer.port network_peer_port 对端端口
network.transport network_transport 网络协议
messaging.system messaging_system 消息系统
messaging.operation messaging_operation 消息动作
messaging.message messaging_message 消息
messaging.destination messaging_destination 消息目标
rpc.service rpc_service RPC 服务名
rpc.system rpc_system RPC 系统
error error 是否错误
error.message error_message 错误信息
error.stack error_stack 堆栈
error.type error_type 错误类型
project project 项目
version version 版本
env env 环境
host host 主机
pod_name pod_name Pod 名
pod_namespace pod_namespace Pod 命名空间
telemetry.sdk.language sdk_language SDK 语言
telemetry.sdk.name sdk_name SDK 名
telemetry.sdk.version sdk_version SDK 版本

添加自定义资源标签:

-Dotel.resource.attributes=service.name=app,version=1.1.0,env=prod
Span Kind
  • unspecified:未指定
  • internal:内部 span
  • server:服务端 span
  • client:客户端 span
  • producer:消息生产者
  • consumer:消息消费者

指标采集

Java Agent 通过内置 JMX 支持上报 JVM 和相关中间件指标。可在应用中通过:

  • otel.jmx.enabled=true/false(默认开启)控制是否上报 JMX;
  • otel.jmx.discovery.delay 调整探测间隔(单位:毫秒)。

更多 JMX 扩展支持见: GitHub OTEL JMX Metric

直方图指标转换

OTEL 直方图会转换为 Prometheus 直方图语义:

输入区间:

[0, 10), [10, 50), [50, 100)

转换后的指标:

my_histogram_bucket{le="10"} 100
my_histogram_bucket{le="50"} 200
my_histogram_bucket{le="100"} 250

并补齐:

my_histogram_count 250
my_histogram_max 100
my_histogram_min 50
my_histogram_sum 12345.67

_bucket 结尾的指标应对应同一维度的 _count_sum_min_max

日志采集

Version-1.33.0

OTEL logs 通过 OTLP 上报到 DataKit。OTEL V1 默认不采集日志,需显式开启:

# 环境变量
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<DataKit Addr>:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
java -jar app.jar

# 命令行
java -javaagent:/path/to/agent.jar \
  -Dotel.logs.exporter=otlp \
  -Dotel.exporter.otlp.endpoint=http://<DataKit Addr>:4317 \
  -Dotel.exporter.otlp.protocol=grpc \
  -jar app.jar

如果你使用 V2 的 HTTP/Protobuf,需要设置: -Dotel.exporter.otlp.protocol=http/protobuf-Dotel.exporter.otlp.logs.endpoint=http://<DataKit Addr>:9529/otel/v1/logs

默认 message 字段最大 500KB,超出部分会截断,tag 最大 32KB。

文档评价

文档内容是否对您有帮助? ×