OpenTelemetry

OpenTelemetry (hereinafter referred to as OTEL) is an observability project under CNCF (Cloud Native Computing Foundation). It aims to provide a standardized solution in the field of observability, addressing standardization issues related to the data model, collection, processing, and export of observability data.

OTEL is a collection of standards and tools designed to manage observability data such as traces, metrics, and logs. This document describes how to configure and enable OTEL data ingestion on DataKit, as well as best practices for Java and Go.

Configuration¶

[[inputs.opentelemetry]]
  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to center is "project_name"
  # customer_tags = ["sink_project", "custom.otel.tag", "reg:key_*"]

  ## If set to true, all Attributes will be extracted and message.Attributes will be empty.
  # customer_tags_all = false

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
  # compatible_ddtrace=false

  ## split service.name form xx.system.
  ## see: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md
  split_service_name = true

  ## delete trace message
  # del_message = true

  ## logging message data max length,default is 500kb
  log_max = 500

  ## JSON marshaler: set JSON marshaler. available marshaler are:
  ##   gojson/jsoniter/protojson
  ##
  ## For better performance, gojson and jsoniter is better than protojson,
  ## for compatible reason we still use protojson as default.
  jmarshaler = "protojson"

  ## cleaned the top-level fields in message. Default true
  clean_message = true

  ## tracing_metric_enable: trace_hits trace_hits_by_http_status trace_latency trace_errors trace_errors_by_http_status trace_apdex.
  ## Extract the above metrics from the collection traces.
  # tracing_metric_enable = true

  ## Blacklist of metric tags: There are many labels in the metric: "tracing_metrics".
  ## If you want to remove certain tag, you can use the blacklist to remove them.
  ## By default, it includes: source,span_name,env,service,status,version,resource,http_status_code,http_status_class
  ## and "customer_tags", k8s related tags, and others service.
  # tracing_metric_tag_blacklist = ["resource", "operation", "tag_a", "tag_b"]

  ## White list of metric tags: There are many labels in the metric: "tracing_metrics".
  # tracing_metric_tag_whitelist = []

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.opentelemetry.close_resource]
    # service1 = ["resource1", "resource2", ...]
    # service2 = ["resource1", "resource2", ...]
    # "*" = ["close_resource_under_all_services"]
    # ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.opentelemetry.sampler]
    # sampling_rate = 1.0

  # [inputs.opentelemetry.tags]
    # key1 = "value1"
    # key2 = "value2"
    # ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.opentelemetry.threads]
    # buffer = 100
    # threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.opentelemetry.storage]
    # path = "./otel_storage"
    # capacity = 5120

  ## OTEL agent HTTP config for trace and metrics
  ## If enable set to be true, trace and metrics will be received on path respectively, by default is:
  ## trace : /otel/v1/traces
  ## metric: /otel/v1/metrics
  ## and the client side should be configured properly with Datakit listening port(default: 9529)
  ## or custom HTTP request path.
  ## for example http://127.0.0.1:9529/otel/v1/traces
  ## The acceptable http_status_ok values will be 200 or 202.
  [inputs.opentelemetry.http]
   http_status_ok = 200
   trace_api = "/otel/v1/traces"
   metric_api = "/otel/v1/metrics"
   logs_api = "/otel/v1/logs"

  ## OTEL agent GRPC config for trace and metrics.
  ## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
  ## add is the listening on address for GRPC server.
  [inputs.opentelemetry.grpc]
   addr = "127.0.0.1:4317"
   max_payload = 16777216 # default 16MiB

  ## If 'expected_headers' is well configed, then the obligation of sending certain wanted HTTP headers is on the client side,
  ## otherwise HTTP status code 400(bad request) will be provoked.
  ## Note: expected_headers will be effected on both trace and metrics if setted up.
  # [inputs.opentelemetry.expected_headers]
  # ex_version = "1.2.3"
  # ex_name = "env_resource_name"
  # ...

The customer_tags parameter supports regular expressions but requires a fixed prefix format reg:. For example, reg:key_* matches all keys starting with key_.

Notes¶

1. It is recommended to use the gRPC protocol, as gRPC offers advantages such as high compression rate, fast serialization, and higher efficiency.
2. Starting from DataKit version 1.10.0, the routes for the HTTP protocol are configurable. The default request paths (for Trace/Metric) are /otel/v1/traces, /otel/v1/logs, and /otel/v1/metrics respectively.
3. For float/double type data, a maximum of two decimal places will be retained.
4. Both HTTP and gRPC support the gzip compression format. You can configure an environment variable in the exporter to enable it: OTEL_EXPORTER_OTLP_COMPRESSION = gzip; gzip is disabled by default.
5. The HTTP protocol request format supports both JSON and Protobuf serialization formats. However, gRPC only supports the Protobuf format.

Note the environment variable configuration when using the OTEL HTTP exporter. Since the default configuration of DataKit uses /otel/v1/traces, /otel/v1/logs, and /otel/v1/metrics, you need to configure trace and metric separately if you want to use the HTTP protocol.

Agent V2 Version¶

The V2 version uses otlp exporter by default, changing the previous grpc to http/protobuf. You can set it via the command -Dotel.exporter.otlp.protocol=grpc, or use the default http/protobuf.

If using HTTP, the path for each exporter needs to be explicitly configured. For example:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.logs.endpoint=http://localhost:9529/otel/v1/logs \
  -Dotel.exporter.otlp.traces.endpoint=http://localhost:9529/otel/v1/traces \
  -Dotel.exporter.otlp.metrics.endpoint=http://localhost:9529/otel/v1/metrics \
  -Dotel.service.name=app \
  -jar app.jar

If using the gRPC protocol, explicit configuration is required; otherwise, the default HTTP protocol will be used:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
  -Dotel.service.name=app \
  -jar app.jar

Logging is enabled by default. To disable log collection, set the exporter configuration to empty: -Dotel.logs.exporter=none.

For more major changes in the V2 version, refer to the official documentation or GitHub release notes: Github-v2.0.0

Common Commands¶

The following configurations are commonly used when starting an application:

You can pass the otel.javaagent.debug=true parameter to the Agent to view debug logs. Note that these logs are quite verbose; use them with caution in production environments.

Trace Sampling¶

You can use head-based sampling or tail-based sampling. For details, refer to the two best practice documents:

• Tail-based sampling with collector: OpenTelemetry Sampling Best Practices
• Head-based sampling on the Agent side: OpenTelemetry Java Agent Sampling Strategy

Tag Extraction¶

Starting from DataKit version 1.22.0, the blacklist function is deprecated. A fixed tag list is added, and only tags in this list will be extracted into top-level tags. The fixed list is as follows:

To add custom tags, use the following environment variable:

# Add custom tags via startup parameters
-Dotel.resource.attributes=username=myName,env=1.1.0

Span kind¶

All spans have the span_kind tag, which has 6 attributes:

• unspecified: Not set.
• internal: Internal span or child span type.
• server: WEB service, RPC service, etc.
• client: Client type.
• producer: Message producer.
• consumer: Message consumer.

Metric Collection¶

The OpenTelemetry Java Agent obtains MBean metric information from applications via the JMX protocol. The Java Agent reports selected JMX metrics through the internal SDK, which means all metrics are configurable.

You can enable or disable JMX metric reporting using the command otel.jmx.enabled=true/false (enabled by default). To control the time interval between MBean detection attempts, use the otel.jmx.discovery.delay command. This attribute defines the interval in milliseconds between the first and subsequent detection cycles.

In addition, the Agent has built-in collection configurations for some third-party software. For details, refer to: GitHub OTEL JMX Metric

We have implemented special handling for Histogram metrics:

• OpenTelemetry histogram buckets are directly mapped to Prometheus histogram buckets.

• The count of each bucket is converted to the Prometheus cumulative count format. For example, OpenTelemetry buckets [0, 10), [10, 50), [50, 100) are converted to Prometheus _bucket metrics with the le tag:

  my_histogram_bucket{le="10"} 100
  my_histogram_bucket{le="50"} 200
  my_histogram_bucket{le="100"} 250

• The total number of observations in the OpenTelemetry histogram is converted to the Prometheus _count metric.

• The sum of the OpenTelemetry histogram is converted to the Prometheus _sum metric, and _max and _min are also added.

  my_histogram_count 250
  my_histogram_max 100
  my_histogram_min 50
  my_histogram_sum 12345.67

All metrics ending with _bucket are histogram data, and there must be corresponding metrics ending with _max, _min, _count, and sum.

You can use the le (less than or equal) tag to categorize histogram data and filter based on tags. For all metrics and tags, refer to OpenTelemetry Metrics.

This conversion enables seamless integration of histogram data collected by OpenTelemetry into Prometheus, allowing you to leverage Prometheus' powerful query and visualization capabilities for analysis.

Log Collection¶

Version-1.33.0

Currently, the JAVA Agent supports collecting stdout logs and sending them to DataKit via the otlp protocol using the Standard output method.

By default, log collection is disabled for OTEL Agent V1. Explicit commands are required to enable it. The enabling methods are as follows:

# env
export OTEL_LOGS_EXPORTER=OTLP
export OTEL_EXPORTER_OTLP.ENDPOINT=http://<DataKit Addr>:4317
java -jar app.jar

# command
java -javaagent:/path/to/agnet.jar \
  -otel.logs.exporter=otlp \
  -Dotel.exporter.otlp.endpoint=http://<DataKit Addr>:4317 \
  -jar app.jar

By default, the maximum length of log content is 500KB. Content exceeding this limit will be split into multiple logs. The maximum length of log tags is 32KB (this field is not configurable), and content exceeding this limit will be truncated.

The source of logs collected via OTEL is the service name. You can also customize it by adding a tag: log.source. For example: -Dotel.resource.attributes="log.source=source_name".

Note: If the app runs in a container environment (e.g., k8s), DataKit will automatically collect logs by default. Enabling log collection again will result in duplicate collection. It is recommended to manually disable DataKit's independent log collection before enabling OTEL log collection.

For more languages, refer to the official documentation.

Collection Field Description¶

Tracing¶

opentelemetry¶

Following is tags/fields of tracing data

Metrics¶

otel_service¶

OpenTelemetry JVM Metrics

tracing_metrics¶

Based on OpenTelemetry's span data, we count span count, span cost metrics

Deleted Tags in Metrics¶

In the otel_service metric set, there are many useless tags in the originally reported metrics. These tags are of String type and are discarded due to high memory and bandwidth consumption. The discarded tags are as follows:

process.command_line
process.executable.path
process.runtime.description
process.runtime.name
process.runtime.version
telemetry.distro.name
telemetry.distro.version
telemetry.sdk.language
telemetry.sdk.name
telemetry.sdk.version

Examples¶

DataKit currently provides best practices for the following two languages:

• Golang
• Java

More Documents¶

• Golang SDK
• Official User Guide
• Environment Variable Configuration
• Sampling Strategy Notes for DDTrace and OpenTelemetry Integration