OpenTelemetry

OpenTelemetry (hereinafter referred to as OTEL) is a CNCF observability project aimed at providing standardized solutions in the observability domain, addressing standardization issues of observable data models, collection, processing, and export.

OTEL is a collection of standards and tools designed to manage observable data such as traces, metrics, and logs.

This document aims to introduce how to configure and enable OTEL data ingestion on Datakit, along with best practices for Java and Go.

Configuration¶

[[inputs.opentelemetry]]
  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to Guance center is "project_name"
  # customer_tags = ["sink_project", "custom.otel.tag"]

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not present in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## Compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
  # compatible_ddtrace=false

  ## Split service.name from xx.system.
  ## see: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md
  spilt_service_name = true

  ## Delete trace message
  # del_message = true

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.opentelemetry.close_resource]
    # service1 = ["resource1", "resource2", ...]
    # service2 = ["resource1", "resource2", ...]
    # "*" = ["close_resource_under_all_services"]
    # ...

  ## Sampler config used to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.opentelemetry.sampler]
    # sampling_rate = 1.0

  # [inputs.opentelemetry.tags]
    # key1 = "value1"
    # key2 = "value2"
    # ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.opentelemetry.threads]
    # buffer = 100
    # threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.opentelemetry.storage]
    # path = "./otel_storage"
    # capacity = 5120

  ## OTEL agent HTTP config for trace and metrics
  ## If enable set to be true, trace and metrics will be received on path respectively, by default is:
  ## trace : /otel/v1/traces
  ## metric: /otel/v1/metrics
  ## and the client side should be configured properly with Datakit listening port(default: 9529)
  ## or custom HTTP request path.
  ## for example http://127.0.0.1:9529/otel/v1/traces
  ## The acceptable http_status_ok values will be 200 or 202.
  [inputs.opentelemetry.http]
   http_status_ok = 200
   trace_api = "/otel/v1/traces"
   metric_api = "/otel/v1/metrics"
   logs_api = "/otel/v1/logs"

  ## OTEL agent GRPC config for trace and metrics.
  ## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
  ## add is the listening on address for GRPC server.
  [inputs.opentelemetry.grpc]
   addr = "127.0.0.1:4317"

  ## If 'expected_headers' is well configured, then the obligation of sending certain wanted HTTP headers is on the client side,
  ## otherwise HTTP status code 400(bad request) will be provoked.
  ## Note: expected_headers will be effected on both trace and metrics if setted up.
  # [inputs.opentelemetry.expected_headers]
  # ex_version = "1.2.3"
  # ex_name = "env_resource_name"
  # ...

Precautions¶

1. It is recommended to use gRPC protocol, which has advantages such as high compression ratio, fast serialization, and higher efficiency.
2. Since Datakit 1.10.0, HTTP protocol routes are configurable. Default request paths (Trace/Metric) are /otel/v1/traces, /otel/v1/logs, and /otel/v1/metrics.
3. When dealing with float/double type data, only two decimal places will be retained.
4. Both HTTP and gRPC support gzip compression format. In exporters, environment variables can be configured to enable it: OTEL_EXPORTER_OTLP_COMPRESSION = gzip, which is disabled by default.
5. HTTP protocol supports both JSON and Protobuf serialization formats. However, gRPC only supports Protobuf.

When using the OTEL HTTP exporter, pay attention to the environment variable configuration. Since Datakit's default configuration is /otel/v1/traces /otel/v1/logs and /otel/v1/metrics, if you want to use the HTTP protocol, you need to configure trace and metric separately,

Agent V2 Version¶

The V2 version defaults to using otlp exporter changing the previous grpc to http/protobuf. This can be set using the command -Dotel.exporter.otlp.protocol=grpc, or use the default http/protobuf.

If using HTTP, each exporter path needs explicit configuration, for example:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.logs.endpoint=http://localhost:9529/otel/v1/logs \
  -Dotel.exporter.otlp.traces.endpoint=http://localhost:9529/otel/v1/traces \
  -Dotel.exporter.otlp.metrics.endpoint=http://localhost:9529/otel/v1/metrics \
  -Dotel.service.name=app \
  -jar app.jar

If using gRPC protocol, it must be explicitly configured; otherwise, it defaults to HTTP protocol:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317
  -Dotel.service.name=app \
  -jar app.jar

Logs are enabled by default. To disable log collection, set the exporter configuration to empty: -Dotel.logs.exporter=none

For more significant changes in V2, please refer to the official documentation or GitHub Guance release notes: Github-GuanCe-v2.11.0

Common Commands¶

You can pass the parameter otel.javaagent.debug=true to the Agent to view debug logs. Please note that these logs are quite verbose and should be used cautiously in production environments.

Tracing¶

A trace (trace) is a chain of information composed of multiple spans. Whether it's a single service or a service cluster, trace information provides a complete set of paths between all services involved in a request from start to finish.

Datakit only accepts OTLP data. OTLP has three data types: gRPC, http/protobuf, and http/json. Specific configurations can be found below:

# OpenTelemetry sends data to Datakit using the gPRC protocol by default
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.exporter.otlp.endpoint=http://datakit-endpoint:4317

# Using http/protobuf method
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/protobuf \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/traces \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metrics 

# Using http/json method
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/json \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/traces \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metrics

Trace Sampling¶

You can adopt head-based or tail-based sampling. For detailed best practices, refer to the following articles:

• Tail-based sampling with Collector: OpenTelemetry Sampling Best Practices
• Head-based sampling on the Agent side: OpenTelemetry Java Agent Sampling Strategies

Tags¶

Starting from DataKit version 1.22.0, the blacklist feature has been deprecated. A fixed tag list has been added, and only items in this list will be extracted into first-level tags. Below is the fixed list:

To add custom tags, you can use environment variables:

# Add custom tags through startup parameters
-Dotel.resource.attributes=username=myName,env=1.1.0

Modify the whitelist in the configuration file so that custom tags appear in the first-level tags of the trace details in Guance:

customer_tags = ["sink_project", "username","env"]

Kind¶

All Spans have a span_kind tag, with six attributes:

• unspecified: Unset.
• internal: Internal span or sub-span type.
• server: WEB services, RPC services, etc.
• client: Client type.
• producer: Message producer.
• consumer: Message consumer.

Metrics¶

OpenTelemetry Java Agent collects JMX metrics from applications through the JMX protocol. The Java Agent reports selected JMX metrics via the internal SDK, meaning all metrics are configurable.

You can enable and disable JMX metric collection using the command otel.jmx.enabled=true/false, which is enabled by default.

To control the interval between MBean detection attempts, you can use the otel.jmx.discovery.delay command, defining the milliseconds between the first and next detection cycles.

Additionally, the Agent has built-in configurations for collecting metrics from third-party software. Refer to: GitHub OTEL JMX Metric

When converting Histogram metrics to Guance, some special handling is applied:

• OpenTelemetry histogram buckets are directly mapped to Prometheus histogram buckets.
• Each bucket count is converted to Prometheus cumulative count format.
• For example, OpenTelemetry buckets [0, 10), [10, 50), [50, 100) are converted to Prometheus _bucket metrics with le labels:

  my_histogram_bucket{le="10"} 100
  my_histogram_bucket{le="50"} 200
  my_histogram_bucket{le="100"} 250

• The total number of observations in the OpenTelemetry histogram is converted to the Prometheus _count metric.
• The sum of the OpenTelemetry histogram is converted to the Prometheus _sum metric, with _max and _min added.

  my_histogram_count 250
  my_histogram_max 100
  my_histogram_min 50
  my_histogram_sum 12345.67

Any metric ending in _bucket is histogram data and will have corresponding _max, _min, _count, and sum metrics.

In histogram data, you can classify using the le(less or equal) label and filter based on the label. For more information, refer to OpenTelemetry Metrics for all metrics and labels.

This conversion allows OpenTelemetry-collected histogram data to seamlessly integrate into Prometheus and leverage its powerful query and visualization capabilities for analysis.

Data Field Explanation¶

Metrics¶

• Tags

• Metrics

Tracing¶

• Tags

• Metrics

Deleted Tags in Metrics¶

Many useless tags are reported by OTEL metrics. These are string types that occupy too much memory and bandwidth, so they are deleted and not uploaded to the Guance center.

These tags include:

process.command_line
process.executable.path
process.runtime.description
process.runtime.name
process.runtime.version
telemetry.distro.name
telemetry.distro.version
telemetry.sdk.language
telemetry.sdk.name
telemetry.sdk.version

Logging¶

Version-1.33.0

Currently, JAVA Agent supports collecting stdout logs and uses the Standard output method to send logs to DataKit via the otlp protocol.

By default, OTEL Agent does not enable log collection and must be explicitly enabled through the command otel.logs.exporter. The method is as follows:

# env
export OTEL_LOGS_EXPORTER=OTLP
export OTEL_EXPORTER_OTLP.ENDPOINT=http://<DataKit Addr>:4317
# other env
java -jar app.jar

# command
java -javaagent:/path/to/agnet.jar \
  -otel.logs.exporter=otlp \
  -Dotel.exporter.otlp.endpoint=http://<DataKit Addr>:4317 \
  -jar app.jar

Logs collected via OTEL have a source of the service name, which can be customized by adding tags: log.source, for example: -Dotel.resource.attributes="log.source=source_name".

Note: If the app is running in a container environment (such as k8s), Datakit automatically collects logs automatically (default behavior). Collecting logs again will result in duplicate collection. It is recommended to manually disable Datakit's automatic log collection before enabling log collection.

For more languages, refer to the official documentation

Examples¶

Datakit currently provides best practices for the following two languages:

• [- Golang
• Java

More Documentation¶

• Golang SDK
• Official User Manual
• Environment Variable Configuration
• Guance Custom Development Version

This completes the translation of the provided content. If you need any further assistance or additional sections translated, please let me know!