OpenTelemetry

OpenTelemetry (hereinafter referred to as OTEL) is an observability project of CNCF, which aims to provide a standardization scheme in the field of observability and solve the standardization problems of data model, collection, processing and export of observation data.

OTEL is a collection of standards and tools for managing observational data, such as trace, metrics, logs, etc. (new observational data types may appear in the future).

OTEL provides vendor-independent implementations that export observation class data to different backends, such as open source Prometheus, Jaeger, DataKit, or cloud vendor services, depending on the user's needs.

The purpose of this article is to introduce how to configure and enable OTEL data access on DataKit, and the best practices of Java and Go.

Configuration¶

Collector Configuration¶

[[inputs.opentelemetry]]
  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to GuanCe center is "project_name"
  # customer_tags = ["sink_project", "custom.otel.tag"]

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
  # compatible_ddtrace=false

  ## spilt service.name form xx.system.
  ## see: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md
  spilt_service_name = true

  ## delete trace message
  # del_message = true

  ## logging message data max length,default is 500kb
  log_max = 500

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.opentelemetry.close_resource]
    # service1 = ["resource1", "resource2", ...]
    # service2 = ["resource1", "resource2", ...]
    # "*" = ["close_resource_under_all_services"]
    # ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.opentelemetry.sampler]
    # sampling_rate = 1.0

  # [inputs.opentelemetry.tags]
    # key1 = "value1"
    # key2 = "value2"
    # ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.opentelemetry.threads]
    # buffer = 100
    # threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.opentelemetry.storage]
    # path = "./otel_storage"
    # capacity = 5120

  ## OTEL agent HTTP config for trace and metrics
  ## If enable set to be true, trace and metrics will be received on path respectively, by default is:
  ## trace : /otel/v1/traces
  ## metric: /otel/v1/metrics
  ## and the client side should be configured properly with Datakit listening port(default: 9529)
  ## or custom HTTP request path.
  ## for example http://127.0.0.1:9529/otel/v1/traces
  ## The acceptable http_status_ok values will be 200 or 202.
  [inputs.opentelemetry.http]
   http_status_ok = 200
   trace_api = "/otel/v1/traces"
   metric_api = "/otel/v1/metrics"
   logs_api = "/otel/v1/logs"

  ## OTEL agent GRPC config for trace and metrics.
  ## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
  ## add is the listening on address for GRPC server.
  [inputs.opentelemetry.grpc]
   addr = "127.0.0.1:4317"

  ## If 'expected_headers' is well configed, then the obligation of sending certain wanted HTTP headers is on the client side,
  ## otherwise HTTP status code 400(bad request) will be provoked.
  ## Note: expected_headers will be effected on both trace and metrics if setted up.
  # [inputs.opentelemetry.expected_headers]
  # ex_version = "1.2.3"
  # ex_name = "env_resource_name"
  # ...

Notes¶

1. It is recommended to use grpc protocol, which has the advantages of high compression ratio, fast serialization and higher efficiency.
2. The route of the http protocol is configurable and the default request path is trace: /otel/v1/traces, metric:/otel/v1/metrics,logs:/otel/v1/logs
3. When data of type float double is involved, a maximum of two decimal places are reserved.
4. Both http and grpc support the gzip compression format. You can configure the environment variable in exporter to turn it on: OTEL_EXPORTER_OTLP_COMPRESSION = gzip; gzip is not turned on by default.
5. The http protocol request format supports both JSON and Protobuf serialization formats. But grpc only supports Protobuf.

Pay attention to the configuration of environment variables when using OTEL HTTP exporter. Since the default configuration of DataKit is /otel/v1/traces and /otel/v1/metrics, if you want to use the HTTP protocol, you need to configure trace and trace separately metric,

The default request routes of OTLP are /otel/v1/logs v1/traces and v1/metrics, which need to be configured separately for these two. If you modify the routing in the configuration file, just replace the routing address below.

Agent V2 version¶

The default OTLP protocol has been changed from grpc to http/protobuf in order to align with the specification. You can switch to the grpc protocol using OTEL_EXPORTER_OTLP_PROTOCOL=grpc or -Dotel.exporter.otlp.protocol=grpc.

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.logs.endpoint=http://localhost:9529/otel/v1/logs \
  -Dotel.exporter.otlp.traces.endpoint=http://localhost:9529/otel/v1/traces \
  -Dotel.exporter.otlp.metrics.endpoint=http://localhost:9529/otel/v1/metrics \
  -Dotel.service.name=app \
  -jar app.jar

Use gPRC:

java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent-2.5.0.jar \
  -Dotel.exporter=otlp \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317
  -Dotel.service.name=app \
  -jar app.jar

The default log is enabled. If you want to turn off log collection, the exporter configuration can be empty: -Dotel.logs.exporter=none

For more major changes in the V2 version, please check the official documentation or GitHub GuanCe Cloud version notes

General SDK Configuration¶

You can pass the 'otel.javaagent.debug=true' parameter to the agent to view debugging logs. Please note that these logs are quite lengthy and should be used with caution in production environments.

Tracing¶

DataKit only accepts OTLP data. OTLP has clear data types: gRPC, http/protobuf and http/json. For specific configuration, please refer to:

# OpenTelemetry Agent default is gRPC
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.exporter.otlp.endpoint=http://datakit-endpoint:4317

# use http/protobuf
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/protobuf \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/traces \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metrics 

# use http/json
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/json \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/traces \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metrics

Tag¶

Starting from DataKit version 1.22.0 ,ignore_tags is deprecated. Add a fixed tags, only those in this list will be extracted into the tag. The following is the fixed list:

If you want to add custom labels, you can use environment variables:

-Dotel.resource.attributes=username=myName,env=1.1.0

And modify the whitelist in the configuration file so that a custom label can appear in the first level label of the Guance link details.

customer_tags = ["sink_project", "username","env"]

Kind¶

All Span has span_kind tag,

• unspecified: unspecified.
• internal: internal span.
• server: WEB server or RPC server.
• client: HTTP client or RPC client.
• producer: message producer.
• consumer: message consumer.

Best Practices¶

DataKit currently provides Go language、Java languages, with other languages available later.

Metric¶

The OpenTelemetry Java Agent obtains the MBean's indicator information from the application through the JMX protocol, and the Java Agent reports the selected JMX indicator through the internal SDK, which means that all indicators are configurable.

You can enable and disable JMX metrics collection by command otel.jmx.enabled=true/false, which is enabled by default.

To control the time interval between MBean detection attempts, one can use the OTEL.jmx.discovery.delay property, which defines the number of milliseconds to elapse between the first and the next detection cycle.

In addition, the acquisition configuration of some third-party software built in the Agent. For details, please refer to: JMX Metric Insight

When exporting Histogram metrics to Observability Cloud, some metrics undergo special processing:

• OpenTelemetry histogram buckets are directly mapped to Prometheus histogram buckets.
• The count for each bucket is converted to Prometheus' cumulative count format.
• For example, OpenTelemetry buckets [0, 10), [10, 50), and [50, 100) are converted into Prometheus _bucket metrics with an le label:

  my_histogram_bucket{le="10"} 100
  my_histogram_bucket{le="50"} 200
  my_histogram_bucket{le="100"} 250

• The total number of observations in the OpenTelemetry histogram is converted into the Prometheus _count metric.
• The sum of the OpenTelemetry histogram is converted into the Prometheus _sum metric, and _max and _min are also added.

  my_histogram_count 250
  my_histogram_max 100
  my_histogram_min 50
  my_histogram_sum 12345.67

Any metric ending with _bucket is histogram data, and it will always have corresponding metrics ending with _max, _min, _count, and _sum.

In histogram data, the le (less or equal) label can be used for classification, and filtering can be performed based on labels. You can refer to OpenTelemetry Metrics for all metrics and labels.

This conversion enables seamless integration of OpenTelemetry-collected histogram data into Prometheus, leveraging Prometheus' powerful querying and visualization capabilities for analysis.

Delete Metric Tags¶

There are many useless tags in the indicators reported by OTEL. These are all of String type. They have been deleted because they occupy too much memory and bandwidth and will not be uploaded to the GuanCe cloud center.

These tags include:

process.command_line
process.executable.path
process.runtime.description
process.runtime.name
process.runtime.version
telemetry.distro.name
telemetry.distro.version
telemetry.sdk.language
telemetry.sdk.name
telemetry.sdk.version

otel_service¶

OpenTelemetry JVM Metrics

• Tags

• Metrics

opentelemetry¶

This is the field description for the trace.

• Tags

• Metrics

Logging¶

Version-1.33.0

“Standard output” LogRecord Exporter is a LogRecord Exporter which outputs the logs to stdout/console.

If a language provides a mechanism to automatically configure a LogRecordProcessor to pair with the associated exporter (e.g., using the OTEL_LOGS_EXPORTER environment variable), by default the standard output exporter SHOULD be paired with a simple processor.

The source of the logs collected through OTEL is the service.name, and it can also be customized by adding tags such as log.source, for example: -Dotel.resource.attributes="log.source=sourcename".

You can View logging documents

By default, the maximum length of log content is 500kb, and the excess length will be divided into multiple logs. The maximum length of the log label is 32KB. This field is not configurable, and the excess length will be cut off.

Note: If the app is running in a container environment (such as k8s), DataKit will automatically collect logs. If otel collects logs again, there will be a problem of duplicate collection. It is recommended to manually turn off DataKit's autonomous log collection behavior before enabling otel to collect logs.

More Docs¶

• Go open source address OpenTelemetry-go
• Official user manual: opentelemetry-io-docs
• Environment variable configuration: sdk-extensions
• GitHub GuanceCloud version OpenTelemetry-Java-instrumentation