OpenTelemetry

OpenTelemetry (hereinafter referred to as OTEL) is an observability project of CNCF, which aims to provide a standardization scheme in the field of observability and solve the standardization problems of data model, collection, processing and export of observation data.

OTEL is a collection of standards and tools for managing observational data, such as trace, metrics, logs, etc. (new observational data types may appear in the future).

OTEL provides vendor-independent implementations that export observation class data to different backends, such as open source Prometheus, Jaeger, Datakit, or cloud vendor services, depending on the user's needs.

The purpose of this article is to introduce how to configure and enable OTEL data access on Datakit, and the best practices of Java and Go.

Version Notes: Datakit currently only accesses OTEL v1 version of OTLP data.

Configuration¶

Collector Configuration¶

[[inputs.opentelemetry]]
  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to GuanCe center is "project_name"
  # customer_tags = ["sink_project", "custom.otel.tag"]

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible ddtrace: It is possible to compatible OTEL Trace with DDTrace trace
  # compatible_ddtrace=false

  ## delete trace message
  # del_message = true

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.opentelemetry.close_resource]
    # service1 = ["resource1", "resource2", ...]
    # service2 = ["resource1", "resource2", ...]
    # "*" = ["close_resource_under_all_services"]
    # ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.opentelemetry.sampler]
    # sampling_rate = 1.0

  # [inputs.opentelemetry.tags]
    # key1 = "value1"
    # key2 = "value2"
    # ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.opentelemetry.threads]
    # buffer = 100
    # threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.opentelemetry.storage]
    # path = "./otel_storage"
    # capacity = 5120

  ## OTEL agent HTTP config for trace and metrics
  ## If enable set to be true, trace and metrics will be received on path respectively, by default is:
  ## trace : /otel/v1/trace
  ## metric: /otel/v1/metric
  ## and the client side should be configured properly with Datakit listening port(default: 9529)
  ## or custom HTTP request path.
  ## for example http://127.0.0.1:9529/otel/v1/trace
  ## The acceptable http_status_ok values will be 200 or 202.
  [inputs.opentelemetry.http]
   enable = true
   http_status_ok = 200
   trace_api = "/otel/v1/trace"
   metric_api = "/otel/v1/metric"
   logs_api = "/otel/v1/logs"

  ## OTEL agent GRPC config for trace and metrics.
  ## GRPC services for trace and metrics can be enabled respectively as setting either to be true.
  ## add is the listening on address for GRPC server.
  [inputs.opentelemetry.grpc]
   trace_enable = true
   metric_enable = true
   addr = "127.0.0.1:4317"

  ## If 'expected_headers' is well configed, then the obligation of sending certain wanted HTTP headers is on the client side,
  ## otherwise HTTP status code 400(bad request) will be provoked.
  ## Note: expected_headers will be effected on both trace and metrics if setted up.
  # [inputs.opentelemetry.expected_headers]
  # ex_version = "1.2.3"
  # ex_name = "env_resource_name"
  # ...

Notes¶

1. It is recommended to use grpc protocol, which has the advantages of high compression ratio, fast serialization and higher efficiency.
2. The route of the http protocol is configurable and the default request path is trace: /otel/v1/trace, metric:/otel/v1/metric
3. When data of type float double is involved, a maximum of two decimal places are reserved.
4. Both http and grpc support the gzip compression format. You can configure the environment variable in exporter to turn it on: OTEL_EXPORTER_OTLP_COMPRESSION = gzip; gzip is not turned on by default.
5. The http protocol request format supports both JSON and Protobuf serialization formats. But grpc only supports Protobuf.

Pay attention to the configuration of environment variables when using OTEL HTTP exporter. Since the default configuration of Datakit is /otel/v1/trace and /otel/v1/metric, if you want to use the HTTP protocol, you need to configure trace and trace separately metric,

The default request routes of OTLP are v1/traces and v1/metrics, which need to be configured separately for these two. If you modify the routing in the configuration file, just replace the routing address below.

General SDK Configuration¶

You can pass the 'otel.javaagent.debug=true' parameter to the agent to view debugging logs. Please note that these logs are quite lengthy and should be used with caution in production environments.

Tracing¶

Datakit only accepts OTLP data. OTLP has clear data types: gRPC, http/protobuf and http/json. For specific configuration, please refer to:

# OpenTelemetry Agent default is gRPC
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.exporter.otlp.endpoint=http://datakit-endpoint:4317

# use http/protobuf
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/protobuf \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/trace \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metric 

# use http/json
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/json \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/trace \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metric

Tag¶

Starting from DataKit version 1.22.0 ,ignore_tags is deprecated. Add a fixed tags, only those in this list will be extracted into the tag. The following is the fixed list:

If you want to add custom labels, you can use environment variables:

-Dotel.resource.attributes=username=myName,env=1.1.0

And modify the whitelist in the configuration file so that a custom label can appear in the first level label of the Guance Cloud link details.

customer_tags = ["sink_project", "username","env"]

Kind¶

All Span has span_kind tag,

• unspecified: unspecified.
• internal: internal span.
• server: WEB server or RPC server.
• client: HTTP client or RPC client.
• producer: message producer.
• consumer: message consumer.

Best Practices¶

Datakit currently provides Go language、Java languages, with other languages available later.

Metric¶

The OpenTelemetry Java Agent obtains the MBean's indicator information from the application through the JMX protocol, and the Java Agent reports the selected JMX indicator through the internal SDK, which means that all indicators are configurable.

You can enable and disable JMX metrics collection by command otel.jmx.enabled=true/false, which is enabled by default.

To control the time interval between MBean detection attempts, one can use the OTEL.jmx.discovery.delay property, which defines the number of milliseconds to elapse between the first and the next detection cycle.

In addition, the acquisition configuration of some third-party software built in the Agent. For details, please refer to: JMX Metric Insight

opentelemetry¶

• tag

• metric list

``¶

• tag

• metric list

Logging¶

Version-1.33.0

“Standard output” LogRecord Exporter is a LogRecord Exporter which outputs the logs to stdout/console.

If a language provides a mechanism to automatically configure a LogRecordProcessor to pair with the associated exporter (e.g., using the OTEL_LOGS_EXPORTER environment variable), by default the standard output exporter SHOULD be paired with a simple processor.

The source of the logs collected through OTEL is the service.name, and it can also be customized by adding tags such as log.source, for example: -Dotel.resource.attributes="log.source=sourcename".

You can View logging documents

Note: If the app is running in a container environment (such as k8s), Datakit will automatically collect logs. If otel collects logs again, there will be a problem of duplicate collection. It is recommended to manually turn off Datakit's autonomous log collection behavior before enabling otel to collect logs.

More Docs¶

• Go open source address OpenTelemetry-go
• Official user manual: opentelemetry-io-docs
• Environment variable configuration: sdk-extensions
• GitHub GuanceCloud version OpenTelemetry-Java-instrumentation