Skip to content

OpenTelemetry Agent for Java Configuration

Configuration Conventions Using Environment Variables

In some environments, configuration via environment variables is preferred. Any setting that can be configured using a system property can also be configured using an environment variable. Many of the settings below include both options, but they do not follow the steps below to determine the correct name mapping for the required system property:

  1. Convert the system property to uppercase.
  2. Replace all . and - characters with _. For example, otel.instrumentation.common.default-enabled will be converted to OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED.

Startup

-javaagent:/home/liurui/agent/opentelemetry-javaagent-guance.jar \
-Dotel.traces.exporter=otlp \
-Dotel.exporter.otlp.endpoint=http://localhost:4317 \
-Dotel.resource.attributes=service.name=springboot \
-jar springboot-server.jar

Disable the Agent

The agent is enabled by default. To disable it, set to false.

-Dotel.javaagent.enabled=false

Enable debug Logging

-Dotel.javaagent.debug=true

[otel.javaagent 2024-02-19 10:25:25:909 +0800] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.26.1-guance
[otel.javaagent 2024-02-19 10:25:25:911 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.VersionLogger - Running on Java 11.0.18. JVM Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 11.0.18+9-LTS-195
[otel.javaagent 2024-02-19 10:25:25:911 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.AgentInstaller - io.opentelemetry.javaagent.tooling.AgentInstaller loaded on io.opentelemetry.javaagent.bootstrap.AgentClassLoader@6cc4c815
[otel.javaagent 2024-02-19 10:25:25:924 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.SafeServiceLoader - Unable to load instrumentation class: io/opentelemetry/javaagent/shaded/instrumentation/javaagent/runtimetelemetryjfr/RuntimeMetricsInstallerJfr has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
[otel.javaagent 2024-02-19 10:25:26:098 +0800] [main] DEBUG io.opentelemetry.sdk.internal.JavaVersionSpecific - Using the APIs optimized for: Java 9+
[otel.javaagent 2024-02-19 10:25:26:584 +0800] [main] DEBUG io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder - Global OpenTelemetry set to OpenTelemetrySdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={host.arch="amd64", host.name="liurui", os.description="Linux 5.15.0-92-generic", os.type="linux", process.command_line="/home/liurui/middleware/jdk/jdk-11.0.18/bin/java -javaagent:/home/liurui/agent/opentelemetry-javaagent-1.26.1-guance.jar -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dotel.resource.attributes=service.name=springboot -Dotel.javaagent.debug=true -XX:TieredStopAtLevel=1 -Xverify:none -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dmanagement.endpoints.jmx.exposure.include=* -javaagent:/home/liurui/middleware/idea-IU-231.9161.38/lib/idea_rt.jar=39689:/home/liurui/middleware/idea-IU-231.9161.38/bin -Dfile.encoding=UTF-8 com.zy.observable.server.ServerApplication", process.executable.path="/home/liurui/middleware/jdk/jdk-11.0.18/bin/java", process.pid=11464, process.runtime.description="Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 11.0.18+9-LTS-195", process.runtime.name="Java(TM) SE Runtime Environment", process.runtime.version="11.0.18+9-LTS-195", service.name="springboot", telemetry.auto.version="1.26.1-guance", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.26.0"}}, spanLimitsSupplier=SpanLimitsValue{maxNumberOfAttributes=128, maxNumberOfEvents=128, maxNumberOfLinks=128, maxNumberOfAttributesPerEvent=128, maxNumberOfAttributesPerLink=128, maxAttributeValueLength=2147483647}, sampler=ParentBased{root:AlwaysOnSampler,remoteParentSampled:AlwaysOnSampler,remoteParentNotSampled:AlwaysOffSampler,localParentSampled:AlwaysOnSampler,localParentNotSampled:AlwaysOffSampler}, spanProcessor=MultiSpanProcessor{spanProcessorsStart=[io.opentelemetry.javaagent.tooling.AddThreadDetailsSpanProcessor@74287ea3], spanProcessorsEnd=[BatchSpanProcessor{spanExporter=io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter@7d7758be, scheduleDelayNanos=5000000000, maxExportBatchSize=512, exporterTimeoutNanos=30000000000}, SimpleSpanProcessor{spanExporter=io.opentelemetry.exporter.logging.LoggingSpanExporter@2bdd8394}], spanProcessorsAll=[BatchSpanProcessor{spanExporter=io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter@7d7758be, scheduleDelayNanos=5000000000, maxExportBatchSize=512, exporterTimeoutNanos=30000000000}, io.opentelemetry.javaagent.tooling.AddThreadDetailsSpanProcessor@74287ea3, SimpleSpanProcessor{spanExporter=io.opentelemetry.exporter.logging.LoggingSpanExporter@2bdd8394}]}}, meterProvider=SdkMeterProvider{clock=SystemClock{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={host.arch="amd64", host.name="liurui", os.description="Linux 5.15.0-92-generic", os.type="linux", process.command_line="/home/liurui/middleware/jdk/jdk-11.0.18/bin/java -javaagent:/home/liurui/agent/opentelemetry-javaagent-1.26.1-guance.jar -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dotel.resource.attributes=service.name=springboot -Dotel.javaagent.debug=true -XX:TieredStopAtLevel=1 -Xverify:none -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dmanagement.endpoints.jmx.exposure.include=* -javaagent:/home/liurui/middleware/idea-IU-231.9161.38/lib/idea_rt.jar=39689:/home/liurui/middleware/idea-IU-231.9161.38/bin -Dfile.encoding=UTF-8 com.zy.observable.server.ServerApplication", process.executable.path="/home/liurui/middleware/jdk/jdk-11.0.18/bin/java", process.pid=11464, process.runtime.description="Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 11.0.18+9-LTS-195", process.runtime.name="Java(TM) SE Runtime Environment", process.runtime.version="11.0.18+9-LTS-195", service.name="springboot", telemetry.auto.version="1.26.1-guance", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.26.0"}}, metricReaders=[PeriodicMetricReader{exporter=io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter@68746f22, intervalNanos=60000000000}], views=[]}, loggerProvider=SdkLoggerProvider{clock=SystemClock{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={host.arch="amd64", host.name="liurui", os.description="Linux 5.15.0-92-generic", os.type="linux", process.command_line="/home/liurui/middleware/jdk/jdk-11.0.18/bin/java -javaagent:/home/liurui/agent/opentelemetry-javaagent-1.26.1-guance.jar -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dotel.resource.attributes=service.name=springboot -Dotel.javaagent.debug=true -XX:TieredStopAtLevel=1 -Xverify:none -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dmanagement.endpoints.jmx.exposure.include=* -javaagent:/home/liurui/middleware/idea-IU-231.9161.38/lib/idea_rt.jar=39689:/home/liurui/middleware/idea-IU-231.9161.38/bin -Dfile.encoding=UTF-8 com.zy.observable.server.ServerApplication", process.executable.path="/home/liurui/middleware/jdk/jdk-11.0.18/bin/java", process.pid=11464, process.runtime.description="Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 11.0.18+9-LTS-195", process.runtime.name="Java(TM) SE Runtime Environment", process.runtime.version="11.0.18+9-LTS-195", service.name="springboot", telemetry.auto.version="1.26.1-guance", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.26.0"}}, logLimits=LogLimits{maxNumberOfAttributes=128, maxAttributeValueLength=2147483647}, logRecordProcessor=io.opentelemetry.sdk.logs.NoopLogRecordProcessor@7fd7a283}, propagators=DefaultContextPropagators{textMapPropagator=MultiTextMapPropagator{textMapPropagators=[W3CTraceContextPropagator, W3CBaggagePropagator]}}} by autoconfiguration
[otel.javaagent 2024-02-19 10:25:26:873 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.AgentInstaller - Loading extension instrumentation-loader [class io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader]
[otel.javaagent 2024-02-19 10:25:26:959 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.SafeServiceLoader - Unable to load instrumentation class: io/opentelemetry/javaagent/instrumentation/spring/jms/v6_0/SpringJmsInstrumentationModule has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
[otel.javaagent 2024-02-19 10:25:27:010 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.SafeServiceLoader - Unable to load instrumentation class: io/opentelemetry/javaagent/instrumentation/spring/webmvc/v6_0/SpringWebMvcInstrumentationModule has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
[otel.javaagent 2024-02-19 10:25:27:011 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation opentelemetry-instrumentation-annotations [class io.opentelemetry.javaagent.instrumentation.instrumentationannotations.AnnotationInstrumentationModule]
[otel.javaagent 2024-02-19 10:25:27:031 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation opentelemetry-api [class io.opentelemetry.javaagent.instrumentation.opentelemetryapi.v1_15.OpenTelemetryApiInstrumentationModule]
[otel.javaagent 2024-02-19 10:25:27:032 +0800] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation opentelemetry-api [class io.opentelemetry.javaagent.instrumentation.opentelemetryapi.v1_10.OpenTelemetryApiInstrumentationModule]

After enabling debug, logs are output to the console and to the application's configured log file (if the application has configured log file output).

Database Statement Sanitization

The agent sanitizes all database queries/statements before setting the db.statement semantic attribute. All values (strings, numbers) in the query string are replaced with a question mark (?).

Note: JDBC bind parameters are not captured in db.statement. If you are looking to capture bind parameters, see the corresponding issue.

Examples:

  • The SQL query SELECT a from b where password="secret" will appear as SELECT a from b where password=? in the exported span;
  • The Redis command HSET map password "secret" will appear as HSET map password ? in the exported span.

This behavior is enabled by default for all database instrumentation. Use the following property to disable it:

System property: otel.instrumentation.common.db-statement-sanitizer.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED Default value: true Description: Enables DB statement sanitization.

Span Suppression

It is recommended to enable the span-kind strategy.

Some libraries instrumented by the agent internally use lower-level libraries that are also instrumented. This often results in nested spans containing duplicate telemetry data. For example:

  • Spans generated by the Reactor Netty HTTP client instrumentation will have duplicate HTTP client spans generated by the Netty instrumentation;
  • Dynamo DB spans generated by the AWS SDK instrumentation will have child HTTP client spans generated by its internal HTTP client library (which is also instrumented);
  • Spans generated by the Tomcat instrumentation will have duplicate HTTP server spans generated by the generic Servlet API instrumentation.

The javaagent prevents these situations by detecting and suppressing nested spans with duplicate telemetry data. The suppression behavior can be configured using the following configuration option:

System property: otel.instrumentation.experimental.span-suppression-strategy Description: The javaagent span suppression strategy. The following 3 strategies are supported:

  • semconv: The agent will suppress spans with duplicate semantic conventions. This is the default behavior of the javaagent.
  • span-kind: The agent will suppress spans with the same kind (except INTERNAL).
  • none: The agent will not suppress anything. We do not recommend using this option for anything other than debugging purposes, as it will generate a large amount of duplicate telemetry data.

For example, suppose we instrument a database client that internally uses the Reactor Netty HTTP client, which in turn uses Netty.

Using the default semconv suppression strategy will result in 2 nested CLIENT spans:

  • A CLIENT span with database client semantic attributes emitted by the database client instrumentation;
  • A CLIENT span with HTTP client semantic attributes emitted by the Reactor Netty instrumentation. The Netty instrumentation will be suppressed because it duplicates the Reactor Netty HTTP client instrumentation.

Using the span-kind suppression strategy will produce only one span:

  • A CLIENT span with database client semantic attributes emitted by the database client instrumentation. Both the Reactor Netty and Netty instrumentation will be suppressed because they also emit CLIENT spans.

Finally, using the none suppression strategy will produce 3 spans:

  • A CLIENT span with database client semantic attributes emitted by the database client instrumentation;
  • A CLIENT span with HTTP client semantic attributes emitted by the Reactor Netty instrumentation;
  • A CLIENT span with HTTP client semantic attributes emitted by the Netty instrumentation.

Propagator Configuration

Propagators are used for compatibility with multiple types of APM framework trace data. The propagator type can be adjusted using parameters or environment variables:

  • -Dotel.propagators: The default propagator is tracecontext,baggage (W3C).
  • OTEL_PROPAGATORS

The following types of propagators are currently supported:

  • tracecontext: W3C Trace Context (add baggage as well to include W3C baggage)
  • baggage: W3C Baggage
  • b3: B3 Single
  • b3multi: B3 Multi
  • jaeger: Jaeger (includes Jaeger baggage)
  • xray: AWS X-Ray
  • ottrace: OT Trace

For more information on the propagator principle, refer to the document Link Propagation Mechanism and Use Cases.

Method Tracing

If you cannot modify the code, OpenTelemetry provides the following configuration to capture spans of specific methods in the Java agent.

  • Parameter method: -Dotel.instrumentation.methods.include
  • Environment variable method: OTEL_INSTRUMENTATION_METHODS_INCLUDE

Note that OpenTelemetry does not support wildcard configuration.

Example: -Dotel.instrumentation.methods.include="com.zy.observable.server.service.TestService[users]" means adding tracing to the users method of TestService.

How to Use APM to Trace Complete Class Method Calls

Method Parameter Tracing

By default, the trace can capture the method call, but not the method arguments. The following configuration can be used to capture the method arguments:

-Dotel.instrumentation.methods.attribute.enabled, default value is false.

Export Protocol

OpenTelemetry supports three data protocols: gRPC, http/protobuf, and http/json. The protocol can be adjusted using the following:

  • -Dotel.exporter.otlp.protocol=grpc
  • OTEL_EXPORTER_OTLP_PROTOCOL

DataKit parameter configuration varies depending on the export protocol.

# OpenTelemetry gRPC protocol sent to DataKit
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.exporter.otlp.endpoint=http://datakit-endpoint:4317

# Using http/protobuf method
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/protobuf \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/trace \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metric 

# Using http/json method
-Dotel.exporter=otlp \
-Dotel.exporter.otlp.protocol=http/json \
-Dotel.exporter.otlp.traces.endpoint=http://datakit-endpoint:9529/otel/v1/trace \
-Dotel.exporter.otlp.metrics.endpoint=http://datakit-endpoint:9529/otel/v1/metric

The export protocol may differ between OpenTelemetry versions; it is recommended to configure it manually.

Reference Documents

OpenTelemetry Java Configuration

OpenTelemetry Java Extension Configuration

Opentelemetry Agent Enhanced Version

Feedback

Is this page helpful?