Propagation Among Multiple Tracing Stack¶

This article mainly introduces the products of multiple tracing stacks, and how to realize tracing information propagation them in distributed services.

The transparent transmission protocol, also known as the propagation protocol, is implemented by adding specific header information (generally referring to HTTP headers) in service requests and responses. When a service requests another service, it carries specific request headers. When the next hop receives the request, it obtains the specific link information from the request header and inherits it, and continues to propagate backward until the end of the link. In this way, the entire call chain can be correlated.

Common propagation protocols¶

The following is a brief introduction to the differences between these transparent transmission protocols in the HTTP header:

Trace Context¶

Trace Context is a trace protocol standardized by W3C, which defines two HTTP header fields: traceparent and tracestate:

• traceparent contains basic information about the current trace, such as SpanID and ParentSpanID, etc., for example: traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
• tracestate is used to pass metadata related to the trace. For example: tracestate: congo=t61rcWkgMzE

B3/B3Multi¶

B3 is a popular tracking protocol that defines several HTTP header fields to identify tracking information. The B3Multi transparent transmission protocol is an extension of the B3 protocol. The commonly used fields are: X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled, X -B3-Flags etc.

Jaeger¶

Jaeger is a distributed tracing system that defines several HTTP header fields for passing trace information. Commonly used fields are: uber-trace-id, jaeger-baggage, etc.

OpenTracing¶

OpenTracing is a transparent transmission protocol of OpenTelemetry, which defines multiple HTTP header fields for transmitting link information:

• ot-tracer-traceid: used to pass the link ID, indicating a complete request link
• ot-tracer-spanid: used to pass the ID of the current span, representing a single operation or event
• ot-tracer-sampled: used to indicate whether to sample the request to decide whether to record the trace information of the request

Datadog¶

Datadog is a distributed tracing system that defines several HTTP header fields for passing trace information. Commonly used fields are: x-datadog-trace-id, x-datadog-parent-id, etc.

Baggage¶

Baggage is a concept introduced by the Jaeger tracking system, which is used to transfer business-related context information. Baggage is passed through the HTTP header field x-b3-baggage-<key>, where key is the key of the business context.

The real meaning of Baggage is to propagate key-value pairs of the key:value nature, which is often used to propagate AppID, Host-Name, Host-IP, etc.

Link manufacturers and product introduction¶

Products and manufacturers:

The open source address of the product:

• OpenTelemetry is a product under CNCF. At the same time, Observation Cloud also extended it
• Jaeger also belongs to CNCF
• Datadog is a multilingual link tool, in which Observation Cloud has extended
• SkyWalking is an open source product under the Apache Foundation
• Zipkin There are link tools in multiple languages.

Product transparent transmission protocol¶

OpenTelemetry¶

List of tracing transparent transmission protocols supported by OTEL:

Example format of distributed link header information in transparent transmission:

# Example of command line injection (multiple communication protocols are separated by commas)
-Dotel.propagators="tracecontext,baggage"

# Environment variable injection example (Linux)
export OTEL_PROPAGATORS="tracecontext, baggage"

# Environment variable injection example (Windows)
$env:OTEL_PROPAGATORS="tracecontext,baggage"

Datadog¶

Here none means that tracing protocol transparent transmission is not set.

DD_TRACE_PROPAGATION_STYLE¶

Datadog tracing can make inbound settings on the behavior of protocol transparent transmission, that is, whether to inherit the upstream protocol and whether to transparently transmit its own protocol to the downstream. It is controlled separately by the following two environment variables:

• Inbound control: export DD_TRACE_PROPAGATION_STYLE_EXTRACT=<XXX>
• Outbound control: export DD_TRACE_PROPAGATION_STYLE_INJECT=<YYY>
• It is also possible to control both inbound and outbound via a single ENV: export DD_TRACE_PROPAGATION_STYLE="tracecontext,datadog"

Example:

# Inbound will inherit X-Datadog-* and X-B3-* headers (if any),
# X-Datadog-* and X-B3-* request headers will be carried when outbound
$ export DD_TRACE_PROPAGATION_STYLE="datadog,b3" ...

For more language examples, see here.

SkyWalking¶

SkyWalking's own protocol (SW8)

Zipkin¶

see here

Jaeger¶

All supported protocols:

• Jaeger Propagation Format
• B3 propagation -W3C Trace-Context

Multi-link series¶

Request header and vendor support list:

	W3C	b3multi	Jaeger	OpenTracing	Datadog	sw8
HTTP Header	tracecontext/tracestate	X-B3-*	uber-trace-id	ot-tracer-*	x-datadog-*	xxx-xxx-xxx-xxx
OpenTelemetry
Datadog
SkyWalking
Zipkin
Jaeger

According to the specific manufacturer's tools used, the corresponding transparent transmission protocol can be used to realize link series connection and ensure link integrity.

Concatenation Example¶

Here is an example to illustrate the concatenation of DDTrace and OpenTelemetry link data. It can be seen from the above table that both DDTrace and OpenTelemetry support the W3C Trace Context protocol, and link concatenation can be realized through this protocol.

• TraceID in DDTrace is a 64-bit int string, and SpanID and ParentID are also 64-bit int
• TraceID in OTEL is a 128-bit hexadecimal int string, and SpanID and ParentID are 64-bit int strings

If the two want to associate TraceID, DDTrace needs to be upgraded to 128 bits.

No matter which one is the initiator of the request, DDTrace needs to enable 128bit TraceID support (dd.trace.128.bit.traceid.generation.enabled):

# DDTrace start example
$ java -javaagent:/usr/local/ddtrace/dd-java-agent.jar\
  -Ddd.service.name=client \
  -Ddd.trace.128.bit.traceid.generation.enabled=true \
  -Ddd.trace.propagation.style=tracecontext\
  -jar springboot-client.jar

# OTEL start example
$ java -javaagent:/usr/local/ddtrace/opentelemetry-javaagent.jar\
  -dotel.service.name=server\
  -jar springboot-server.jar

The client will send an HTTP request to the server, and DDTrace will pass the link information in the tracecontext request header to the server

However, in the "service call relationship", the data from the two tools cannot be connected. This is because the SpanIDs of both parties are not uniform. DDTrace is a decimal string of numbers, while OpenTelemetry is a hexadecimal number character. string. To do this, you need to modify the configuration in the ddtrace collector and release compatible_otel in ddtrace.conf:

  ## compatible otel: It is possible to compatible OTEL Trace with DDTrace trace.
  ## make span_id and parent_id to hex encoding.
  compatible_otel=true

After compatible_otel=true, all DDTrace span_id and parent_id will become hexadecimal numeric strings.

# convert string to int64
fn parse_int(val: str, base: int) int64

# convert int64 to string
fn format_int(val: int64, base: int) str

So far, DDTrace and OTEL have been connected in series on the link, and the service call relationship and logs can also be connected in series: