DDTrace

DDTrace is an open-source APM product by DataDog. The DDTrace Agent embedded in DataKit receives, processes, and analyzes data following the DataDog Tracing protocol.

DDTrace Documentation and Examples¶

Configuration¶

[[inputs.ddtrace]]
  ## DDTrace Agent endpoints register by version respectively.
  ## Endpoints can be skipped listen by remove them from the list.
  ## NOTE: DO NOT EDIT.
  endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _, like this :
  ## "project.name" to send to Guance center is "project_name"
  # customer_tags = ["sink_project", "custom_dd_tag"]

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough (not present in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible otel: It is possible to compatible OTEL Trace with DDTrace trace.
  ## make span_id and parent_id to hex encoding.
  # compatible_otel=true

  ##  It is possible to compatible B3/B3Multi TraceID with DDTrace.
  # trace_id_64_bit_hex=true

  ## delete trace message
  # del_message = true

  ## max spans limit on each trace. default 100000 or set to -1 to remove this limit.
  # trace_max_spans = 100000

  ## max trace body(Content-Length) limit. default 32MiB or set to -1 to remove this limit.
  # max_trace_body_mb = 32

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.ddtrace.close_resource]
  #   service1 = ["resource1", "resource2", ...]
  #   service2 = ["resource1", "resource2", ...]
  #   "*" = ["close_resource_under_all_services"]
  #   ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.ddtrace.sampler]
  #   sampling_rate = 1.0

  # [inputs.ddtrace.tags]
  #   key1 = "value1"
  #   key2 = "value2"
  #   ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.ddtrace.threads]
  #   buffer = 100
  #   threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.ddtrace.storage]
  #   path = "./ddtrace_storage"
  #   capacity = 5120

Multi-line Propagation Tool Considerations¶

DDTrace currently supports propagation protocols: datadog/b3multi/tracecontext. Two cases need attention:

• When using tracecontext, since the trace ID is 128 bits, you need to enable the compatible_otel=true switch in the configuration.
• When using b3multi, pay attention to the length of the trace_id. If it is 64-bit hex encoded, you need to enable trace_id_64_bit_hex=true in the configuration file.
• For more information on propagation protocols and tool usage, see: Multi-link Propagation

Inject Pod and Node Information¶

When deploying applications in Kubernetes or other container environments, we can append Pod/Node information to the final Span data by modifying the application's Yaml. Below is an example of a Kubernetes Deployment yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
        service: my-service
    spec:
      containers:
        - name: my-app
          image: my-app:v0.0.1
          env:
            - name: POD_NAME    # <------
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['service']
            - name: DD_TAGS
              value: pod_name:$(POD_NAME),host:$(NODE_NAME)

Note that POD_NAME and NODE_NAME must be defined first, then embedded into the DDTrace-specific environment variables.

After starting the application, enter the corresponding Pod to verify if the ENV variables are effective:

$ env | grep DD_
...

Once injected successfully, the final Span data will include the Pod and Node names where the Span resides.

endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

# [inputs.ddtrace.sampler]
# sampling_rate = 1.0

HTTP Settings¶

If Trace data is sent across machines, you need to configure DataKit's HTTP settings.

If DDTrace data is sent to DataKit, you can see it in DataKit's monitor:

Enable Disk Cache¶

If the volume of Trace data is large, to avoid consuming too much host resources, you can temporarily cache Trace data to disk for delayed processing:

[inputs.ddtrace.storage]
  path = "/path/to/ddtrace-disk-storage"
  capacity = 5120

DDtrace SDK Configuration¶

After configuring the collector, additional configurations can be made on the DDtrace SDK side.

Environment Variable Settings¶

• DD_TRACE_ENABLED: Enable global tracer (supported by some language platforms)
• DD_AGENT_HOST: DDtrace agent host address
• DD_TRACE_AGENT_PORT: DDtrace agent host port
• DD_SERVICE: Service name
• DD_TRACE_SAMPLE_RATE: Set sampling rate
• DD_VERSION: Application version (optional)
• DD_TRACE_STARTUP_LOGS: DDtrace logger
• DD_TRACE_DEBUG: DDtrace debug mode
• DD_ENV: Application environment values
• DD_TAGS: Application tags

In addition to setting project name, environment name, and version during application initialization, you can also set these through the following methods:

• Inject environment variables via command line

DD_TAGS="project:your_project_name,env=test,version=v1" ddtrace-run python app.py

• Configure custom tags directly in _ddtrace.conf. This affects all data sent to the DataKit tracing service and should be carefully considered:

# tags is ddtrace configured key-value pairs
[inputs.ddtrace.tags]
  some_tag = "some_value"
  more_tag = "some_other_value"

APMTelemetry¶

Version-1.35.0 · Experimental

After the DDTrace probe starts, it continuously reports service-related information through additional interfaces, such as startup configuration, heartbeat, loaded probes list, etc. This data can be viewed in Guance's Infrastructure -> Resource Catalog. The displayed data helps troubleshoot startup commands and third-party library version issues. It also includes host information, service information, and the number of generated Spans.

Different languages and versions may result in significant differences in the data. Refer to the actual received data.

DdTrace APM Telemetry¶

Collect service, host, process APM telemetry messages.

• Tags (String type)

• Metric List (Non-string type or long string type)

Fixed Extraction of Tags¶

Starting from DataKit version 1.21.0, the blacklist feature has been deprecated, and not all Span.Mate fields are promoted to top-level tags anymore but selectively extracted.

The following is a list of potentially extracted tags:

In the Tracing interface of Guance, tags not listed here can still be filtered.

Starting from DataKit version 1.22.0, the whitelist feature has been restored. If certain tags must be extracted to the top-level list, they can be configured in customer_tags. If the configured whitelist tags are in the original message.meta, the . separator will be replaced with _.

Trace Field Explanation¶

ddtrace¶

• Tags (String type)

• Metrics (Non-string type or long string type)

Further Reading¶

• DataKit Tracing Field Definitions
• General Tracing Data Collection Instructions for DataKit
• Correctly Using Regular Expressions for Configuration
• Multi-link Propagation
• Java Integration and Exception Handling