DDTrace

DDTrace is an open-source APM (Application Performance Monitoring) product by DataDog. The DDTrace Agent embedded in DataKit is used to receive, process, and analyze data in the DataDog Tracing protocol.

DDTrace Documentation and Examples¶

Configuration¶

[[inputs.ddtrace]]
  ## DDTrace Agent endpoints register by version respectively.
  ## Endpoints can be skipped listen by remove them from the list.
  ## NOTE: DO NOT EDIT.
  endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to center is "project_name"
  # customer_tags = ["sink_project", "custom_dd_tag", "reg:key_*"]

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible otel: It is possible to compatible OTEL Trace with DDTrace trace.
  ## make span_id and parent_id to hex encoding.
  # compatible_otel=true

  ##  It is possible to compatible B3/B3Multi TraceID with DDTrace.
  # trace_id_64_bit_hex=true

  ## api:/telemetry/proxy/api/v2/apmtelemetry is collect jvm metadata.
  ## data is: app-dependencies-loaded,app-client-configuration-change,app-integrations-change ...
  ## default is true.
  # apmtelemetry_route_enable = true

  ## When true, the tracer generates 128 bit Trace IDs, 
  ## and encodes Trace IDs as 32 lowercase hexadecimal characters with zero padding.
  ## default is true.
  # trace_128_bit_id = true

  ## delete trace message
  # del_message = true

  ## max spans limit on each trace. default 100000 or set to -1 to remove this limit.
  # trace_max_spans = 100000

  ## max trace body(Content-Length) limit. default 32MiB or set to -1 to remove this limit.
  # max_trace_body_mb = 32

  ## tracing_metric_enable: trace_hits trace_hits_by_http_status trace_latency trace_errors trace_errors_by_http_status trace_apdex.
  ## Extract the above metrics from the collection traces.
  # tracing_metric_enable = true

  ## Blacklist of metric tags: There are many labels in the metric: "tracing_metrics".
  ## If you want to remove certain tag, you can use the blacklist to remove them.
  ## By default, it includes: source,span_name,env,service,status,version,resource,http_status_code,http_status_class
  ## and "customer_tags", k8s related tags, and others service.
  # tracing_metric_tag_blacklist = ["resource","operation","tag_x"]

  ## Whitelist of metric tags: There are many labels in the metric: "tracing_metrics".
  # tracing_metric_tag_whitelist = []

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.ddtrace.close_resource]
  #   service1 = ["resource1", "resource2", ...]
  #   service2 = ["resource1", "resource2", ...]
  #   "*" = ["close_resource_under_all_services"]
  #   ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.ddtrace.sampler]
  #   sampling_rate = 1.0

  # [inputs.ddtrace.tags]
  #   key1 = "value1"
  #   key2 = "value2"
  #   ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.ddtrace.threads]
  #   buffer = 100
  #   threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.ddtrace.storage]
  #   path = "./ddtrace_storage"
  #   capacity = 5120

The customer_tags parameter supports regular expressions but requires a fixed prefix format reg:. For example, reg:key_* matches all keys starting with key_.

Notes on Multi-Tool Tracing Propagation¶

The TraceID in the DDTrace data structure is of uint64 type. When using the tracecontext propagation protocol, a _dd.p.tid:67c573cf00000000 field is added inside the DDTrace trace details. This is because the trace_id in the tracecontext protocol is a 128-bit hexadecimal-encoded string, and this high-bit tag is added for compatibility purposes.

Currently, DDTrace supports the following propagation protocols: datadog/b3multi/tracecontext. Note the following two scenarios: - When using tracecontext, since the trace ID is 128-bit, you need to enable the compatible_otel=true and trace_128_bit_id switches in the configuration. - When using b3multi, pay attention to the length of the trace_id. If it is a 64-bit hexadecimal encoding, you need to enable trace_id_64_bit_hex=true in the configuration file. - For more propagation protocols and tool usage, refer to: Multi-Tracing Propagation

Inject Pod and Node Information¶

When the application is deployed in a container environment such as Kubernetes, you can append Pod/Node information to the final Span data by modifying the application's YAML file. Below is an example YAML for a Kubernetes Deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
        service: my-service
    spec:
      containers:
        - name: my-app
          image: my-app:v0.0.1
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: DD_TAGS
              value: pod_name:$(POD_NAME),host:$(NODE_NAME)
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['service']

Note that you need to first define POD_NAME and NODE_NAME, then embed them into the DDTrace-specific environment variables.

After the application starts, enter the corresponding Pod and verify if the ENV is in effect:

$ env | grep DD_
...

Once injected successfully, you can see the Pod and Node names where the Span is located in the final Span data.

endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

# [inputs.ddtrace.sampler]
# sampling_rate = 1.0

HTTP Settings¶

If Trace data is sent from a remote machine, you need to configure the HTTP settings of DataKit.

If DDTrace data is sent to DataKit, you can view it on the DataKit monitor:

Enable Disk Cache¶

If the volume of Trace data is large, to avoid excessive resource consumption on the host, you can temporarily cache Trace data to disk for delayed processing:

[inputs.ddtrace.storage]
  path = "/path/to/ddtrace-disk-storage"
  capacity = 5120

DDTrace SDK Configuration¶

After configuring the collector, you can also make additional configurations on the DDTrace SDK side.

Environment Variable Settings¶

• DD_TRACE_ENABLED: Enable global tracer (supported by some language platforms)
• DD_AGENT_HOST: DDTrace agent host address
• DD_TRACE_AGENT_PORT: DDTrace agent host port
• DD_SERVICE: Service name
• DD_TRACE_SAMPLE_RATE: Set sampling rate
• DD_VERSION: Application version (optional)
• DD_TRACE_STARTUP_LOGS: DDTrace logger
• DD_TRACE_DEBUG: DDTrace debug mode
• DD_ENV: Application environment value
• DD_TAGS: Application tags

In addition to setting the project name, environment name, and version number during application initialization, you can also set them in the following two ways:

• Inject environment variables via the command line

DD_TAGS="project:your_project_name,env=test,version=v1" ddtrace-run python app.py

• Configure custom tags directly in ddtrace.conf. This method affects all data sent to the DataKit tracing service, so use it with caution:

# tags are key-value pairs configured for ddtrace
[inputs.ddtrace.tags]
  some_tag = "some_value"
  more_tag = "some_other_value"

APMTelemetry¶

Version-1.35.0 · Experimental

After the DDTrace agent starts, it continuously reports service-related information through an additional interface, such as startup configuration, heartbeat, and the list of loaded agents. You can view this information in Guance Infrastructure -> Resource Directory. The displayed data is helpful for troubleshooting issues related to startup commands and versions of referenced third-party libraries. It also includes host information, service information, and the number of Spans generated.

Data may vary significantly across different languages and versions; please refer to the actual received data.

Fixed Tag Extraction¶

Starting from DataKit version 1.21.0, the blacklist function is deprecated, and not all fields in Span.Meta are extracted into top-level tags anymore—only selected fields are extracted.

The following is a list of tags that may be extracted:

In the Studio tracing interface, tags not in the list can also be used for filtering.

Starting from DataKit version 1.22.0, the whitelist function is restored. If there are tags that must be extracted into the top-level tag list, you can configure them in customer_tags. If the whitelisted tags are in the original message.meta, the collector will use . as a separator and convert . to _ during extraction.

Collected Data Field Description¶

Tracing¶

ddtrace¶

Following is tags/fields of tracing data

Metrics¶

tracing_metrics¶

Based on DDTrace's span data, we count span count, span cost metrics

Custom Objects¶

After DDTrace starts, it reports its own configuration information, integration list, dependencies, and service-related information to DataKit. Currently, only Java Agent is supported. The following is a description of each field:

• app_client_configuration_change: Contains the agent's configuration information
• app_dependencies_loaded: Dependency list (including package names and version information)
• app_integrations_change: Integration list (including package names and whether the agent is enabled)
• Other host information, service information, etc.

tracing_service¶

Collect service, host, process APM telemetry message.

More Readings¶

• DataKit Tracing Field Definition
• DataKit General Tracing Data Collection Description
• Proper Use of Regular Expressions for Configuration
• Multi-Tracing Propagation
• Java Integration and Exception Description
• DDTrace Sampling Strategy and Notes on Multi-Tool Tracing Propagation