DDTrace

DDTrace Agent embedded in Datakit is used to receive, calculate and analyze DataDog Tracing protocol data.

DDTrace Documentation and Examples¶

Configuration¶

Collector Configuration¶

[[inputs.ddtrace]]
  ## DDTrace Agent endpoints register by version respectively.
  ## Endpoints can be skipped listen by remove them from the list.
  ## NOTE: DO NOT EDIT.
  endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to GuanCe center is "project_name"
  # customer_tags = ["sink_project", "custom_dd_tag"]

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible otel: It is possible to compatible OTEL Trace with DDTrace trace.
  ## make span_id and parent_id to hex encoding.
  # compatible_otel=true

  ##  It is possible to compatible B3/B3Multi TraceID with DDTrace.
  # trace_id_64_bit_hex=true

  ## delete trace message
  # del_message = true

  ## max spans limit on each trace. default 100000 or set to -1 to remove this limit.
  # trace_max_spans = 100000

  ## max trace body(Content-Length) limit. default 32MiB or set to -1 to remove this limit.
  # max_trace_body_mb = 32

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.ddtrace.close_resource]
  #   service1 = ["resource1", "resource2", ...]
  #   service2 = ["resource1", "resource2", ...]
  #   "*" = ["close_resource_under_all_services"]
  #   ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.ddtrace.sampler]
  #   sampling_rate = 1.0

  # [inputs.ddtrace.tags]
  #   key1 = "value1"
  #   key2 = "value2"
  #   ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.ddtrace.threads]
  #   buffer = 100
  #   threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.ddtrace.storage]
  #   path = "./ddtrace_storage"
  #   capacity = 5120

Notes on Linking Multiple Line Tools¶

DDTrace currently supports the following propagation protocols: datadog/b3multi/tracecontext. There are two things to note:

• When using tracecontext, the compatible_otel=true switch needs to be turned on in the configuration because the link ID is 128 bits.
• When using b3multi, pay attention to the length of trace_id. If it is 64-bit hex encoding, the trace_id_64_bit_hex=true needs to be turned on in the configuration file.
• For more propagation protocol and tool usage, please refer to: Multi-Link Concatenation

Add Pod and Node tags¶

When your service deployed on Kubernetes, we can add Pod/Node tags to Span, edit your Pod yaml, here is a Deployment yaml example:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
        service: my-service
    spec:
      containers:
        - name: my-app
          image: my-app:v0.0.1
          env:
            - name: POD_NAME    # <------
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['service']
            - name: DD_TAGS
              value: pod_name:$(POD_NAME),host:$(NODE_NAME)

Here we must define POD_NAME and NODE_NAME before reference them in dedicated environment keys of DDTrace:

After your Pod started, enter the Pod, we can check if environment applied:

$ env | grep DD_
...

Once environment set, the Pod/Node name will attached to related Span tags.

endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

# [inputs.ddtrace.sampler]
# sampling_rate = 1.0

HTTP Settings¶

If Trace data is sent across machines, you need to set HTTP settings for DataKit.

If you have ddtrace data sent to the DataKit, you can see it on DataKit's monitor:

Turn on Disk Cache¶

If the amount of Trace data is large, in order to avoid causing a lot of resource overhead to the host, you can temporarily cache the Trace data to disk and delay processing:

[inputs.ddtrace.storage]
  path = "/path/to/ddtrace-disk-storage"
  capacity = 5120

DDtrace SDK Configuration¶

After configuring the collector, you can also do some configuration on the DDtrace SDK side.

Environment Variables Setting¶

• DD_TRACE_ENABLED: Enable global tracer (Partial language platform support)
• DD_AGENT_HOST: DDtrace agent host address
• DD_TRACE_AGENT_PORT: DDtrace agent host port
• DD_SERVICE: Service name
• DD_TRACE_SAMPLE_RATE: Set sampling rate
• DD_VERSION: Application version (optional)
• DD_TRACE_STARTUP_LOGS: DDtrace logger
• DD_TRACE_DEBUG: DDtrace debug mode
• DD_ENV: Application env values
• DD_TAGS: Application

In addition to setting the project name, environment name, and version number when initialization is applied, you can also set them in the following two ways:

• Inject environment variables from the command line

DD_TAGS="project:your_project_name,env=test,version=v1" ddtrace-run python app.py

• Configure custom tags directly in ddtrace. conf. This approach affects all data sends to the DataKit tracing service and should be considered carefully:

# tags is ddtrace configed key value pairs
[inputs.ddtrace.tags]
  some_tag = "some_value"
  more_tag = "some_other_value"

APMTelemetry¶

Version-1.35.0 · Experimental

This information can be viewed in the Guance Cloud central infrastructure, and the presented data is helpful for troubleshooting issues related to startup commands and referenced third-party library versions. It also includes host information, service information, and the number of spans created and ended.

There may be significant differences in data due to different languages and versions.

DdTrace APM Telemetry¶

Collect service,host,process APM Telemetry message.

• tag

• metric list

Add a Business Tag to your Code¶

Starting from DataKit version 1.21.0, do not include All in Span.Mate are advanced to the first level label and only select following list labels:

In the link interface of the Guance Cloud, tags that are not in the list can also be filtered.

Restore whitelist functionality from DataKit version 1.22.0. If there are labels that must be extracted from the first level label list, they can be found in the customer_tags.

If the configured whitelist label is in the native message.meta, Will convert to replace . with _.

Tracing¶

ddtrace¶

• tag

• metric list

More Readings¶

• DataKit Tracing Field definition
• DataKit general Tracing data collection instructions
• Proper use of regular expressions to configure