コンテンツにスキップ

Dataway Tail Sampling


Introduction

Dataway performs tail sampling on grouped tracing, logging, and rum data. When a group expires, Dataway applies sampling rules and sends the retained data upstream.

Dataway 1.22.2 adds Spill to temporary disk storage for groups awaiting a decision when the memory budget is exhausted. See the ACK + ESSD deployment example.

Regular mode sends batches with Zstd compression. Downstream KodoX Point Workers must support Content-Encoding: zstd. Memory budgets cover specific buffers, not total process memory.

Working Modes

Tail sampling shares the aggregator_mode setting with aggregation.

Standalone

The current node receives data, looks up sampling rules by token + data_type, processes expired groups, and sends retained data upstream.

Proxy

The current node forwards data to backend nodes and broadcasts /v1/tail_sampling_config to all backends. Configure aggregator_endpoint and send a valid Guance-Pick-Key so the same group reaches the same backend.

For Kubernetes backends, use a StatefulSet and stable Pod DNS addresses in aggregator_endpoint.

Local Configuration

Tail sampling uses the same mode settings as aggregation:

aggregator_mode: standalone
aggregator_endpoint:
  - http://dataway-0:9528
  - http://dataway-1:9528

Environment variables:

DW_AGGREGATOR_MODE=standalone
DW_AGGREGATOR_ENDPOINTS=http://dataway-0:9528,http://dataway-1:9528

Backend addresses are used only in proxy mode.

Temporary Tail-Sampling Spill

Spill is disabled by default and applies only to standalone nodes. See the Spill guide for setup, configuration, and storage deployment.

  • Use a separate writable data-volume directory for each instance.
  • Exit or restart clears temporary data without recovery, even on a persistent volume.
  • Retained data uses the same Sinker routing rules. Spill retries transient downstream failures without resending to successful destinations.

Sampling Config Delivery

Sampling rules are delivered by API instead of being written in dataway.yaml:

POST /v1/tail_sampling_config

The JSON request uses trace, logging, or rum fields for the corresponding data types. Complete examples follow. Configuring derived_metrics returns an error because it is unsupported.

Tracing Example

{
  "version": 1,
  "trace": {
    "version": 1,
    "data_ttl": "5m",
    "group_key": "trace_id",
    "pipelines": [
      {
        "name": "keep-all",
        "type": "probabilistic",
        "rate": 1
      }
    ],
    "builtin_metrics": [
      {
        "name": "trace_total_count",
        "enabled": true
      }
    ]
  }
}

Notes:

  • trace.group_key currently only supports trace_id
  • trace.data_ttl defaults to 5m when empty
  • pipelines support condition and probabilistic
    • condition uses action=keep/drop
    • probabilistic uses rate=0~1

Logging Example

Logging and RUM use group_dimensions for grouping. Their data_ttl defaults to 1m when omitted.

{
  "version": 1,
  "logging": {
    "version": 1,
    "data_ttl": "1m",
    "group_dimensions": [
      {
        "group_key": "service",
        "pipelines": [
          {
            "name": "keep-all",
            "type": "probabilistic",
            "rate": 1
          }
        ]
      }
    ]
  }
}

RUM Example

{
  "version": 1,
  "rum": {
    "version": 1,
    "data_ttl": "1m",
    "group_dimensions": [
      {
        "group_key": "session_id",
        "pipelines": [
          {
            "name": "keep-all",
            "type": "probabilistic",
            "rate": 1
          }
        ]
      }
    ]
  }
}

Data APIs

DataKit Aggregator automatically sends tail-sampling packets through HTTP POST; you do not need to construct them manually. Use the table below to check client and protocol compatibility. The compression header is Guance-Tail-Sampling-Payload-Compression.

Client scenario Request path Compression header PayloadCompression Payload
Datakit versions earlier than 2.10 /v1/tail_sampling None 0 Raw PBPoints
Datakit 2.10 compatibility path /v1/tail_sampling None 1 Zstd PBPoints
Default path for Datakit 2.11 and later /v2/tail_sampling zstd 1 Zstd PBPoints
Fallback path for Datakit 2.11 and later /v1/tail_sampling None 0 Raw PBPoints

The legacy /v1/tail_sampling_v2 path accepts only raw data without a compression header, with PayloadCompression set to 0.

Common response status codes:

Status Meaning
200 Data was accepted; downstream delivery is not yet confirmed
400 Invalid data format or fields
412 Configuration is unavailable and data is cached. Send configuration without resending data
413 Request or group limit exceeded. Reduce data size before retrying
415 Compression method, path, header, or compression field does not match
503 Capacity exhausted, read timeout, or disk write failure. Data was not accepted; retry with backoff

For Spill limits and failures, see responses and failure handling.

Compression Protocol and Rolling Upgrades

Datakit 2.11 and later use /v2/tail_sampling by default. After 404 or 415, they retry with uncompressed data at /v1/tail_sampling and cache that node's legacy protocol capability for 10 minutes. This allows rolling upgrades with Dataway versions supporting the protocols above.

Datakit 2.10 requires a Dataway version that accepts Zstd packets without a negotiation header. Earlier Datakit versions continue to use the uncompressed V1 protocol.

Retained Data Delivery and Shutdown Recovery

With Spill disabled, retained data enters the send queue:

  • Failed sends retry with backoff up to 3 times, then count as failure/drop.
  • A full memory queue spills to disk; delivery resumes after restart.
  • Shutdown persists pending sends to disk first. If disk is unavailable, network retries share a 5-second budget. Undelivered data is recorded in failure metrics and logs.

Spill temporary data is not recovered after restart; see data lifecycle.

412 and Pending Cache

In standalone mode, when configuration for token + data_type is unavailable, Dataway stores data in memory and returns 412. Continue sending /v1/tail_sampling_config and do not resend accepted data. Sampling starts automatically when the configuration arrives.

Regular mode limits the cache to 100000 packets and 64 MiB. With Spill enabled, the limits are max_groups packets and 8 MiB. A full cache returns 503, meaning the data was not accepted.

Tail-Sampling Metric Set (tail_sampling)

Sampling statistics are written to the central tail_sampling metric set, controlled by builtin_metrics. An empty configuration enables all built-in metrics for the data type. The trace_duration metric is a duration distribution; the others are counters.

Tracing

  • trace_total_count
  • trace_kept_count
  • trace_dropped_count
  • trace_error_count
  • span_total_count
  • trace_duration

Logging

  • logging_total_count
  • logging_error_count
  • logging_kept_count
  • logging_dropped_count

RUM

  • rum_total_count
  • rum_kept_count
  • rum_dropped_count

Automatic Dataway Metrics (Metric Set dataway_aggregate)

Dataway reports runtime metrics to dataway_aggregate every minute using its default token. These metrics cover tail-sampling processing, delivery, and backlog. The token label is fixed to redacted.

Dedicated Spill metrics are available both at local /metrics and in central dataway_aggregate; see Spill metrics.

The current tail-sampling-related metrics are:

Metric Type Labels Description
dataway_http_api_body_size_bytes_total Counter api, token Accumulated request body bytes for tail-sampling APIs
dataway_http_tail_sampling_trace_total Counter token Received tracing group count
dataway_http_tail_sampling_span_total Counter token Received tracing span count
dataway_http_tail_sampling_packet_stage_total Counter token, data_type, stage, result Packet count by stage (receive/ingest/decision/submit/kodo) and result
dataway_http_tail_sampling_point_stage_total Counter token, data_type, stage, result Point count by stage and result
dataway_http_tail_sampling_rule_packet_total Counter token, data_type, rule_name, rule_index, rule_type, action, result Packet count by matched sampling rule and decision
dataway_http_tail_sampling_rule_point_total Counter Same as above Point count by matched rule and decision
dataway_http_tail_sampling_packet_send_total Counter token, data_type, result Send result count, where result includes success, failure, and drop
dataway_http_tail_sampling_submit_queue_event_total Counter token, data_type, result Submit queue enqueue result (memory/wait/overflow/disk/drop/closed)
dataway_http_tail_sampling_submit_queue_depth Gauge - Current submit queue depth (backlog)
dataway_http_tail_sampling_submit_queue_capacity Gauge - Submit queue capacity
dataway_http_tail_sampling_submit_worker_total Gauge - Current worker count
dataway_http_tail_sampling_submit_worker_busy Gauge - Current busy worker count
dataway_http_tail_sampling_submit_disk_depth Gauge - Disk overflow queue depth
dataway_http_tail_sampling_submit_queue_wait_seconds Summary source Queue wait duration before subsequent handling (source: memory/wait/overflow/disk)

Counters and summaries reset after each report, except for performance and Spill metrics, whose cumulative values are retained from process startup. All gauges retain current values. Tracing packets without RawGroupId may be absent from Kodo-stage trace ID metrics, without affecting delivery.

The tail_sampling metric set records sampling statistics per token; dataway_aggregate records Dataway instance runtime status.

Performance Diagnostics Metrics

These metrics are available both at local /metrics and in central dataway_aggregate. Central reporting identifies instances by dataway_id (instance UUID); local collection uses instance. Ensure UUIDs are unique across deployments.

Metric Type Labels Description
dataway_http_tail_sampling_phase_seconds Summary (sum/count only) stage, listed below Cumulative duration and execution count per stage, without quantiles
dataway_http_tail_sampling_batch_pending_bytes Gauge None Reserved decoded bytes for decoding, batching, queued and in-flight sends
dataway_http_tail_sampling_batch_pending_limit_bytes Gauge None Send byte budget per standalone instance, currently 64 MiB
dataway_http_tail_sampling_batch_payload_bytes_total Counter encoding=raw\|zstd Cumulative batch bytes before and after compression
stage Timing scope
ingest Standalone request reception, validation, and admission; includes failures, excludes proxy forwarding
decision Expired-group checks and sampling decisions; includes empty polls, excludes send-task submission waits
decode Payload decoding; includes failures, excludes waits for decoder capacity
compress Zstd batch compression and output copying
send Kodo queueing, HTTP delivery, and failure caching; excludes batcher retry backoff
budget_wait Waiting for send byte budget or batch bucket capacity
sink Decoding, rule grouping, and repacking for routing; excludes subsequent compression and sending

Stage durations include waits and I/O. They do not measure CPU time and cannot be summed into end-to-end latency. Calculate per-instance deltas or rates for counters and summaries, accounting for restarts; read gauges as current values.

Useful PromQL (exclude zero denominators for average duration and compression ratio):

# Average duration per stage in seconds, retaining instance/stage and other labels
rate(dataway_http_tail_sampling_phase_seconds_sum[5m])
  / rate(dataway_http_tail_sampling_phase_seconds_count[5m])

# Reserved send payload bytes as a fraction of the budget
dataway_http_tail_sampling_batch_pending_bytes
  / dataway_http_tail_sampling_batch_pending_limit_bytes

# Compressed output / input byte ratio; lower means better compression
rate(dataway_http_tail_sampling_batch_payload_bytes_total{encoding="zstd"}[5m])
  / ignoring(encoding) rate(dataway_http_tail_sampling_batch_payload_bytes_total{encoding="raw"}[5m])

Check the following first:

  • Higher CPU: correlate ingress volume and stage duration; use pprof if needed.
  • Budget near its limit with increasing budget_wait and send: inspect downstream responses and the Kodo queue.
  • Rising RSS with stable send reservations: inspect other buffers, heap, and GC. The send budget does not represent total memory.
  • Compression byte counts include retries and routing overhead and cannot establish data loss.

フィードバック

このページは役に立ちましたか?