Dataway Tail Sampling¶

Feature Overview¶

Dataway provides tail sampling capabilities with the following external interfaces:

• /v1/tail_sampling (raw payload; compatible with Datakit 2.10's zstd payload without header)
• /v1/tail_sampling_v2 (legacy raw compatibility path)
• /v2/tail_sampling (zstd payload)
• /v1/tail_sampling_config

Tail sampling first receives data grouped at the Dataway side, then decides whether to retain or discard based on sampling rules, and finally writes the retained data to the center.

Currently, three types of data are supported:

• tracing
• logging
• rum

The basic processing flow is as follows:

sequenceDiagram
autonumber

participant dk as Datakit/Client
participant dw as Dataway
participant ts as TailSamplingProcessor
participant kodo as Kodo

dk ->> dw: POST /v2/tail_sampling（zstd）或 /v1/tail_sampling（raw）
alt config ready
    dw ->> ts: ingest packet
    ts ->> dw: kept packets
    dw ->> kodo: write tracing/logging/rum
else config not ready
    dw ->> dw: pending cache
    dw -->> dk: 412 Precondition Failed
    dk ->> dw: POST /v1/tail_sampling_config
    dw ->> ts: update config and drain pending
end

Working Modes¶

Tail sampling and aggregation share the same mode configuration:

• standalone
• proxy

standalone¶

In standalone mode, the current Dataway directly processes tail sampling data:

• Receives protobuf-encoded aggregate.DataPacket
• Looks up the tail sampling configuration by token + data_type
• When the configuration is ready, writes directly to TailSamplingProcessor
• Periodically extracts expired groups and sends them to the corresponding data type write interface

In the current implementation:

• The sampling window advancement period is 1 second
• The derived metric refresh period is 1 minute
• The sending phase uses a worker pool for asynchronous writes

proxy¶

In proxy mode, the current Dataway does not retain local tail sampling state:

• /v1/tail_sampling, /v1/tail_sampling_v2, and /v2/tail_sampling are forwarded to backend nodes
• /v1/tail_sampling_config is broadcast to all backend nodes

Therefore, in proxy mode:

• aggregator_endpoint must be configured
• The client must carry a valid Guance-Pick-Key
• The backend node is responsible for actual sampling and state maintenance

Local Configuration¶

Dataway does not have a separate tail sampling YAML configuration item. Tail sampling uses the same mode configuration as aggregation:

aggregator_mode: standalone
aggregator_endpoint:
  - http://dataway-0:9528
  - http://dataway-1:9528

Environment variables:

DW_AGGREGATOR_MODE=standalone
DW_AGGREGATOR_ENDPOINTS=http://dataway-0:9528,http://dataway-1:9528

Explanation:

• standalone: The current node holds its own tail sampling state
• proxy: The current node only forwards or broadcasts

In Kubernetes, if the front-end Dataway serves as the entry layer and the back-end Dataway is responsible for actual tail sampling, the back-end nodes are better suited for StatefulSet deployment, and the stable addresses of the StatefulSet Pods should be written into aggregator_endpoint.

Sampling Configuration Delivery¶

Tail sampling rules are delivered via an API, not written in dataway.yaml:

POST /v1/tail_sampling_config

The request body is JSON, with the top-level structure as follows:

{
  "version": 1,
  "trace": {},
  "logging": {},
  "rum": {}
}

Where:

• trace corresponds to tracing tail sampling configuration
• logging corresponds to logging tail sampling configuration
• rum corresponds to RUM tail sampling configuration

Tracing Configuration Example¶

{
  "version": 1,
  "trace": {
    "version": 1,
    "data_ttl": "5m",
    "group_key": "trace_id",
    "pipelines": [
      {
        "name": "keep-all",
        "type": "probabilistic",
        "rate": 1
      }
    ],
    "builtin_metrics": [
      {
        "name": "trace_total_count",
        "enabled": true
      }
    ]
  }
}

Explanation:

• trace.group_key can only be trace_id
• trace.data_ttl defaults to 5m when empty
• pipelines supports condition and probabilistic
• condition uses action=keep/drop
• probabilistic uses rate=0~1

Logging Configuration Example¶

{
  "version": 1,
  "logging": {
    "version": 1,
    "data_ttl": "1m",
    "group_dimensions": [
      {
        "group_key": "service",
        "pipelines": [
          {
            "name": "keep-all",
            "type": "probabilistic",
            "rate": 1
          }
        ]
      }
    ]
  }
}

RUM Configuration Example¶

{
  "version": 1,
  "rum": {
    "version": 1,
    "data_ttl": "1m",
    "group_dimensions": [
      {
        "group_key": "session_id",
        "pipelines": [
          {
            "name": "keep-all",
            "type": "probabilistic",
            "rate": 1
          }
        ]
      }
    ]
  }
}

Data Reporting API¶

Tail sampling data interfaces:

POST /v1/tail_sampling
POST /v1/tail_sampling_v2
POST /v2/tail_sampling

Explanation:

• /v1/tail_sampling receives uncompressed PBPoints payload; additionally, it is compatible with the zstd packet sent by Datakit 2.10 without compression negotiation header and PayloadCompression=1
• /v1/tail_sampling_v2 is a legacy raw compatibility path, sharing the same processing logic as /v1/tail_sampling; it is not a zstd protocol path
• /v2/tail_sampling only receives zstd payload; the request must satisfy both:
• header Guance-Tail-Sampling-Payload-Compression: zstd
• aggregate.DataPacket.PayloadCompression=1
• In standalone mode, the request body must be a protobuf-encoded aggregate.DataPacket
• In proxy mode, the request is forwarded to the backend node

Clients should use the following protocol combinations:

Common response status codes:

Compression Protocol and Rolling Upgrade¶

Datakit 2.11 and later send zstd payload via /v2/tail_sampling by default. Dataway explicitly validates the path, header, and packet compression fields:

• Older Dataway versions do not recognize /v2/tail_sampling and return 404
• Newer Dataway versions return 415 Unsupported Media Type when receiving an unsupported compression method, missing header, raw v2 packet, or a v1 zstd packet that does not belong to the Datakit 2.10 compatibility combination
• Upon receiving 404 or 415, Datakit reverts the current packet to raw, retries with /v1/tail_sampling, and caches the endpoint's legacy capability for 10 minutes
• Before fallback sending, Datakit splits the packet based on the decompressed protobuf body size to ensure each splittable raw packet does not exceed Dataway's MaxRawBodySize, preventing high-compression-ratio packets from being rejected with 413 on older nodes
• Datakit 2.10 sends zstd packets directly to /v1/tail_sampling without a negotiation header; the new Dataway maintains precise compatibility for this released version
• Datakit versions earlier than 2.10 continue to send raw v1 data; the new Dataway recalculates span predicates and, before entering the timing wheel, decides whether to compress based on benefit

With the above bidirectional compatibility, upgrades to Datakit 2.11+ and the new Dataway can be performed with mixed rolling. Datakit 2.10 is still incompatible with older Dataway versions that do not support this compatibility; one end should be upgraded to a version with compatibility logic first.

Kept Packet Delivery and Exit Recovery¶

Dataway uses a bounded worker pool and a disk overflow queue to send packets that have been decided to be retained:

• On send failure, it backs off and retries up to 3 times; only after retries are exhausted is the packet counted as failure/drop
• When the memory queue is full, it first enters an asynchronous overflow channel, then writes to the disk queue; after Dataway restarts, it actively opens the existing queue and continues sending
• Upon process exit, packets in the overflow, memory queue, and backoff are written to healthy disk first
• When disk is unavailable, the network fallback retries during exit share a 5-second budget; packets that exhaust the budget are explicitly counted as failure metrics and output a summary log

412 and Pending Cache¶

In standalone mode, if Dataway has just started and the sampling configuration for the corresponding token + data_type has not yet been delivered:

• Dataway first places this batch of data into the local pending cache
• Then returns 412 Precondition Failed

Current behavior:

• The pending cache is an in-memory cache
• Stored by token + data_type
• After the configuration is successfully delivered, available data is automatically drained into TailSamplingProcessor
• The default maximum cache size is 100000 packets

Contractual behavior:

• Upon receiving 412, the client considers this batch of data as accepted by Dataway
• The client only needs to continue sending /v1/tail_sampling_config
• The client does not need to resend this batch of data

Exception:

• If the pending cache is full, Dataway returns 503
• In this case, the request can no longer be considered as received

Tail Sampling Measurement (tail_sampling)¶

The tail sampling configuration supports builtin_metrics. These metrics are generated by the tail sampling processor during the sampling process and written to the center during periodic refresh. After being written to the center, the measurement name is tail_sampling, for example, in Guance you can query the field trace_dropped_count and tags stage/decision/data_type.

The current built-in metrics are as follows.

Tracing¶

• trace_total_count
• trace_kept_count
• trace_dropped_count
• trace_error_count
• span_total_count
• trace_duration

Where:

• trace_duration is a duration distribution metric
• Others are count metrics

Logging¶

• logging_total_count
• logging_error_count
• logging_kept_count
• logging_dropped_count

RUM¶

• rum_total_count
• rum_kept_count
• rum_dropped_count

Explanation:

• When builtin_metrics is empty, all built-in metrics supported by that data type are enabled by default
• These metrics originate from the tail sampling process itself, not from Dataway's own operational metrics

Dataway Auto-Reported Metrics (Measurement dataway_aggregate)¶

In addition to the sampler's own builtin_metrics (written to the tail_sampling measurement), apis/metrics_special.go automatically maintains a set of Dataway self-observation metrics to describe the processing status of the tail sampling API. These metrics are aggregated and enter the center's dataway_aggregate measurement, with field prefixes dataway_http_tail_sampling_*.

Currently, the tail-sampling-related metrics include:

These metrics will:

• Be collected every 1 minute
• Be converted to dataway_aggregate metric points
• Be reported to /v1/write/metric using the Dataway default token
• Reset the current accumulated value after reporting
• The token tag in the metrics is fixed to redacted, and no fragment of the original token is retained

This set of metrics reflects the operational status of Dataway's own processing of tail sampling traffic, not the business statistics of the sampling rules themselves.

Division of responsibilities between the two central measurements:

• tail_sampling: Business statistics of sampling rules (one per token), such as trace_kept_count / trace_dropped_count
• dataway_aggregate: Operational status of the Dataway processing process (aggregated by the Dataway default token), such as per-stage counts, send/backlog, and worker status