Dataway Tail Sampling¶

Introduction¶

Dataway provides tail-sampling APIs:

• /v1/tail_sampling (raw payload; also accepts headerless zstd payloads from Datakit 2.10)
• /v1/tail_sampling_v2 (legacy raw compatibility path)
• /v2/tail_sampling (zstd payload)
• /v1/tail_sampling_config

Tail sampling receives grouped data on Dataway first, applies sampling rules, and sends the kept data upstream.

Three data types are currently supported:

• tracing
• logging
• rum

The basic flow is:

sequenceDiagram
autonumber

participant dk as Datakit/Client
participant dw as Dataway
participant ts as TailSamplingProcessor
participant kodo as Kodo

dk ->> dw: POST /v2/tail_sampling (zstd) or /v1/tail_sampling (raw)
alt config ready
    dw ->> ts: ingest packet
    ts ->> dw: kept packets
    dw ->> kodo: write tracing/logging/rum
else config not ready
    dw ->> dw: pending cache
    dw -->> dk: 412 Precondition Failed
    dk ->> dw: POST /v1/tail_sampling_config
    dw ->> ts: update config and drain pending
end

Working Modes¶

Tail sampling uses the same mode settings as aggregate:

• standalone
• proxy

standalone¶

In standalone mode, the current Dataway handles tail-sampling data directly:

• receive protobuf-encoded aggregate.DataPacket
• look up sampling config by token + data_type
• ingest the packet into TailSamplingProcessor when config is ready
• periodically flush expired groups to the target write API

In the current implementation:

• the sampling loop advances every 1 second
• derived metrics are flushed every 1 minute
• a worker pool is used for async delivery

proxy¶

In proxy mode, the current Dataway does not keep local tail-sampling state:

• /v1/tail_sampling, /v1/tail_sampling_v2, and /v2/tail_sampling are forwarded to backend nodes
• /v1/tail_sampling_config is broadcast to all backend nodes

That means in proxy mode:

• aggregator_endpoint is required
• the client must send a valid Guance-Pick-Key
• backend nodes keep the actual sampling state

Local Configuration¶

There is no separate local YAML section for tail sampling. Dataway uses the same mode settings as aggregate:

aggregator_mode: standalone
aggregator_endpoint:
  - http://dataway-0:9528
  - http://dataway-1:9528

Environment variables:

DW_AGGREGATOR_MODE=standalone
DW_AGGREGATOR_ENDPOINTS=http://dataway-0:9528,http://dataway-1:9528

Notes:

• standalone: the current node keeps tail-sampling state locally
• proxy: the current node only forwards or broadcasts

In Kubernetes, when a front Dataway acts as the ingress layer and backend Dataway nodes perform the actual tail sampling, StatefulSet is the better backend deployment model and its stable Pod addresses should be used in aggregator_endpoint.

Sampling Config Delivery¶

Sampling rules are delivered by API instead of being written in dataway.yaml:

POST /v1/tail_sampling_config

The request body is JSON with this top-level structure:

{
  "version": 1,
  "trace": {},
  "logging": {},
  "rum": {}
}

Where:

• trace is the tracing tail-sampling config
• logging is the logging tail-sampling config
• rum is the RUM tail-sampling config

Tracing Example¶

{
  "version": 1,
  "trace": {
    "version": 1,
    "data_ttl": "5m",
    "group_key": "trace_id",
    "pipelines": [
      {
        "name": "keep-all",
        "type": "probabilistic",
        "rate": 1
      }
    ],
    "builtin_metrics": [
      {
        "name": "trace_total_count",
        "enabled": true
      }
    ]
  }
}

Notes:

• trace.group_key currently only supports trace_id
• trace.data_ttl defaults to 5m when empty
• pipelines support condition and probabilistic
• condition uses action=keep/drop
• probabilistic uses rate=0~1

Logging Example¶

{
  "version": 1,
  "logging": {
    "version": 1,
    "data_ttl": "1m",
    "group_dimensions": [
      {
        "group_key": "service",
        "pipelines": [
          {
            "name": "keep-all",
            "type": "probabilistic",
            "rate": 1
          }
        ]
      }
    ]
  }
}

RUM Example¶

{
  "version": 1,
  "rum": {
    "version": 1,
    "data_ttl": "1m",
    "group_dimensions": [
      {
        "group_key": "session_id",
        "pipelines": [
          {
            "name": "keep-all",
            "type": "probabilistic",
            "rate": 1
          }
        ]
      }
    ]
  }
}

Data APIs¶

Tail-sampling data APIs:

POST /v1/tail_sampling
POST /v1/tail_sampling_v2
POST /v2/tail_sampling

Notes:

• /v1/tail_sampling accepts an uncompressed PBPoints payload; as a narrow compatibility exception, it also accepts a zstd packet from Datakit 2.10 when the negotiation header is absent and PayloadCompression=1
• /v1/tail_sampling_v2 is a legacy raw compatibility path and uses the same handler as /v1/tail_sampling; it is not the zstd protocol path
• /v2/tail_sampling accepts only zstd payloads; each request must include both:
• header Guance-Tail-Sampling-Payload-Compression: zstd
• aggregate.DataPacket.PayloadCompression=1
• in standalone mode, the request body must be protobuf-encoded aggregate.DataPacket
• in proxy mode, the request is forwarded to a backend node

Clients should use one of the following protocol combinations:

Common response status codes:

Compression Protocol And Rolling Upgrades¶

Datakit 2.11 and later send zstd payloads to /v2/tail_sampling by default. Dataway validates the path, header, and packet compression field explicitly:

• an old Dataway does not recognize /v2/tail_sampling and returns 404
• a new Dataway returns 415 Unsupported Media Type for an unsupported compression method, a missing header, a raw v2 packet, or a v1 zstd packet that does not match the Datakit 2.10 compatibility exception
• after receiving 404 or 415, Datakit converts the current packet back to raw, retries /v1/tail_sampling, and caches the endpoint's legacy capability for 10 minutes
• before fallback, Datakit splits the packet by its decoded protobuf body size so every splittable raw packet stays within Dataway's MaxRawBodySize; this prevents a highly compressible packet from being rejected by an old node with 413
• Datakit 2.10 sends zstd packets directly to /v1/tail_sampling without a negotiation header; a new Dataway accepts this exact combination for compatibility with that released version
• Datakit versions earlier than 2.10 continue sending raw v1 data; a new Dataway computes missing span predicates and compresses the payload before ingestion when compression is beneficial

With both compatibility paths available, Datakit 2.11+ and the new Dataway can be rolled out in either order. Datakit 2.10 remains incompatible with an old Dataway that lacks this exception, so one side must first be upgraded to a version that provides the compatibility bridge.

Kept-packet Delivery And Shutdown Recovery¶

Dataway sends packets that have been kept by the sampling decision through a bounded worker pool and a disk overflow queue:

• a failed send is retried with backoff up to 3 times; only exhausted retries count as failure/drop
• when the memory queue is full, packets enter an asynchronous overflow channel and are then persisted to disk; Dataway opens an existing queue at startup and resumes delivery
• during shutdown, packets in overflow, the memory queue, and retry backoff are persisted to a healthy disk first
• when disk persistence is unavailable, shutdown network fallback retries share a 5-second budget; packets left after the budget are counted explicitly as failures and summarized in the log

412 And Pending Cache¶

In standalone mode, if Dataway has just started and the sampling config for the current token + data_type has not arrived yet:

• Dataway stores the packet in the local pending cache first
• then returns 412 Precondition Failed

Current behavior:

• the pending cache is in memory
• packets are grouped by token + data_type
• after config delivery succeeds, matching packets are drained into TailSamplingProcessor
• the current default limit is 100000 packets

Expected client behavior:

• when the client receives 412, it should treat the packet as already accepted by Dataway
• the client should continue sending /v1/tail_sampling_config
• the client should not resend the same data packet

Failure case:

• if the pending cache is full, Dataway returns 503
• in that case the request must not be treated as accepted

Tail-sampling Metric Set (tail_sampling)¶

Tail-sampling config supports builtin_metrics. These metrics are generated by the tail-sampling processor itself and flushed upstream periodically. Once flushed, the measurement (metric set) name is tail_sampling — for example, query field trace_dropped_count with tags stage/decision/data_type in Guance Cloud.

The current built-in metrics are:

tracing¶

• trace_total_count
• trace_kept_count
• trace_dropped_count
• trace_error_count
• span_total_count
• trace_duration

Where:

• trace_duration is a duration-distribution metric
• the others are count metrics

logging¶

• logging_total_count
• logging_error_count
• logging_kept_count
• logging_dropped_count

rum¶

• rum_total_count
• rum_kept_count
• rum_dropped_count

Notes:

• when builtin_metrics is empty, all supported built-in metrics for that data type are enabled by default
• these metrics come from the sampling process itself, not from Dataway runtime observation

Automatic Dataway Metrics (metric set dataway_aggregate)¶

Besides sampling builtin_metrics (written to the tail_sampling metric set), apis/metrics_special.go also tracks Dataway runtime metrics for the tail-sampling APIs. These metrics are aggregated into the dataway_aggregate metric set with field prefix dataway_http_tail_sampling_*.

The current tail-sampling-related metrics are:

These metrics are:

• gathered every 1 minute
• converted into dataway_aggregate points
• sent to /v1/write/metric with the default Dataway token
• reset after a reporting round
• the token label is always set to redacted; no part of the original token is retained

These metrics describe Dataway's own tail-sampling runtime behavior rather than the business-level sampling outcome.

Division of labor between the two central metric sets:

• tail_sampling: business-level sampling statistics (one copy per token), such as trace_kept_count / trace_dropped_count
• dataway_aggregate: Dataway processing runtime status (aggregated under the default Dataway token), such as per-stage counters, send/backlog, and worker state