Dataway Tail Sampling¶
Introduction¶
Dataway performs tail sampling on grouped tracing, logging, and rum data. When a group expires, Dataway applies sampling rules and sends the retained data upstream.
Dataway 1.22.2 adds Spill to temporary disk storage for groups awaiting a decision when the memory budget is exhausted. See the ACK + ESSD deployment example.
Regular mode sends batches with Zstd compression. Downstream KodoX Point Workers must support Content-Encoding: zstd. Memory budgets cover specific buffers, not total process memory.
Working Modes¶
Tail sampling shares the aggregator_mode setting with aggregation.
Standalone¶
The current node receives data, looks up sampling rules by token + data_type, processes expired groups, and sends retained data upstream.
Proxy¶
The current node forwards data to backend nodes and broadcasts /v1/tail_sampling_config to all backends. Configure aggregator_endpoint and send a valid Guance-Pick-Key so the same group reaches the same backend.
For Kubernetes backends, use a StatefulSet and stable Pod DNS addresses in aggregator_endpoint.
Local Configuration¶
Tail sampling uses the same mode settings as aggregation:
Environment variables:
Backend addresses are used only in proxy mode.
Temporary Tail-Sampling Spill¶
Spill is disabled by default and applies only to standalone nodes. See the Spill guide for setup, configuration, and storage deployment.
- Use a separate writable data-volume directory for each instance.
- Exit or restart clears temporary data without recovery, even on a persistent volume.
- Retained data uses the same Sinker routing rules. Spill retries transient downstream failures without resending to successful destinations.
Sampling Config Delivery¶
Sampling rules are delivered by API instead of being written in dataway.yaml:
The JSON request uses trace, logging, or rum fields for the corresponding data types. Complete examples follow. Configuring derived_metrics returns an error because it is unsupported.
Tracing Example¶
{
"version": 1,
"trace": {
"version": 1,
"data_ttl": "5m",
"group_key": "trace_id",
"pipelines": [
{
"name": "keep-all",
"type": "probabilistic",
"rate": 1
}
],
"builtin_metrics": [
{
"name": "trace_total_count",
"enabled": true
}
]
}
}
Notes:
trace.group_keycurrently only supportstrace_idtrace.data_ttldefaults to5mwhen emptypipelinessupportconditionandprobabilisticconditionusesaction=keep/dropprobabilisticusesrate=0~1
Logging Example¶
Logging and RUM use group_dimensions for grouping. Their data_ttl defaults to 1m when omitted.
{
"version": 1,
"logging": {
"version": 1,
"data_ttl": "1m",
"group_dimensions": [
{
"group_key": "service",
"pipelines": [
{
"name": "keep-all",
"type": "probabilistic",
"rate": 1
}
]
}
]
}
}
RUM Example¶
{
"version": 1,
"rum": {
"version": 1,
"data_ttl": "1m",
"group_dimensions": [
{
"group_key": "session_id",
"pipelines": [
{
"name": "keep-all",
"type": "probabilistic",
"rate": 1
}
]
}
]
}
}
Data APIs¶
DataKit Aggregator automatically sends tail-sampling packets through HTTP POST; you do not need to construct them manually. Use the table below to check client and protocol compatibility. The compression header is Guance-Tail-Sampling-Payload-Compression.
| Client scenario | Request path | Compression header | PayloadCompression |
Payload |
|---|---|---|---|---|
| Datakit versions earlier than 2.10 | /v1/tail_sampling |
None | 0 |
Raw PBPoints |
| Datakit 2.10 compatibility path | /v1/tail_sampling |
None | 1 |
Zstd PBPoints |
| Default path for Datakit 2.11 and later | /v2/tail_sampling |
zstd |
1 |
Zstd PBPoints |
| Fallback path for Datakit 2.11 and later | /v1/tail_sampling |
None | 0 |
Raw PBPoints |
The legacy /v1/tail_sampling_v2 path accepts only raw data without a compression header, with PayloadCompression set to 0.
Common response status codes:
| Status | Meaning |
|---|---|
200 |
Data was accepted; downstream delivery is not yet confirmed |
400 |
Invalid data format or fields |
412 |
Configuration is unavailable and data is cached. Send configuration without resending data |
413 |
Request or group limit exceeded. Reduce data size before retrying |
415 |
Compression method, path, header, or compression field does not match |
503 |
Capacity exhausted, read timeout, or disk write failure. Data was not accepted; retry with backoff |
For Spill limits and failures, see responses and failure handling.
Compression Protocol and Rolling Upgrades¶
Datakit 2.11 and later use /v2/tail_sampling by default. After 404 or 415, they retry with uncompressed data at /v1/tail_sampling and cache that node's legacy protocol capability for 10 minutes. This allows rolling upgrades with Dataway versions supporting the protocols above.
Datakit 2.10 requires a Dataway version that accepts Zstd packets without a negotiation header. Earlier Datakit versions continue to use the uncompressed V1 protocol.
Retained Data Delivery and Shutdown Recovery¶
With Spill disabled, retained data enters the send queue:
- Failed sends retry with backoff up to 3 times, then count as
failure/drop. - A full memory queue spills to disk; delivery resumes after restart.
- Shutdown persists pending sends to disk first. If disk is unavailable, network retries share a 5-second budget. Undelivered data is recorded in failure metrics and logs.
Spill temporary data is not recovered after restart; see data lifecycle.
412 and Pending Cache¶
In standalone mode, when configuration for token + data_type is unavailable, Dataway stores data in memory and returns 412. Continue sending /v1/tail_sampling_config and do not resend accepted data. Sampling starts automatically when the configuration arrives.
Regular mode limits the cache to 100000 packets and 64 MiB. With Spill enabled, the limits are max_groups packets and 8 MiB. A full cache returns 503, meaning the data was not accepted.
Tail-Sampling Metric Set (tail_sampling)¶
Sampling statistics are written to the central tail_sampling metric set, controlled by builtin_metrics. An empty configuration enables all built-in metrics for the data type. The trace_duration metric is a duration distribution; the others are counters.
Tracing¶
trace_total_counttrace_kept_counttrace_dropped_counttrace_error_countspan_total_counttrace_duration
Logging¶
logging_total_countlogging_error_countlogging_kept_countlogging_dropped_count
RUM¶
rum_total_countrum_kept_countrum_dropped_count
Automatic Dataway Metrics (Metric Set dataway_aggregate)¶
Dataway reports runtime metrics to dataway_aggregate every minute using its default token. These metrics cover tail-sampling processing, delivery, and backlog. The token label is fixed to redacted.
Dedicated Spill metrics are available both at local /metrics and in central dataway_aggregate; see Spill metrics.
The current tail-sampling-related metrics are:
| Metric | Type | Labels | Description |
|---|---|---|---|
dataway_http_api_body_size_bytes_total |
Counter | api, token |
Accumulated request body bytes for tail-sampling APIs |
dataway_http_tail_sampling_trace_total |
Counter | token |
Received tracing group count |
dataway_http_tail_sampling_span_total |
Counter | token |
Received tracing span count |
dataway_http_tail_sampling_packet_stage_total |
Counter | token, data_type, stage, result |
Packet count by stage (receive/ingest/decision/submit/kodo) and result |
dataway_http_tail_sampling_point_stage_total |
Counter | token, data_type, stage, result |
Point count by stage and result |
dataway_http_tail_sampling_rule_packet_total |
Counter | token, data_type, rule_name, rule_index, rule_type, action, result |
Packet count by matched sampling rule and decision |
dataway_http_tail_sampling_rule_point_total |
Counter | Same as above | Point count by matched rule and decision |
dataway_http_tail_sampling_packet_send_total |
Counter | token, data_type, result |
Send result count, where result includes success, failure, and drop |
dataway_http_tail_sampling_submit_queue_event_total |
Counter | token, data_type, result |
Submit queue enqueue result (memory/wait/overflow/disk/drop/closed) |
dataway_http_tail_sampling_submit_queue_depth |
Gauge | - | Current submit queue depth (backlog) |
dataway_http_tail_sampling_submit_queue_capacity |
Gauge | - | Submit queue capacity |
dataway_http_tail_sampling_submit_worker_total |
Gauge | - | Current worker count |
dataway_http_tail_sampling_submit_worker_busy |
Gauge | - | Current busy worker count |
dataway_http_tail_sampling_submit_disk_depth |
Gauge | - | Disk overflow queue depth |
dataway_http_tail_sampling_submit_queue_wait_seconds |
Summary | source |
Queue wait duration before subsequent handling (source: memory/wait/overflow/disk) |
Counters and summaries reset after each report, except for performance and Spill metrics, whose cumulative values are retained from process startup. All gauges retain current values. Tracing packets without RawGroupId may be absent from Kodo-stage trace ID metrics, without affecting delivery.
The tail_sampling metric set records sampling statistics per token; dataway_aggregate records Dataway instance runtime status.
Performance Diagnostics Metrics¶
These metrics are available both at local /metrics and in central dataway_aggregate. Central reporting identifies instances by dataway_id (instance UUID); local collection uses instance. Ensure UUIDs are unique across deployments.
| Metric | Type | Labels | Description |
|---|---|---|---|
dataway_http_tail_sampling_phase_seconds |
Summary (sum/count only) | stage, listed below |
Cumulative duration and execution count per stage, without quantiles |
dataway_http_tail_sampling_batch_pending_bytes |
Gauge | None | Reserved decoded bytes for decoding, batching, queued and in-flight sends |
dataway_http_tail_sampling_batch_pending_limit_bytes |
Gauge | None | Send byte budget per standalone instance, currently 64 MiB |
dataway_http_tail_sampling_batch_payload_bytes_total |
Counter | encoding=raw\|zstd |
Cumulative batch bytes before and after compression |
stage |
Timing scope |
|---|---|
ingest |
Standalone request reception, validation, and admission; includes failures, excludes proxy forwarding |
decision |
Expired-group checks and sampling decisions; includes empty polls, excludes send-task submission waits |
decode |
Payload decoding; includes failures, excludes waits for decoder capacity |
compress |
Zstd batch compression and output copying |
send |
Kodo queueing, HTTP delivery, and failure caching; excludes batcher retry backoff |
budget_wait |
Waiting for send byte budget or batch bucket capacity |
sink |
Decoding, rule grouping, and repacking for routing; excludes subsequent compression and sending |
Stage durations include waits and I/O. They do not measure CPU time and cannot be summed into end-to-end latency. Calculate per-instance deltas or rates for counters and summaries, accounting for restarts; read gauges as current values.
Useful PromQL (exclude zero denominators for average duration and compression ratio):
# Average duration per stage in seconds, retaining instance/stage and other labels
rate(dataway_http_tail_sampling_phase_seconds_sum[5m])
/ rate(dataway_http_tail_sampling_phase_seconds_count[5m])
# Reserved send payload bytes as a fraction of the budget
dataway_http_tail_sampling_batch_pending_bytes
/ dataway_http_tail_sampling_batch_pending_limit_bytes
# Compressed output / input byte ratio; lower means better compression
rate(dataway_http_tail_sampling_batch_payload_bytes_total{encoding="zstd"}[5m])
/ ignoring(encoding) rate(dataway_http_tail_sampling_batch_payload_bytes_total{encoding="raw"}[5m])
Check the following first:
- Higher CPU: correlate ingress volume and stage duration; use
pprofif needed. - Budget near its limit with increasing
budget_waitandsend: inspect downstream responses and the Kodo queue. - Rising RSS with stable send reservations: inspect other buffers, heap, and GC. The send budget does not represent total memory.
- Compression byte counts include retries and routing overhead and cannot establish data loss.