Dataway Tail-Sampling Spill Guide¶
Applicability¶
Supported starting with Dataway 1.22.2.
Spill is disabled by default and only applies to aggregator_mode: standalone. Sampling rules and TTL are still delivered through /v1/tail_sampling_config; see sampling configuration.
Data Lifecycle¶
When the memory budget is exhausted, Spill stores groups awaiting a decision on disk as raw data. Appends refresh TTL. Expired groups are read, sampled, and reclaimed; even groups dropped by rules can incur disk I/O.
- Exit or restart clears temporary data without recovery, even on a persistent PVC.
- Processing and retries retain group capacity. Persistent downstream failures cause backlog.
- Use a separate writable directory per instance. Raw data is stored in the
dataway-tail-spillsubdirectory underdir; relative paths resolve against the Dataway work directory.
Configuration and Limits¶
YAML *_bytes fields take integer byte counts; durations use values such as 30s.
tail_sampling_spill field |
Default | Description |
|---|---|---|
enabled |
false |
Standalone only |
dir |
tail-sampling-spill |
Must be usable when the disk budget is positive |
memory_bytes |
134217728 (128 MiB) |
Resident fragment budget, not an RSS limit |
disk_bytes |
2147483648 (2 GiB) |
Records and write reservations; does not preallocate disk |
max_group_bytes |
67108864 (64 MiB) |
Total raw payload, metadata, and record headers per group; also limited to 4,096 fragments |
max_groups |
100000 |
Admitted groups, including backlog and processing; separately limits pending packets |
ingress_concurrent |
2 |
In-flight Spill requests |
ingress_max_bytes |
0 (automatic) |
Ingress byte budget; automatic value is the greater of 64 MiB and the per-request limit after capping it at max_group_bytes |
ingress_read_timeout |
30s |
Deadline for reading the whole body; 0 uses the default |
workers |
2 |
Workers processing expired groups concurrently |
When configuring limits:
memory_bytesanddisk_bytescannot both be zero. Group count, concurrency, and workers must be positive.- The request limit is the smallest of positive
max_http_body_bytes,max_group_bytes, and the effective ingress byte budget. Unknown-length requests reserve this limit; compressed requests also need capacity for decoded data. - Group size includes raw data and record overhead. With the Kodo queue enabled, group payload is also capped by
queue_max_bytes. - Budgets do not represent total memory. Leave headroom for decoding, processing, and sending.
Environment variables use DW_TAIL_SAMPLING_SPILL_ plus the uppercase field name. Byte values accept MiB / GiB, for example DW_TAIL_SAMPLING_SPILL_MEMORY_BYTES=2GiB. A legacy dataway.yaml, if present, takes precedence.
Configuration Reference¶
This example uses one instance on a 16c/32g host with an SSD data volume. Retain existing token, remote_host, and other settings, and adjust for your workload.
The example sends sampling results through the Kodo queue: tail_sampling_spill.workers controls group processing concurrency, while kodo_queue controls send concurrency and queue capacity.
aggregator_mode: standalone
max_http_body_bytes: 67108864 # 64 MiB
tail_sampling_spill:
enabled: true
dir: /data/dataway/spill
memory_bytes: 2147483648 # 2 GiB
disk_bytes: 128849018880 # 120 GiB
max_group_bytes: 67108864 # 64 MiB, including record headers and metadata
max_groups: 100000
ingress_concurrent: 16 # Start at 16; consider 32 after observation
ingress_max_bytes: 536870912 # 512 MiB
ingress_read_timeout: 30s
workers: 8
kodo_queue:
enabled: true
workers: 32
queue_size: 128
queue_max_bytes: 1073741824 # 1 GiB shared by queued and in-flight bodies
enqueue_timeout: 100ms
Start ingress concurrency at 16 to leave room for other requests. The kodo_queue.workers: 32 setting also limits global HTTP concurrency for all requests with bodies.
| Item | Constraint with these settings |
|---|---|
| Ingress | 512 MiB holds at most eight requests each charged 64 MiB; unknown-length requests also reserve 64 MiB |
| Memory | Eight workers processing maximum-sized groups need about 512 MiB, plus decoding, routing, and send buffers. Monitor actual RSS and container memory |
| Sending | 1 GiB holds about sixteen 64 MiB bodies, potentially binding before the 128 queue slots; accounting includes in-flight tasks |
| Disk | The Spill budget is 120 GiB. Keeping 20% free requires at least 120 / 0.8 = 150 GiB, before other data; start with a dedicated 200 GiB volume |
| Groups | With a five-minute TTL and no appends or backlog, 100,000 groups accommodates about 333 new groups/s; TTL refresh and downstream waits reduce headroom |
Size capacity from raw data volume before sampling and residence time. Low sampling rates do not proportionally reduce storage needs. Adjust for group sizes, TTL, ingress rate, and SSD latency.
Kubernetes Storage Selection¶
Use a dedicated SSD data volume in production to isolate container and log I/O on the system disk. An existing dedicated SSD is suitable.
| Volume type | Considerations |
|---|---|
| SSD cloud-disk PVC | Select the performance tier through a StorageClass; check both disk and node IOPS/throughput limits |
| Local PV backed by SSD | Requires node affinity and ties the Pod to that node's availability |
Regular emptyDir |
May use the system disk; sizeLimit does not reserve a separate disk. Verify backing storage |
emptyDir.medium: Memory |
Consumes container memory; unsuitable for disk Spill intended to relieve memory pressure |
See Kubernetes volumes. This example uses StatefulSet volumeClaimTemplates for a separate PVC per Pod; see storage behavior.
Alibaba Cloud ACK + ESSD Example¶
For ACK ECS worker nodes with cloud-disk CSI installed: one standalone replica, 200 GiB ESSD PL1, and WaitForFirstConsumer to match the Pod's availability zone. See ACK cloud-disk parameters.
Replace UUID, token, and the Kodo address, verify disk support for the ECS instance and zone, and save this as dataway-spill-ack.yaml. Use a registered instance's UUID. Scaling requires separate UUIDs and stable routing of each sampling group to the same backend.
apiVersion: v1
kind: Namespace
metadata:
name: dataway
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: dataway-spill-essd
provisioner: diskplugin.csi.alibabacloud.com
parameters:
type: cloud_essd
fstype: ext4
performanceLevel: PL1
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain
---
apiVersion: v1
kind: Secret
metadata:
name: dataway-tail-credentials
namespace: dataway
type: Opaque
stringData:
DW_UUID: "REPLACE_WITH_REGISTERED_DATAWAY_UUID"
DW_TOKEN: "REPLACE_WITH_DATAWAY_TOKEN"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: dataway-tail-env
namespace: dataway
data:
DW_REMOTE_HOST: "http://kodo.example.internal:9527"
DW_BIND: "0.0.0.0:9528"
DW_PROM_LISTEN: "0.0.0.0:9090"
DW_AGGREGATOR_MODE: "standalone"
DW_MAX_HTTP_BODY_BYTES: "67108864"
DW_TAIL_SAMPLING_SPILL_ENABLED: "true"
DW_TAIL_SAMPLING_SPILL_DIR: "/usr/local/cloudcare/dataflux/dataway/data/spill"
DW_TAIL_SAMPLING_SPILL_MEMORY_BYTES: "2GiB"
DW_TAIL_SAMPLING_SPILL_DISK_BYTES: "120GiB"
DW_TAIL_SAMPLING_SPILL_MAX_GROUP_BYTES: "64MiB"
DW_TAIL_SAMPLING_SPILL_MAX_GROUPS: "100000"
DW_TAIL_SAMPLING_SPILL_INGRESS_CONCURRENT: "16"
DW_TAIL_SAMPLING_SPILL_INGRESS_MAX_BYTES: "512MiB"
DW_TAIL_SAMPLING_SPILL_INGRESS_READ_TIMEOUT: "30s"
DW_TAIL_SAMPLING_SPILL_WORKERS: "8"
DW_KODO_QUEUE_ENABLED: "true"
DW_KODO_QUEUE_WORKERS: "32"
DW_KODO_QUEUE_SIZE: "128"
DW_KODO_QUEUE_MAX_BYTES: "1GiB"
DW_KODO_QUEUE_ENQUEUE_TIMEOUT: "100ms"
---
apiVersion: v1
kind: Service
metadata:
name: dataway-tail
namespace: dataway
spec:
clusterIP: None
selector:
app: dataway-tail
ports:
- name: http
port: 9528
targetPort: http
- name: metrics
port: 9090
targetPort: metrics
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dataway-tail
namespace: dataway
spec:
serviceName: dataway-tail
replicas: 1
selector:
matchLabels:
app: dataway-tail
template:
metadata:
labels:
app: dataway-tail
spec:
containers:
- name: dataway
image: pubrepo.guance.com/dataflux/dataway:1.22.2
imagePullPolicy: IfNotPresent
command: ["/usr/local/cloudcare/dataflux/dataway/dataway"]
args: ["--docker"]
envFrom:
- configMapRef:
name: dataway-tail-env
- secretRef:
name: dataway-tail-credentials
ports:
- name: http
containerPort: 9528
- name: metrics
containerPort: 9090
readinessProbe:
tcpSocket:
port: http
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: "8"
memory: 16Gi
limits:
cpu: "12"
memory: 24Gi
volumeMounts:
- name: data
mountPath: /usr/local/cloudcare/dataflux/dataway/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
volumeMode: Filesystem
storageClassName: dataway-spill-essd
resources:
requests:
storage: 200Gi
disk_bytesis the Spill disk budget; it does not expand the PVC.- StatefulSets retain PVCs by default;
Retainalso keeps the cloud disk after PVC deletion. Arrange cleanup separately. - The 12-core / 24 GiB Pod limit leaves space for system components on a 16c/32g host. Check node allocatable resources and other workloads.
Deployment and Mount Checks¶
After replacing the placeholders, run:
kubectl apply -f dataway-spill-ack.yaml
kubectl -n dataway rollout status statefulset/dataway-tail
kubectl -n dataway get pods,pvc
kubectl -n dataway describe pvc data-dataway-tail-0
kubectl -n dataway exec dataway-tail-0 -- df -h /usr/local/cloudcare/dataflux/dataway/data
Verify a Bound 200 GiB PVC, an ESSD PL1 backing disk, and a data-volume mount in df. Pending before Pod scheduling is normal; persistent Pending warrants checking CSI, zone, disk quotas, and node resources.
A front proxy can use http://dataway-tail-0.dataway-tail.dataway.svc.cluster.local:9528, adjusted for the cluster DNS domain. Sampling rules must still be delivered.
To inspect metrics locally, forward the port in one terminal and query in another:
For continuous collection, scrape port 9090 on each Pod and retain instance identity. ConfigMap environment changes require Pod recreation; plan changes according to the data lifecycle.
Responses and Failure Handling¶
| Response | Meaning and action |
|---|---|
200 |
Admitted; downstream delivery is not yet confirmed |
412 |
Accepted into pending. Send configuration without resending the packet; pending is limited to 8 MiB and max_groups packets |
413 |
Request, group, fragment-count, or Kodo byte limit exceeded. Adjust data size instead of retrying unchanged. Already accepted fragments remain eligible for processing |
503 + Retry-After: 1 |
Ingress/capacity exhausted, read timeout, disk write failure, or similar. Diagnose with metrics and retry with backoff |
400 / 415 |
Content or compression protocol error; see ingestion protocol |
Spill retries Kodo network errors, 408, 429, and 5xx itself, without using the HTTP failure cache. The default interval is one second; Retry-After can extend it to 60 seconds. Backoff and queue waits do not block other destinations in the group, and successful destinations are not resent. A non-retryable error ends delivery only to the affected destination.
Retry waits release Kodo queue capacity but retain the group, worker, and request-body memory outside the queue budget. The enqueue_timeout setting limits only one queue admission wait. Check retry metrics and logs for ongoing failures; send_error records final errors only when group processing finishes.
Failed deletion pauses new disk writes until background cleanup succeeds. Read failures may lose data; check logs and disk health.
Spill Metrics¶
Spill metrics are exposed at local /metrics and reported to central dataway_aggregate every minute. See Prometheus / Datakit Prom collection for local collection.
Counters accumulate from startup; gauges show current values. Neither resets after reporting. Central reporting identifies instances by the unique UUID label dataway_id; local collection uses instance. Select one collection path to avoid double counting. Missing samples do not mean zero.
Resource Usage¶
dataway_tail_sampling_spill_usage (Gauge). No budget gauges exist; use instance configuration as utilization denominators.
resource |
Unit | Scope |
|---|---|---|
ingress_bytes |
Bytes | In-flight reservation and growth for decoding, not total request buffer memory |
memory_bytes |
Bytes | Accounted resident fragments, excluding some read/merge/routing copies and indexes |
disk_bytes |
Bytes | Records, write reservations, and undeleted files, not filesystem usage |
pending_bytes |
Bytes | Accounted fragments awaiting configuration, fixed 8 MiB limit |
groups |
Groups | Admitted groups waiting, processing, or retrying; excludes pending |
processing_groups |
Groups | Groups held by workers, including I/O and retry waits, not CPU-active threads |
Activity and Errors¶
dataway_tail_sampling_spill_events_total (Counter). Events can overlap and do not sum to requests or lost data.
event |
Scope |
|---|---|
write |
Successful disk admissions/appends, not bytes |
read |
Successfully read groups; not one-to-one with multiple write appends |
ingress_rejected |
Insufficient ingress concurrency or byte reservation, excluding global HTTP rejection |
ingress_read_timeout |
Timeout reading the full body |
capacity_rejected |
Group admission returned 503, including group, disk, write, and pending capacity issues |
write_error |
Failed write; may also increment capacity_rejected |
read_error |
Failed read/validation; previously admitted data may be lost |
cleanup_error |
Failed deletion; retries on one file can count repeatedly |
send_error |
Sending finally returned an error; ongoing transient retries may not increment it |
kodo_queue_retry |
Entered waiting for Kodo queue capacity |
kodo_send_retry |
Retried after downstream transient-error backoff |
Coverage Compared with Existing Metrics¶
Use dedicated metrics for Spill backlog and capacity. Regular-mode submit_queue_*, batch_pending_*, and batch_payload_bytes_total do not apply. Spill sends raw PBPoints; regular mode sends Zstd-compressed data.
The ingest, sink, and send stages of phase_seconds can help diagnose issues; send includes retry waits. Other stages and rule counters do not fully cover Spill and cannot establish data loss. Check logs and downstream data as well.
Queries and Performance Troubleshooting¶
Denominators match the example configuration. Update them with configuration changes and retain per-instance dimensions.
# Disk and group utilization
dataway_tail_sampling_spill_usage{resource="disk_bytes"} / 128849018880
dataway_tail_sampling_spill_usage{resource="groups"} / 100000
# Busy worker fraction (including waits) and process RSS (GiB)
dataway_tail_sampling_spill_usage{resource="processing_groups"} / 8
dataway_mem_stat{type="rss"} / 1073741824
# Error, rejection, and retry rates
sum by (instance, event) (
rate(dataway_tail_sampling_spill_events_total{event=~".*error|.*rejected|ingress_read_timeout|kodo_.*retry"}[5m])
)
| Symptom | Check first |
|---|---|
| Higher CPU | dataway_cpu_usage, ingress volume, ingest/sink duration, and pprof; busy workers do not imply busy CPUs |
| Rising RSS with stable memory_bytes | dataway_mem_stat{type="rss"}, concurrent large groups, read/routing copies, heap, and GC; also host available memory and page cache |
| Growing disk or groups | Ingress rate, TTL, SSD latency, inodes, and Kodo 429/5xx |
| Processing groups stay full | kodo_send_retry / kodo_queue_retry, dataway_kodo_queue_bytes / dataway_kodo_queue_depth; empty queues during backoff do not imply a healthy downstream |
| More 503 responses | Distinguish global dataway_http_concurrency_requests_total, ingress_rejected, ingress_read_timeout, and capacity_rejected |
| More read, write, or cleanup errors | Logs, space, inodes, permissions, and disk health |
dataway_process_io_bytes_total (type="r" / "w") covers the whole process; correlate device latency and utilization. Trial runs should cover at least one complete effective TTL, large-group bursts, downstream throttling, and recovery. Check that backlog drains before adjusting concurrency.