콘텐츠로 이동

Dataway Tail-Sampling Spill Guide


Applicability

Supported starting with Dataway 1.22.2.

Spill is disabled by default and only applies to aggregator_mode: standalone. Sampling rules and TTL are still delivered through /v1/tail_sampling_config; see sampling configuration.

Data Lifecycle

When the memory budget is exhausted, Spill stores groups awaiting a decision on disk as raw data. Appends refresh TTL. Expired groups are read, sampled, and reclaimed; even groups dropped by rules can incur disk I/O.

  • Exit or restart clears temporary data without recovery, even on a persistent PVC.
  • Processing and retries retain group capacity. Persistent downstream failures cause backlog.
  • Use a separate writable directory per instance. Raw data is stored in the dataway-tail-spill subdirectory under dir; relative paths resolve against the Dataway work directory.

Configuration and Limits

YAML *_bytes fields take integer byte counts; durations use values such as 30s.

tail_sampling_spill field Default Description
enabled false Standalone only
dir tail-sampling-spill Must be usable when the disk budget is positive
memory_bytes 134217728 (128 MiB) Resident fragment budget, not an RSS limit
disk_bytes 2147483648 (2 GiB) Records and write reservations; does not preallocate disk
max_group_bytes 67108864 (64 MiB) Total raw payload, metadata, and record headers per group; also limited to 4,096 fragments
max_groups 100000 Admitted groups, including backlog and processing; separately limits pending packets
ingress_concurrent 2 In-flight Spill requests
ingress_max_bytes 0 (automatic) Ingress byte budget; automatic value is the greater of 64 MiB and the per-request limit after capping it at max_group_bytes
ingress_read_timeout 30s Deadline for reading the whole body; 0 uses the default
workers 2 Workers processing expired groups concurrently

When configuring limits:

  • memory_bytes and disk_bytes cannot both be zero. Group count, concurrency, and workers must be positive.
  • The request limit is the smallest of positive max_http_body_bytes, max_group_bytes, and the effective ingress byte budget. Unknown-length requests reserve this limit; compressed requests also need capacity for decoded data.
  • Group size includes raw data and record overhead. With the Kodo queue enabled, group payload is also capped by queue_max_bytes.
  • Budgets do not represent total memory. Leave headroom for decoding, processing, and sending.

Environment variables use DW_TAIL_SAMPLING_SPILL_ plus the uppercase field name. Byte values accept MiB / GiB, for example DW_TAIL_SAMPLING_SPILL_MEMORY_BYTES=2GiB. A legacy dataway.yaml, if present, takes precedence.

Configuration Reference

This example uses one instance on a 16c/32g host with an SSD data volume. Retain existing token, remote_host, and other settings, and adjust for your workload.

The example sends sampling results through the Kodo queue: tail_sampling_spill.workers controls group processing concurrency, while kodo_queue controls send concurrency and queue capacity.

aggregator_mode: standalone
max_http_body_bytes: 67108864       # 64 MiB

tail_sampling_spill:
  enabled: true
  dir: /data/dataway/spill
  memory_bytes: 2147483648          # 2 GiB
  disk_bytes: 128849018880          # 120 GiB
  max_group_bytes: 67108864         # 64 MiB, including record headers and metadata
  max_groups: 100000
  ingress_concurrent: 16           # Start at 16; consider 32 after observation
  ingress_max_bytes: 536870912      # 512 MiB
  ingress_read_timeout: 30s
  workers: 8

kodo_queue:
  enabled: true
  workers: 32
  queue_size: 128
  queue_max_bytes: 1073741824       # 1 GiB shared by queued and in-flight bodies
  enqueue_timeout: 100ms

Start ingress concurrency at 16 to leave room for other requests. The kodo_queue.workers: 32 setting also limits global HTTP concurrency for all requests with bodies.

Item Constraint with these settings
Ingress 512 MiB holds at most eight requests each charged 64 MiB; unknown-length requests also reserve 64 MiB
Memory Eight workers processing maximum-sized groups need about 512 MiB, plus decoding, routing, and send buffers. Monitor actual RSS and container memory
Sending 1 GiB holds about sixteen 64 MiB bodies, potentially binding before the 128 queue slots; accounting includes in-flight tasks
Disk The Spill budget is 120 GiB. Keeping 20% free requires at least 120 / 0.8 = 150 GiB, before other data; start with a dedicated 200 GiB volume
Groups With a five-minute TTL and no appends or backlog, 100,000 groups accommodates about 333 new groups/s; TTL refresh and downstream waits reduce headroom

Size capacity from raw data volume before sampling and residence time. Low sampling rates do not proportionally reduce storage needs. Adjust for group sizes, TTL, ingress rate, and SSD latency.

Kubernetes Storage Selection

Use a dedicated SSD data volume in production to isolate container and log I/O on the system disk. An existing dedicated SSD is suitable.

Volume type Considerations
SSD cloud-disk PVC Select the performance tier through a StorageClass; check both disk and node IOPS/throughput limits
Local PV backed by SSD Requires node affinity and ties the Pod to that node's availability
Regular emptyDir May use the system disk; sizeLimit does not reserve a separate disk. Verify backing storage
emptyDir.medium: Memory Consumes container memory; unsuitable for disk Spill intended to relieve memory pressure

See Kubernetes volumes. This example uses StatefulSet volumeClaimTemplates for a separate PVC per Pod; see storage behavior.

Alibaba Cloud ACK + ESSD Example

For ACK ECS worker nodes with cloud-disk CSI installed: one standalone replica, 200 GiB ESSD PL1, and WaitForFirstConsumer to match the Pod's availability zone. See ACK cloud-disk parameters.

Replace UUID, token, and the Kodo address, verify disk support for the ECS instance and zone, and save this as dataway-spill-ack.yaml. Use a registered instance's UUID. Scaling requires separate UUIDs and stable routing of each sampling group to the same backend.

apiVersion: v1
kind: Namespace
metadata:
  name: dataway
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dataway-spill-essd
provisioner: diskplugin.csi.alibabacloud.com
parameters:
  type: cloud_essd
  fstype: ext4
  performanceLevel: PL1
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain
---
apiVersion: v1
kind: Secret
metadata:
  name: dataway-tail-credentials
  namespace: dataway
type: Opaque
stringData:
  DW_UUID: "REPLACE_WITH_REGISTERED_DATAWAY_UUID"
  DW_TOKEN: "REPLACE_WITH_DATAWAY_TOKEN"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: dataway-tail-env
  namespace: dataway
data:
  DW_REMOTE_HOST: "http://kodo.example.internal:9527"
  DW_BIND: "0.0.0.0:9528"
  DW_PROM_LISTEN: "0.0.0.0:9090"
  DW_AGGREGATOR_MODE: "standalone"
  DW_MAX_HTTP_BODY_BYTES: "67108864"
  DW_TAIL_SAMPLING_SPILL_ENABLED: "true"
  DW_TAIL_SAMPLING_SPILL_DIR: "/usr/local/cloudcare/dataflux/dataway/data/spill"
  DW_TAIL_SAMPLING_SPILL_MEMORY_BYTES: "2GiB"
  DW_TAIL_SAMPLING_SPILL_DISK_BYTES: "120GiB"
  DW_TAIL_SAMPLING_SPILL_MAX_GROUP_BYTES: "64MiB"
  DW_TAIL_SAMPLING_SPILL_MAX_GROUPS: "100000"
  DW_TAIL_SAMPLING_SPILL_INGRESS_CONCURRENT: "16"
  DW_TAIL_SAMPLING_SPILL_INGRESS_MAX_BYTES: "512MiB"
  DW_TAIL_SAMPLING_SPILL_INGRESS_READ_TIMEOUT: "30s"
  DW_TAIL_SAMPLING_SPILL_WORKERS: "8"
  DW_KODO_QUEUE_ENABLED: "true"
  DW_KODO_QUEUE_WORKERS: "32"
  DW_KODO_QUEUE_SIZE: "128"
  DW_KODO_QUEUE_MAX_BYTES: "1GiB"
  DW_KODO_QUEUE_ENQUEUE_TIMEOUT: "100ms"
---
apiVersion: v1
kind: Service
metadata:
  name: dataway-tail
  namespace: dataway
spec:
  clusterIP: None
  selector:
    app: dataway-tail
  ports:
    - name: http
      port: 9528
      targetPort: http
    - name: metrics
      port: 9090
      targetPort: metrics
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dataway-tail
  namespace: dataway
spec:
  serviceName: dataway-tail
  replicas: 1
  selector:
    matchLabels:
      app: dataway-tail
  template:
    metadata:
      labels:
        app: dataway-tail
    spec:
      containers:
        - name: dataway
          image: pubrepo.guance.com/dataflux/dataway:1.22.2
          imagePullPolicy: IfNotPresent
          command: ["/usr/local/cloudcare/dataflux/dataway/dataway"]
          args: ["--docker"]
          envFrom:
            - configMapRef:
                name: dataway-tail-env
            - secretRef:
                name: dataway-tail-credentials
          ports:
            - name: http
              containerPort: 9528
            - name: metrics
              containerPort: 9090
          readinessProbe:
            tcpSocket:
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: "8"
              memory: 16Gi
            limits:
              cpu: "12"
              memory: 24Gi
          volumeMounts:
            - name: data
              mountPath: /usr/local/cloudcare/dataflux/dataway/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        volumeMode: Filesystem
        storageClassName: dataway-spill-essd
        resources:
          requests:
            storage: 200Gi
  • disk_bytes is the Spill disk budget; it does not expand the PVC.
  • StatefulSets retain PVCs by default; Retain also keeps the cloud disk after PVC deletion. Arrange cleanup separately.
  • The 12-core / 24 GiB Pod limit leaves space for system components on a 16c/32g host. Check node allocatable resources and other workloads.

Deployment and Mount Checks

After replacing the placeholders, run:

kubectl apply -f dataway-spill-ack.yaml
kubectl -n dataway rollout status statefulset/dataway-tail
kubectl -n dataway get pods,pvc
kubectl -n dataway describe pvc data-dataway-tail-0
kubectl -n dataway exec dataway-tail-0 -- df -h /usr/local/cloudcare/dataflux/dataway/data

Verify a Bound 200 GiB PVC, an ESSD PL1 backing disk, and a data-volume mount in df. Pending before Pod scheduling is normal; persistent Pending warrants checking CSI, zone, disk quotas, and node resources.

A front proxy can use http://dataway-tail-0.dataway-tail.dataway.svc.cluster.local:9528, adjusted for the cluster DNS domain. Sampling rules must still be delivered.

To inspect metrics locally, forward the port in one terminal and query in another:

kubectl -n dataway port-forward pod/dataway-tail-0 9090:9090
curl -fsS http://127.0.0.1:9090/metrics | grep '^dataway_tail_sampling_spill_'

For continuous collection, scrape port 9090 on each Pod and retain instance identity. ConfigMap environment changes require Pod recreation; plan changes according to the data lifecycle.

Responses and Failure Handling

Response Meaning and action
200 Admitted; downstream delivery is not yet confirmed
412 Accepted into pending. Send configuration without resending the packet; pending is limited to 8 MiB and max_groups packets
413 Request, group, fragment-count, or Kodo byte limit exceeded. Adjust data size instead of retrying unchanged. Already accepted fragments remain eligible for processing
503 + Retry-After: 1 Ingress/capacity exhausted, read timeout, disk write failure, or similar. Diagnose with metrics and retry with backoff
400 / 415 Content or compression protocol error; see ingestion protocol

Spill retries Kodo network errors, 408, 429, and 5xx itself, without using the HTTP failure cache. The default interval is one second; Retry-After can extend it to 60 seconds. Backoff and queue waits do not block other destinations in the group, and successful destinations are not resent. A non-retryable error ends delivery only to the affected destination.

Retry waits release Kodo queue capacity but retain the group, worker, and request-body memory outside the queue budget. The enqueue_timeout setting limits only one queue admission wait. Check retry metrics and logs for ongoing failures; send_error records final errors only when group processing finishes.

Failed deletion pauses new disk writes until background cleanup succeeds. Read failures may lose data; check logs and disk health.

Spill Metrics

Spill metrics are exposed at local /metrics and reported to central dataway_aggregate every minute. See Prometheus / Datakit Prom collection for local collection.

Counters accumulate from startup; gauges show current values. Neither resets after reporting. Central reporting identifies instances by the unique UUID label dataway_id; local collection uses instance. Select one collection path to avoid double counting. Missing samples do not mean zero.

Resource Usage

dataway_tail_sampling_spill_usage (Gauge). No budget gauges exist; use instance configuration as utilization denominators.

resource Unit Scope
ingress_bytes Bytes In-flight reservation and growth for decoding, not total request buffer memory
memory_bytes Bytes Accounted resident fragments, excluding some read/merge/routing copies and indexes
disk_bytes Bytes Records, write reservations, and undeleted files, not filesystem usage
pending_bytes Bytes Accounted fragments awaiting configuration, fixed 8 MiB limit
groups Groups Admitted groups waiting, processing, or retrying; excludes pending
processing_groups Groups Groups held by workers, including I/O and retry waits, not CPU-active threads

Activity and Errors

dataway_tail_sampling_spill_events_total (Counter). Events can overlap and do not sum to requests or lost data.

event Scope
write Successful disk admissions/appends, not bytes
read Successfully read groups; not one-to-one with multiple write appends
ingress_rejected Insufficient ingress concurrency or byte reservation, excluding global HTTP rejection
ingress_read_timeout Timeout reading the full body
capacity_rejected Group admission returned 503, including group, disk, write, and pending capacity issues
write_error Failed write; may also increment capacity_rejected
read_error Failed read/validation; previously admitted data may be lost
cleanup_error Failed deletion; retries on one file can count repeatedly
send_error Sending finally returned an error; ongoing transient retries may not increment it
kodo_queue_retry Entered waiting for Kodo queue capacity
kodo_send_retry Retried after downstream transient-error backoff

Coverage Compared with Existing Metrics

Use dedicated metrics for Spill backlog and capacity. Regular-mode submit_queue_*, batch_pending_*, and batch_payload_bytes_total do not apply. Spill sends raw PBPoints; regular mode sends Zstd-compressed data.

The ingest, sink, and send stages of phase_seconds can help diagnose issues; send includes retry waits. Other stages and rule counters do not fully cover Spill and cannot establish data loss. Check logs and downstream data as well.

Queries and Performance Troubleshooting

Denominators match the example configuration. Update them with configuration changes and retain per-instance dimensions.

# Disk and group utilization
dataway_tail_sampling_spill_usage{resource="disk_bytes"} / 128849018880
dataway_tail_sampling_spill_usage{resource="groups"} / 100000

# Busy worker fraction (including waits) and process RSS (GiB)
dataway_tail_sampling_spill_usage{resource="processing_groups"} / 8
dataway_mem_stat{type="rss"} / 1073741824

# Error, rejection, and retry rates
sum by (instance, event) (
  rate(dataway_tail_sampling_spill_events_total{event=~".*error|.*rejected|ingress_read_timeout|kodo_.*retry"}[5m])
)
Symptom Check first
Higher CPU dataway_cpu_usage, ingress volume, ingest/sink duration, and pprof; busy workers do not imply busy CPUs
Rising RSS with stable memory_bytes dataway_mem_stat{type="rss"}, concurrent large groups, read/routing copies, heap, and GC; also host available memory and page cache
Growing disk or groups Ingress rate, TTL, SSD latency, inodes, and Kodo 429/5xx
Processing groups stay full kodo_send_retry / kodo_queue_retry, dataway_kodo_queue_bytes / dataway_kodo_queue_depth; empty queues during backoff do not imply a healthy downstream
More 503 responses Distinguish global dataway_http_concurrency_requests_total, ingress_rejected, ingress_read_timeout, and capacity_rejected
More read, write, or cleanup errors Logs, space, inodes, permissions, and disk health

dataway_process_io_bytes_total (type="r" / "w") covers the whole process; correlate device latency and utilization. Trial runs should cover at least one complete effective TTL, large-group bursts, downstream throttling, and recovery. Check that backlog drains before adjusting concurrency.

문서 평가

이 페이지가 도움이 되었나요?