Dataway Aggregate¶

Introduction¶

Dataway provides an aggregate upload capability through the /v1/aggregate API.

This feature is used to receive aggregated metric payloads on Dataway first, then organize them by time window and forward the resulting metric points to the center. Two working modes are currently supported:

• standalone: the current Dataway receives aggregate payloads, stores them in a local aggregate cache, and flushes expired windows to Kodo through /v1/write/metric
• proxy: the current Dataway does not aggregate locally and only forwards /v1/aggregate requests to backend Dataway nodes

The basic flow is:

sequenceDiagram
autonumber

participant dk as Datakit/Client
participant dw as Dataway
participant cache as Aggregate Cache
participant kodo as Kodo

dk ->> dw: POST /v1/aggregate
alt standalone
    dw ->> cache: write aggregate batch
    cache ->> dw: expired windows
    dw ->> kodo: POST /v1/write/metric
else proxy
    dw ->> dw: pick backend endpoint
    dw ->> kodo: forward request
end

Working Modes¶

standalone¶

In standalone mode, Dataway decodes the request body as aggregate.Batchs and writes it into the local aggregate.Cache.

In the current implementation:

• the aggregate cache window is 1 minute
• Dataway checks expired windows every 1 second
• window data is grouped by workspace token
• one /v1/write/metric request is sent for each token

This mode fits deployments where Dataway is responsible for turning aggregate payloads into final metric writes.

proxy¶

In proxy mode, Dataway does not process the aggregate content itself. It selects a backend endpoint by the Guance-Pick-Key request header:

target = aggregator_endpoint[pick_key % len(aggregator_endpoint)]

That means in proxy mode:

• aggregator_endpoint is required
• the client must send a valid Guance-Pick-Key
• the current node only forwards requests and does not hold aggregate window state

This mode fits an ingress layer that only distributes traffic while backend nodes keep the aggregate state.

Configuration¶

Aggregate-related configuration:

aggregator_mode: standalone
aggregator_endpoint:
  - http://dataway-0:9528
  - http://dataway-1:9528

Field description:

• aggregator_mode
• allowed values: standalone, proxy
• defaults to proxy when empty
• aggregator_endpoint
• backend node list used in proxy mode
• optional in standalone mode

Equivalent environment variables:

DW_AGGREGATOR_MODE=standalone
DW_AGGREGATOR_ENDPOINTS=http://dataway-0:9528,http://dataway-1:9528

Examples¶

Single-node aggregate mode:

aggregator_mode: standalone

Ingress forwarding to backend aggregate nodes:

aggregator_mode: proxy
aggregator_endpoint:
  - http://dataway-0.dataway:9528
  - http://dataway-1.dataway:9528

In Kubernetes, these addresses usually map to the stable DNS names of StatefulSet Pods.

API¶

Aggregate API:

POST /v1/aggregate

Notes:

• authentication follows the same token validation flow as other Dataway write APIs
• in standalone mode, the request body must be protobuf-encoded aggregate.Batchs
• in proxy mode, the client must also provide the Guance-Pick-Key header

Response behavior:

• in standalone mode, success is returned after the payload is accepted into the local aggregate cache
• in proxy mode, the request is forwarded to the selected backend and the backend response decides the final status code

Built-in Metrics¶

Dataway maintains a small set of built-in metrics while handling aggregate requests. The metrics directly related to aggregate are:

Label description:

• api: API path, usually /v1/aggregate for this feature
• token: workspace token of the current data

These metrics are useful for observing aggregate ingress volume, payload size, and point distribution by workspace.

Automatic Metric Reporting¶

apis/metrics_special.go periodically converts the built-in counters into metric points and reports them through Dataway itself.

Current behavior:

• gather accumulated values every 1 minute
• convert them into the dataway_aggregate measurement
• send them to /v1/write/metric with the default Dataway token
• reset the counters after a reporting round succeeds

Field mapping rules:

• Counter metrics:
• the metric name becomes the field name
• Summary metrics:
• <metric>_sum
• <metric>_count
• <metric>_quantile_<quantile>

Tag mapping rules:

• Prometheus metric labels are copied directly as point tags

For example, dataway_http_api_body_size_bytes_total{api="/v1/aggregate",token="tkn_xxx"} is converted into one dataway_aggregate point whose field name is dataway_http_api_body_size_bytes_total.

Typical Use Cases¶

• aggregate metrics near the ingress layer before sending final writes upstream
• separate aggregate ingress traffic from the nodes that keep aggregate state
• observe aggregate request volume, request body size, and written point counts