Dataway Aggregate¶
Introduction¶
Dataway provides an aggregate upload capability through the /v1/aggregate API.
This feature is used to receive aggregated metric payloads on Dataway first, then organize them by time window and forward the resulting metric points to the center. Two working modes are currently supported:
standalone: the current Dataway receives aggregate payloads, stores them in a local aggregate cache, and flushes expired windows to Kodo through/v1/write/metricproxy: the current Dataway does not aggregate locally and only forwards/v1/aggregaterequests to backend Dataway nodes
The basic flow is:
sequenceDiagram
autonumber
participant dk as Datakit/Client
participant dw as Dataway
participant cache as Aggregation Window
participant backend as Backend Dataway
participant kodo as Data Platform
dk ->> dw: POST /v1/aggregate
alt standalone
dw ->> cache: write aggregate batch
cache ->> dw: expired windows
dw ->> kodo: POST /v1/write/metric
else proxy
dw ->> dw: pick backend endpoint
dw ->> backend: forward request
end
Working Modes¶
Standalone¶
In standalone mode, Dataway receives aggregate data from DataKit, combines it by time window, and reports the results.
Processing behavior:
- The aggregate cache window is 1 minute
- Dataway checks expired windows every 1 second
- Window data is grouped by workspace
token - One
/v1/write/metricrequest is sent for eachtoken
This mode fits deployments where Dataway is responsible for turning aggregate payloads into final metric writes.
Proxy¶
In proxy mode, Dataway does not process the aggregate content itself. It selects a backend endpoint by the Guance-Pick-Key request header:
As long as the backend list stays unchanged, the same group is forwarded to the same node. All ingress nodes must use backend lists with identical length and order. Coordinate changes to the list and allow in-flight data to finish processing.
That means in proxy mode:
aggregator_endpointis required- The client must send a valid
Guance-Pick-Key - The current node only forwards requests and does not hold aggregate window state
This mode fits an ingress layer that only distributes traffic while backend nodes keep the aggregate state.
Warning
In Kubernetes, if the front Dataway needs to forward /v1/aggregate traffic to fixed backend nodes, aggregator_endpoint must contain stable backend addresses. StatefulSet is a better fit for the backend Dataway deployment because each Pod keeps a stable network identity, which lets the front Dataway forward to fixed endpoints.
Configuration¶
Aggregate-related configuration:
Field description:
aggregator_mode- Allowed values:
standalone,proxy - Defaults to
proxywhen empty
- Allowed values:
aggregator_endpoint- Backend node list used in
proxymode - Optional in
standalonemode
- Backend node list used in
Equivalent environment variables:
Warning
If aggregator_mode is empty, Dataway falls back to proxy. If aggregator_endpoint is also empty in that case, the aggregate feature is not initialized and the /v1/aggregate route is not enabled.
Examples¶
Single-node aggregate mode:
Ingress forwarding to backend aggregate nodes:
aggregator_mode: proxy
aggregator_endpoint:
- http://dataway-0.dataway:9528
- http://dataway-1.dataway:9528
In Kubernetes, these addresses usually map to the stable DNS names of StatefulSet Pods.
API¶
Aggregate API:
Notes:
- Authentication follows the same token validation flow as other Dataway write APIs
- DataKit Aggregator generates the required Protobuf aggregate payload automatically; you do not need to construct request bodies manually
- In
proxymode, the client must also provide theGuance-Pick-Keyheader
Response behavior:
- In
standalonemode, success is returned after the payload is accepted into the local aggregate cache - In
proxymode, the request is forwarded to the selected backend and the backend response decides the final status code
Before admission, standalone validates field names, methods, and quantile options. Field names must not be empty or whitespace-only. Supported methods are sum, avg, count, min, max, histogram (including its alias merge_histogram), stdev, quantiles, count_distinct, last, and first; method names are case-insensitive and ignore surrounding whitespace. Missing, unknown, or unsupported methods (such as expo_histogram) return 400. For quantiles, percentiles must be non-empty and every value must be finite and within [0,1]; missing parameters, NaN, Inf, or out-of-range values also return 400. Any validation failure rejects the entire payload without partial admission.
Malformed or incomplete payloads also return 400 and do not participate in aggregation. If calculation fails for a window, its data is discarded while other windows continue processing. Check Dataway error logs and investigate field and method settings. Discarded data is not recovered automatically.
Built-in Metrics¶
Dataway maintains a small set of built-in metrics while handling aggregate requests. The metrics directly related to aggregate are:
| Metric | Type | Labels | Description |
|---|---|---|---|
dataway_http_api_body_size_bytes_total |
Counter | api, token |
Accumulated request body bytes for /v1/aggregate |
dataway_http_aggr_point_total |
Counter | api, token |
Accumulated point count written from aggregate payloads |
Label description:
api: API path, usually/v1/aggregatefor this featuretoken: workspace token of the current data
These metrics are useful for observing aggregate ingress volume, payload size, and point distribution by workspace.
Automatic Metric Reporting¶
Dataway periodically collects and reports these built-in metrics so you can query aggregation traffic and volume in the platform.
Current behavior:
- Gather accumulated values every 1 minute
- Convert them into the
dataway_aggregatemeasurement - Send them to
/v1/write/metricwith the default Dataway token - Reset the counters after a reporting round succeeds
Field mapping rules:
Countermetrics:- The metric name becomes the field name
Summarymetrics:<metric>_sum<metric>_count<metric>_quantile_<quantile>
Tag mapping rules:
- Prometheus metric labels are copied directly as point tags
For example, dataway_http_api_body_size_bytes_total{api="/v1/aggregate",token="tkn_xxx"} is converted into one dataway_aggregate point whose field name is dataway_http_api_body_size_bytes_total.
Info
Automatic reporting supports Counter and Summary. Gauge, Histogram, and other types are not yet supported for automatic reporting.
Typical Use Cases¶
- Aggregate metrics near the ingress layer before sending final writes upstream
- Separate aggregate ingress traffic from the nodes that keep aggregate state
- Observe aggregate request volume, request body size, and written point counts