Data Aggregation and Tail Sampling¶
Aggregation and tail sampling process data before it reaches the platform. Use them to produce summary metrics and reduce the amount of raw data retained according to your business requirements.
- Data aggregation calculates metrics such as counts, sums, and averages by time window and dimension. Aggregation and retention of the original data are separate configuration decisions.
- Tail sampling collects a group of data before deciding whether to keep or drop it. For tracing, grouping by
trace_idallows you to retain error traces preferentially and sample ordinary traces at a configured rate.
Tail sampling cannot recover data already discarded by SDK head sampling, collector filters, or Pipelines. To make decisions from complete traces, first ensure that the relevant spans can reach the processing node.
Choose a Processing Method¶
| Comparison | Using Dataway | Using EdgeFlow |
|---|---|---|
| Processing component | Dataway aggregation and tail sampling | EdgeFlow deployed separately between DataKit and Dataway |
| DataKit integration | Configure Aggregator to send aggregation or tail sampling packets | Configure Aggregator to send trace data to EdgeFlow |
| Rules | DataKit Aggregator settings and Dataway configuration | DataKit controls data delivery; actual sampling rules are maintained and applied from local DSL files on EdgeFlow |
| Coverage in this section | Metric aggregation, tail sampling, and temporary spill storage | Trace tail sampling, host and Docker deployment, and local rule management |
| Typical use | Add processing to an existing Dataway data path | Deploy and manage trace sampling nodes separately |
Follow the instructions for the method you choose. Dataway's aggregator_mode and EdgeFlow's full proxy and sampling-only integration modes have different meanings. Their rule formats are not interchangeable.
Using Dataway¶
- Prepare the processing nodes using Dataway Installation and Usage.
- Configure collection-side delivery and rules using DataKit Aggregator.
- Enable metric aggregation or tail sampling as required.
- To temporarily spill groups awaiting a decision to disk, see Tail Sampling Spill.
Using EdgeFlow¶
Follow EdgeFlow Tail Sampling in this order: choose an integration mode, install EdgeFlow, apply local DSL rules, configure DataKit, and verify the results.
The number of retained traces is not a direct measure of the real request count. Preferential error retention and targeted drops change the sample distribution. Use independent metrics collected before sampling for request counts, error rates, and latency distributions, and state how they are calculated.