Ingestion Canary
This collector is used to detect data ingestion availability and latency. It automatically generates probe data (metrics, logs, traces) and then verifies data collection success through DQL queries, measuring the latency from data sending to queryable.
Requirements¶
- DataWay must be configured for data reporting and DQL queries
- If
result_workspaceis configured, ensure the workspace URL is accessible
Configuration¶
Go to the conf.d/samples directory under the DataKit installation directory, copy ingestion_canary.conf.sample and name it ingestion_canary.conf. Examples are as follows:
[[inputs.ingestion_canary]]
## Collect interval, default 10m
interval = "10m"
## Query timeout, should be less than interval, default 5m
query_timeout = "5m"
## Poll interval for DQL query, default 500ms
poll_interval = "500ms"
## Max retry count for query errors, default 10
error_retries = 10
## Result workspace URL to report metrics
# result_workspace = "https://openway.guance.com?token=xxx"
## Data categories to collect (metric, logging, tracing)
## If not specified, all categories will be collected
categories = ["metric", "logging", "tracing"]
## Logging configuration
[inputs.ingestion_canary.logging]
storage_index = "default"
## Enable election mode
election = true
## Extra tags
[inputs.ingestion_canary.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
Once configured, restart DataKit.
Can be turned on by ConfigMap Injection Collector Configuration or Config ENV_DATAKIT_INPUTS.
Measurements¶
This collector generates two types of data:
- Probe Data: Probe data points (metrics, logs, traces) used to test data ingestion availability
- Result Metric: Test result metrics containing latency and test status
Probe data points do not carry global tags, only include the data point's own fields and tags, plus tags specified in configuration.
Probe Data¶
ingestion_canary (Metric)¶
Synthetic metric points emitted by the collector per probe round to verify metric ingestion path.
| Tags & Fields | Description |
|---|---|
| test_type ( tag) |
Test type: collect (collector) or cmd (CLI tool) |
| round | Round number of the ingestion canary probe. Type: int | (gauge) Unit: count Tagged by: test_type |
ingestion_canary (Logging)¶
Synthetic logging payload emitted by the canary to verify logging ingestion path.
| Tags & Fields | Description |
|---|---|
| test_type ( tag) |
Test type: collect (collector) or cmd (CLI tool) |
| message | Synthetic log message emitted by the ingestion canary probe. Type: string Unit: N/A Tagged by: test_type |
| round | Round number of the ingestion canary probe. Type: int Unit: count Tagged by: test_type |
ingestion_canary (Tracing)¶
Synthetic tracing payload emitted by the canary to verify trace collection and forwarding.
| Tags & Fields | Description |
|---|---|
| service ( tag) |
Service name |
| source ( tag) |
Source name |
| span_type ( tag) |
Span type |
| test_type ( tag) |
Test type: collect (collector) or cmd (CLI tool) |
| duration | Trace span duration in microseconds. Type: int Unit: time,μs Tagged by: service, source, span_type, test_type |
| parent_id | Parent span identifier emitted by the ingestion canary probe. Type: string Unit: N/A Tagged by: service, source, span_type, test_type |
| resource | Synthetic resource name emitted by the ingestion canary probe. Type: string Unit: N/A Tagged by: service, source, span_type, test_type |
| round | Round number of the ingestion canary probe. Type: int Unit: count Tagged by: service, source, span_type, test_type |
| span_id | Span identifier emitted by the ingestion canary probe. Type: string Unit: N/A Tagged by: service, source, span_type, test_type |
| start | Trace span start time as a Unix timestamp in microseconds. Type: int Unit: timeStamp,usec Tagged by: service, source, span_type, test_type |
| status | Synthetic span status emitted by the ingestion canary probe. Type: string Unit: N/A Tagged by: service, source, span_type, test_type |
| trace_id | Trace identifier emitted by the ingestion canary probe. Type: string Unit: N/A Tagged by: service, source, span_type, test_type |
Result Metric¶
ingestion_canary_result¶
Result metrics of each ingestion canary probe round, including category, status, storage destination and latency.
| Tags & Fields | Description |
|---|---|
| category ( tag) |
Data category: M (metric), L (logging), T (tracing) |
| status ( tag) |
Test status: ok, timeout, error |
| storage_index ( tag) |
Storage index for logging data (optional) |
| latency_ms | Latency from feeding a canary point until it becomes queryable, in milliseconds. Type: int | (gauge) Unit: time,ms Tagged by: category, status, storage_index, test_type |
CLI Tool¶
In addition to the collector mode, a CLI tool is provided for one-time testing:
# Use default configuration
datakit tool --ingestion-canary
# Specify storage index for logging data
datakit tool --ingestion-canary --ingestion-canary-index my_index
Options:
--ingestion-canary: Enable ingestion canary test tool--ingestion-canary-index: Specify storage index for logging data, default is "default" (only applies to logging data)
Description:
The tool generates one round of probe data (metrics, logs, traces), sends it to DataWay, then continuously queries until data is found or user interrupts, and outputs latency for each data type. The tool runs continuously with 10 second interval between rounds.