Log Streaming
Start an HTTP Server, receive the log text data and report it to Guance.
HTTP URL is fixed as: /v1/write/logstreaming, that is, http://Datakit_IP:PORT/v1/write/logstreaming
Note: If DataKit is deployed in Kubernetes as a daemonset, it can be accessed as a Service at http://datakit-service.datakit:9529
Configuration¶
Collector Configuration¶
Go to the conf.d/samples directory under the DataKit installation directory, copy logstreaming.conf.sample and name it logstreaming.conf. Examples are as follows:
[inputs.logstreaming]
ignore_url_tags = false
json_as_fields = false
## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
## buffer is the size of jobs' buffering of worker channel.
## threads is the total number fo goroutines at running time.
# [inputs.logstreaming.threads]
# buffer = 100
# threads = 8
## Storage config a local storage space in hard dirver to cache trace data.
## path is the local file path used to cache data.
## capacity is total space size(MB) used to store data.
# [inputs.logstreaming.storage]
# path = "./log_storage"
# capacity = 5120
Once configured, restart DataKit.
The collector can now be turned on by ConfigMap Injection Collector Configuration.
JSON Fields Mode¶
Set json_as_fields = true in [inputs.logstreaming] to convert the JSON root object of each line for the default text type into fields. See JSON Fields Mode for the conversion rules. This setting does not apply to the influxdb, firelens, or firehose types and is not an HTTP URL parameter.
Support Parameter¶
logstreaming supports adding parameters to the HTTP URL to manipulate log data. The list of parameters is as follows:
type: Data format; supportsinfluxdb,firelens, andfirehose.- When
typeisinfluxdb(/v1/write/logstreaming?type=influxdb), the data itself is in row protocol format (default precision iss), and only built-in Tags will be added and nothing else will be done - When
typeisfirelens(/v1/write/logstreaming?type=firelens), the data format should be multiple logs in JSON format - When
typeisfirehose(/v1/write/logstreaming?type=firehose), the data format is AWS Firehose data source types. - When this value is empty, the data will be processed such as branching and Pipeline
- When
source: Identify the source of the data, that is, the measurement of the line protocol. Such asnginxorredis(/v1/write/logstreaming?source=nginx). This value is not valid whentypeisinfluxdb. Default isdefault.service: Add a service label field, such as (/v1/write/logstreaming?service=nginx_service). Default tosourcevalue.pipeline: Specify the Pipeline name required for the data, such asnginx.p(/v1/write/logstreaming?pipeline=nginx.p)tags: Add custom tags, split by,, such askey1=value1andkey2=value2(/v1/write/logstreaming?tags=key1=value1,key2=value2)storage_index: Set log index name(The index name must begin with a letter and can only contain lowercase letters, numbers or the underscores character.) Version-1.76.0
FireLens data source types¶
The log, source, and date fields in this type of data will be treated specially. Data example:
[
{
"date": 1705485197.93957,
"container_id": "xxxxxxxxxxx-xxxxxxx",
"container_name": "nginx_demo",
"source": "stdout",
"log": "127.0.0.1 - - [19/Jan/2024:11:49:48 +0800] \"GET / HTTP/1.1\" 403 162 \"-\" \"curl/7.81.0\"",
"ecs_cluster": "Cluster_demo"
},
{
"date": 1705485197.943546,
"container_id": "f68a9aeb3d64493595e89f8821fa3f86-4093234565",
"container_name": "javatest",
"source": "stdout",
"log": "2024/01/19 11:49:48 [error] 1316#1316: *1 directory index of \"/var/www/html/\" is forbidden, client: 127.0.0.1, server: _, request: \"GET / HTTP/1.1\", host: \"localhost\"",
"ecs_cluster": "Cluster_Demo"
}
]
After extracting the two logs in the list, log will be used as the message field of the data, date will be converted to the time of the log, and source will be renamed to firelens_source.
AWS Firehose Data Source Type¶
Send Firehose-formatted data with POST /v1/write/logstreaming?type=firehose. Each records[].data value is the Base64 encoding of the original record bytes. For example:
{
"requestId": "ed46c141-8086-455a-85bc-6644837c7875",
"timestamp": 1677654321000,
"records": [
{
"data": "dGhpcyBpcyBsb2cgbWVzc2FnZQ=="
},
{
"data": "bG9nIGRhdGEuLi4="
}
]
}
The decoded messages are this is log message and log data.... Each record produces one log; newlines inside a record remain in message.
Set source and service to help select the Pipeline. Firehose common attributes are sent in the X-Amz-Firehose-Common-Attributes header; its source takes precedence over the URL parameter.
For cloud ingestion and optional newline splitting, see AWS Firehose HTTP Endpoint. That page's /v1/input/firehose is the cloud ingestion endpoint. Its firehose_split=newline common attribute does not apply to this DataKit endpoint, which does not split records based on that attribute.
Usage¶
- Fluentd uses Influxdb Output doc
- Fluentd uses HTTP Output doc
- Logstash uses Influxdb Output doc
- Logstash uses HTTP Output doc
Simply configure Output Host as a logstreaming URL (http://Datakit_IP:PORT/v1/write/logstreaming)and add corresponding parameters.
Metric¶
default¶
Remote log entries received through the logstreaming HTTP endpoint. The point name comes from the source query parameter and defaults to default.
| Tags & Fields | Description |
|---|---|
| collector_source_ip ( tag) |
Remote client IP address resolved from the HTTP request. |
| ip_or_hostname ( tag) |
Hostname part of the logstreaming request URL. |
| service ( tag) |
Service name. Using the service parameter in the URL. |
| message | Log message text parsed from the request body or protocol-specific payload. Type: string Unit: N/A |
| status | Log status string assigned to the streamed log entry before pipeline processing. Type: string Unit: N/A |