AWS Firehose HTTP Endpoint
This page describes sending logs from AWS Firehose to the Guance cloud endpoint. Use a common attribute to split multiple newline-delimited events in one Firehose record into separate logs.
Endpoint¶
In the Firehose HTTP endpoint destination, enter the ingestion address provided for your site or private deployment, using a workspace token:
category defaults to logging. AWS Firehose requires HTTPS on port 443 for HTTP endpoint destinations.
| Receiver | Endpoint Path | Splitting Within Records |
|---|---|---|
| Cloud ingestion endpoint | /v1/input/firehose?token=<workspace-token>&category=logging |
Supports firehose_split; deploy a log processing service that supports it before enabling |
| DataKit Logstreaming | /v1/write/logstreaming?type=firehose |
One log per record; the splitting attribute on this page is unsupported |
For DataKit configuration, see Logstreaming. The following attributes and response behavior apply to the cloud endpoint.
Configure Common Attributes¶
Add these common attributes to the Firehose HTTP endpoint destination:
| Attribute | Example Value | Purpose |
|---|---|---|
source |
cloudtrail |
Log source used to select the Pipeline |
service |
audit |
Service tag |
firehose_split |
newline |
Optional; split each record on actual newlines |
Firehose sends the attributes in a request header, for example:
X-Amz-Firehose-Common-Attributes: {"commonAttributes":{"source":"cloudtrail","service":"audit","firehose_split":"newline"}}
firehose_split is a common attribute, not a URL parameter or a field in the business log JSON. Missing, empty, or unsupported values preserve one log per record. Unsupported values produce a configuration warning.
The splitting and synchronous error responses described here require a supporting server release. Before enabling them, confirm that the ingestion service and all kodox nodes consuming Firehose logs support this contract. For private deployments, upgrade these services before setting newline. Older nodes do not split records and store the attribute as an ordinary log tag.
Splitting Rules and Example¶
The request body uses Firehose JSON, with record bytes Base64-encoded in records[].data. When HTTP gzip compression is enabled, the body is decompressed before JSON parsing, Base64 decoding, and record splitting.
With firehose_split=newline:
- Split each record on actual LF or CRLF, preserving record and line order.
- Skip empty lines created by delimiters and preserve spaces around non-empty lines. An escaped
\ninside a JSON string is not a delimiter. - Each line inherits the request timestamp,
source,request_id,arn, protocol version, and ordinary common attributes. The control attributefirehose_splitis not stored as a log tag. - JSON arrays and CloudWatch
logEventsarrays are not expanded. Business JSON fields are still extracted by the Pipeline.
For example, this request contains one record with two JSON lines:
{
"requestId": "ed46c141-8086-455a-85bc-6644837c7875",
"timestamp": 1677654321000,
"records": [
{
"data": "eyJldmVudElEIjoib25lIn0KeyJldmVudElEIjoidHdvIn0K"
}
]
}
The decoded content is:
By default, this produces one log containing both lines. With splitting enabled, it produces two logs, allowing the Pipeline to extract each eventID separately. Splitting increases the number of logs and subsequent processing volume.
HTTP Responses and Retries¶
| Result | HTTP Status |
|---|---|
| Request accepted and successfully queued | 200 |
| Invalid token, permissions, parameters, request size, or other sender errors detected synchronously | Corresponding 4xx, such as 400 or 403 |
| Ingestion service, internal dependency, or queue publication failure | Corresponding 5xx, such as 500 or 503 |
Responses use Firehose JSON containing requestId and a millisecond timestamp, plus errorMessage on failure. A 200 means the request was accepted and queued; it does not confirm parsing or storage. Later format, Pipeline, or storage failures do not change the response already returned.
AWS Firehose treats responses other than 200 and 413 as retriable errors, including other 4xx responses. A 413 is a permanent failure, and the batch is not sent to the configured error bucket. Retries can produce duplicate deliveries; a successful response does not guarantee end-to-end deduplication. See the AWS request and response specification.