Skip to content

AWS Firehose HTTP Endpoint

This page describes sending logs from AWS Firehose to the Guance cloud endpoint. Use a common attribute to split multiple newline-delimited events in one Firehose record into separate logs.

Endpoint

In the Firehose HTTP endpoint destination, enter the ingestion address provided for your site or private deployment, using a workspace token:

https://<endpoint-host>/v1/input/firehose?token=<workspace-token>&category=logging

category defaults to logging. AWS Firehose requires HTTPS on port 443 for HTTP endpoint destinations.

Receiver Endpoint Path Splitting Within Records
Cloud ingestion endpoint /v1/input/firehose?token=<workspace-token>&category=logging Supports firehose_split; deploy a log processing service that supports it before enabling
DataKit Logstreaming /v1/write/logstreaming?type=firehose One log per record; the splitting attribute on this page is unsupported

For DataKit configuration, see Logstreaming. The following attributes and response behavior apply to the cloud endpoint.

Configure Common Attributes

Add these common attributes to the Firehose HTTP endpoint destination:

Attribute Example Value Purpose
source cloudtrail Log source used to select the Pipeline
service audit Service tag
firehose_split newline Optional; split each record on actual newlines

Firehose sends the attributes in a request header, for example:

X-Amz-Firehose-Common-Attributes: {"commonAttributes":{"source":"cloudtrail","service":"audit","firehose_split":"newline"}}

firehose_split is a common attribute, not a URL parameter or a field in the business log JSON. Missing, empty, or unsupported values preserve one log per record. Unsupported values produce a configuration warning.

The splitting and synchronous error responses described here require a supporting server release. Before enabling them, confirm that the ingestion service and all kodox nodes consuming Firehose logs support this contract. For private deployments, upgrade these services before setting newline. Older nodes do not split records and store the attribute as an ordinary log tag.

Splitting Rules and Example

The request body uses Firehose JSON, with record bytes Base64-encoded in records[].data. When HTTP gzip compression is enabled, the body is decompressed before JSON parsing, Base64 decoding, and record splitting.

With firehose_split=newline:

  • Split each record on actual LF or CRLF, preserving record and line order.
  • Skip empty lines created by delimiters and preserve spaces around non-empty lines. An escaped \n inside a JSON string is not a delimiter.
  • Each line inherits the request timestamp, source, request_id, arn, protocol version, and ordinary common attributes. The control attribute firehose_split is not stored as a log tag.
  • JSON arrays and CloudWatch logEvents arrays are not expanded. Business JSON fields are still extracted by the Pipeline.

For example, this request contains one record with two JSON lines:

{
  "requestId": "ed46c141-8086-455a-85bc-6644837c7875",
  "timestamp": 1677654321000,
  "records": [
    {
      "data": "eyJldmVudElEIjoib25lIn0KeyJldmVudElEIjoidHdvIn0K"
    }
  ]
}

The decoded content is:

{"eventID":"one"}
{"eventID":"two"}

By default, this produces one log containing both lines. With splitting enabled, it produces two logs, allowing the Pipeline to extract each eventID separately. Splitting increases the number of logs and subsequent processing volume.

HTTP Responses and Retries

Result HTTP Status
Request accepted and successfully queued 200
Invalid token, permissions, parameters, request size, or other sender errors detected synchronously Corresponding 4xx, such as 400 or 403
Ingestion service, internal dependency, or queue publication failure Corresponding 5xx, such as 500 or 503

Responses use Firehose JSON containing requestId and a millisecond timestamp, plus errorMessage on failure. A 200 means the request was accepted and queued; it does not confirm parsing or storage. Later format, Pipeline, or storage failures do not change the response already returned.

AWS Firehose treats responses other than 200 and 413 as retriable errors, including other 4xx responses. A 413 is a permanent failure, and the batch is not sent to the configured error bucket. Retries can produce duplicate deliveries; a successful response does not guarantee end-to-end deduplication. See the AWS request and response specification.

Feedback

Is this page helpful?