AWS Firehose HTTP Endpoint

This guide provides detailed instructions on how to send Logging and Metrics data to AWS Kinesis Firehose through three methods and configure a custom HTTP Endpoint to receive this data.

Sending Kinesis Stream Data to Firehose¶

Creating a Data Stream¶

Create a data stream in Amazon Kinesis and enter a stream name. If a stream already exists, there is no need to create it again.

Creating a Firehose Stream¶

In Amazon Kinesis Data Firehose, select "Create Firehose delivery stream".

• Source: Amazon Kinesis Data Streams
• Destination: HTTP Endpoint

HTTP Endpoint Configuration Notes:

• Firehose Name: Customizable or use the default.
• Source Settings: Select the previously created data stream.
• Destination Settings:
  • HTTP Endpoint URL: Must be HTTPS protocol on port 443, e.g., https://your-endpoint/v1/input/firehose?category=logging.
  • Access Key: Fill in the user token, e.g., tkn_xxxxxx.
  • Parameters: Set source for subsequent Pipeline operations, e.g., source=my_source or service=my_service.

Subsequent Pipeline Processing¶

After logging data is sent to the observability platform, it will be displayed in plaintext. Pipeline scripts can be created for processing based on the preconfigured source.

Sending CloudWatch Log Data to Firehose¶

Creating a Log Group and Log Stream¶

Create a log group and log stream via the AWS CLI or console.

Creating a Firehose Stream and IAM Role¶

Use the AWS CLI to create a Firehose stream and IAM role, and assign necessary permissions. For detailed steps, refer to the AWS Official Documentation.

Configuring CloudWatch Subscription Filters¶

Select a log group from the log group list, click "Actions," and select "Subscription Filters." Then, choose "Create Kinesis subscription filter."

• Destination: Select the data stream created via the CLI.
• Grant Permissions: Select the role name created via the CLI.

Creating Firehose with Lambda¶

When creating a Firehose stream, select the following configuration:

• Source: Amazon Kinesis Data Streams
• Destination: HTTP Endpoint
• Firehose Stream Name: Default is acceptable
• Kinesis Data Stream: The stream created via the CLI
• Transform Records (Lambda): Create a function and select "Common Amazon Data Firehose processing." Search for Process records sent to an Firehose stream when creating the function and select the function name to create. Then, return to the Firehose creation page, refresh, and select the created function.

HTTP Endpoint Configuration:

• Destination Settings: HTTPS protocol on port 443, e.g., https://your-endpoint/v1/input/firehose?category=logging.
• Access Key: User token.
• Add Parameters: Define source and service.

Logs are compressed when sent from the log stream to the Kinesis stream, so decompression is required before sending to the HTTP endpoint. This is why Lambda is configured.

CloudWatch Metric to Firehose¶

Creating a Firehose Stream¶

When selecting a source, choose "Direct PUT."

When selecting a destination, choose "HTTP Endpoint."

HTTP Endpoint Configuration:

• Destination Settings: HTTPS protocol on port 443, e.g., https://your-endpoint/v1/input/firehose?category=metric.
• Access Key: User token.
• Add Parameters: Define source and service.

Creating a Metric Stream¶

In CloudWatch Metrics, select "Streams" and choose "Create Metric Stream."

Notes:

1. Destination: Select custom settings for Firehose.
2. Firehose Stream: Select the name with Direct PUT as the source just created.
3. (Optional) Service Role: Configure as needed.
4. (Required) Change Output Format: OpenTelemetry 1.0.
5. Metrics to Stream: Choose as desired.

After creation, metrics will be streamed to the Firehose destination.

Additional Reference Documents¶

• What is Amazon Data Firehose
• HTTP Endpoint Request and Response Data
• Using Metric Streams
• OpenTelemetry 1.0.0

This is an optimized and formatted version of the original Markdown document, including the addition of a table of contents, detailed steps, adjusted heading levels, and formatting enhancements. I hope this is helpful!