Skip to content

AWS Firehose

Collect AWS Firehose Metrics

Configuration

Install Func

We recommend enabling Guance Integrations - Extensions - Hosted Func: all prerequisites are installed automatically. Continue with script installation.

To deploy Func yourself, see Self-hosted Func.

Install Script

Note: Please prepare the required Amazon AK in advance (for simplicity, you can grant global read-only permissions ReadOnlyAccess)

Activate Script in DataFlux Func (Automata)

  1. Log in to the Guance console
  2. Click the 【Integration】 menu and select 【Cloud Account Management】
  3. Click 【Add Cloud Account】, select 【AWS】, and fill in the required information on the interface. If you have already configured the cloud account information before, ignore this step
  4. Click 【Test】, and if the test is successful, click 【Save】. If the test fails, please check if the relevant configuration information is correct and test again
  5. Click 【Cloud Account Management】, and you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page
  6. Click on the 【Integration】 button on the cloud account details page, find AWS Firehose under the Not Installed list, and click on the 【Install】 button to pop up the installation interface and install it.

Activate Script Manually

  1. Log in to the Func console, click on 【Script Market】, enter the Guance script market, and search for aws_firehose

  2. Click 【Install】, then enter the corresponding parameters: AWS AK ID, AK Secret, and account name.

  3. Click 【Deploy Startup Script】, the system will automatically create the Startup script set and automatically configure the corresponding startup script.

  4. After activation, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to execute it immediately without waiting for the scheduled time. After a while, you can view the execution task records and corresponding logs.

Verification

  1. Under [Management / Automatic Trigger Configuration], confirm that an automatic trigger configuration exists for the task, and check the task records and logs for errors.
  2. In Guance, check [Infrastructure / Custom] for asset information.
  3. In Guance, check [Metrics] for the corresponding monitoring data.

Metrics

After configuring Amazon CloudWatch, the default Measurement is as follows. You can collect more Metrics through configuration:

Amazon CloudWatch AWS Firehose Metrics Details

AWS Firehose

Delivery Stream Management

Metric Description
DescribeDeliveryStream.Latency DescribeDeliveryStream operation duration. Statistics: Minimum, Maximum, Average, Samples. Unit: milliseconds.
DescribeDeliveryStream.Requests DescribeDeliveryStream request count. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: count.
ListDeliveryStreams.Latency ListDeliveryStreams operation duration. Statistics: Minimum, Maximum, Average, Samples. Unit: milliseconds.
ListDeliveryStreams.Requests ListDeliveryStreams request count. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: count.

Data Ingestion

Metric Description
PutRecordBatch.Bytes Bytes written to the delivery stream through PutRecordBatch. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: bytes.
PutRecordBatch.Latency PutRecordBatch operation duration. Statistics: Minimum, Maximum, Average, Samples. Unit: milliseconds.

Delivery to Amazon Redshift

Metric Description
DeliveryToRedshift.Bytes The number of bytes copied to Amazon Redshift during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToRedshift.Records The number of records copied to Amazon Redshift during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToRedshift.Success The total number of successful Amazon Redshift COPY commands.

Delivery to Amazon S3 (Primary Destination)

Metric Description
DeliveryToS3.Bytes The number of bytes delivered to Amazon S3 during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Bytes
DeliveryToS3.DataFreshness The age of the oldest record in Amazon Data Firehose (from the time it entered Amazon Data Firehose to now). Any record older than this age has been delivered to the S3 bucket. Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
DeliveryToS3.Records The number of records delivered to Amazon S3 during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToS3.Success The total number of successful Amazon S3 put commands.
BackupToS3.Bytes The number of bytes delivered to Amazon S3 for backup during the specified time period. This metric is emitted by Amazon Data Firehose when backup is enabled (only if data transformation is also enabled). Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
BackupToS3.DataFreshness The age of the oldest record in Amazon Data Firehose (from the time it entered Amazon Data Firehose to now). Any record older than this age has been delivered to the Amazon S3 bucket for backup. This metric is emitted by Amazon Data Firehose when backup is enabled (only if data transformation is also enabled). Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
BackupToS3.Records The number of records delivered to Amazon S3 for backup during the specified time period. This metric is emitted by Amazon Data Firehose when backup is enabled (only if data transformation is also enabled). Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
BackupToS3.Success The total number of successful Amazon S3 backup put commands. This metric is emitted by Amazon Data Firehose when backup is enabled (only if data transformation is also enabled).

Delivery to Snowflake

Metric Description
DeliveryToSnowflake.Bytes The number of bytes delivered to Snowflake during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Bytes
DeliveryToSnowflake.DataFreshness The age of the oldest record in Firehose (from the time it entered Firehose to now). Any record older than this age has been delivered to Snowflake. Note that it may take a few seconds for data to be committed to Snowflake after a successful Firehose insert call. For the time required to commit data to Snowflake, see the DeliveryToSnowflake.DataCommitLatency metric. Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
DeliveryToSnowflake.DataCommitLatency The time required to commit data to Snowflake after a successful Firehose insert call. Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
DeliveryToSnowflake.Records The number of records delivered to Snowflake during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToSnowflake.Success The total number of successful insert calls to Snowflake.
DeliveryToS3.Bytes (Snowflake Backup Failure) The number of bytes delivered to Amazon S3 during the specified time period. This metric is available only when delivery to Snowflake fails and Firehose attempts to back up the failed data to S3. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Bytes
DeliveryToS3.Records (Snowflake Backup Failure) The number of records delivered to Amazon S3 during the specified time period. This metric is available only when delivery to Snowflake fails and Firehose attempts to back up the failed data to S3. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToS3.Success (Snowflake Backup Failure) The total number of successful Amazon S3 put commands. This metric is available only when delivery to Snowflake fails and Firehose attempts to back up the failed data to S3.
BackupToS3.DataFreshness (Snowflake Full Backup) The age of the oldest record in Firehose (from the time it entered Firehose to now). Any record older than this age has been backed up to the Amazon S3 bucket. This metric is available when the Firehose stream is configured to back up all data. Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
BackupToS3.Records (Snowflake Full Backup) The number of records delivered to Amazon S3 for backup during the specified time period. This metric is available when the Firehose stream is configured to back up all data. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
BackupToS3.Bytes (Snowflake Full Backup) The number of bytes delivered to Amazon S3 for backup during the specified time period. This metric is available when the Firehose stream is configured to back up all data. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
BackupToS3.Success (Snowflake Full Backup) The total number of successful Amazon S3 backup put commands. This metric is emitted by Firehose when the Firehose stream is configured to back up all data.

Delivery to Splunk

Metric Description
DeliveryToSplunk.Bytes The number of bytes delivered to Splunk during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Bytes
DeliveryToSplunk.DataAckLatency The approximate time required for Amazon Data Firehose to receive an acknowledgment from Splunk after sending data to Splunk. The trend of increase or decrease in this metric is more useful than the absolute approximation. A growing trend may indicate slower indexing and acknowledgment by the Splunk indexer. Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
DeliveryToSplunk.DataFreshness The age of the oldest record in Amazon Data Firehose (from the time it entered Amazon Data Firehose to now). Any record older than this age has been delivered to Splunk. Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
DeliveryToSplunk.Records The number of records delivered to Splunk during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToSplunk.Success The total number of successfully indexed records.
DeliveryToS3.Success (Splunk Backup) The total number of successful Amazon S3 put commands. This metric is emitted when backup to Amazon S3 is enabled.
BackupToS3.Bytes (Splunk Backup) The number of bytes delivered to Amazon S3 for backup during the specified time period. This metric is emitted by Amazon Data Firehose when the Firehose stream is configured to back up all documents. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count

Delivery to HTTP Endpoint

Metric Description
DeliveryToHttpEndpoint.Bytes The number of bytes successfully delivered to the HTTP endpoint. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Bytes
DeliveryToHttpEndpoint.Records The number of records successfully delivered to the HTTP endpoint. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToHttpEndpoint.DataFreshness The age of the oldest record in Amazon Data Firehose. Statistics: Minimum, Maximum, Average, Samples. Unit: Seconds
DeliveryToHttpEndpoint.Success The total number of all requests that successfully delivered data to the HTTP endpoint. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToHttpEndpoint.ProcessedBytes The number of bytes attempted to be processed, including retries.
DeliveryToHttpEndpoint.ProcessedRecords The number of records attempted to be processed, including retries.

Delivery to OpenSearch Service

Metric Description
DeliveryToAmazonOpenSearchService.Bytes The number of bytes indexed to OpenSearch Service during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Bytes
DeliveryToAmazonOpenSearchService.DataFreshness The age of the oldest record in Amazon Data Firehose (from the time it entered Amazon Data Firehose to now). Any record older than this age has been delivered to OpenSearch Service. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Seconds
DeliveryToAmazonOpenSearchService.Records The number of records indexed to OpenSearch Service during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToAmazonOpenSearchService.Success The total number of successfully indexed records.
DeliveryToAmazonOpenSearchService.AuthFailure Authentication/authorization error. Verify the OS/ES cluster policy and role permissions. 0 indicates no problem. 1 indicates authentication failure.
DeliveryToAmazonOpenSearchService.DeliveryRejected Delivery rejected error. Verify the OS/ES cluster policy and role permissions. 0 indicates no problem. 1 indicates delivery failure.

Amazon S3

Metric Description
DeliveryToS3.Bytes The number of bytes delivered to Amazon S3 during the specified time period. This metric is emitted by Amazon Data Firehose only when backup is enabled for all documents. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToS3.DataFreshness The age of the oldest record in Amazon Data Firehose (from the time it entered Amazon Data Firehose to now). Any record older than this age has been delivered to the S3 bucket. This metric is emitted by Amazon Data Firehose only when backup is enabled for all documents. Unit: Seconds
DeliveryToS3.Records The number of records delivered to Amazon S3 during the specified time period. This metric is emitted by Amazon Data Firehose only when backup is enabled for all documents. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToS3.Success The total number of successful Amazon S3 put commands. Amazon Data Firehose always emits this metric, whether backup is enabled for failed documents only or for all documents.

Delivery to OpenSearch Serverless

Metric Description
DeliveryToAmazonOpenSearchServerless.Bytes The number of bytes indexed to OpenSearch Serverless during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Bytes
DeliveryToAmazonOpenSearchServerless.DataFreshness The age of the oldest record in Amazon Data Firehose (from the time it entered Amazon Data Firehose to now). Any record older than this age has been delivered to OpenSearch Serverless. Unit: Seconds
DeliveryToAmazonOpenSearchServerless.Records The number of records indexed to OpenSearch Serverless during the specified time period. Statistics: Minimum, Maximum, Average, Sum, Samples. Unit: Count
DeliveryToAmazonOpenSearchServerless.Success The total number of successfully indexed records.
DeliveryToAmazonOpenSearchServerless.AuthFailure Authentication/authorization error. Verify the OS/ES cluster policy and role permissions. 0 indicates no problem. 1 indicates authentication failure.
DeliveryToAmazonOpenSearchServerless.DeliveryRejected Delivery rejected error. Verify the OS/ES cluster policy and role permissions. 0 indicates no problem. 1 indicates delivery failure.

Feedback

Is this page helpful?