Data Forwarding to AWS S3¶

1. Complete the rule name and filter condition settings;
2. Proceed to configure archiving.

Configure¶

1. Select AWS S3 as the archive type to save matched data to this object storage;

2. Select a storage format;

   • If you select Apache Iceberg, additional configuration is required;

3. Choose whether to encrypt the storage;

4. Select an access type;
5. Click OK to create the rule.

Apache Iceberg¶

Apache Iceberg is an open table format. When you select Apache Iceberg as the storage format, Guance writes data as Parquet files to your Iceberg table via the REST Catalog. The data can be directly read by data lake engines such as Spark, Trino, and Flink, and supports table capabilities like Schema evolution and snapshots.

1. Select Apache Iceberg as the storage format (❗️after selection, the "Encrypt Storage" and "Data Wait Time" configuration items are no longer displayed);

2. In Iceberg Catalog, select the Catalog Provider, and fill in the Catalog URI and Warehouse (both required). The value formats differ by Provider:

   Catalog Provider	Catalog URI	Warehouse Value
   Generic REST Catalog	REST Catalog address, e.g., https://catalog.example.com. Do not manually append /v1	Warehouse storage address, e.g., s3://warehouse/observability
   AWS S3 Tables	https://s3tables.<region>.amazonaws.com/iceberg	Full Table Bucket ARN, e.g., arn:aws:s3tables:<region>:<account-id>:bucket/<bucket-name> (ARN prefix for China regions is arn:aws-cn)
   AWS Glue	https://glue.<region>.amazonaws.com/iceberg	12-digit AWS main account ID (not IAM user ID), e.g., 294654068288

   • If you select Generic REST Catalog, you can optionally fill in the Endpoint: To forward data to a private cloud or custom S3 endpoint, fill in the endpoint; for public cloud scenarios, it is recommended to leave this blank, and the system will automatically determine whether to use internal network forwarding. If an endpoint is filled, data forwarding will be billed at public network rates.

3. In Target Table, enter the Namespace (single-level namespace, cannot contain .) and Table (table name); if the table does not exist, it will be created automatically, and data will be partitioned by hour(date);

4. Select Catalog Authentication and fill in the corresponding credentials:

   • Generic REST Catalog supports the following authentication methods:

     • No Authentication: No credentials required;
     • Basic Auth: Fill in the Credential (username:password);
     • Bearer Token: Fill in the Token;
     • OAuth2: Fill in the Credential (client-id:client-secret), optionally fill in the OAuth2 Server URI (if not provided, the Catalog's default address is used);

   • AWS S3 Tables / AWS Glue: Uses SigV4 by default; no additional configuration is needed; the AWS access credentials configured above are reused automatically.

5. Complete the authorization and archive information configuration according to the access type instructions below;

6. Click Test Connection to verify connectivity between the storage bucket and the Catalog. After the test passes, click OK to save the current rule.

Access Type¶

Role Authorization¶

1. Use the external ID generated by Guance by default to configure third-party access to AWS resources;

2. After configuring the Guance IAM role in AWS, fill in the archive information: AWS account ID, AWS role name, region, and bucket name;

3. Enter a storage path to distinguish and locate the specific data forwarding location later;

4. Click Test Connection. If the information meets the specifications, the test connection succeeds. Click OK to save the current rule.

Storage Path Naming Convention¶

1. Create single or multi-level folders; a slash (/) indicates a multi-level folder.
2. Folder names cannot start or end with a slash (/).
3. Cannot contain two or more consecutive slashes (/).
4. Applies to all access types.

Access Keys¶

1. Click to download the AWS resource authorization template, then configure the Guance IAM policy in AWS;
2. After configuration, fill in the account information: AWS Access Key & Secret Key, region, and bucket name;
3. Enter a storage path to distinguish and locate the specific data forwarding location later;
4. Click Test Connection. If the information meets the specifications, the test connection succeeds.

Account Authorization¶

1. AWS provides cross-account authorization. Use the Guance-specific account ID and add a cross-account access authorization policy according to the configuration instructions;

2. After configuration, select the region and enter the bucket name;

3. Enter a storage path to distinguish and locate the specific data forwarding location later;

4. Click Test Connection. If the information meets the specifications, the test connection succeeds. Click OK to save the current rule.

Data Wait Time¶

Data is first written to local temporary files. When the file size reaches 256 MB or the wait time reaches the set value, the system automatically compresses and uploads the data. You can adjust the wait time as needed: 15 minutes, 30 minutes, or 1 hour.

• Shorter times reduce downstream data latency but generate more files
• Longer times reduce the number of files but increase downstream data latency

Next Steps¶

Continue to configure the data access permissions for the current forwarding rule.