Data Forward¶
Release Notes
November 2, 2023:
- Support saving data to storage repositories such as OSS, S3 and OBS inside Guance.
- Data Forward: The navigation position has been adjusted to the Management module, and you can still be accessed through the secondary menu of the original Logs, Real User Monitoring and Application Performance Monitoring.
September 26, 2023: RUM and APM data are supported in Data forwarding rule query.
September 21, 2023: Add an entry for querying external storage forwarding rule data; enable/disable forwarding rules is supported.
September 7, 2023: The original Backup Logs has been officially renamed to Data Forward.
Guance offers the functionality to save logs, traces and user access data to the object storage of Guance and forward it to external storage. You are free to choose the storage objects and manage data forwarding flexibly.
On the data forwarding page, you can quickly query stored data (including Guance backup logs, AWS S3, Huawei Cloud OBS, Alibaba Cloud OSS, and Kafka message queues) by setting query time and data forwarding rules. You can also view the historical backup logs of Guance and SLS Query Logstore data.
Precondition¶
Only Guance Commercial Plan users can use the data forwarding function, Experience Plan users need to upgrade to the Commercial Plan.
Create¶
On the Data Forward page, click Forward rules > Create.
Note: After creating the data forwarding rule, the rule validation will be executed every 5 minutes.
Enter Rule Name¶
In the pop-up dialog box, enter a name to add a new rule.
| Field | Description |
|---|---|
| Rule Name | Limit the input to a maximum of 30 characters. |
| Include extend fields | If the option is selected, the entire log data that meets the criteria will be forwarded. The application performance and user access data will be forwarded by default, regardless of this option. If multiple data forwarding rules are created, the rule that includes extended fields will be prioritized. In other words, if different rules match the same data, the entire log data will be displayed according to the logic of synchronously including extended fields. |
Define Filter Conditions¶
Data Source: Logs, Trace and RUM.
Filtering Conditions: Support custom logical operations between conditions. You can choose All Conditions or Any Condition:
- All Conditions: Only log data that meets all filtering conditions will be saved to data forwarding.
- Any Condition: Log data that meets any of the filtering conditions will be saved to data forwarding.
Note: Not adding any filtering conditions means saving all log data. You can add multiple filtering conditions.
The operators for conditions are shown in the table below:
| Conditional Operators | Matching Types |
|---|---|
| in, not in | Exact Match: Support multiple values (comma-separated) |
| match, not match | Fuzzy Match: Support input regular expressions |
Select Archive Type¶
To provide a more comprehensive data forwarding and storage method, Guance supports the following five storage paths:
Note: All five types of archives are open to the entire site.
Guance¶
When the data forwarding storage object is set to Observation Cloud, the matched log data will be saved in the OSS, S3 and OBS object storage on Guance.
The minimum storage duration for log data under this rule is 180 days by default. Once the rule is created, it cannot be canceled, and fees will be charged daily during the storage period. You can go to Management > Settings > Change Data Storage Policy to modify the data forwarding storage policy.
AWS S3¶
I. Select the archival type as AWS S3, which means that the matched log data will be saved in the S3 object storage;
II. Choose the access type:
You need to configure third-party access to AWS resources using the external ID generated by Guance.
After configuring the Guance IAM role in AWS, fill in the archive information with your AWS account ID, AWS role name, region and bucket name.
Click Test Connection. If the above information meets the requirements, a successful connection test will be prompted.
If the test fails, you need to verify the following:
- Whether the external ID is invalid;
- Whether the account ID is correct;
- Whether the account role exists;
- Whether the bucket exists;
- Whether the region is consistent.
When this situation occurs, please note:
- If you click on Regenerate External ID, the previous ID will become invalid after 24 hours. Please go to the AWS console as soon as possible to replace it.
- Do not generate the external ID multiple times, please proceed with caution!
Click to download the AWS Resource Authorization Template, go to AWS to configure the Guance IAM policy.
Once the configuration is complete, fill in the account information, enter the AWS Account ID, AWS AK & SK, region and Bucket name.
Click Test Connection. If the above information meets the requirements, a successful connection test will be prompted.
If the test fails, please check:
- Is the Account ID correct?
- Do the AK/SK exist?
- Does the bucket exist?
- Are the regions consistent?
AWS provides cross-account authorization capabilities. You need to use the account ID specific to Guance and add the cross-account access authorization policy according to the configuration instructions.
Note: The account ID for overseas sites is different from that of the Chinese site, please make a distinction:
| Sites | ID |
|---|---|
| HongKong(China) | 588271335135 |
| Oregon | 521643107266 |
| Singapore | 521643107266 |
| Frankfurt | 521643107266 |
After the configuration is complete, select the region and enter the bucket name.
Click Test Connection. If the above information meets the requirements, a successful connection test will be prompted.
If the test fails, you need to confirm:
- Whether the account ID is correct;
- Whether the storage bucket exists;
- Whether the region is inconsistent.
III. Click Confirm to create successfully.
Note: If there are changes in the archive type information, a confirmation dialog will pop up to confirm the creation rule.
Huawei Cloud OBS¶
I. Select the Huawei Cloud OBS as the archive type, which means that the matched logs will be automatically forwarded to external OBS.
II. In order to configure the access authorization for Huawei Cloud resources, you need to use the exclusive Huawei Cloud account ID f000ee4d7327428da2f53a081e7109bd provided by Guance. Go here to add cross-account access authorization policies.
Note: The account ID for overseas sites is different from that of the Chinese site, please make a distinction:
| Sites | ID |
|---|---|
| HongKong(China) | 25507c35fe7e40aeba77f7309e94dd77 |
| Oregon | 25507c35fe7e40aeba77f7309e94dd77 |
| Singapore | 25507c35fe7e40aeba77f7309e94dd77 |
| Frankfurt | 25507c35fe7e40aeba77f7309e94dd77 |
III. Select Region.
IV. In the Bucket, enter the name of your bucket in Huawei Cloud.
V. Click Confirm to create successfully.
Alibaba Cloud OSS¶
I. Select the Alibaba Cloud OSS as the archive type, which means that the matched log data will be saved to Alibaba Cloud OSS object storage.
II. Choose access type:
You need to use the external ID generated by Guance to configure role authorization.
After configuring the authorized role in the Alibaba Cloud console, fill in the archive information including the Alibaba Cloud account ID, OSS role name, region, and bucket name.
Click Test Connection. If the above information meets the requirements, a successful connection test will be prompted.
If the test fails, you need to confirm:
- Whether the authorization is successful;
- Whether the account ID is correct;
- Whether the bucket exists;
- Whether the region is inconsistent.
When this situation occurs, please note:
- If you click on Regenerate External ID, the previous ID will become invalid after 24 hours. Please go to the Alibaba console as soon as possible to replace it.
- Do not generate the external ID multiple times, please proceed with caution!
You need to configure the OBS RAM policy in Alibaba Cloud. After configuring it, fill in the account information, including your Alibaba Cloud Account ID, Alibaba Cloud AK & SK, region and Bucket name.
Click Test Connection. If the above information meets the requirements, a successful connection test will be prompted.
If the test fails, you need to confirm:
- Whether the authorization is successful;
- Whether the account ID is correct;
- Whether the bucket exists;
- Whether the region is inconsistent.
Alibaba Cloud provides cross-account authorization capabilities. You need to use the account ID exclusive to Guance and add a cross-account access authorization policy according to the configuration instructions here.
Note: The account ID for overseas sites is different from that of the Chinese site, please make a distinction:
| Sites | ID |
|---|---|
| HongKong(China) | 1702505505232494 |
| Oregon | 218475797167922022 |
| Singapore | 218475797167922022 |
| Frankfurt | 218475797167922022 |
After the configuration is complete, select the region and enter the bucket name.
Click Test Connection. If the above information meets the requirements, a successful connection test will be prompted.
If the test fails, you need to confirm:
- Whether the account ID is correct;
- Whether the storage bucket exists;
- Whether the region is inconsistent.
III. Click Confirm to create successfully.
Note: If there are changes in the archive type information, a confirmation dialog will pop up to confirm the creation rule.
Kafka Message Queue¶
I. Server Address: Host:Port; multiple nodes are separated by commas.
II. Topic: the name of the topic.
III. Security protocol:
On the Kafka side, SASL can use either the PLAINTEXT or SSL protocol as the transport layer. Correspondingly, you can use SASL_PLAINTEXT or SASL_SSL security protocol. If using SASL_SSL security protocol, SSL certificate must be configured.
No need for any security verification, you can directly test the connection.
The authentication method is set to PLAIN by default, with the optional choices of SCRAM-SHA-256 and SCRAM-SHA-512.
Please provide the username/password for authentication on the Kafka side and then test the connection.
Here, [SSL certificate needs to be uploaded](https://kafka.apachecn.org/documentation.html#security_ssl.
The default authentication method is PLAIN, with the options of SCRAM-SHA-256 and SCRAM-SHA-512.
Enter the username/password for security authentication on the Kafka side and then test the connection.
Click Test Connection. If the above information meets the requirements, a successful connection test will be prompted.
If the test fails, you need to confirm:
- Whether the address is correct;
- Whether the message topic name is correct;
- Whether the SSL certificate is correct;
- Whether the username is correct;
- Whether the password is correct.
V. Click Confirm to create successfully.
View Forwarding Rules¶
After the rule is created, it will automatically enter the forwarding rule list:
You can search by entering the rule name.
You can choose to enable or disable the current rule.
Click the , edit button and button on the right side of the rule to perform the corresponding operation.
Note: Data that has been forwarded will not be deleted after the rule is deleted, but no new forwarding data will be generated.
You can select multiple rules for batch operations.
Forwarding Rule Explorer¶
Go back to the Data Forward page, which defaults to the Forward Rules tab. First, select a rule from the dropdown menu. You can customize the time range for the query, selecting multiple dates and defining a start and end time. The time will be accurate to the hour:
Note:
- You can enter keywords to search for matching data.
- The time control is initially empty, and you can select the hour after selecting a date. Clickable hour options will be listed based on the forwarding rule.
- If you select a future time range, the system will automatically correct it to the current date.
- Guance will retrieve file search matching data in batches based on the selected time. Each batch returns 50 pieces of data. If no data is found in the first query, or the returned data does not meet the requirement of 50 pieces per page, you can manually click Continue Query until the scan is complete.
- Since the retrieved data is in a disordered state, you can sort the data based on the listed time range. This action will not affect the data query results.
Under Index, you can view Guance historical backup logs and SLS Query Logstore data:
For specific operations of Explorer, see The Power of the Explorer.