Configure Data Forwarding for Deployment Plan¶

Introduction¶

This document demonstrates how to configure data forwarding for the Guance Deployment Plan.

Prerequisites¶

• Guance has been fully initialized.
• You have an account with the corresponding cloud provider that has permissions to configure users and storage buckets.

Configuration Steps¶

Step 1: Configure Storage Bucket and Account¶

Huawei Cloud¶

• Create a storage user
• Save the ak and sk for later configuration in the service.
• Create a parallel file system
• Configure file system access control

Alibaba Cloud¶

• Create a storage user
• Save the ak and sk for later configuration in the service.
• Create an OSS bucket
• Grant OSS bucket permissions to the created user.

AWS¶

• Create an S3 bucket
• Create a user permission policy

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Sid": "VisualEditor0",
              "Effect": "Allow",
              "Action": "s3:*",
              "Resource": [
                  "arn:aws:s3:::bucket-name",
                  "arn:aws:s3:::bucket-name/*"
              ]
          }
      ]
  }
  

  
• Create a storage user
• Bind the policy to the newly created user.
• Create and save the ak and sk for later configuration in the service.

MinIO¶

• Create a bucket
• Create a policy

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:*"
              ],
              "Resource": [
                  "arn:aws:s3:::bucketname",
                  "arn:aws:s3:::bucketname/*"
              ]
          }
      ]
  }
  

• Create a user and save the user ak and sk for later configuration in the service.

Step 2: Modify Service Configuration¶

Modify the configuration of application services like kodo, kodo-x, etc., to make the backup configuration effective.

• Modify the configmap resources named kodo, kodo-x, kodo-inner in the forethought-kodo namespace. Add the following content.

Public cloud storage bucket configuration

backup_log:
  ${store_type}:
    ak: "LTAI5tMxxxxxxxxFroj"
    sk: "6MpS1gxxxxxxxxxxxxxxxxUoH6"
    region: "cn-northwest-1"
    bucket: "guance-backuplog"
  guance:
    store_type: "obs"

1. Choose one from oss, s3, obs for ${store_type}, corresponding to Alibaba Cloud, AWS, and Huawei Cloud storage bucket services respectively.
2. guance.store_type determines which vendor's storage service is actually used. This value should be the same as ${store_type}.
3. When ${store_type} is s3, there is an additional configuration item: partition. For AWS in China, this value is aws-cn; for AWS outside China, it is aws.

1. Choose one from oss, s3, obs, minio for ${store_type}, corresponding to Alibaba Cloud, AWS, Huawei Cloud storage bucket, and MinIO storage services respectively.
2. guance.store_type determines which vendor's storage service is actually used. This value should be the same as ${store_type}.
3. endpoint: Configure the access address of the storage bucket service.
   • General Rule: Do not include the http:// or https:// protocol prefix when configuring. The system will use the HTTPS protocol by default when connecting.
   • MinIO: If your MinIO service does not support HTTPS and only supports HTTP, you must explicitly add the http:// prefix here (e.g., http://minio.example.com); otherwise, the connection will fail.
4. region: Required. If the storage service itself does not have the concept of a region (such as a privately deployed MinIO), you can fill in any non-empty string (e.g., "minio" in the example) to meet the SDK requirements.

5. use_path_style: Boolean type, default value is false. This configuration item controls the format of the access URL generated by the S3 SDK. When set to true, the format <ENDPOINT>/<BUCKET>/<KEY> (path style) is used; when set to false, the format <BUCKET>.<ENDPOINT>/<KEY> (virtual hosted style) is used.

6. After the modification is complete, restart the kodo, kodo-x, kodo-inner, and kodo-x-backuplog services.

Step 3: Data Forwarding Configuration¶

After the above steps are completed, you need to log in to Guance to configure data forwarding. After configuration, you can refer to Data Forwarding for verification.