Best Practices for Backing Up Log Data to OSS (Function)¶

Prerequisites¶

Install Datakit¶

1. Click the [Integration] module, [Datakit], and select the appropriate installation command based on your operating system and system type.

1. Copy the Datakit installation command and run it directly on the server that needs to be monitored.

2. Installation directory: /usr/local/datakit/

3. Log directory: /var/log/datakit/
4. Main configuration file: /usr/local/datakit/conf.d/datakit.conf
5. Plugin configuration directory: /usr/local/datakit/conf.d/

Install Function¶

1. Click the [Integration] module, [Funciton], download the installation package, and execute the installation command.

1. After installation, access the initialization interface via a browser at http://Server IP Address:8088.

1. Log in to the system using the default username/password admin.

RAM Access Control¶

1. Log in to the RAM console https://ram.console.aliyun.com/users

2. Create a new user: User Management - Users - Create User

1. Save or download the CSV file containing the AccessKey ID and AccessKey Secret (this will be used in the configuration).

2. Authorize the user (grant permissions to manage Object Storage Service (OSS)).

Script Development¶

Log Backup to OSS¶

1. Management - Experimental Features - Enable PIP Tool Module

1. Install Alibaba Cloud SDK dependency package (oss2)

1. Create a new script set and add a script

1. Write the code, filling in the AccessKey ID, AccessKey Secret, Bucket Name, and Filename.

Refer to the documentation:

• <Simple Upload to OSS>
• <Function Development Guide - DataKit.Query>

import time
import json
import oss2

@DFF.API('Run DQL via DataKit')
def run_dql_via_datakit():
    datakit = DFF.SRC('datakit')
    # Use the time_range parameter to limit data from the last 10 minutes
    time_range = [
        int(time.time() - 600) * 1000,
        int(time.time()) * 1000,
    ]
    # DQL query statement
    status_code, result = datakit.query(dql='L::`apache`', time_range=time_range, raw=True)
    result = json.dumps(result, indent=2)
    # Configure OSS information
    auth = oss2.Auth('AccessKey ID', 'AccessKey Secret')
    bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'Bucket Name')
    file = bucket.put_object('Filename.txt', result)

1. Log in to the OSS console to view the file