Best Practice for Log Data Backup to OSS (Function)¶

Prerequisites¶

Install DataKit¶

1. Click the [Integration] module, then [DataKit], and select the appropriate installation command based on your operating system and system type.

1. Copy the DataKit installation command and run it directly on the server to be monitored.

2. Installation directory: /usr/local/datakit/

3. Log directory: /var/log/datakit/

4. Main configuration file: /usr/local/datakit/conf.d/datakit.conf

5. Plugin configuration directory: /usr/local/datakit/conf.d/

Install Function¶

1. Click the [Integration] module, then [Function], download the installation package and execute the installation command.

1. After installation, use a browser to access http://服务器IP地址:8088 to perform the initialization interface.

1. Log in to the system using the default username/password: admin.

RAM Access Control¶

1. Log in to the RAM console https://ram.console.aliyun.com/users

2. Create a user: Identity Management → Users → Create User.

1. Save or download the CSV file containing the AccessKey ID and AccessKey Secret (will be used in the configuration file).

2. Grant user permissions (manage OSS permissions).

Script Development¶

Log Backup to OSS¶

1. Administration → Experimental Features → Enable the PIP tool module.

1. Install the Alibaba Cloud SDK dependency package (oss2).

1. Create a script set and add a script.

1. Write the code. You need to fill in the AccessKey ID, AccessKey Secret, Bucket Name, and Filename.

Reference documents

• <Simple Upload to OSS>

• <Function Development Guide - DataKit.Query>

import time
import json
import oss2

@DFF.API('Run DQL via DataKit')
def run_dql_via_datakit():
    datakit = DFF.SRC('datakit')
    # Use the time_range parameter to limit data to the last 10 minutes
    time_range = [
        int(time.time() - 600) * 1000,
        int(time.time()) * 1000,
    ]
    # DQL query statement
    status_code, result = datakit.query(dql='L::`apache`', time_range=time_range, raw=True)
    result = (json.dumps(result, indent=2))
    # Configure OSS information
    auth = oss2.Auth('AccessKey ID', 'AccessKey Secret')
    bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'Bucket Name')
    file = bucket.put_object('Filename.txt', result)

1. Log in to the OSS console to view the file.