Skip to content

Best Practices for Getting Started with the Pythond Collector


Pythond is a complete solution for periodically triggering user-defined Python collection scripts. This document uses "the number of users who log in each hour" as an example of a metric reported to the center.

Demo Overview

The business process is roughly as follows:
Collect data from the database (Python script) -> Pythond collector periodically triggers the script to report data (DataKit) -> View metrics from the center (web).

The database has a table named customers with the following fields:

  • name: name (string)
  • last_logined_time: login time (timestamp)

The table creation statement is as follows:

create table customers
(
  `id`                BIGINT(20)  not null AUTO_INCREMENT COMMENT '自增 ID',
  `last_logined_time` BIGINT(20)  not null DEFAULT 0      COMMENT '登录时间 (时间戳)',
  `name`              VARCHAR(48) not null DEFAULT ''     COMMENT '姓名',

  primary key(`id`),
  key idx_last_logined_time(last_logined_time)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Insert test data into the table above:

INSERT INTO customers (id, last_logined_time, name) VALUES (1, 1645600127, 'zhangsan');
INSERT INTO customers (id, last_logined_time, name) VALUES (2, 1645600127, 'lisi');
INSERT INTO customers (id, last_logined_time, name) VALUES (3, 1645600127, 'wangwu');

Use the following SQL statement to get "the number of users logging in each hour":

select count(1) from customers where last_logined_time>=(unix_timestamp()-3600);

Report the above data to the center as a metric.

The detailed steps to implement this scenario are described below.

Prerequisites

Python Environment

Currently in alpha stage, compatible with Python 3+ only. Tested versions:

  • 3.10.1

Python Dependencies

The following dependencies must be installed:

  • requests (used for network operations and reporting metrics)
  • pymysql (used for MySQL database operations; connects to the database to retrieve business data)

Install them as follows:

# Python 3
python3 -m pip install requests
python3 -m pip install pymysql

The commands above require pip. If you do not have pip, refer to the following methods (source: here):

# Linux/MacOS
python3 -m ensurepip --upgrade

# Windows
py -m ensurepip --upgrade

Installation and Deployment

1. Write a User-Defined Script

You need to subclass the DataKitFramework class and override its run method. The source code of the DataKitFramework class is in datakit_framework.py, located at datakit/python.d/core/datakit_framework.py.

For usage details, refer to the source file datakit/python.d/core/demo.py.

Based on the requirements above, this example uses the following Python script named hellopythond.py:

hellopythond.py
from datakit_framework import DataKitFramework
import pymysql
import re
import logging

class MysqlConn():
    def __init__(self, logger, config):
        self.logger = logger
        self.config = config
        self.re_errno = re.compile(r'^\((\d+),')

        try:
            self.conn = pymysql.Connect(**self.config)
            self.logger.info("pymysql.Connect() ok, {0}".format(id(self.conn)))
        except Exception as e:
            raise e

    def __del__(self):
        self.close()

    def close(self):
        if self.conn:
            self.logger.info("conn.close() {0}".format(id(self.conn)))
            self.conn.close()


    def execute_query(self, sql_str, sql_params=(), first=True):
        res_list = None
        cur = None
        try:
            cur = self.conn.cursor()
            cur.execute(sql_str, sql_params)
            res_list = cur.fetchall()
        except Exception as e:
            err = str(e)
            self.logger.error('execute_query: {0}'.format(err))
            if first:
                retry = self._deal_with_network_exception(err)
                if retry:
                    return self.execute_query(sql_str, sql_params, False)
        finally:
            if cur is not None:
                cur.close()
        return res_list

    def execute_write(self, sql_str, sql_params=(), first=True):
        cur = None
        n = None
        err = None
        try:
            cur = self.conn.cursor()
            n = cur.execute(sql_str, sql_params)
        except Exception as e:
            err = str(e)
            self.logger.error('execute_query: {0}'.format(err))
            if first:
                retry = self._deal_with_network_exception(err)
                if retry:
                    return self.execute_write(sql_str, sql_params, False)
        finally:
            if cur is not None:
                cur.close()
        return n, err

    def _deal_with_network_exception(self, stre):
        errno_str = self._get_errorno_str(stre)
        if errno_str != '2006' and errno_str != '2013' and errno_str != '0':
            return False
        try:
            self.conn.ping()
        except Exception as e:
            return False
        return True

    def _get_errorno_str(self, stre):
        searchObj = self.re_errno.search(stre)
        if searchObj:
            errno_str = searchObj.group(1)
        else:
            errno_str = '-1'
        return errno_str

    def _is_duplicated(self, stre):
        errno_str = self._get_errorno_str(stre)
        # 1062: duplicate field value, insert failed
        # 1169: duplicate field value, update failed
        if errno_str == "1062" or errno_str == "1169":
            return True
        return False

class HelloPythond(DataKitFramework):
    __name = 'HelloPythond'
    interval = 10 # Collect and report every 10 seconds. Adjust this based on actual business needs; this is only for demonstration.

    # if your datakit ip is 127.0.0.1 and port is 9529, you won't need use this,
    # just comment it.
    # def __init__(self, **kwargs):
    #     super().__init__(ip = '127.0.0.1', port = 9529)

    def run(self):
        config = {
            "host": "172.16.2.203",
            "port": 30080,
            "user": "root",
            "password": "Kx2ADer7",
            "db": "df_core",
            "autocommit": True,
            # "cursorclass": pymysql.cursors.DictCursor,
            "charset": "utf8mb4"
        }

        mysql_conn = MysqlConn(logging.getLogger(''), config)
        query_str = "select count(1) from customers where last_logined_time>=(unix_timestamp()-%s)"
        sql_params = ('3600')
        n = mysql_conn.execute_query(query_str, sql_params)

        data = [
        {
            "measurement": "hour_logined_customers_count", # Metric name.
            "tags": {
                "tag_name": "tag_value", # Custom tag. Fill in the value you want to use; this is just a placeholder.
            },
            "fields": {
                "count": n[0][0], # Metric: the number of users logged in each hour.
            },
        },
        ]

        in_data = {
            'M':data,
            'input': "pyfromgit"
        }

        return self.report(in_data) # you must call self.report here

2. Place the User-Defined Script in the Correct Location

Under the python.d directory in the DataKit installation directory, create a new folder named hellopythond. This folder name must be the same as the class name defined above, i.e., hellopythond.

Then put the hellopythond.py script written above into this folder. The final directory structure is as follows:

├── ...
├── datakit
└── python.d
    ├── core
    │   ├── datakit_framework.py
    │   └── demo.py
    └── hellopythond
        └── hellopythond.py

Note: The core folder above is the Pythond core folder. Do not modify it.

The above applies when the gitrepos feature is disabled. If gitrepos is enabled, the directory structure is as follows:

├── ...
├── datakit
├── python.d
├── gitrepos
│   └── yourproject
│       ├── conf.d
│       ├── pipeline
│       └── python.d
│           └── hellopythond
│               └── hellopythond.py

3. Enable the Pythond Configuration File

Copy the Pythond configuration file.
In the conf.d/pythond directory, copy pythond.conf.sample to pythond.conf, and then configure it as follows:

[[inputs.pythond]]

    # Python collector name
    name = 'some-python-inputs'  # required

    # Environment variables required for running the Python collector
    #envs = ['LD_LIBRARY_PATH=/path/to/lib:$LD_LIBRARY_PATH',]

    # Path to the Python collector executable (use an absolute path if possible)
    cmd = "python3" # required. python3 is recommended.

    # Relative path of user scripts (enter the folder name; after it is entered, modules and .py files in the next-level directory under this folder will be applied)
    dirs = ["hellopythond"] # Enter the folder name here, i.e., the class name

4. Restart DataKit

sudo datakit --restart

Results

If everything goes smoothly, you can see the metric chart in the center within about 1 minute.

14.pythond.png

Reference Documentation

<Official Manual: Manage Configuration Files with Git>

Feedback

Is this page helpful?