Collector Configuration¶

The collector configuration in DataKit is in the Toml format, and all collector configurations are in the conf.d directory:

• Linux/Mac：/usr/local/datakit/conf.d/
• Windows：C:\Program Files\datakit\conf.d\

Each collection is categorized and located in the lower subdirectory of conf.d. You can refer to the specific collector configuration instructions to find the corresponding subdirectory.

A typical configuration collector file has the following structure:

[[inputs.some_name]] # The line is required to indicate which collector configuration this toml file is
  key = value
  ...

[[inputs.some_name.other_options]] # The line is optional, and some collectors are configured with this line, while others are not
  key = value
  ...

How to Modify Collector Configuration¶

At present, some collectors can be turned on without configuration, while others need to edit the configuration manually.

Enable Multiple Collections with the Same Collector¶

Taking MySQL as an example, if you want to configure multiple different MySQL collections, there are two ways:

1. Add a new conf file, such as mysql-2.conf, which can be placed in the same directory as the existing mysql.conf.
2. In the existing mysql.conf, add a paragraph like this:

# The first MySQL collection
[[inputs.mysql]]
  host = "localhost"
  user = "datakit"
  pass = "<PASS>"
  port = 3306

  interval = "10s"

  [inputs.mysql.log]
    files = ["/var/log/mysql/*.log"]

  [inputs.mysql.tags]

    # Omit other configuration items...

#-----------------------------------------
# Another MySQL collection
#-----------------------------------------
[[inputs.mysql]]
  host = "localhost"
  user = "datakit"
  pass = "<PASS>"
  port = 3306

  interval = "10s"

  [inputs.mysql.log]
    files = ["/var/log/mysql/*.log"]

  [inputs.mysql.tags]

    # Omit other configuration items...

#-----------------------------------------
# Continue to add another one below
#-----------------------------------------
[[inputs.mysql]]
  ...

The second method is probably simpler to manage, which manages all collectors with the same name with the same conf, and the first method may lead to confusion in the configuration directory.

To sum up, the structure of the second multi-acquisition configuration is as follows:

[[inputs.some-name]]
   ...
[[inputs.some-name]]
   ...
[[inputs.some-name]]
   ...

This is actually a Toml array structure, the structure is suitable for multiple configurations of all collectors.

Single Instance Collector¶

Some collectors only allow a single instance to run, and even if multiple copies are configured, only a single instance will run. These single instance collectors are listed as follows:

Close the Specific Collector¶

Sometimes, we want to temporarily shut down a collector, and there are two ways:

1. Rename the corresponding collector conf, such as mysql.conf to mysql.conf.bak. Just make sure the file suffix is not conf
2. In conf, comment out the corresponding collection configuration, such as:

# Comment out the first MySQL collection
#[[inputs.mysql]]
#  host = "localhost"
#  user = "datakit"
#  pass = "<PASS>"
#  port = 3306
#  
#  interval = "10s"
#  
#  [inputs.mysql.log]
#    files = ["/var/log/mysql/*.log"]
#  
#  [inputs.mysql.tags]
#  
#    # Omit other configuration items...
#

# Keep this MySQL collection
[[inputs.mysql]]
  host = "localhost"
  user = "datakit"
  pass = "<PASS>"
  port = 3306

  interval = "10s"

  [inputs.mysql.log]
    files = ["/var/log/mysql/*.log"]

  [inputs.mysql.tags]

    # Omit other configuration items...

In contrast, the first approach is more crude and simple, and the second one needs to be carefully modified, which may lead to Toml configuration errors.

Regular Expressions in Collector Configuration¶

When editing the collector configuration, some regular expressions may need to be configured.

Since DataKit is mostly developed using Golang, the regular wild match used in the configuration section is also implemented using Golang's own regular implementation. As there are some differences in the regular systems of different languages, it is difficult to write the configuration correctly at one time.

We recommend an online tool to debug our regular wildcard. As shown in the following figure:

In addition, since Toml is used in the configuration of DataKit, it is recommended that you fill in the regular form by using '''Here is a specific regular expression''' (that is, three English single quotation marks are used on both sides of the regular form), so as to avoid some complicated escapes.

Collector Turned on by Default¶

After DataKit is installed, a batch of collectors will be turned on by default without manual opening. These collectors are generally related to the host, and the list is as follows:

Password Encoding¶

In configuring connection strings, special characters in passwords, such as @#*, need to be encoded to ensure the link string is correctly interpreted. Below is a list of encodings for these special characters:

Note: Not all special characters (like ~_-.) require encoding, but they are listed here for reference.

Assuming we have the following Git connection string:

http://username:pa55w#rd@github.com/path/to/repository.git 

We need to convert the # in the password to its URL-encoded form %23:

http://username:pa55w%23rd@github.com/path/to/repository.git 

For More Readings¶

• DataKit K8s Installation and Configuration
• Manage collector configuration through Git