DataKit Election¶

When there is only one target for data collection (e.g., Kubernetes) in a cluster, but multiple DataKits are deployed in bulk with identical configurations and all of them have collection enabled for that central target, the election feature can be activated in DataKit to avoid duplicate data collection.

Currently, DataKit only supports a "self-election" mode. Under the same election namespace, one DataKit instance will be elected as the leader and take charge of all data collection tasks, while the other instances remain on standby.

The advantages and disadvantages of this mode are as follows:

• Advantages: Simple configuration, with no need to deploy additional components.
• Disadvantages: The elected DataKit leader bears a higher load, as all collection tasks are concentrated on this single instance, which may lead to a significant increase in its system resource usage.

DataKit Self Election¶

Election Configuration¶

[election]
  # Open the election
  enable = false

  # Set the namespace of the election (default)
  namespace = "default"

  # tag that allows election space to be appended to data
  enable_namespace_tag = false

  ## election.tags: Election-related global tags
  [election.tags]
    #  project = "my-project"
    #  cluster = "my-cluster"

Viewing Election Status¶

After the election is configured, you can check the current election status of DataKit by viewing the monitor. In the Basic Info section, there will be a line like this:

Elected default::success|MacBook-Pro.local(elected: 4m40.554909s)

Here's what each part means:

• default indicates the election-namespace in which the current DataKit participates in the election. A workspace can have multiple election-namespaces dedicated to elections.
• success indicates that the current DataKit has election enabled and has been chosen as the leader.
• MacBook-Pro.local shows the hostname of the DataKit that was elected in the current namespace. If this hostname is the same as the current DataKit, the duration for which it has been the leader will be displayed afterward (elected: 4m40.554909s) Version-1.5.8

If it is displayed as follows, it means that the current DataKit was not elected, but it will show which one was elected:

Elected default::defeat|host-abc

Here's the breakdown:

• default indicates the namespace in which the current DataKit is participating in the election, as explained above.

• defeat indicates that the current DataKit has election enabled but was not successful. In addition to this, there are several other possible statuses:

  • disabled: The election feature is not enabled.
  • success: The election was successfully completed.
  • banned: The election feature is enabled, but it is not on the whitelist allowed for election Version-1.35.0

• host-abc shows the hostname of the DataKit that was elected in the current namespace.

Election Principle¶

Take MySQL as an example. In the same cluster (such as k8s cluster), suppose there are 10 DataKits, 2 MySQL instances, and all DataKits have elections turned on (in DaemonSet mode, the configuration of each DataKit is the same) and MySQL collector:

• Once a DataKit is elected, it collects data from every configured MySQL target; the same winner-takes-all rule applies to other election-capable collectors. Other DataKit instances remain on standby.
• Guance uses heartbeats to determine whether the current leader is healthy. If it becomes unhealthy, one of the standby DataKit instances replaces it.
• A DataKit instance with election disabled still collects from its configured MySQL targets without election constraints, even if it is outside the current cluster.
• Elections are scoped by workspace + election-namespace. Only one DataKit can be the leader within each such scope.
  • With regard to workspaces, in datakit.conf, it is represented by the token URL parameter in the DataWay address string, and each workspace has its corresponding token.
  • The election namespace is represented by the namespace setting in datakit.conf. One workspace can contain multiple namespaces used by different DataKit groups.

Election Class Collector's Global Tag Settings¶

[election]
  [election.tags]
    # project = "my-project"
    # cluster = "my-cluster"

[global_host_tags]
  ip         = "__datakit_ip"
  host       = "__datakit_hostname"

Election Whitelist¶

Version-1.35.0

[election]

  # election whitelist. If list empty, all host/node are allowed for election.
  node_whitelist = ["host-name-1", "host-name-2", "..."]

Collection List Supporting Election¶

The list of collectors currently supporting elections is as follows:

• Apache
• ElasticSearch
• GitLab
• InfluxDB
• Container
• MongoDB
• MySQL
• NSQ
• Nginx
• PostgreSQL
• Prom
• RabbitMQ
• Redis
• Solr
• TDengine

In fact, there are more collectors that support elections, and this information may not be up-to-date. Please refer to the specific documentation of the collector for the most accurate information.

FAQ¶

host Field Problem¶

For objects collected by collectors participating in elections, such as MySQL, because the DataKit collecting their data may change (election rotation occurs), by default, the data collected by such collectors will not take the tag host to avoid timeline growth. We recommend adding an additional tags field to the MySQL collector configuration:

[inputs..tags]
  host = "real-mysql-instance-name"

This way, the host field configured in tags will continue to be used when the DataKit has an election rotation.