Best Practices for Deploying DataKit on Rancher¶

Author: Liu Yujie

Introduction¶

Rancher is an open-source platform for managing Kubernetes clusters at the enterprise level. To simplify the deployment of DataKit in Kubernetes clusters, DataKit supports deployment via the Rancher application marketplace. Additionally, DataKit provides a feature to manage collector configuration files using Git repositories.

Deploying DataKit on Rancher and managing collector configurations with Git is one of the best practices for deploying DataKit in Kubernetes clusters. By deploying DataKit using DaemonSet, the Container collector is enabled by default. For more flexible configuration of the Container collector, this guide will use a custom approach to enable the Container collector, specifically by managing the container.conf file through a Git repository, then deploying DataKit via the Rancher application marketplace, and finally verifying the collected metrics to ensure that the collector configuration from the Git repository has taken effect.

Prerequisites¶

• Install Kubernetes 1.18+
• Install Rancher, with permissions to operate Kubernetes clusters
• Install GitLab
• Install Helm 3.0+

Procedure¶

Step 1: Create the dk-config Repository¶

Log in to GitLab, click on Create blank project

Enter datakit-conf as the Project name, and click Create Project.

Navigate into the dk-config project, create a new container.conf file with the following content. This configuration enables metric collection for containers and disables stdout log collection for all images.

[inputs.container]
  docker_endpoint = "unix:///var/run/docker.sock"
  containerd_address = "/var/run/containerd/containerd.sock"

  enable_container_metric = true
  enable_k8s_metric = true
  enable_pod_metric = true

  ## Containers logs to include and exclude, default collect all containers. Globs accepted.
  container_include_log = []
  container_exclude_log = ["image:*"]

  exclude_pause_container = true

  ## Removes ANSI escape codes from text strings
  logging_remove_ansi_escape_codes = false

  kubernetes_url = "https://kubernetes.default:443"

  ## Authorization level:
  ##   bearer_token -> bearer_token_string -> TLS
  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## linux at:   /run/secrets/kubernetes.io/serviceaccount/token
  ## windows at: C:\var\run\secrets\kubernetes.io\serviceaccount\token
  bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  # bearer_token_string = "<your-token-string>"

  [inputs.container.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

Then click Clone - Clone with HTTP, and obtain the git address http://120.26.37.252/root/dk-config.git.

Step 2: Create Namespace¶

For easier management, install DataKit in the datakit namespace.
Log in to Rancher - Cluster - Projects/Namespaces, and click Create Namespace.

Enter datakit as the name, and click Create.

Step 3: Add Chart Repository¶

Log in to Rancher - Cluster - Marketplace - Chart Repositories, and click Create. Enter datakit as the name, and input https://pubrepo.guance.com/chartrepo/datakit as the URL, then click Create.

Step 4: Deploy DataKit¶

Log in to Rancher - Cluster - Marketplace - Charts, select datakit, and you will see an icon labeled DataKit. Click to enter.

Click Install.

Select the datakit namespace, and click Next.

Click DataKit Configuration, replace <your-token> with Guance's token. Since we are using a custom container collector configuration, remove container from Enable the default Inputs.

Click Git Management Configurations to set up the Git repository information as shown below.

• Select Enable Use Git Management Configurations
• Enter the Git URL for dk-config http://120.26.37.252/root/dk-config.git, and append the Git username root and password xxxxxx after http://.
• Enter main for The Git Branch.

Then click Install.

Note: If you select Enable git SSH key, it means pulling the Git repository configuration via SSH.

Click Kube-State-Metrics, and select Deployment KubeState Metrics Deployment to install the kube-state-metrics component; if not needed, leave it unchecked.

Click metrics-server, and select Deployment kubeState Metrics Server Deployment to install the metrics-server component; if not needed, leave it unchecked.

Finally, click Install.

Step 5: Resource Limits¶

To limit the CPU and memory usage of DataKit, you can restrict DataKit's resource consumption through Rancher.

Navigate to Cluster - Workloads - DaemonSets, click on the right side of the datakit row, select Edit Configuration, and click Resource Limits and Reservations. Allocate resources reasonably based on your server's capacity, then click Save.

Step 6: Verify Metric Collection¶

Log in to Rancher, click on the command line icon in the upper-right corner, and execute the following commands to enter datakit.
Check if there is a container.conf file under the gitrepos directory, which indicates that the Git repository configuration has been successfully pulled.

 kubectl get pods -n datakit
 kubectl exec -it datakit-qc58m -n datakit bash
 cd gitrepos/
 ls

Log in to Guance - Metrics, select the kubernetes Measurement, and verify that the metrics have data, indicating successful metric collection.

Step 7: Upgrade¶

When a new version of DataKit is available, the Rancher UI will show the upgradeable version.

Navigate to Rancher - Cluster - Marketplace - Installed Apps, and in the Upgrade column of the datakit row, you will see a pending upgrade version 1.4.6. Click to enter the upgrade interface.

Select version 1.4.6, and click Next.

Click Upgrade.

After the upgrade is complete, you will see that the DataKit version is now 1.4.6.