Best Practices for Deploying DataKit with Rancher¶

Author: Liu Yujie

Introduction¶

Rancher is an open-source platform for managing Kubernetes clusters at the enterprise level. To simplify the deployment of DataKit in Kubernetes clusters, DataKit supports deployment through the Rancher App Marketplace. Additionally, DataKit provides the ability to manage collector configuration files using a Git repository.

Deploying DataKit with Rancher and managing collector configurations with Git is the best deployment approach for DataKit in Kubernetes clusters. By deploying DataKit as a DaemonSet, the Container collector is enabled by default. To achieve more flexible configuration of the Container collector, this article adopts a custom approach: the container.conf file is managed through a Git repository, DataKit is deployed via the Rancher App Marketplace, and finally the collected metrics are checked to verify that the collector configuration in the Git repository takes effect.

Prerequisites¶

• Install Kubernetes 1.18+
• Install Rancher with permissions to operate the Kubernetes cluster
• Install Gitlab
• Install Helm 3.0+

Procedure¶

Step 1: Create the dk-config Repository¶

Log in to Gitlab, and click Create blank project.

Enter datakit-conf as the Project name, and click Create Project.

Enter the dk-config project, create a new file named container.conf with the following content. The Container collector enables metric collection and disables stdout log collection for all images.

[inputs.container]
  docker_endpoint = "unix:///var/run/docker.sock"
  containerd_address = "/var/run/containerd/containerd.sock"

  enable_container_metric = true
  enable_k8s_metric = true
  enable_pod_metric = true

  ## Containers logs to include and exclude, default collect all containers. Globs accepted.
  container_include_log = []
  container_exclude_log = ["image:*"]

  exclude_pause_container = true

  ## Removes ANSI escape codes from text strings
  logging_remove_ansi_escape_codes = false

  kubernetes_url = "https://kubernetes.default:443"

  ## Authorization level:
  ##   bearer_token -> bearer_token_string -> TLS
  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## linux at:   /run/secrets/kubernetes.io/serviceaccount/token
  ## windows at: C:\var\run\secrets\kubernetes.io\serviceaccount\token
  bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  # bearer_token_string = "<your-token-string>"

  [inputs.container.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

Then click Clone → Clone with HTTP to get the Git URL: http://120.26.37.252/root/dk-config.git.

Step 2: Create a Namespace¶

For easier management, install DataKit in the datakit namespace.
Log in to Rancher, go to Cluster → Projects/Namespaces, and click Create Namespace.

Enter datakit as the Name, and click Create.

Step 3: Add the Chart Repository¶

Log in to Rancher, go to Cluster → App Marketplace → Chart Repositories, and click Create.
Enter datakit as the Name, enter https://pubrepo.guance.com/chartrepo/datakit as the URL, and click Create.

Step 4: Deploy DataKit¶

Log in to Rancher, go to Cluster → App Marketplace → Charts, select datakit. The icon labeled DataKit appears; click it.

Click Install.

Select the datakit namespace, and click Next.

Click Datakit Configuration, replace <your-token> with the token of Guance. Since we are using a custom Container collector configuration, uncheck container under Enable the default Inputs.

Click Git Management Configurations to set up the Git repository information, as shown in the figure below.

• Check Enable Use Git Management Configurations
• The Git Hub URL: fill in the Git URL of the dk-config repository: http://120.26.37.252/root/dk-config.git. Place the Git username root and password xxxxxx after http://.
• The Git Branch: fill in the branch main.

Then click Install.

Note: If you check Enable git SSH key, the configuration will be pulled from the Git repository via SSH.

Click Kube-State-Metrics, check Deployment KubeState Metrics Deployment to install the kube-state-metrics component. If not needed, you can leave it unchecked.

Click metrics-server, check Deployment kubeState Metrics Server Deployment to install the metrics-server component. If not needed, you can leave it unchecked.

Finally, click Install.

Step 5: Resource Limits¶

To limit the CPU and memory used by DataKit, you can restrict resource usage through Rancher.

Go to Cluster → Workloads → DaemonSets, click the ⋮ on the right side of the datakit row, select Edit Config, then click Resource Limits & Reservations, allocate resources appropriately based on server capacity, and click Save.

Step 6: Verify Metric Collection¶

Log in to Rancher, click the command line icon in the top-right corner, and execute the following commands to enter the DataKit pod.
Check that the gitrepos directory contains the container.conf file, which indicates that the Git repository configuration has been pulled successfully.

 kubectl get pods -n datakit
 kubectl exec -it datakit-qc58m -n datakit bash
 cd gitrepos/
 ls

Log in to Guance, go to Metrics, select the kubernetes measurement, and verify that metrics are present, confirming that metric collection is working correctly.

Step 7: Upgrade¶

When a new version of DataKit is available, you can see the upgradable version on the Rancher UI.

Go to Rancher Cluster → App Marketplace → Installed Apps. In the Upgrade column of the datakit row, you will see the version 1.4.6 available for upgrade. Click it to enter the upgrade interface.

Select version 1.4.6, and click Next.

Click Upgrade.

After the upgrade is complete, the DataKit version will be 1.4.6.