Best Practices for Collecting Metrics from Multiple Kubernetes Clusters¶

Overview¶

When connecting multiple Kubernetes clusters to a single workspace, you need to set the ENV_NAMESPACE environment variable to a non-empty string, and the value must be different for each cluster.

- name: ENV_NAMESPACE
  value: xxx

To collect metrics from multiple Kubernetes clusters in one workspace, Guance provides a method using global tags to distinguish them. When there is only one collection target in a cluster, such as collecting Kubernetes API Server metrics, the number of DataKit instances in the cluster may be greater than one. To avoid duplicate metric collection, DataKit enables the election feature. In this case, the way to distinguish clusters is to add ENV_GLOBAL_ELECTION_TAGS.

- name: ENV_GLOBAL_ELECTION_TAGS
  value: cluster_name_k8s=k8s-prod

For non-election metric collection, such as collecting metrics by adding annotations to Pods, Guance provides a method to add global tags in the ENV_GLOBAL_HOST_TAGS environment variable.

- name: ENV_GLOBAL_HOST_TAGS
  value: host=__datakit_hostname,host_ip=__datakit_ip,cluster_name_k8s=k8s-prod

All collected measurements will include the cluster_name_k8s tag with a value of k8s-prod, which is the principle for distinguishing clusters.

The following takes collecting Kubernetes API Server metrics from multiple clusters as an example to provide detailed instructions.

Prerequisites¶

Install Kubernetes 1.18+
Guance account

Steps¶

Step 1: Deploy DataKit in the Staging Cluster¶

1.1 Download datakit.yaml¶

Log in to the Guance console, click Integration > DataKit > Kubernetes, and download datakit.yaml.

1.2 Replace the Token¶

Log in to the Guance console, navigate to the Workspace Management module, copy the token from Basic Settings, and replace <your-token> in the value of the ENV_DATAWAY environment variable in the datakit.yaml file.

1.3 Add Global Tags¶

Append cluster_name_k8s=k8s-test to the value of the ENV_GLOBAL_HOST_TAGS environment variable in the datakit.yaml file.
Then add the environment variable ENV_GLOBAL_ELECTION_TAGS, so that the staging cluster is identified as k8s-test.
Set the ENV_NAMESPACE environment variable to k8s-test. This enables DataKit election, ensuring that only one DataKit in the workspace + this namespace collects Kubernetes API Server metrics.

- name: ENV_NAMESPACE
  value: k8s-test
- name: ENV_GLOBAL_ELECTION_TAGS
  value: cluster_name_k8s=k8s-test

1.4 Configure Kubernetes API Server Metric Collection¶

See the Kubernetes API Server Integration Documentation for details.

1.5 Deploy DataKit¶

Upload datakit.yaml to the Master node of the staging cluster and run the deployment command.

kubectl apply -f datakit.yaml

Step 2: Deploy DataKit in the Production Cluster¶

2.1 Modify datakit.yaml¶

Use the datakit.yaml from Step 1, change k8s-test to k8s-prod, so that the production cluster is identified as k8s-prod.
Also modify the url in api-server.conf.

2.2 Deploy DataKit¶

Upload datakit.yaml to the Master node of the production cluster and run the deployment command.

kubectl apply -f datakit.yaml

Step 3: View Preview¶

3.1 Metrics Preview¶

Log in to the Guance console, click Metrics, and search for prom_api_server. Under the cluster_name_k8s tag, you should see the names of the two clusters.

3.2 Create a Dashboard¶

Log in to the Guance console, click Scenarios > Create Dashboard, and select the Kubernetes API Server Monitoring View. The cluster name dropdown now allows you to distinguish between clusters.