Faq

faq.md# FAQ

1 Problems Occurred During Initial Installation, Need to Clean Up and Reinstall - What to Do?¶

Note: The following cleanup steps apply only to scenarios where problems occurred during the initial installation and you need to remove everything and reinstall. Please confirm carefully before executing them!

If an installation problem occurs and you need to remove everything and reinstall, you must clean up the following three areas before you can reinstall Guance from Launcher:

1.1 Clean Up Installed Guance Application Services¶

To clean up the various Guance application services installed in Kubernetes, on the operations host, enter the Launcher container and run the cleanup script bundled with Launcher:

kubectl exec -it launcher-xxxxxxxx-xxx -n launcher /bin/bash

1.2 Clean Up Databases Automatically Created in MySQL¶

You can enter the Launcher container, which has a mysql client tool bundled. Use the following command to connect to the Guance MySQL instance:

mysql -h <mysql 实例 host> -u root -P <mysql 端口> -p  

drop database df_core;
drop user df_core;
drop database df_message_desk;
drop user df_message_desk;
drop database df_func;
drop user df_func;
drop database df_dialtesting;
drop user df_dialtesting;

1.3 Clean Up Users Automatically Created in InfluxDB¶

If the time-series engine uses InfluxDB, use the influx client to connect to InfluxDB and run the following two user cleanup commands:

drop user user_wr;
drop user user_ro;

2 Deployment Considerations¶

2.1 After Deployment, Can I Manually Modify the Kubernetes Resources Automatically Generated by the Installer?¶

Manual modification is not allowed, because after the next new version release, upgrading with Launcher will regenerate resources such as Deployment, Service, Ingress based on the configuration information filled in during installation (except for Configmap; the information in Configmap configuration items can be manually modified at will, but arbitrary modification may cause abnormal program operation).

3 Certificate Renewal for Rancher Server Running in a Standalone Container¶

3.1 What to Do If the Certificate Has Not Expired¶

Rancher Server can run normally. After upgrading to Rancher v2.0.14+, v2.1.9+, or v2.2.2+, the certificate validity period is checked automatically. If the certificate is about to expire, a new certificate will be generated automatically. Therefore, for a Rancher Server running in a standalone container, you only need to upgrade the Rancher version to one that supports automatic SSL certificate renewal before the certificate expires; no other action is required.

3.2 What to Do If the Certificate Has Expired¶

Rancher Server cannot run normally. Even after upgrading to Rancher v2.0.14+, v2.1.9+, or v2.2.2+, certificate errors may still be reported. If this occurs, handle it with the following operations:

1. Normally upgrade the Rancher version to v2.0.14+, v2.1.9+, or v2.2.2+;

2. Run the following commands:

3. For versions 2.0 or 2.1

docker exec -ti <rancher_server_id>mv /var/lib/rancher/management-state/certs/bundle.json /var/lib/rancher/management-state/certs/bundle.json-bak

• For 2.2+

docker exec -ti <rancher_server_id>mv /var/lib/rancher/management-state/tls/localhost.crt /var/lib/rancher/management-state/tls/localhost.crt-bak

• For 2.3+

 docker exec -ti <rancher_server_id>mv /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/tlsbak

 # Run it twice: the first time to request the certificate, and the second time to load the certificate and start
 docker restart <rancher_server_id>

• For 2.4+

  a. Exec into the Rancher Server

kubectl --insecure-skip-tls-verify -n kube-system delete secrets k3s-serving
kubectl --insecure-skip-tls-verify delete secret serving-cert -n cattle-system
rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json

  b. Restart Rancher Server

docker restart <rancher_server_id>

  c. Run the following command to refresh the parameters

curl --insecure -sfL https://server-url/v3

1. Restart the Rancher Server container

docker restart <rancher_server_id>

4 Handling Inability to Manage k8s Clusters Caused by Expired Rancher Server Certificate¶

If the cluster certificate has expired, the certificate cannot be rotated even after upgrading to Rancher v2.0.14, v2.1.9, or higher versions. Rancher updates certificates through the Agent; if the certificate has expired, it cannot connect to the Agent.

4.1 Solution¶

You can manually set the node time and adjust it backward a bit. Because the Agent only communicates with the K8S master and Rancher Server, if the Rancher Server certificate has not expired, you only need to adjust the K8S master node time. Adjustment commands:

# Disable NTP synchronization; otherwise the time will be updated automatically
timedatectl set-ntp false
# Modify the node time
timedatectl set-time '2019-01-01 00:00:00'

Then upgrade the Rancher Server, and after the certificate rotation is complete, restore time synchronization.

timedatectl set-ntp true

openssl x509 -in /etc/kubernetes/ssl/kube-apiserver.pem -noout -dates

5 Why Is the DataWay I Created Not Visible in the Frontend¶

5.1 Common Cause Analysis¶

• The Dataway service is not running normally after being deployed to the server.
• The Dataway service configuration file is incorrect; the correct listening address and Workspace token information are not configured.
• The Dataway service runtime configuration is incorrect. This can be located by checking the Dataway logs.
• The server where Dataway is deployed cannot communicate with the kodo service. (This also includes the case where the correct resolution for the df-kodo service has not been added to the hosts file on the Dataway server.)
• The kodo service is abnormal. This can be confirmed by checking the kodo service logs.
• The df-kodo ingress service is not correctly configured. Specifically, you are unable to access http|https://df-kodo.<xxxx>:<port>.

6 Why Can't I Use Synthetic Monitoring?¶

6.1 Cause Analysis¶

• The deployed Guance application is in an offline environment, and the network of the physical nodes cannot access the internet. (This is relatively common.)
• The network of the self-hosted probe nodes is abnormal.
• The network of the regional provider is abnormal.
• The Synthetic Test was created incorrectly.

7 Common Deployment Issues and Solutions¶

7.1 describe pods Reports the unbound immediate PersistentVolumeClaims Error¶

• Check the PVCs

NAMESPACE    NAME                                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
default      opensearch-single-opensearch-single-0    Bound     pvc-0da2cb6f-1cb9-4630-b0ab-512ce57743a8   16Gi       RWO            openebs-hostpath   19d
launcher     persistent-data                          Pending                                                                        df-nfs-storage     6m3s
middleware   data-es-cluster-0                        Bound     pvc-36e48f5a-37b3-4c28-ad14-059265ee3009   50Gi       RWO            openebs-hostpath   18d

You will find that the status of persistent-data is Pending.

• Check the status of the nfs-subdir-external-provisioner container

kubectl get pods -n kube-system  | grep nfs-subdir-external-provisioner
nfs-provisioner-nfs-subdir-external-provisioner-58b7cdf6f5dr5vr   0/1     ContainerCreating   0             7h7m

• Check the information of nfs-provisioner-nfs-subdir-external-provisioner-58b7cdf6f5dr5vr

kubectl describe  -n kube-system pods nfs-provisioner-nfs-subdir-external-provisioner-58b7cdf6f5dr5vr
....
  Type     Reason       Age                     From     Message
  ----     ------       ----                    ----     -------
  Warning  FailedMount  30m (x49 over 6h53m)    kubelet  Unable to attach or mount volumes: unmounted volumes=[nfs-subdir-external-provisioner-root], unattached volumes=[kube-api-access-5p4qn nfs-subdir-external-provisioner-root]: timed out waiting for the condition
  Warning  FailedMount  5m27s (x136 over 7h4m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[nfs-subdir-external-provisioner-root], unattached volumes=[nfs-subdir-external-provisioner-root kube-api-access-5p4qn]: timed out waiting for the condition
  Warning  FailedMount  74s (x217 over 7h6m)    kubelet  MountVolume.SetUp failed for volume "nfs-subdir-external-provisioner-root" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 10.200.14.112:/nfsdata /var/lib/kubelet/pods/3970ff5f-5dbf-419e-a6af-3080508d2524/volumes/kubernetes.io~nfs/nfs-subdir-external-provisioner-root
Output: mount: wrong fs type, bad option, bad superblock on 10.200.14.112:/nfsdata,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

The wrong fs type, bad option error is caused by nfs-utils not being installed.

• Install nfs-utils

Run the following command on the host:

yum install nfs-utils