Skip to content

Troubleshooting Data Gaps

Introduction

This article describes how to troubleshoot data gaps in Guance logs, traces, and metrics.

Architecture Diagram

The data flow of Guance is as follows:

  1. DataKit pushes metrics and logs to the Guance DataWay cluster.
  2. DataWay pushes data to the kodo service for processing.
  3. kodo pushes the processed data to the nsqd message queue service.
  4. kodo-x requests the nsqd message queue service to consume data.
  5. kodo-x pushes the consumed data to the corresponding storage engine.

Data Gap Troubleshooting Steps

Step 1: Check Host Time

Confirm the following:

  • The time on the Guance cluster host matches the current time.
  • The time on the DataKit collector host matches the current time.

Run the command to check:

date

If the host time does not match the current time, correct it as follows:

# Install ntpdate
yum install ntpdate -y

# Sync local time
ntpdate time.windows.com

# Sync with a network source
ntpdate cn.pool.ntp.org

sudo date -s "2022-01-01 10:30:00"

Note: Adjust the time accordingly.

Step 2: Agent Troubleshooting

Refer to DataKit Data Gap Troubleshooting

Step 3: Check DataWay Service Logs

Execute the following:

# Log into the container
kubectl exec -ti -n  <Namespace> <dataway pod name> bash
# View logs
cd /usr/local/cloudcare/dataflux/dataway
# Search for error logs
grep -Ei error log

Step 4: Check the Running Status of Each Service

  • Check whether the cluster node status is normal
kubectl get node
  • Check whether the status of all services under forethought-kodo is normal
kubectl get pods -n forethought-kodo
  • Check whether the nsqd service status is normal
kubectl get pods -n middleware | grep nsqd
  • Check whether the storage engine is normal
kubectl get pods -n middleware

Step 5: Check kodo Service Logs

Note

Checking the kodo service logs helps determine whether Guance successfully pushed data to the consumption queue.

  • Namespace: forethought-kodo

  • Deployment: kodo

  • Log path: /logdata/log

If the kodo service is normal, execute the following:

# Log into the container
kubectl exec -ti -n  forethought-kodo <kodo pod name> bash
# View logs
cd /logdata
# Search for error logs
grep -Ei error log

If the kodo service is abnormal, you will not be able to log into the container. You can first adjust the kodo log output mode, then check the container logs.

  • Modify the kodo log output mode
kubectl get configmap kodo -n forethought-kodo -o yaml | \
       sed "s/\/logdata\/log/stdout/g" | \
       kubectl apply -f -
  • Restart the kodo container
kubectl rollout restart -n forethought-kodo deploy kodo
  • View the kodo container logs
kubectl logs -f -n forethought-kodo <kodo pod name>

Step 6: Check kodo-x Service Logs

Note

Checking the kodo-x service logs helps determine whether Guance successfully wrote data, and whether there are issues such as log rate limiting or slow write logging.

  • Namespace: forethought-kodo
  • Deployment: kodo-x
  • Log path: /logdata/log

If the kodo-x service is normal, execute the following:

# Log into the container
kubectl exec -ti -n  forethought-kodo <kodo-x pod name> bash
# View logs
cd /logdata
# Search for error logs
grep -Ei error log

If the kodo service is abnormal, you will not be able to log into the container. You can first adjust the kodo log output mode, then check the container logs.

  • Modify the kodo log output mode
kubectl get configmap kodo-x -n forethought-kodo -o yaml | \
       sed "s/\/logdata\/log/stdout/g" | \
       kubectl apply -f -
  • Restart the kodo container
kubectl rollout restart -n forethought-kodo deploy kodo-x
  • View the kodo container logs
kubectl logs -f -n forethought-kodo <kodo-x pod name>

Feedback

Is this page helpful?