Skip to content

Reliability Verification

This article validates the reliability of a centralized deployment through a complete data pipeline:

  • Ingesting large amounts of data using specific Datakit collectors
  • Uploading data to Kodo via Dataway
  • Kodo writes the data to log-class storage based on the established workflow

Environment Preparation

  • A basic Linux machine with Datakit installed and the logstreaming collector enabled
  • Prepare a test script to push data to the Datakit logstreaming collector
  • Activate an unlimited workspace to accommodate a large volume of log data

Implementation

  • Download the test data set, which contains 10,000 log entries, each 1 KB in size:
wget https://static.guance.com/testing-data/10000-1kb.log
  • Modify the script below, fill in the Datakit IP prepared above, and save the script as curl-log-streaming.sh:
#!/bin/bash

# Check if a parameter is provided
if [ $# -eq 0 ]; then
    echo "Usage: $0 <number_of_iterations>"
    exit 1
fi

# Read the command-line argument as the number of iterations
num_iterations=$1
file=$2

# Use a for loop to execute the specified number of iterations
for ((i=1; i<=num_iterations; i++)); do
        curl -v http://<YOUR-DATAKIT-IP>:9529/v1/write/logstreaming?source=drop-testing --data-binary "@$2"
        sleep 2.5 # Sleep for 2.5s
done
  • Execute the script above. The script will push 40,000 requests to Datakit, lasting about 28 hours:
bash curl-log-streaming.sh 40000 10000-1kb.log

Viewing Results

After completing the steps above, the logstreaming/drop-testing collection can be seen in the Datakit monitor:

datakit monitor -MI

The request latency from Datakit to Dataway can be viewed with the following command:

datakit monitor -MW

In the Guance Log Explorer, select the log source (source) as drop-testing. The height of each bar (interval of 1 minute) in the status distribution chart should be approximately 220,000 to 240,000 (60s/2.5s*10000). Under normal conditions, the data should be very uniform, with no sudden spikes or drops (since data is written at a fixed frequency).

During this process, you can search for "dataway" in the built-in views to check the metrics of Dataway itself. You should also check the explorers for components such as NSQ, GuanceDB, Kodo, and Kodo-X to ensure they are deployed successfully.

Feedback

Is this page helpful?