Log Engine Capacity Planning¶
Before using Elasticsearch/OpenSearch as the storage backend for Guance logs/traces, first evaluate the storage capacity and cluster specifications. This document provides an estimation algorithm based on community content and actual testing.
Warning
- Due to the license change of Elasticsearch, OpenSearch is the preferred recommendation.
- It is recommended to first test the data generation rate of monitoring data using a smaller Elasticsearch/OpenSearch cluster, and then plan the cluster based on the test results.
Data Compression¶
The following are test results of Elasticsearch compression algorithms, using the official nyc_taxis dataset from esrally, with a size of 74 GB.
| codec configuration | compression algorithm | compressed size (GB) | compression ratio |
|---|---|---|---|
| default | LZ4 | 35.5 | 0.48 |
| best_compression | DEFLATE | 26.4 | 0.36 |
The following are test results of two compression algorithms in OpenSearch, using the eventdata dataset from opensearch-benchmark, with a size of 15 GB.
| codec configuration | compression algorithm | compressed size (GB) | compression ratio |
|---|---|---|---|
| default | LZ4 | 5.1 | 0.34 |
| best_compression | DEFLATE | 3.7 | 0.25 |
Disk Capacity Estimation¶
Assume the following conditions:
- 1 TB of monitoring source data generated per day
- Compression ratio of 0.5
- 1 replica per index
- Monitoring data retained for the latest 7 days
The storage calculation formula is based on the Tencent Cloud Elasticsearch storage estimation.
| Parameter | Reference Value |
|---|---|
| Data expansion / index overhead | 10% |
| Internal task overhead | 20% |
| OS reservation | 5% |
| Safety reservation | 15% |
According to the Guance data retention policy, we increase the safety reservation to 30%. The original formula is as follows:
Actual space = source data × compression ratio × (1 + number of replicas) × (1 + data expansion) / (1 - internal task overhead) / (1 - OS reservation) × (1 + reserved space)
Substituting the reference values:
If retaining the latest 7 days of data, the possible peak is:
Cluster Planning¶
Data Nodes¶
The following is the Elasticsearch cluster planning. Higher single-node configurations support larger cluster sizes.
| Single-node specification | Maximum number of cluster nodes | Maximum disk per node |
|---|---|---|
| 2C4GB | 10 | 200 GB |
| 2C8GB | 10 | 400 GB |
| 4C16GB | 20 | 800 GB |
| 8C32GB | 40 | 1.5 TB |
| 16C64GB | 80 | 3 TB |
If using 4C16GB nodes, based on the capacity estimation above, the recommended cluster data node size is:
If using 8C32GB nodes, based on the capacity estimation above, the recommended cluster data node size is:
If using 16C64GB nodes, based on the capacity estimation above, the recommended cluster data node size is:
Coordinating Nodes¶
It is recommended to add dedicated coordinating nodes at a ratio of 1:5 (starting from 2 nodes), with a CPU:Memory ratio of 1:4 or 1:8. For example, for 10 data nodes of 8C32GB, it is recommended to configure 2 dedicated coordinating nodes of 8C32GB. The cluster manager nodes should be deployed in a 2n+1 high-availability configuration to prevent split-brain.