RustFS
RustFS reports metrics, traces, and logs to DataKit through OpenTelemetry OTLP.
Configuration¶
Prerequisites¶
- DataKit is installed, and RustFS can access DataKit's HTTP API port
9529.
Enable DataKit OpenTelemetry ingestion¶
cd /usr/local/datakit/conf.d
sudo cp samples/opentelemetry.conf.sample opentelemetry.conf
sudo vim opentelemetry.conf
Restart DataKit:
Start RustFS¶
The following example runs RustFS and DataKit on the same host:
docker pull rustfs/rustfs:latest
docker run -d \
--name rustfs \
--restart unless-stopped \
--network host \
-v rustfs-data:/data \
-e RUSTFS_VOLUMES=/data \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001 \
-e RUSTFS_CONSOLE_ENABLE=true \
-e RUSTFS_ACCESS_KEY='<YOUR_ACCESS_KEY>' \
-e RUSTFS_SECRET_KEY='<YOUR_SECRET_KEY>' \
-e RUSTFS_OBS_ENDPOINT=http://127.0.0.1:9529/otel \
-e RUSTFS_OBS_SERVICE_NAME=rustfs-local-validation \
-e RUSTFS_OBS_ENVIRONMENT=local \
-e RUSTFS_OBS_SAMPLE_RATIO=1.0 \
-e RUSTFS_OBS_METER_INTERVAL=10 \
-e RUSTFS_OBS_LOGGER_LEVEL=info \
-e RUSTFS_OBS_LOG_STDOUT_ENABLED=true \
rustfs/rustfs:latest
When they run on different hosts, set RUSTFS_OBS_ENDPOINT to http://<DATAKIT_HOST>:9529/otel, which must be reachable from RustFS.
Verification¶
- Perform S3 upload and download operations. In Guance, query
otel_service, filter byservice_name=rustfs-local-validation, and verify thatrustfs_process_uptime_secondsandrustfs_s3_put_object_totalcontinue to be reported. - Query traces with
service=rustfs-local-validationand logs withsource=rustfs-local-validationto verify data reporting.
Metrics¶
RustFS metrics are written to otel_service through OTLP. The following table lists the metrics used by this integration.
| MetricName | MetricDescribe | Dimensions | Unit |
|---|---|---|---|
rustfs_bucket_usage_objects_total |
Logical object count in the bucket (cluster-level snapshot) | bucket, deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_bucket_usage_usage_bytes |
Total logical object capacity in the bucket (cluster-level snapshot) | bucket, deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | B |
rustfs_bytes_pool_hit_rate |
Byte buffer pool hit rate; raw value ranges from 0 to 100 | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version, tier | percent |
rustfs_bytes_pool_hits_total |
Cumulative byte buffer pool hit count | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version, tier | - |
rustfs_bytes_pool_misses_total |
Cumulative byte buffer pool miss count | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version, tier | - |
rustfs_cluster_buckets_total |
Total number of buckets in the cluster | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_cluster_capacity_free_bytes |
Free space in the cluster's usable capacity | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | B |
rustfs_cluster_capacity_missing_drives |
Number of drives missing capacity observations | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_cluster_capacity_stale_drives |
Number of drives whose capacity data comes from stale snapshots | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_cluster_capacity_used_bytes |
Used cluster capacity, consistent with total usable and free capacity; not equal to logical object capacity | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | B |
rustfs_cluster_erasure_set_read_health |
Erasure set read health: 1 healthy, 0 unhealthy | collection_scope, deployment_environment_name, network_local_address, observer, pool_id, rustfs_node, service_instance_id, service_name, service_version, set_id | - |
rustfs_cluster_erasure_set_write_health |
Erasure set write health: 1 healthy, 0 unhealthy | collection_scope, deployment_environment_name, network_local_address, observer, pool_id, rustfs_node, service_instance_id, service_name, service_version, set_id | - |
rustfs_cluster_health_drives_offline_count |
Number of offline drives in the cluster | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_cluster_health_drives_online_count |
Number of online drives in the cluster | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_cluster_objects_total |
Total logical object count in the cluster | collection_scope, deployment_environment_name, network_local_address, observer, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_cluster_usage_objects_since_last_update_seconds |
Time since the object usage snapshot was last updated | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | s |
rustfs_cluster_usage_objects_total_bytes |
Total logical capacity of cluster objects | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | B |
rustfs_cpu_usage_percent |
CPU utilization exported by RustFS | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | percent |
rustfs_http_server_active_requests |
Current number of active HTTP server requests | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_http_server_failures_total |
Cumulative failed HTTP requests, grouped by status class | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version, status_class | - |
rustfs_http_server_requests_total |
Cumulative HTTP server request count | deployment_environment_name, method, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_io_latency_p95_ms |
Native RustFS I/O latency P95; the upstream aggregation window is unconfirmed and this is not S3 API P95 | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | ms |
rustfs_io_latency_p99_ms |
Native RustFS I/O latency P99; the upstream aggregation window is unconfirmed and this is not S3 API P99 | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | ms |
rustfs_io_queue_utilization_percent |
I/O queue utilization | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | percent |
rustfs_memory_process_resident_bytes |
RustFS process resident memory | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | B |
rustfs_memory_usage_percent |
Memory utilization exported by RustFS | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | percent |
rustfs_process_uptime_seconds |
RustFS process uptime | deployment_environment_name, network_local_address, rustfs_node, server, service_instance_id, service_name, service_version | s |
rustfs_s3_get_object_duration_ms_count |
Cumulative sample count in the GET object duration histogram | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_s3_get_object_duration_ms_sum |
Cumulative duration in the GET object duration histogram | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | ms |
rustfs_s3_get_object_size_bytes_sum |
Cumulative bytes in the GET object size histogram | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | B |
rustfs_s3_get_object_total |
Cumulative successful GET object count | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_s3_http_requests_total |
Cumulative S3 HTTP requests, grouped by method, operation, and outcome | deployment_environment_name, method, network_local_address, op, outcome, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_s3_operations_total |
Cumulative S3 API operation count | deployment_environment_name, network_local_address, op, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_s3_put_object_duration_ms_count |
Cumulative sample count in the PUT object duration histogram | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_s3_put_object_duration_ms_sum |
Cumulative duration in the PUT object duration histogram | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | ms |
rustfs_s3_put_object_size_bytes_sum |
Cumulative bytes in the PUT object size histogram | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | B |
rustfs_s3_put_object_total |
Cumulative successful PUT object count | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_scanner_last_cycle_duration_seconds |
Duration of the most recently completed scan cycle | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | s |
rustfs_scanner_last_cycle_result |
Latest scan cycle result: 0 unknown, 1 succeeded, 2 error, 3 partially completed, 4 superseded, 5 delayed | deployment_environment_name, network_local_address, rustfs_node, service_instance_id, service_name, service_version | - |
rustfs_system_drive_api_latency_micros |
Average disk storage API latency over the last minute; raw unit is microseconds | collection_scope, deployment_environment_name, drive, network_local_address, observer, rustfs_node, server, service_instance_id, service_name, service_version | μs |
rustfs_system_drive_health |
Drive health: 0 offline, 1 healthy, 2 healing | collection_scope, deployment_environment_name, drive, network_local_address, observer, rustfs_node, server, service_instance_id, service_name, service_version | - |
rustfs_system_drive_waiting_io |
Current number of I/O operations waiting on the drive | collection_scope, deployment_environment_name, drive, network_local_address, observer, rustfs_node, server, service_instance_id, service_name, service_version | - |
rustfs_system_process_file_descriptor_open_total |
Current number of open file descriptors in the RustFS process (Gauge) | deployment_environment_name, network_local_address, rustfs_node, server, service_instance_id, service_name, service_version | - |
rustfs_system_process_io_read_bytes |
Cumulative bytes read by the process from underlying storage; corresponds to /proc/[pid]/io read_bytes | deployment_environment_name, network_local_address, rustfs_node, server, service_instance_id, service_name, service_version | B |
rustfs_system_process_io_write_bytes |
Cumulative bytes written by the process to underlying storage; corresponds to /proc/[pid]/io write_bytes | deployment_environment_name, network_local_address, rustfs_node, server, service_instance_id, service_name, service_version | B |