Datakit Exported Metrics¶
Foe better self-observability, Datakit has exported many Prometheus metrics among various submodules. We can use these metrics to trouble shooting errors or bugs during Datakit running.
Exported Metrics¶
Since Datakit Version 1.5.9, we can visit http://localhost:9529/metrics
to see these metrics(Different release may got some new or renamed metrics).
We have used these metrics in Datakit monitor, and we can playing these metrics with some tricks like this(refresh CPU usage of Datakit):
# play CPU usage of Datakit on every 3 seconds
$ watch -n 3 'curl -s http://localhost:9529/metrics | grep -a datakit_cpu_usage'
# HELP datakit_cpu_usage Datakit CPU usage(%)
# TYPE datakit_cpu_usage gauge
datakit_cpu_usage 4.9920266849857144
We can also playing other metrics too(change the grep
string), all available metrics list below(current Datakit version is 1.38.0):
TYPE | NAME | LABELS | HELP |
---|---|---|---|
GAUGE | datakit_config_datakit_ulimit |
status |
Datakit ulimit |
COUNTER | datakit_dns_domain_total |
N/A |
DNS watched domain counter |
COUNTER | datakit_dns_ip_updated_total |
domain |
Domain IP updated counter |
COUNTER | datakit_dns_watch_run_total |
interval |
Watch run counter |
SUMMARY | datakit_dns_cost_seconds |
domain,status |
DNS IP lookup cost |
COUNTER | datakit_election_pause_total |
id,namespace |
Input paused count when election failed |
COUNTER | datakit_election_resume_total |
id,namespace |
Input resume count when election OK |
GAUGE | datakit_election_status |
elected_id,id,namespace,status |
Datakit election status, if metric = 0, meas not elected, or the elected time(unix timestamp second) |
GAUGE | datakit_election_inputs |
namespace |
Datakit election input count |
SUMMARY | datakit_election_seconds |
namespace,status |
Election latency |
GAUGE | datakit_goroutine_alive |
name |
Alive Goroutine count |
COUNTER | datakit_goroutine_recover_total |
name |
Recovered Goroutine count |
COUNTER | datakit_goroutine_stopped_total |
name |
Stopped Goroutine count |
COUNTER | datakit_goroutine_crashed_total |
name |
Crashed goroutines count |
GAUGE | datakit_goroutine_groups |
N/A |
Goroutine group count |
SUMMARY | datakit_goroutine_cost_seconds |
name |
Goroutine running duration |
SUMMARY | datakit_http_api_elapsed_seconds |
api,method,status |
API request cost |
SUMMARY | datakit_http_api_req_size_bytes |
api,method,status |
API request body size |
COUNTER | datakit_http_api_total |
api,method,status |
API request counter |
GAUGE | datakit_http_api_global_tags_last_updated |
api,method,status |
Global tag updated timestamp, in second |
SUMMARY | datakit_httpcli_got_first_resp_byte_cost_seconds |
from |
Got first response byte cost |
COUNTER | datakit_httpcli_tcp_conn_total |
from,remote,type |
HTTP TCP connection count |
COUNTER | datakit_httpcli_conn_reused_from_idle_total |
from |
HTTP connection reused from idle count |
SUMMARY | datakit_httpcli_conn_idle_time_seconds |
from |
HTTP connection idle time |
SUMMARY | datakit_httpcli_dns_cost_seconds |
from |
HTTP DNS cost |
SUMMARY | datakit_httpcli_tls_handshake_seconds |
from |
HTTP TLS handshake cost |
SUMMARY | datakit_httpcli_http_connect_cost_seconds |
from |
HTTP connect cost |
SUMMARY | datakit_io_build_body_cost_seconds |
category,encoding |
Build point HTTP body cost |
SUMMARY | datakit_io_build_body_batches |
category,encoding |
Batch HTTP body batches |
SUMMARY | datakit_io_build_body_batch_bytes |
category,encoding,gzip |
Batch HTTP body size |
SUMMARY | datakit_io_build_body_batch_points |
category,encoding,gzip |
Batch HTTP body points |
COUNTER | datakit_io_dataway_point_total |
category,status |
Dataway uploaded points, partitioned by category and send status(HTTP status) |
COUNTER | datakit_io_dataway_point_bytes_total |
category,enc,status |
Dataway uploaded points bytes, partitioned by category and pint send status(HTTP status) |
COUNTER | datakit_io_dataway_http_drop_point_total |
category,error |
Dataway write drop points |
SUMMARY | datakit_io_dataway_api_latency_seconds |
api,status |
Dataway HTTP request latency partitioned by HTTP API(method@url) and HTTP status |
COUNTER | datakit_io_http_retry_total |
api,status |
Dataway HTTP retried count |
SUMMARY | datakit_io_grouped_request |
category |
Grouped requests under sinker |
SUMMARY | datakit_io_flush_failcache_bytes |
category |
IO flush fail-cache bytes(in gzip) summary |
COUNTER | datakit_filter_update_total |
N/A |
Filters(remote) updated count |
GAUGE | datakit_filter_last_update_timestamp_seconds |
N/A |
Filter last update time |
COUNTER | datakit_filter_point_total |
category,filters,source |
Filter points of filters |
GAUGE | datakit_filter_parse_error |
error,filters |
Filter parse error |
COUNTER | datakit_filter_point_dropped_total |
category,filters,source |
Dropped points of filters |
SUMMARY | datakit_filter_pull_latency_seconds |
status |
Filter pull(remote) latency |
SUMMARY | datakit_filter_latency_seconds |
category,filters,source |
Filter latency of these filters |
GAUGE | datakit_io_queue_points |
category |
IO module queued(cached) points |
COUNTER | datakit_io_input_filter_point_total |
name,category |
Input filtered point total |
COUNTER | datakit_io_feed_total |
name,category |
Input feed total |
GAUGE | datakit_io_last_feed_timestamp_seconds |
name,category |
Input last feed time(according to Datakit local time) |
SUMMARY | datakit_input_collect_latency_seconds |
name,category |
Input collect latency |
GAUGE | datakit_io_chan_usage |
category |
IO channel usage(length of the channel) |
GAUGE | datakit_io_chan_capacity |
category |
IO channel capacity |
SUMMARY | datakit_io_feed_cost_seconds |
category,from |
IO feed waiting(on block mode) seconds |
SUMMARY | datakit_io_feed_point |
name,category |
Input feed point |
GAUGE | datakit_io_flush_workers |
category |
IO flush workers |
COUNTER | datakit_io_flush_total |
category |
IO flush total |
COUNTER | datakit_error_total |
source,category |
Total errors, only count on error source, not include error message |
GAUGE | datakit_goroutines |
N/A |
Goroutine count within Datakit |
GAUGE | datakit_heap_alloc_bytes |
N/A |
Datakit memory heap bytes |
GAUGE | datakit_sys_alloc_bytes |
N/A |
Datakit memory system bytes |
GAUGE | datakit_cpu_usage |
N/A |
Datakit CPU usage(%) |
GAUGE | datakit_open_files |
N/A |
Datakit open files(only available on Linux) |
GAUGE | datakit_cpu_cores |
N/A |
Datakit CPU cores |
GAUGE | datakit_uptime_seconds |
auto_update,docker,hostname,lite,elinker,resource_limit,version=?,build_at=?,branch=?,os_arch=? |
Datakit uptime |
GAUGE | datakit_data_overuse |
N/A |
Does current workspace's data(metric/logging) usage(if 0 not beyond, or with a unix timestamp when overuse occurred) |
COUNTER | datakit_process_ctx_switch_total |
type |
Datakit process context switch count(Linux only) |
COUNTER | datakit_process_io_count_total |
type |
Datakit process IO count |
COUNTER | datakit_process_io_bytes_total |
type |
Datakit process IO bytes count |
COUNTER | datakit_pipeline_offload_point_total |
category,exporter,remote |
Pipeline offload processed total points |
COUNTER | datakit_pipeline_offload_error_point_total |
category,exporter,remote |
Pipeline offload processed total error points |
SUMMARY | datakit_pipeline_offload_cost_seconds |
category,exporter,remote |
Pipeline offload total cost |
GAUGE | datakit_input_container_kubernetes_fetch_error |
namespace,resource,error |
Kubernetes resource fetch error |
SUMMARY | datakit_input_container_kubernetes_collect_cost_seconds |
category |
Kubernetes collect cost |
SUMMARY | datakit_input_container_kubernetes_collect_resource_cost_seconds |
category,kind,fieldselector |
Kubernetes collect resource cost |
COUNTER | datakit_input_container_kubernetes_collect_pts_total |
category |
Kubernetes collect point total |
COUNTER | datakit_input_container_kubernetes_pod_metrics_query_total |
target |
Kubernetes query pod metrics count |
SUMMARY | datakit_input_container_collect_cost_seconds |
category |
Container collect cost |
COUNTER | datakit_input_container_collect_pts_total |
category |
Container collect point total |
SUMMARY | datakit_input_container_total_collect_cost_seconds |
category |
Total container collect cost |
SUMMARY | datakit_dialtesting_task_run_cost_seconds |
region,protocol |
Task run time |
SUMMARY | datakit_dialtesting_task_exec_time_interval_seconds |
region,protocol |
Task execution time interval |
GAUGE | datakit_dialtesting_worker_job_chan_number |
type |
The number of the channel for the jobs |
GAUGE | datakit_dialtesting_worker_job_number |
N/A |
The number of the jobs to send data in parallel |
GAUGE | datakit_dialtesting_worker_cached_points_number |
region,protocol |
The number of cached points |
GAUGE | datakit_dialtesting_worker_send_points_number |
region,protocol,status |
The number of the points which have been sent |
SUMMARY | datakit_dialtesting_worker_send_cost_seconds |
region,protocol |
Time cost to send points |
GAUGE | datakit_dialtesting_task_number |
region,protocol |
The number of tasks |
GAUGE | datakit_dialtesting_dataway_send_failed_number |
region,protocol,dataway |
The number of failed sending for each Dataway |
SUMMARY | datakit_dialtesting_pull_cost_seconds |
region,is_first |
Time cost to pull tasks |
COUNTER | datakit_dialtesting_task_synchronized_total |
region,protocol |
Task synchronized number |
COUNTER | datakit_dialtesting_task_invalid_total |
region,protocol,fail_reason |
Invalid task number |
SUMMARY | datakit_dialtesting_task_check_cost_seconds |
region,protocol,status |
Task check time |
COUNTER | datakit_input_kafkamq_consumer_message_total |
topic,partition,status |
Kafka consumer message numbers from Datakit start |
COUNTER | datakit_input_kafkamq_group_election_total |
N/A |
Kafka group election count |
SUMMARY | datakit_input_kafkamq_process_message_nano |
topic |
kafkamq process message nanoseconds duration |
COUNTER | datakit_input_kubernetesprometheus_resource_collect_pts_total |
role,name |
The number of the points which have been sent |
GAUGE | datakit_input_kubernetesprometheus_forked_worker_number |
role,name |
The number of the worker |
GAUGE | datakit_inputs_instance |
input |
Input instance count |
COUNTER | datakit_inputs_crash_total |
input |
Input crash count |
GAUGE | datakit_input_ploffload_chan_capacity |
channel_name |
PlOffload channel capacity |
GAUGE | datakit_input_ploffload_chan_usage |
channel_name |
PlOffload channel usage |
COUNTER | datakit_input_ploffload_point_total |
category |
PlOffload processed total points |
SUMMARY | datakit_input_promremote_collect_points |
source |
Total number of promremote collection points |
SUMMARY | datakit_input_promremote_time_diff_in_second |
source |
Time diff with local time |
COUNTER | datakit_input_promremote_no_time_points_total |
source |
Total number of promremote collection no time points |
GAUGE | api_elapsed_seconds |
N/A |
Proxied API elapsed seconds |
COUNTER | api_post_bytes_total |
api,status |
Proxied API post bytes total |
SUMMARY | api_latency_seconds |
api,status |
Proxied API latency |
COUNTER | datakit_input_proxy_connect_total |
client_ip |
Proxy connect(method CONNECT) |
COUNTER | datakit_input_proxy_api_total |
api,method |
Proxy API total |
SUMMARY | datakit_input_proxy_api_latency_seconds |
api,method,status |
Proxy API latency |
COUNTER | datakit_input_rum_session_replay_drop_total |
app_id,env,version,service |
statistics the total count of session replay points which have been filtered by rules |
COUNTER | datakit_input_rum_session_replay_drop_bytes_total |
app_id,env,version,service |
statistics the total bytes of session replay points which have been filtered by rules |
COUNTER | datakit_input_rum_locate_statistics_total |
app_id,ip_status,locate_status |
locate by ip addr statistics |
COUNTER | datakit_input_rum_source_map_total |
app_id,sdk_name,status,remark |
source map result statistics |
GAUGE | datakit_input_rum_loaded_zips |
platform |
RUM source map currently loaded zip archive count |
SUMMARY | datakit_input_rum_source_map_duration_seconds |
sdk_name,app_id,env,version |
statistics elapsed time in RUM source map(unit: second) |
SUMMARY | datakit_input_rum_session_replay_upload_latency_seconds |
app_id,env,version,service,status_code |
statistics elapsed time in session replay uploading |
COUNTER | datakit_input_rum_session_replay_upload_failure_total |
app_id,env,version,service,status_code |
statistics count of session replay points which which have unsuccessfully uploaded |
COUNTER | datakit_input_rum_session_replay_upload_failure_bytes_total |
app_id,env,version,service,status_code |
statistics the total bytes of session replay points which have unsuccessfully uploaded |
SUMMARY | datakit_input_rum_session_replay_read_body_delay_seconds |
app_id,env,version,service |
statistics the duration of reading session replay body |
SUMMARY | datakit_input_snmp_discovery_cost |
profile_type |
Discovery cost(in second) |
SUMMARY | datakit_input_snmp_collect_cost |
N/A |
Every loop collect cost(in second) |
SUMMARY | datakit_input_snmp_device_collect_cost |
class |
Device collect cost(in second) |
GAUGE | datakit_input_snmp_alive_devices |
class |
Alive devices |
SUMMARY | datakit_input_prom_collect_points |
mode,source |
Total number of prom collection points |
SUMMARY | datakit_input_prom_http_get_bytes |
mode,source |
HTTP get bytes |
SUMMARY | datakit_input_prom_http_latency_in_second |
mode,source |
HTTP latency(in second) |
GAUGE | datakit_input_prom_stream_size |
mode,source |
Stream size |
SUMMARY | datakit_input_statsd_collect_points |
N/A |
Total number of statsd collection points |
SUMMARY | datakit_input_statsd_accept_bytes |
N/A |
Accept bytes from network |
COUNTER | datakit_input_logging_socket_feed_message_count_total |
network |
socket feed to IO message count |
SUMMARY | datakit_input_logging_socket_log_length |
network |
record the length of each log line |
COUNTER | datakit_tailer_collect_multiline_state_total |
source,filepath,multilinestate |
Tailer multiline state total |
COUNTER | datakit_tailer_file_rotate_total |
source,filepath |
Tailer rotate total |
COUNTER | datakit_tailer_buffer_force_flush_total |
source,filepath |
Tailer force flush total |
COUNTER | datakit_tailer_parse_fail_total |
source,filepath,mode |
Tailer parse fail total |
GAUGE | datakit_tailer_open_file_num |
mode |
Tailer open file total |
COUNTER | datakit_input_logging_socket_connect_status_total |
network,status |
connect and close count for net.conn |
COUNTER | datakit_input_tracing_total |
input,service |
The total links number of Trace processed by the trace module |
COUNTER | datakit_input_sampler_total |
input,service |
The sampler number of Trace processed by the trace module |
COUNTER | diskcache_put_bytes_total |
path |
Cache Put() bytes count |
COUNTER | diskcache_get_total |
path |
Cache Get() count |
COUNTER | diskcache_wakeup_total |
path |
Wakeup count on sleeping write file |
COUNTER | diskcache_seek_back_total |
path |
Seek back when Get() got any error |
COUNTER | diskcache_get_bytes_total |
path |
Cache Get() bytes count |
GAUGE | diskcache_capacity |
path |
Current capacity(in bytes) |
GAUGE | diskcache_max_data |
path |
Max data to Put(in bytes), default 0 |
GAUGE | diskcache_batch_size |
path |
Data file size(in bytes) |
GAUGE | diskcache_size |
path |
Current cache size(in bytes) |
GAUGE | diskcache_open_time |
no_fallback_on_error,no_lock,no_pos,no_sync,path |
Current cache Open time in unix timestamp(second) |
GAUGE | diskcache_last_close_time |
path |
Current cache last Close time in unix timestamp(second) |
GAUGE | diskcache_datafiles |
path |
Current un-read data files |
SUMMARY | diskcache_get_latency |
path |
Get() time cost(micro-second) |
SUMMARY | diskcache_put_latency |
path |
Put() time cost(micro-second) |
COUNTER | diskcache_dropped_bytes_total |
path |
Dropped bytes during Put() when capacity reached. |
COUNTER | diskcache_dropped_total |
path,reason |
Dropped files during Put() when capacity reached. |
COUNTER | diskcache_rotate_total |
path |
Cache rotate count, mean file rotate from data to data.0000xxx |
COUNTER | diskcache_remove_total |
path |
Removed file count, if some file read EOF, remove it from un-read list |
COUNTER | diskcache_put_total |
path |
Cache Put() count |
COUNTER | pointpool_chan_get_total |
N/A |
Get count from reserved channel |
COUNTER | pointpool_chan_put_total |
N/A |
Put count to reserved channel |
COUNTER | pointpool_pool_get_total |
N/A |
Get count from reserved channel |
COUNTER | pointpool_pool_put_total |
N/A |
Put count to reserved channel |
COUNTER | pointpool_reserved_capacity |
N/A |
Reserved capacity of the pool |
COUNTER | pointpool_malloc_total |
N/A |
New object malloc from pool |
COUNTER | pointpool_escaped |
N/A |
Points that not comes from pool |
Golang Runtime Metrics¶
On API /metrics
, Datakit exported Golang runtime metrics, it seems like this:
go_cgo_go_to_c_calls_calls_total 8447
go_gc_cycles_automatic_gc_cycles_total 10
go_gc_cycles_forced_gc_cycles_total 0
go_gc_cycles_total_gc_cycles_total 10
go_gc_duration_seconds{quantile="0"} 3.4709e-05
go_gc_duration_seconds{quantile="0.25"} 3.9917e-05
go_gc_duration_seconds{quantile="0.5"} 0.000138459
go_gc_duration_seconds{quantile="0.75"} 0.000211333
go_gc_duration_seconds{quantile="1"} 0.000693833
go_gc_duration_seconds_sum 0.001920708
go_gc_duration_seconds_count 10
go_gc_heap_allocs_by_size_bytes_bucket{le="8.999999999999998"} 16889
go_gc_heap_allocs_by_size_bytes_bucket{le="24.999999999999996"} 221293
go_gc_heap_allocs_by_size_bytes_bucket{le="64.99999999999999"} 365672
go_gc_heap_allocs_by_size_bytes_bucket{le="144.99999999999997"} 475633
go_gc_heap_allocs_by_size_bytes_bucket{le="320.99999999999994"} 507361
go_gc_heap_allocs_by_size_bytes_bucket{le="704.9999999999999"} 516511
go_gc_heap_allocs_by_size_bytes_bucket{le="1536.9999999999998"} 521176
go_gc_heap_allocs_by_size_bytes_bucket{le="3200.9999999999995"} 522802
go_gc_heap_allocs_by_size_bytes_bucket{le="6528.999999999999"} 524529
go_gc_heap_allocs_by_size_bytes_bucket{le="13568.999999999998"} 525164
go_gc_heap_allocs_by_size_bytes_bucket{le="27264.999999999996"} 525269
go_gc_heap_allocs_by_size_bytes_bucket{le="+Inf"} 525421
go_gc_heap_allocs_by_size_bytes_sum 7.2408264e+07
go_gc_heap_allocs_by_size_bytes_count 525421
go_gc_heap_allocs_bytes_total 7.2408264e+07
go_gc_heap_allocs_objects_total 525421
go_gc_heap_frees_by_size_bytes_bucket{le="8.999999999999998"} 11081
go_gc_heap_frees_by_size_bytes_bucket{le="24.999999999999996"} 168291
go_gc_heap_frees_by_size_bytes_bucket{le="64.99999999999999"} 271749
go_gc_heap_frees_by_size_bytes_bucket{le="144.99999999999997"} 352424
go_gc_heap_frees_by_size_bytes_bucket{le="320.99999999999994"} 378481
go_gc_heap_frees_by_size_bytes_bucket{le="704.9999999999999"} 385700
go_gc_heap_frees_by_size_bytes_bucket{le="1536.9999999999998"} 389443
go_gc_heap_frees_by_size_bytes_bucket{le="3200.9999999999995"} 390591
go_gc_heap_frees_by_size_bytes_bucket{le="6528.999999999999"} 392069
go_gc_heap_frees_by_size_bytes_bucket{le="13568.999999999998"} 392565
go_gc_heap_frees_by_size_bytes_bucket{le="27264.999999999996"} 392636
go_gc_heap_frees_by_size_bytes_bucket{le="+Inf"} 392747
go_gc_heap_frees_by_size_bytes_sum 5.3304296e+07
go_gc_heap_frees_by_size_bytes_count 392747
go_gc_heap_frees_bytes_total 5.3304296e+07
go_gc_heap_frees_objects_total 392747
go_gc_heap_goal_bytes 3.6016864e+07
go_gc_heap_objects_objects 132674
go_gc_heap_tiny_allocs_objects_total 36033
go_gc_limiter_last_enabled_gc_cycle 0
go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0
go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0
go_gc_pauses_seconds_bucket{le="9.999999999999998e-08"} 0
go_gc_pauses_seconds_bucket{le="1.0239999999999999e-06"} 0
go_gc_pauses_seconds_bucket{le="1.0239999999999999e-05"} 1
go_gc_pauses_seconds_bucket{le="0.00010239999999999998"} 15
go_gc_pauses_seconds_bucket{le="0.0010485759999999998"} 20
go_gc_pauses_seconds_bucket{le="0.010485759999999998"} 20
go_gc_pauses_seconds_bucket{le="0.10485759999999998"} 20
go_gc_pauses_seconds_bucket{le="+Inf"} 20
go_gc_pauses_seconds_sum 0.000656384
go_gc_pauses_seconds_count 20
go_gc_stack_starting_size_bytes 4096
go_goroutines 102
go_info{version="go1.19.5"} 1
go_memory_classes_heap_free_bytes 8.839168e+06
go_memory_classes_heap_objects_bytes 1.9103968e+07
go_memory_classes_heap_released_bytes 3.530752e+06
go_memory_classes_heap_stacks_bytes 2.4576e+06
go_memory_classes_heap_unused_bytes 8.011552e+06
go_memory_classes_metadata_mcache_free_bytes 3600
go_memory_classes_metadata_mcache_inuse_bytes 12000
go_memory_classes_metadata_mspan_free_bytes 77472
go_memory_classes_metadata_mspan_inuse_bytes 426960
go_memory_classes_metadata_other_bytes 6.201928e+06
go_memory_classes_os_stacks_bytes 0
go_memory_classes_other_bytes 1.931459e+06
go_memory_classes_profiling_buckets_bytes 1.489565e+06
go_memory_classes_total_bytes 5.2086024e+07
go_memstats_alloc_bytes 1.9103968e+07
go_memstats_alloc_bytes_total 7.2408264e+07
go_memstats_buck_hash_sys_bytes 1.489565e+06
go_memstats_frees_total 428780
go_memstats_gc_sys_bytes 6.201928e+06
go_memstats_heap_alloc_bytes 1.9103968e+07
go_memstats_heap_idle_bytes 1.236992e+07
go_memstats_heap_inuse_bytes 2.711552e+07
go_memstats_heap_objects 132674
go_memstats_heap_released_bytes 3.530752e+06
go_memstats_heap_sys_bytes 3.948544e+07
go_memstats_last_gc_time_seconds 1.6992580814748092e+09
go_memstats_lookups_total 0
go_memstats_mallocs_total 561454
go_memstats_mcache_inuse_bytes 12000
go_memstats_mcache_sys_bytes 15600
go_memstats_mspan_inuse_bytes 426960
go_memstats_mspan_sys_bytes 504432
go_memstats_next_gc_bytes 3.6016864e+07
go_memstats_other_sys_bytes 1.931459e+06
go_memstats_stack_inuse_bytes 2.4576e+06
go_memstats_stack_sys_bytes 2.4576e+06
go_memstats_sys_bytes 5.2086024e+07
go_sched_gomaxprocs_threads 10
go_sched_goroutines_goroutines 102
go_sched_latencies_seconds_bucket{le="9.999999999999999e-10"} 4886
go_sched_latencies_seconds_bucket{le="9.999999999999999e-09"} 4886
go_sched_latencies_seconds_bucket{le="9.999999999999998e-08"} 5883
go_sched_latencies_seconds_bucket{le="1.0239999999999999e-06"} 6669
go_sched_latencies_seconds_bucket{le="1.0239999999999999e-05"} 7191
go_sched_latencies_seconds_bucket{le="0.00010239999999999998"} 7531
go_sched_latencies_seconds_bucket{le="0.0010485759999999998"} 7567
go_sched_latencies_seconds_bucket{le="0.010485759999999998"} 7569
go_sched_latencies_seconds_bucket{le="0.10485759999999998"} 7569
go_sched_latencies_seconds_bucket{le="+Inf"} 7569
go_sched_latencies_seconds_sum 0.00988825
go_sched_latencies_seconds_count 7569
go_threads 16