StatsD
The indicator data collected by the DDTrace agent will be sent to port 8125 of the DK through the StatsD data type.
This includes the JVM CPU, memory, threads, and class loading information of the JVM runtime, as well as various collected JMX indicators such as Kafka, Tomcat, RabbitMQ, etc.
Config¶
Preconditions¶
When DDTrace runs as an agent, there is no need for the user to specifically open the jmx port. If no port is opened, the agent will randomly open a local port.
DDTrace will collect JVM information by default. By default, it will be sent to 'localhost: 8125'
if k8s:
You can use dd.jmxfetch.<INTEGRATION_NAME>.enabled=true Enable the specified collector.
for INTEGRATION_NAME, You can check the default supported third-party software before.
For example, Tomcat or Kafka:
Collector Configuration¶
Go to the conf.d/samples directory under the DataKit installation directory, copy statsd.conf.sample and name it statsd.conf. Examples are as follows:
[[inputs.statsd]]
## Collector alias.
# source = "statsd/-/-"
## Collect interval, default is 10 seconds. (optional)
# interval = '10s'
protocol = "udp"
## Address to host unix listener on, linux only
service_unix_address = "/var/run/datakit/statsd.sock"
## Address and port to host UDP listener on
service_address = ":8125"
## Tag request metric. Used for distinguish feed metric name.
## eg, DD_TAGS=source_key:tomcat,host_key:cn-shanghai-sq5ei
## eg, -Ddd.tags=source_key:tomcat,host_key:cn-shanghai-sq5ei
# statsd_source_key = "source_key"
# statsd_host_key = "host_key"
## Indicate whether report tag statsd_source_key and statsd_host_key.
# save_above_key = false
delete_gauges = true
delete_counters = true
delete_sets = true
delete_timings = true
## Counter metric is float in new Datakit version, set true if want be int.
# set_counter_int = false
## Percentiles to calculate for timing & histogram stats
percentiles = [50.0, 90.0, 99.0, 99.9, 99.95, 100.0]
## separator to use between elements of a statsd metric
metric_separator = "_"
## Parses tags in the datadog statsd format
## http://docs.datadoghq.com/guides/dogstatsd/
parse_data_dog_tags = true
## Parses datadog extensions to the statsd format
datadog_extensions = true
## Parses distributions metric as specified in the datadog statsd format
## https://docs.datadoghq.com/developers/metrics/types/?tab=distribution#definition
datadog_distributions = true
## We do not need following tags(they may create tremendous of time-series under influxdb's logic)
# Examples:
# "runtime-id", "metric-type"
drop_tags = [ ]
# All metric-name prefixed with 'jvm_' are set to influxdb's measurement 'jvm'
# All metric-name prefixed with 'stats_' are set to influxdb's measurement 'stats'
# Examples:
# "stats_:stats", "jvm_:jvm", "tomcat_:tomcat",
metric_mapping = [ ]
## Number of UDP messages allowed to queue up, once filled,
## the statsd server will start dropping packets, default is 128.
# allowed_pending_messages = 128
## Number of timing/histogram values to track per-measurement in the
## calculation of percentiles. Raising this limit increases the accuracy
## of percentiles but also increases the memory usage and cpu time.
percentile_limit = 1000
## Max duration (TTL) for each metric to stay cached/reported without being updated.
#max_ttl = "1000h"
[inputs.statsd.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
After configuration, restart DataKit.
The collector can now be turned on by configMap injection collector configuration.
Info
If find lot of Feed: io busy in the log, can configure interval='1s', minimum is 1s.
Tag data sources¶
If you want to tag the hosts captured by DDTrace, you can do so by injecting tags:
- You can use environment variables, i.e.
DD_TAGS, for example:DD_TAGS=source_key:tomcat,host_key:cn-shanghai-sq5ei - You can use the command line, i.e.
dd.tags, for example:-Ddd.tags=source_key:tomcat,host_key:cn-shanghai-sq5ei
In the above example, you need to specify that the key for source is source_key and the key for host is host_key in the DataKit configuration. You can change it to something else, but you have to make sure that the field names in DataKit are the same as the field names in DDTrace.
The end result is that you can see statsd/tomcat/cn-shanghai-sq5ei in the datakit monitor, which distinguishes it from other data sources reported to the statsd collector. If the above configuration is not done, then the default display is seen on the datakit monitor: statsd/-/-.
In addition, there is a configuration switch save_above_key that determine whether the tags corresponding to statsd_source_key and statsd_host_key are reported to the center. The default is not to report(false).
Metric¶
For all of the following data collections, a global tag named host is appended by default (the tag value is the host name of the DataKit), or other tags can be specified in the configuration by [inputs.statsd.tags]:
jvm¶
| Tags & Fields | Description |
|---|---|
| host ( tag) |
Host name. |
| instance ( tag) |
Instance name. |
| jmx_domain ( tag) |
JMX domain. |
| metric_type ( tag) |
Metric type. |
| name ( tag) |
Type name. |
| runtime-id ( tag) |
Runtime id. |
| service ( tag) |
Service name. |
| type ( tag) |
Object type. |
| buffer_pool_direct_capacity | Measure of total memory capacity of direct buffers. Type: float | (gauge) Unit: digital,B |
| buffer_pool_direct_count | Number of direct buffers in the pool. Type: float | (gauge) Unit: count |
| buffer_pool_direct_used | Measure of memory used by direct buffers. Type: float | (gauge) Unit: digital,B |
| buffer_pool_mapped_capacity | Measure of total memory capacity of mapped buffers. Type: float | (gauge) Unit: digital,B |
| buffer_pool_mapped_count | Number of mapped buffers in the pool. Type: float | (gauge) Unit: count |
| buffer_pool_mapped_used | Measure of memory used by mapped buffers. Type: float | (gauge) Unit: digital,B |
| cpu_load_process | Recent CPU utilization for the process. Type: float | (gauge) Unit: percent,percent |
| cpu_load_system | Recent CPU utilization for the whole system. Type: float | (gauge) Unit: percent,percent |
| daemon_code_cache_used | The number of daemon threads. Type: float | (count) Unit: count |
| daemon_thread_count | Daemon thread count. Type: float | (gauge) Unit: count |
| gc_code_cache_used | GC code cache used. Type: float | (gauge) Unit: count |
| gc_eden_size | The 'eden' size in garbage collection. Type: float | (gauge) Unit: digital,B |
| gc_major_collection_count | The rate of major garbage collections. Type: float | (gauge) Unit: count |
| gc_major_collection_time | The fraction of time spent(rate) in major garbage collection. Type: float | (gauge) Unit: time,ms |
| gc_metaspace_size | The metaspace size in garbage collection.Type: float | (gauge) Unit: digital,B |
| gc_minor_collection_count | The rate of minor garbage collections. Type: float | (gauge) Unit: count |
| gc_minor_collection_time | The fraction of time spent(rate) in minor garbage collection. Type: float | (gauge) Unit: time,ms |
| gc_old_gen_size | The ond gen size in garbage collection. Type: float | (gauge) Unit: digital,B |
| gc_survivor_size | The survivor size in garbage collection. Type: float | (gauge) Unit: digital,B |
| heap_memory | The total Java heap memory used. Type: float | (gauge) Unit: digital,B |
| heap_memory_committed | The total Java heap memory committed to be used. Type: float | (gauge) Unit: digital,B |
| heap_memory_init | The initial Java heap memory allocated. Type: float | (gauge) Unit: digital,B |
| heap_memory_max | The maximum Java heap memory available. Type: float | (gauge) Unit: digital,B |
| loaded_classes | Number of classes currently loaded. Type: float | (gauge) Unit: count |
| non_heap_memory | The total Java non-heap memory used. Non-heap memory is: Metaspace + CompressedClassSpace + CodeCache.Type: float | (gauge) Unit: digital,B |
| non_heap_memory_committed | The total Java non-heap memory committed to be used. Type: float | (gauge) Unit: digital,B |
| non_heap_memory_init | The initial Java non-heap memory allocated. Type: float | (gauge) Unit: digital,B |
| non_heap_memory_max | The maximum Java non-heap memory available. Type: float | (gauge) Unit: digital,B |
| os_open_file_descriptors | The number of file descriptors used by this process (only available for processes run as the dd-agent user) Type: float | (gauge) Unit: count |
| peak_thread_count | The peak number of live threads. Type: float | (count) Unit: count |
| thread_count | The number of live threads. Type: float | (count) Unit: count |
| total_thread_count | The number of total threads. Type: float | (count) Unit: count |
jmx¶
| Tags & Fields | Description |
|---|---|
| host ( tag) |
Host name. |
| instance ( tag) |
Instance name. |
| jmx_domain ( tag) |
JMX domain. |
| metric_type ( tag) |
Metric type. |
| name ( tag) |
Type name. |
| runtime-id ( tag) |
Runtime id. |
| service ( tag) |
Service name. |
| type ( tag) |
Object type. |
| gc_cms.count | The total number of garbage collections that have occurred. Type: float | (count) Unit: count |
| gc_major_collection_count | The rate of major garbage collections. Type: float | (gauge) Unit: count |
| gc_major_collection_time | The fraction of time spent in major garbage collection. Set new_gc_metrics: true to receive this metric. Type: float | (gauge) Unit: PPM |
| gc_minor_collection_count | The rate of minor garbage collections. Type: float | (gauge) Unit: count |
| gc_minor_collection_time | The fraction of time spent in minor garbage collection. Set new_gc_metrics: true to receive this metric. Type: float | (gauge) Unit: PPM |
| gc_parnew.time | The approximate accumulated garbage collection time elapsed. Type: float | (gauge) Unit: time,ms |
| heap_memory | The total Java heap memory used. Type: float | (gauge) Unit: digital,B |
| heap_memory_committed | The total Java heap memory committed to be used. Type: float | (gauge) Unit: digital,B |
| heap_memory_init | The initial Java heap memory allocated. Type: float | (gauge) Unit: digital,B |
| heap_memory_max | The maximum Java heap memory available. Type: float | (gauge) Unit: digital,B |
| non_heap_memory | The total Java non-heap memory used. Non-heap memory is calculated as follows: 'Metaspace' + CompressedClassSpace + CodeCache Type: float | (gauge) Unit: digital,B |
| non_heap_memory_committed | The total Java non-heap memory committed to be used. Type: float | (gauge) Unit: digital,B |
| non_heap_memory_init | The initial Java non-heap memory allocated. Type: float | (gauge) Unit: digital,B |
| non_heap_memory_max | The maximum Java non-heap memory available. Type: float | (gauge) Unit: digital,B |
| thread_count | The number of live threads. Type: float | (count) Unit: count |
ddtrace¶
| Tags & Fields | Description |
|---|---|
| endpoint ( tag) |
Endpoint. |
| host ( tag) |
Host name. |
| lang ( tag) |
Lang type. |
| lang_interpreter ( tag) |
Lang interpreter. |
| lang_interpreter_vendor ( tag) |
Lang interpreter vendor. |
| lang_version ( tag) |
Lang version. |
| metric_type ( tag) |
Metric type. |
| priority ( tag) |
Priority. |
| service ( tag) |
Service name. |
| stat ( tag) |
Stat. |
| tracer_version ( tag) |
Tracer version. |
| tracer_agent_discovery_time | Tracer agent discovery time. Type: float | (gauge) Unit: time,ms |
| tracer_api_errors_total | Tracer api errors total. Type: float | (gauge) Unit: count |
| tracer_api_requests_total | Tracer api requests total. Type: float | (gauge) Unit: count |
| tracer_flush_bytes_total | Tracer flush bytes total. Type: float | (gauge) Unit: count |
| tracer_flush_traces_total | Tracer flush traces total. Type: float | (gauge) Unit: count |
| tracer_queue_enqueued_bytes | Tracer queue enqueued bytes. Type: float | (gauge) Unit: count |
| tracer_queue_enqueued_spans | Tracer queue enqueued spans. Type: float | (gauge) Unit: count |
| tracer_queue_enqueued_traces | Tracer queue enqueued traces. Type: float | (gauge) Unit: count |
| tracer_queue_max_length | Tracer queue max length. Type: float | (gauge) Unit: count |
| tracer_scope_activate_count | Tracer scope activate count. Type: float | (gauge) Unit: count |
| tracer_scope_close_count | Tracer scope close count. Type: float | (gauge) Unit: count |
| tracer_span_pending_created | Tracer span pending created. Type: float | (gauge) Unit: count |
| tracer_span_pending_finished | Tracer span pending finished. Type: float | (gauge) Unit: count |
| tracer_trace_agent_discovery_time | Tracer trace agent discovery time. Type: float | (gauge) Unit: count |
| tracer_trace_agent_send_time | Tracer trace agent send time. Type: float | (gauge) Unit: count |
| tracer_trace_pending_created | Tracer trace pending created. Type: float | (gauge) Unit: count |
| tracer_tracer_trace_buffer_fill_time | Tracer trace buffer fill time. Type: float | (gauge) Unit: count |