JVM

Here we provide two methods for collecting JVM metrics, one is DDTrace, and the other method is Jolokia (deprecated). The recommendations for choosing a method are as follows:

• It is recommended to use DDTrace for collecting JVM metrics; Jolokia can also be used but it is more complicated, so it is not recommended.
• If you are collecting metrics from your own developed Java application's JVM, the DDTrace method is recommended. In addition to collecting JVM metrics, it can also collect APM data.

Configuration¶

Collecting JVM Metrics via DDTrace¶

DataKit has a built-in StatsD collector to receive StatsD protocol data sent over the network. Here we use DDTrace to collect JVM metrics and send them to DataKit using the StatsD protocol.

[[inputs.statsd]]
  protocol = "udp"

  ## Address and port to host UDP listener on
  service_address = ":8125"

  ## separator to use between elements of a statsd metric
  metric_separator = "_"

  drop_tags = ["runtime-id"]
  metric_mapping = [
    "jvm_:jvm",
    "datadog_tracer_:ddtrace",
  ]

  # The following configurations do not need attention

  delete_gauges = true
  delete_counters = true
  delete_sets = true
  delete_timings = true

  ## Percentiles to calculate for timing & histogram stats
  percentiles = [50.0, 90.0, 99.0, 99.9, 99.95, 100.0]

  ## Parses tags in the datadog statsd format
  ## http://docs.datadoghq.com/guides/dogstatsd/
  parse_data_dog_tags = true

  ## Parses datadog extensions to the statsd format
  datadog_extensions = true

  ## Parses distributions metric as specified in the datadog statsd format
  ## https://docs.datadoghq.com/developers/metrics/types/?tab=distribution#definition
  datadog_distributions = true

  ## Number of UDP messages allowed to queue up, once filled,
  ## the statsd server will start dropping packets
  allowed_pending_messages = 10000

  ## Number of timing/histogram values to track per-measurement in the
  ## calculation of percentiles. Raising this limit increases the accuracy
  ## of percentiles but also increases the memory usage and cpu time.
  percentile_limit = 1000

  ## Max duration (TTL) for each metric to stay cached/reported without being updated.
  #max_ttl = "1000h"

  [inputs.statsd.tags]
  # some_tag = "your-tag-value"
  # some_other_tag = "your-other-tag-value"

Explanation of the above configuration:

• service_address is set to :8125, indicating that DDTrace sends JVM metrics to this target address.
• drop_tags drops the runtime-id tag because it may cause Time Series explosion. If you need this field, remove it from drop_tags.
• metric_mapping maps the original metrics from DDTrace with prefixes jvm_ and datadog_tracer_ to two categories: jvm and ddtrace.

Starting the Java Application¶

A viable deployment method for JVM is as follows:

java -javaagent:dd-java-agent.jar \
    -Ddd.profiling.enabled=true \
    -Ddd.logs.injection=true \
    -Ddd.trace.sample.rate=1 \
    -Ddd.service.name=my-app \
    -Ddd.env=staging \
    -Ddd.agent.host=localhost \
    -Ddd.agent.port=9529 \
    -Ddd.jmxfetch.enabled=true \
    -Ddd.jmxfetch.check-period=1000 \
    -Ddd.jmxfetch.statsd.host=127.0.0.1  \
    -Ddd.jmxfetch.statsd.port=8125 \
    -Ddd.version=1.0 \
    -jar your-app.jar

Note:

• For downloading the dd-java-agent.jar package, refer to here.

• It is recommended to specify the following fields:

  • service.name indicates which application the JVM data comes from.
  • env indicates which environment the JVM data comes from (e.g., prod/testing/preprod).

• Explanation of several options:

  • -Ddd.jmxfetch.check-period represents the collection frequency in milliseconds.
  • -Ddd.jmxfetch.statsd.host=127.0.0.1 specifies the connection address for the StatsD collector on DataKit.
  • -Ddd.jmxfetch.statsd.port=8125 specifies the UDP port for the StatsD collector on DataKit, defaulting to 8125.
  • -Ddd.trace.health.xxx configures the collection and sending of DDTrace metrics.
  • To enable APM, add the following parameters (DataKit HTTP address):
    • -Ddd.agent.host=localhost
    • -Ddd.agent.port=9529

After enabling, you can collect JVM metrics exposed by DDTrace.

Metrics¶

• Tags

Each metric includes the following tags (actual tags depend on Java startup parameters and StatsD configuration):

• Metric List

Key explanation for these metrics: gc_major_collection_count gc_minor_collection_count gc_major_collection_time gc_minor_collection_time:

These metrics are of type counter, meaning they are counters. During collection, each time a metric is collected, it is subtracted from the previous result and divided by time, indicating the rate of change per second rather than the actual value in the JVM MBean.

Collecting JVM Metrics via Jolokia¶

The JVM collector can gather many metrics through JMX and send them to Guance for analysis of Java runtime conditions.

Configuration¶

Prerequisites¶

Install or download Jolokia. The Jolokia jar file is already downloaded in the data directory under the DataKit installation directory. Start the Java application using the following command:

java -javaagent:/path/to/jolokia-jvm-agent.jar=port=8080,host=localhost -jar your_app.jar

Tested versions:

• JDK 20
• JDK 17
• JDK 11
• JDK 8

Navigate to the conf.d/jvm directory under the DataKit installation directory, copy jvm.conf.sample and rename it to jvm.conf. An example configuration is as follows:

[[inputs.jvm]]
  # default_tag_prefix      = ""
  # default_field_prefix    = ""
  # default_field_separator = "."

  # username = ""
  # password = ""
  # response_timeout = "5s"

  ## Optional TLS config
  # tls_ca   = "/var/private/ca.pem"
  # tls_cert = "/var/private/client.pem"
  # tls_key  = "/var/private/client-key.pem"
  # insecure_skip_verify = false

  ## Monitor Interval
  # interval   = "60s"

  # Add agents URLs to query
  urls = ["http://localhost:8080/jolokia"]

  ## Add metrics to read
  [[inputs.jvm.metric]]
    name  = "java_runtime"
    mbean = "java.lang:type=Runtime"
    paths = ["Uptime"]

  [[inputs.jvm.metric]]
    name  = "java_memory"
    mbean = "java.lang:type=Memory"
    paths = ["HeapMemoryUsage", "NonHeapMemoryUsage", "ObjectPendingFinalizationCount"]

  [[inputs.jvm.metric]]
    name     = "java_garbage_collector"
    mbean    = "java.lang:name=*,type=GarbageCollector"
    paths    = ["CollectionTime", "CollectionCount"]
    tag_keys = ["name"]

  [[inputs.jvm.metric]]
    name  = "java_threading"
    mbean = "java.lang:type=Threading"
    paths = ["TotalStartedThreadCount", "ThreadCount", "DaemonThreadCount", "PeakThreadCount"]

  [[inputs.jvm.metric]]
    name  = "java_class_loading"
    mbean = "java.lang:type=ClassLoading"
    paths = ["LoadedClassCount", "UnloadedClassCount", "TotalLoadedClassCount"]

  [[inputs.jvm.metric]]
    name     = "java_memory_pool"
    mbean    = "java.lang:name=*,type=MemoryPool"
    paths    = ["Usage", "PeakUsage", "CollectionUsage"]
    tag_keys = ["name"]

  [inputs.jvm.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

After configuring, restart DataKit.

Jolokia Metrics¶

By default, all collected data will append a global tag named host (the tag value is the hostname where DataKit is running), and additional tags can be specified in the configuration through [inputs.jvm.tags]:

 [inputs.jvm.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

java_runtime¶

• Tags

• Metric List

java_memory¶

• Tags

• Metric List

java_garbage_collector¶

• Tags

• Metric List

java_threading¶

• Tags

• Metric List

java_class_loading¶

• Tags

• Metric List

java_memory_pool¶

• Tags

• Metric List

Further Reading¶

• DDTrace Java Example
• SkyWalking
• OpenTelemetry Java Example