Hadoop Yarn NodeManager¶

Collect Yarn NodeManager Metrics information.

Installation and Deployment¶

Since NodeManager is developed in Java, it can use the jmx-exporter method to collect metrics information.

1. NodeManager Configuration¶

1.1 Download jmx-exporter¶

Download address: https://github.com/prometheus/jmx_exporter

1.2 Download jmx Script¶

Download address: https://github.com/lrwh/jmx-exporter/blob/main/hadoop-yarn-nodemanager.yml

1.3 Adjust NodeManager Startup Parameters¶

Add to nodemanager startup parameters:

{JAVA_GC_ARGS} -javaagent:/opt/guance/jmx/jmx_exporter-1.0.1.jar=localhost:17108:/opt/guance/jmx/jmx_node_manager.yml

1.4 Restart NodeManager¶

2. DataKit Collector Configuration¶

2.1 Install DataKit¶

2.2 Configure Collector¶

The jmx-exporter can directly expose metrics URL, so you can collect data using the prom collector.

Go to the DataKit installation directory under conf.d/prom, copy prom.conf.sample to nodemanager.conf.

cp prom.conf.sample nodemanager.conf

Adjust the content of nodemanager.conf as follows:

  urls = ["http://localhost:17108/metrics"]
  source = "yarn-nodemanager"
  [inputs.prom.tags]
    component = "yarn-nodemanager" 
  interval = "10s"

Other configurations can be adjusted as needed

Parameter adjustment explanation:

• urls: The jmx-exporter metrics address, fill in the metrics URL exposed by the corresponding component.
• source: Alias for the collector, it's recommended to differentiate them.
• keep_exist_metric_name: Keep metric names unchanged.
• interval: Collection interval.
• inputs.prom.tags: Add extra tags.

3. Restart DataKit¶

Restart DataKit

Metrics¶

Hadoop Metrics Set¶

NodeManager metrics are located under the Hadoop Metrics set. This section mainly introduces the related metrics for NodeManager.