Hadoop Yarn ResourceManager¶

Collect metrics information from Yarn ResourceManager.

Installation and Deployment¶

Since ResourceManager is developed in the JAVA languages, it can use the jmx-exporter method to collect metrics information.

1. ResourceManager Configuration¶

1.1 Download jmx-exporter¶

Download address: https://github.com/prometheus/jmx_exporter

1.2 Download jmx Script¶

Download address: https://github.com/lrwh/jmx-exporter/blob/main/hadoop-yarn-resourcemanager.yml

1.3 Adjust ResourceManager Startup Parameters¶

Add to the startup parameters of resourcemanager:

{JAVA_GC_ARGS} -javaagent:/opt/guance/jmx/jmx_exporter-1.0.1.jar=localhost:17109:/opt/guance/jmx/jmx_resource_manager.yml

1.4 Restart ResourceManager¶

2. DataKit Collector Configuration¶

2.1 Install DataKit¶

2.2 Configure Collector¶

The jmx-exporter can directly expose metrics url, so it can be collected directly through the prom collector.

Enter the DataKit installation directory under conf.d/prom, copy prom.conf.sample as resourcemanager.conf.

cp prom.conf.sample resourcemanager.conf

Adjust the content of resourcemanager.conf as follows:

  urls = ["http://localhost:17109/metrics"]
  source ="yarn-resourcemanager"
  [inputs.prom.tags]
    component = "yarn-resourcemanager" 
  interval = "10s"

Other configurations should be adjusted as needed

, parameter adjustment description:

• urls: The jmx-exporter metrics address, fill in the corresponding component's exposed metrics url here.
• source: Collector alias, it is recommended to make distinctions.
• keep_exist_metric_name: Keep the metric name unchanged.
• interval: Collection interval.
• inputs.prom.tags: Add additional tags.

3. Restart DataKit¶

Restart DataKit

Metrics¶

Hadoop Measurement¶

ResourceManager metrics are located under the Hadoop Measurement set, this mainly introduces the descriptions for ResourceManager-related metrics.