Hadoop HDFS DataNode¶

Collect HDFS datanode metric information.

Installation and deployment¶

Since DataNode is developed in Java language, it can collect metric information using jmx-exporter.

1. DataNode configuration¶

1.1 Download jmx-exporter¶

Download link：https://github.com/prometheus/jmx_exporter

1.2 Download jmx script¶

Download link：https://github.com/lrwh/jmx-exporter/blob/main/hadoop-hdfs-datanode.yml

1.3 DataNode startup parameter adjustment¶

Add startup parameters to the datanode

{{JAVA_GC_ARGS}} -javaagent:/opt/guance/jmx/jmx_exporter-1.0.1.jar=localhost:17106:/opt/guance/jmx/hadoop-hdfs-datanode.yml

1.4 Restart DataNode¶

2. DataKit Collector Configuration¶

2.1 Install DataKit¶

2.2 Configure collector¶

By using jmx exporter, the metrics URL can be directly exposed, so it can be collected directly through the prom collector.

Go to the installation directory of DataKit and copy prom.d/prom.sample to datanode.conf.

cp prom.conf.sample datanode.conf

Adjust the content of datanode.conf as follows:

  urls = ["http://localhost:17106/metrics"]
  source ="hdfs-datanode"
  [inputs.prom.tags]
    component = "hdfs-datanode" 
  interval = "10s"

Adjust other configurations as needed

，parameter adjustment instructions ：

• urls：jmx-exporter metric address, fill in the URL of the metric exposed by the corresponding component here
• source：Collector alias, it is recommended to make a distinction
• keep_exist_metric_name: Maintain metric name
• interval：Collection interval
• inputs.prom.tags: Add additional tags

3. Restart DataKit¶

Restart DataKit

Metric¶

Hadoop Metric Set¶

The DataNode metric is located under the Hadoop metric set, and here we mainly introduce the description of DataNode related metrics