Spark

Spark는 대규모 데이터 처리를 위한 분산 컴퓨팅 엔진입니다. 수집 범위에는 Master의 클러스터 스케줄링 상태, Worker의 리소스 용량, Driver의 스케줄링 상태, Executor의 메모리, GC 및 Shuffle I/O가 포함됩니다.

구성¶

사전 요구 사항¶

• A Spark Standalone cluster is deployed;
• DataKit can access the /metrics endpoints exposed by Spark JVMs;
• A JMX Exporter Java Agent compatible with the JVM version is downloaded;
• Exporter ports are exposed only on trusted networks.

JMX Exporter 배포¶

Use separate Exporter endpoints and rule files for Masters, Workers, Drivers, and Executors. Rules must match the target Spark MBeans and preserve resource ownership tags. When multiple Workers or Executors run on one host, use worker_id and executor_id to distinguish logical instances instead of relying only on host.

Add the Java Agent to the Master and Worker startup options. The following uses the Master and Worker-1 ports from the example mapping:

# Master
export SPARK_MASTER_OPTS="$SPARK_MASTER_OPTS -javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=28099:/opt/jmx/spark-master.yml"

# Worker
export SPARK_WORKER_OPTS="$SPARK_WORKER_OPTS -javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=28199:/opt/jmx/spark-worker.yml"

When submitting an application, add the Java Agent for the Driver 및 Executor:

spark-submit \
  --conf 'spark.driver.extraJavaOptions=-javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=24099:/opt/jmx/spark-driver.yml' \
  --conf 'spark.executor.extraJavaOptions=-javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=28100:/opt/jmx/spark-executor.yml' \
  ...

After restarting the relevant Spark processes, confirm the endpoints are reachable from the DataKit host:

curl -fsS http://127.0.0.1:28099/metrics | head
curl -fsS http://127.0.0.1:28199/metrics | head

DataKit Prometheus 수집기 구성¶

Open conf.d/prom under the DataKit installation directory and copy prom.conf.sample to spark.conf. One configuration file can contain multiple [[inputs.prom]] collection blocks; centrally manage Master, Worker, Driver, and Executor endpoints.

The following table is a localhost port-mapping example for a three-Worker test cluster. In production, replace it with actual service or service-discovery addresses; the number of Executor endpoints changes with application state.

Example spark.conf:

# Master
[[inputs.prom]]
  urls = ["http://127.0.0.1:28099/metrics"]
  source = "spark-master"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "master"
    collector = "jmx-exporter"

# Worker-1; copy this block for Worker-2 and Worker-3 and change url and worker_id
[[inputs.prom]]
  urls = ["http://127.0.0.1:28199/metrics"]
  source = "spark-worker"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "worker"
    collector = "jmx-exporter"
    worker_id = "worker-1"

# Driver
[[inputs.prom]]
  urls = ["http://127.0.0.1:24099/metrics"]
  source = "spark-driver"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "driver"
    collector = "jmx-exporter"

# Executor-1; copy this block for Executor-2 and Executor-3 and change url, worker_id, and executor_id
[[inputs.prom]]
  urls = ["http://127.0.0.1:28100/metrics"]
  source = "spark-executor"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "executor"
    collector = "jmx-exporter"
    worker_id = "worker-1"
    executor_id = "1"

urls points to JMX Exporter metric endpoints; source identifies collectors; and measurement_name writes metrics to the spark measurement. Maintain collection blocks for dynamically created or stopped Executors through service discovery or configuration management, and avoid using one static executor_id for multiple instances.

On the DataKit host, first verify that the endpoints return the expected spark_* metrics:

curl -fsS http://127.0.0.1:28099/metrics | rg '^spark_master_'
curl -fsS http://127.0.0.1:24099/metrics | rg '^spark_driver_'

After configuration, restart DataKit as described in DataKit service management.

메트릭¶

Spark metrics are written to the spark measurement by default. Actual fields are determined by JMX Exporter rules; the following are commonly used operational fields.

클러스터 및 Worker 용량¶

Driver 스케줄링 상태¶

Executor 리소스 및 I/O¶

*_total indicates a cumulative counter. To show throughput or rate, first ensure the counter is continuously reported, then calculate it with rate(); do not place cumulative volume, rate, memory, and task counts on the same axis.