Skip to content

AWS EMR Serverless

Collect AWS EMR Serverless monitoring data.

Configuration

Install Func

We recommend enabling hosted Func under Integration - Extensions in Guance. All prerequisites are installed automatically; continue with the script installation.

If deploying Func manually, refer to Manual Deployment of Func

Activate Script

Note: Prepare an AWS AK that meets the requirements. For simplicity, you can grant the global read-only permission ReadOnlyAccess directly.

Activate Scripts in Hosted Func

  1. Log in to the Guance console
  2. Click the 【Integration】 menu and select 【Cloud Account Management】
  3. Click 【Add Cloud Account】, select 【AWS】, and fill in the required information on the interface. If you have already configured the cloud account information before, ignore this step
  4. Click 【Test】, and if the test is successful, click 【Save】. If the test fails, please check if the relevant configuration information is correct and test again
  5. Click 【Cloud Account Management】, and you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page
  6. Click 【Integration】 on the cloud account details page, find AWS EMR Serverless in the Not Installed list and click 【Install】 to open the installation form.

Manual Activation Script

  1. Log in to the Func console, click 【Script Market】, open the Guance script market and search for integration_aws_emr_serverless.
  2. Click 【Install】 and enter AWS AK ID, AK Secret, and the account name.
  3. Click 【Deploy Startup Script】 to create the Startup script set and configure its startup scripts automatically. Set regions to the actual regions and metrics in cloudwatch_configs.targets to the AWS native metric names to collect. Then publish and enable automatic triggering.
  4. After enabling the script, open 【Management / Automatic Trigger Configuration】 to view its automatic trigger configuration. Click 【Execute】 to run it immediately without waiting for the scheduled time. After a short wait, review the task records and logs.

Verification

  1. Under 【Management / Automatic Trigger Configuration】, confirm that the task has an automatic trigger configuration. Review its task records and logs for errors.
  2. In Guance, check [Infrastructure / Custom] for asset information.
  3. In Guance, check 【Metrics】 for the corresponding monitoring data.

Metrics

After configuring AWS CloudWatch, the default metrics are listed below. You can configure additional metrics. AWS EMR Serverless metric details

The measurement is aws_emrserverless.

MetricName MetricDescribe Dimensions Unit
max_cpuallowed_maximum Maximum vCPUs allowed for the application region_id, application_id, application_name vCPU
max_memory_allowed_maximum Maximum memory allowed for the application region_id, application_id, application_name GB
max_storage_allowed_maximum Maximum storage allowed for the application region_id, application_id, application_name GB
cpuallocated_average vCPUs allocated to the application region_id, application_id, application_name, worker_type, capacity_allocation_type vCPU
idle_worker_count_average Idle workers in the application region_id, application_id, application_name, worker_type, capacity_allocation_type count
memory_allocated_average Memory allocated to the application region_id, application_id, application_name, worker_type, capacity_allocation_type GB
pending_creation_worker_count_average Workers pending creation in the application region_id, application_id, application_name, worker_type, capacity_allocation_type count
running_worker_count_average Workers running in the application region_id, application_id, application_name, worker_type, capacity_allocation_type count
storage_allocated_average Disk storage allocated to the application region_id, application_id, application_name, worker_type, capacity_allocation_type GB
total_worker_count_average Total workers available to the application region_id, application_id, application_name, worker_type, capacity_allocation_type count
submitted_jobs_average Number of jobs in the Submitted state region_id, application_id, application_name count
pending_jobs_average Number of jobs in the Pending state region_id, application_id, application_name count
scheduled_jobs_average Number of jobs in the Scheduled state region_id, application_id, application_name count
running_jobs_average Number of jobs in the Running state region_id, application_id, application_name count
success_jobs_average Number of jobs in the Success state region_id, application_id, application_name count
failed_jobs_average Number of jobs in the Failed state region_id, application_id, application_name count
cancelling_jobs_average Number of jobs in the Cancelling state region_id, application_id, application_name count
cancelled_jobs_average Number of jobs in the Cancelled state region_id, application_id, application_name count
worker_cpu_allocated_sum Total vCPUs allocated to workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type vCPU
worker_cpu_used_sum Total vCPUs used by workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type vCPU
worker_memory_allocated_sum Total memory allocated to workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type GB
worker_memory_used_sum Total memory used by workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type GB
worker_ephemeral_storage_allocated_sum Ephemeral storage allocated to workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type GB
worker_ephemeral_storage_used_sum Ephemeral storage used by workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type GB
worker_storage_read_bytes_sum Bytes read from storage by workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type B
worker_storage_write_bytes_sum Bytes written to storage by workers in the job region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type B
failed_sessions_average Number of sessions in the Failed state region_id, application_name, application_id -
started_sessions_average Number of sessions in the Started state region_id, application_name, application_id -
starting_sessions_average Number of sessions in the Starting state region_id, application_name, application_id -
submitted_sessions_average Number of sessions in the Submitted state region_id, application_name, application_id -
terminated_sessions_average Number of sessions in the Terminated state region_id, application_name, application_id -
terminating_sessions_average Number of sessions in the Terminating state region_id, application_name, application_id -

Feedback

Is this page helpful?