AWS EMR Serverless
Collect AWS EMR Serverless monitoring data.
Configuration¶
Install Func¶
We recommend enabling hosted Func under Integration - Extensions in Guance. All prerequisites are installed automatically; continue with the script installation.
If deploying Func manually, refer to Manual Deployment of Func
Activate Script¶
Note: Prepare an AWS AK that meets the requirements. For simplicity, you can grant the global read-only permission
ReadOnlyAccessdirectly.
Activate Scripts in Hosted Func¶
- Log in to the Guance console
- Click the 【Integration】 menu and select 【Cloud Account Management】
- Click 【Add Cloud Account】, select 【AWS】, and fill in the required information on the interface. If you have already configured the cloud account information before, ignore this step
- Click 【Test】, and if the test is successful, click 【Save】. If the test fails, please check if the relevant configuration information is correct and test again
- Click 【Cloud Account Management】, and you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page
- Click 【Integration】 on the cloud account details page, find
AWS EMR Serverlessin theNot Installedlist and click 【Install】 to open the installation form.
Manual Activation Script¶
- Log in to the Func console, click 【Script Market】, open the Guance script market and search for
integration_aws_emr_serverless. - Click 【Install】 and enter AWS AK ID, AK Secret, and the account name.
- Click 【Deploy Startup Script】 to create the
Startupscript set and configure its startup scripts automatically. Setregionsto the actual regions andmetricsincloudwatch_configs.targetsto the AWS native metric names to collect. Then publish and enable automatic triggering. - After enabling the script, open 【Management / Automatic Trigger Configuration】 to view its automatic trigger configuration. Click 【Execute】 to run it immediately without waiting for the scheduled time. After a short wait, review the task records and logs.
Verification¶
- Under 【Management / Automatic Trigger Configuration】, confirm that the task has an automatic trigger configuration. Review its task records and logs for errors.
- In Guance, check [Infrastructure / Custom] for asset information.
- In Guance, check 【Metrics】 for the corresponding monitoring data.
Metrics¶
After configuring AWS CloudWatch, the default metrics are listed below. You can configure additional metrics. AWS EMR Serverless metric details
The measurement is aws_emrserverless.
| MetricName | MetricDescribe | Dimensions | Unit |
|---|---|---|---|
max_cpuallowed_maximum |
Maximum vCPUs allowed for the application | region_id, application_id, application_name | vCPU |
max_memory_allowed_maximum |
Maximum memory allowed for the application | region_id, application_id, application_name | GB |
max_storage_allowed_maximum |
Maximum storage allowed for the application | region_id, application_id, application_name | GB |
cpuallocated_average |
vCPUs allocated to the application | region_id, application_id, application_name, worker_type, capacity_allocation_type | vCPU |
idle_worker_count_average |
Idle workers in the application | region_id, application_id, application_name, worker_type, capacity_allocation_type | count |
memory_allocated_average |
Memory allocated to the application | region_id, application_id, application_name, worker_type, capacity_allocation_type | GB |
pending_creation_worker_count_average |
Workers pending creation in the application | region_id, application_id, application_name, worker_type, capacity_allocation_type | count |
running_worker_count_average |
Workers running in the application | region_id, application_id, application_name, worker_type, capacity_allocation_type | count |
storage_allocated_average |
Disk storage allocated to the application | region_id, application_id, application_name, worker_type, capacity_allocation_type | GB |
total_worker_count_average |
Total workers available to the application | region_id, application_id, application_name, worker_type, capacity_allocation_type | count |
submitted_jobs_average |
Number of jobs in the Submitted state | region_id, application_id, application_name | count |
pending_jobs_average |
Number of jobs in the Pending state | region_id, application_id, application_name | count |
scheduled_jobs_average |
Number of jobs in the Scheduled state | region_id, application_id, application_name | count |
running_jobs_average |
Number of jobs in the Running state | region_id, application_id, application_name | count |
success_jobs_average |
Number of jobs in the Success state | region_id, application_id, application_name | count |
failed_jobs_average |
Number of jobs in the Failed state | region_id, application_id, application_name | count |
cancelling_jobs_average |
Number of jobs in the Cancelling state | region_id, application_id, application_name | count |
cancelled_jobs_average |
Number of jobs in the Cancelled state | region_id, application_id, application_name | count |
worker_cpu_allocated_sum |
Total vCPUs allocated to workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | vCPU |
worker_cpu_used_sum |
Total vCPUs used by workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | vCPU |
worker_memory_allocated_sum |
Total memory allocated to workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | GB |
worker_memory_used_sum |
Total memory used by workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | GB |
worker_ephemeral_storage_allocated_sum |
Ephemeral storage allocated to workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | GB |
worker_ephemeral_storage_used_sum |
Ephemeral storage used by workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | GB |
worker_storage_read_bytes_sum |
Bytes read from storage by workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | B |
worker_storage_write_bytes_sum |
Bytes written to storage by workers in the job | region_id, job_id, job_name, application_id, application_name, worker_type, capacity_allocation_type | B |
failed_sessions_average |
Number of sessions in the Failed state | region_id, application_name, application_id | - |
started_sessions_average |
Number of sessions in the Started state | region_id, application_name, application_id | - |
starting_sessions_average |
Number of sessions in the Starting state | region_id, application_name, application_id | - |
submitted_sessions_average |
Number of sessions in the Submitted state | region_id, application_name, application_id | - |
terminated_sessions_average |
Number of sessions in the Terminated state | region_id, application_name, application_id | - |
terminating_sessions_average |
Number of sessions in the Terminating state | region_id, application_name, application_id | - |