Skip to content

AWS EMR

Use the "Guance Cloud Sync" series script packages in the script market to synchronize cloud monitoring and cloud assets data to Guance

Configuration

Install Func

It is recommended to activate Guance Integration - Extension - Managed Func: All prerequisites are automatically installed. Please proceed with the script installation.

If you deploy Func yourself, refer to Self-deployed Func

Install Script

Note: Please prepare an Amazon AK that meets the requirements in advance (for simplicity, you can directly grant global read-only permission ReadOnlyAccess)

Managed Version Activation Script

  1. Log in to Guance Console
  2. Click on the 【Integration】 menu, select 【Cloud Account Management】
  3. Click 【Add Cloud Account】, choose 【AWS】, fill in the required information on the interface. If you have already configured cloud account information before, ignore this step.
  4. Click 【Test】, after the test succeeds, click 【Save】. If the test fails, check whether the relevant configuration information is correct and retest.
  5. In the 【Cloud Account Management】 list, you can see the added cloud account. Click the corresponding cloud account to enter the details page.
  6. Click the 【Integration】 button on the cloud account details page. Under the Not Installed list, find AWS EMR, click the 【Install】 button, and a pop-up installation interface will appear for installation.

Manual Activation Script

  1. Log in to the Func console, click 【Script Market】, enter the official script market, search: guance_aws_emr

  2. After clicking 【Install】, input the corresponding parameters: AWS AK ID, AK Secret, and account name.

  3. Click 【Deploy Startup Script】, the system will automatically create a Startup script set and automatically configure the corresponding startup script.

  4. After activation, you can see the corresponding automatic trigger configuration in "Management / Automatic Trigger Configuration". Click 【Execute】, and it can be executed immediately without waiting for the regular time. Wait a moment, and you can view the execution task records and corresponding logs.

Verification

  1. In "Management / Automatic Trigger Configuration", confirm whether the corresponding task has an automatic trigger configuration. You can also check the corresponding task records and logs for any anomalies.
  2. In Guance, "Infrastructure / Custom", check if there is asset information.
  3. In Guance, "Metrics" check if there is corresponding monitoring data.

Metrics

After configuring Amazon-Cloud Monitoring, the default metric sets are as follows. You can collect more metrics through configuration. Amazon Cloud Monitoring Metric Details

Metric Description
IsIdle Indicates that the cluster is no longer executing tasks but remains active and incurs charges. This metric is set to 1 if there are no running or pending tasks; otherwise, it is set to 0. The system checks this value every five minutes, and a value of 1 only indicates that the cluster was idle at the time of the check, not throughout the entire five-minute period. To avoid false alarms, you should raise an alert when multiple consecutive five-minute checks result in a value of 1. For example, raise an alert when this value remains 1 for thirty minutes or longer. Use Case: Monitor cluster performance Unit: Boolean
ContainerAllocated Number of resource containers allocated by ResourceManager. Use Case: Monitor cluster progress Unit: Count
ContainerReserved Number of reserved containers. Use Case: Monitor cluster progress Unit: Count
ContainerPending Number of containers in the queue that have not yet been allocated. Use Case: Monitor cluster progress Unit: Count
AppsCompleted Number of applications submitted to YARN and completed. Use Case: Monitor cluster progress Unit: Count
AppsFailed Number of applications submitted to YARN and failed to complete. Use Case: Monitor cluster progress, Monitor cluster health Unit: Count
AppsKilled Number of applications submitted to YARN and terminated. Use Case: Monitor cluster progress, Monitor cluster health Unit: Count
AppsPending Number of applications submitted to YARN and in a pending state. Use Case: Monitor cluster progress Unit: Count
AppsRunning Number of applications submitted to YARN and currently running. Use Case: Monitor cluster progress Unit: Count
AppsSubmitted Number of applications submitted to YARN. Use Case: Monitor cluster progress Unit: Count
CoreNodesRunning Number of core nodes in a running state. This metric's data points are reported only when the corresponding instance group exists. Use Case: Monitor cluster health Unit: Count
LiveDataNodes Percentage of data nodes receiving tasks from Hadoop. Use Case: Monitor cluster health Unit: Percentage
MRActiveNodes Number of nodes currently running MapReduce tasks or jobs. Equivalent to the YARN metric mapred.resourcemanager.NoOfActiveNodes. Use Case: Monitor cluster progress Unit: Count
MRLostNodes Number of nodes assigned to MapReduce marked as LOST status. Equivalent to the YARN metric mapred.resourcemanager.NoOfLostNodes. Use Case: Monitor cluster health, Monitor cluster progress Unit: Count
MRTotalNodes Number of nodes currently available for MapReduce jobs. Equivalent to the YARN metric mapred.resourcemanager.TotalNodes. Use Case: Monitor cluster progress Unit: Count
MRActiveNodes Number of nodes currently running MapReduce tasks or jobs. Equivalent to the YARN metric mapred.resourcemanager.NoOfActiveNodes. Use Case: Monitor cluster progress Unit: Count
MRRebootedNodes Number of available nodes that have been restarted and marked as "rebooted" status. Equivalent to the YARN metric mapred.resourcemanager.NoOfRebootedNodes. Use Case: Monitor cluster health, Monitor cluster progress Unit: Count
MRUnhealthyNodes Number of nodes available for MapReduce jobs marked as "unhealthy" status. Equivalent to the YARN metric mapred.resourcemanager.NoOfUnhealthyNodes. Use Case: Monitor cluster progress Unit: Count
MRDecommissionedNodes Number of nodes assigned to MapReduce applications marked as decommissioned status. Equivalent to the YARN metric mapred.resourcemanager.NoOfDecommissionedNodes. Use Case: Monitor cluster health, Monitor cluster progress Unit: Count
S3BytesWritten Number of bytes written to Amazon S3. This metric aggregates only MapReduce tasks and does not apply to other workloads on Amazon EMR. Use Case: Analyze cluster performance, Monitor cluster progress Unit: Count
S3BytesRead Number of bytes read from Amazon S3. This metric aggregates only MapReduce tasks and does not apply to other workloads on Amazon EMR. Use Case: Analyze cluster performance, Monitor cluster progress Unit: Count
HDFSUtilization Percentage of currently used HDFS storage. Use Case: Analyze cluster performance Unit: Percentage
TotalLoad Total number of concurrent data transfers. Use Case: Monitor cluster health Unit: Count
MemoryTotalMB Total memory in the cluster. Use Case: Monitor cluster progress Unit: Count
MemoryReservedMB Reserved memory. Use Case: Monitor cluster progress Unit: Count
HDFSBytesRead Number of bytes read from HDFS. This metric aggregates only MapReduce tasks and does not apply to other workloads on Amazon EMR. Use Case: Analyze cluster performance, Monitor cluster progress Unit: Count
HDFSBytesWritten Number of bytes written to HDFS. This metric aggregates only MapReduce tasks and does not apply to other workloads on Amazon EMR. Use Case: Analyze cluster performance, Monitor cluster progress Unit: Count
MissingBlocks Number of data blocks in HDFS with no replicas. These blocks may be corrupted. Use Case: Monitor cluster health Unit: Count
MemoryAvailableMB Available memory for allocation. Use Case: Monitor cluster progress Unit: Count
MemoryAllocatedMB Allocated memory to the cluster. Use Case: Monitor cluster progress Unit: Count
PendingDeletionBlocks Number of data blocks marked for deletion. Use Case: Monitor cluster progress, Monitor cluster health Unit: Count
UnderReplicatedBlocks Number of data blocks that need to be replicated one or more times. Use Case: Monitor cluster progress, Monitor cluster health Unit: Count
DfsPendingReplicationBlocks
CapacityRemainingGB Remaining HDFS disk capacity. Use Case: Monitor cluster progress, Monitor cluster health Unit: Count

Objects

The structure of collected AWS EMR object data can be seen in "Infrastructure - Custom"

{
  "measurement": "aws_emr",
  "tags": {
    "Id"                 : "xxxxx",
    "ClusterName"        : "xxxxx",
    "ClusterArn"         : "xxxx",
    "RegionId"           : "cn-north-1",
    "OutpostArn"         : "xxxx",
  },
  "fields": {
    "Status"               : "{Instance Status JSON Data}",
    "message"              : "{Instance JSON Data}"
  }
}

Note: Fields in tags and fields may change with subsequent updates

Feedback

Is this page helpful? ×