AWS ELB¶

Collect ELB metrics and log information.

Configuration¶

Install Func¶

It is recommended to activate Guance Integration - Extensions - DataFlux Func (Automata): All prerequisites are automatically installed, please proceed with script installation.

If deploying Func manually, refer to Manual Deployment of Func

Install Script¶

Note: Please prepare the required Amazon AK in advance (for simplicity, you can grant global read-only permissions ReadOnlyAccess)

Activate Script in Automata¶

1. Log in to the Guance console
2. Click the [Integration] menu, select [Cloud Account Management]
3. Click [Add Cloud Account], select [AWS], and fill in the required information on the interface. If the cloud account information has been configured before, ignore this step
4. Click [Test], and after successful testing, click [Save]. If the test fails, please check if the relevant configuration information is correct and test again
5. Click the [Cloud Account Management] list to see the added cloud account, click the corresponding cloud account to enter the details page
6. Click the [Integration] button on the cloud account details page, find AWS ELB under the Not Installed list, click the [Install] button, and install it in the pop-up installation interface.

Manual Script Activation¶

1. Log in to the Func console, click 【Script Market】, enter the Guance script market, and search for integration_aws_elb, integration_aws_applicationelb, integration_aws_networkelb, integration_aws_gatewayelb.

2. Click 【Install】, then enter the corresponding parameters: AWS AK ID, AK Secret, and account name.

3. Click 【Deploy Startup Script】, the system will automatically create the Startup script set and configure the corresponding startup scripts.

4. After activation, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to execute once immediately without waiting for the scheduled time. After a while, you can check the execution task records and corresponding logs.

Verification¶

1. In 「Manage / Automatic Trigger Configuration」, confirm whether the corresponding task has the corresponding automatic trigger configuration. At the same time, you can check the corresponding task records and logs to see if there are any exceptions.
2. In Guance, check whether asset information exists in 「Infrastructure / Custom」.
3. In Guance, check whether there is corresponding monitoring data in 「Metrics」.

Metrics¶

After configuring Amazon Cloud Monitoring, the default measurement sets are as follows. More metrics can be collected through configuration:

Amazon Cloud Monitoring Application Load Balancer Metric Details

Amazon Cloud Monitoring Network Load Balancer Metric Details

Amazon Cloud Monitoring Gateway Load Balancer Metric Details

Amazon Cloud Monitoring Classic Load Balancer Metric Details

Application Load Balancer Metrics¶

AWS/ApplicationELB namespace includes the following metrics for targets.

AWS/ApplicationELB namespace includes the following metrics for target group health. For more information, see Target Group Health.

AWS/ApplicationELB namespace includes the following metrics for Lambda functions registered as targets.

AWS/ApplicationELB namespace includes the following metrics for user authentication.

Network Load Balancer Metrics¶

Gateway Load Balancer Metrics¶

Classic Load Balancer Metrics¶

Load Balancer Metric Dimensions¶

To filter metrics for Application / Network / Gateway load balancers, use the following dimensions.

To filter metrics for Classic load balancers, use the following dimensions.

Logging¶

ELB logs are not directly exposed for collection. You need to store the logs in an S3 bucket first, and then use Lambda to fetch the log data and report it to the platform.

Application ELB Log Activation¶

1. Select the corresponding load balancer and enter the details page
2. In the details page, click "Actions", then select "Edit Load Balancer Attributes"
3. Under the "Monitoring" category, enable "Access Logs" and select the corresponding S3 bucket.
4. Save

Lambda Configuration¶

Refer to: Lambda Fetch S3 Log Data

Log Pipeline¶

grok(_, "%{NOTSPACE:protocal} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:LoadBalancer} %{IP:client}:%{POSINT:client_port} %{IP:target}:%{POSINT:target_port} %{NUMBER:request_processing_time} %{NUMBER:target_processing_time} %{NUMBER:response_processing_time} %{NUMBER:elb_status_code} %{NUMBER:target_status_code} %{NUMBER:received_bytes} %{NUMBER:sent_bytes} \"%{NOTSPACE:http_method} %{NOTSPACE:uri} HTTP/%{NUMBER:http_version}\" \"%{DATA:user_agent}\" %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol} %{NOTSPACE:target_group_arn} \"%{NOTSPACE:x_amzn_trace_id}\" \"%{DATA:domain_name}\" \"%{DATA:chosen_cert_arn}\" %{NUMBER:matched_rule_priority} %{TIMESTAMP_ISO8601:request_creation_time} \"%{WORD:actions_executed}\" \"%{DATA:redirect_url}\" \"%{DATA:error_reason}\" \"%{IP:target_host}:%{POSINT:target_host_port_list}\" \"%{NUMBER:target_status_code_list}\" \"%{DATA:classification}\" \"%{DATA:classification_reason}\" %{NOTSPACE:conn_trace_id}")

status = "info"

cast(elb_status_code,"int")

if elb_status_code >=400 {
  status = "error"
}

add_key("aws_log_type","access_log")
add_key(status,status)
aws_json=load_json(aws)

lambda_func_data = aws_json["invoked_function_arn"]

add_key(s3_bucket,aws_json["s3"]["bucket"])

grok(lambda_func_data, "arn:aws:lambda:%{DATA:region}:%{NUMBER:account_id}:function:%{NOTSPACE:lambda_func}")

Object¶

The collected AWS ELB object data structure can be seen in [Infrastructure - Custom]

{
  "measurement": "aws_aelb",
  "tags": {
    "name"                 : "app/openway/8e8d762xxxxxx",
    "RegionId"             : "cn-northwest-1",
    "LoadBalancerArn"      : "arn:aws-cn:elasticloadbalancing:cn-northwest-1:588271xxxxx:loadbalancer/app/openway/8e8d762xxxxxx",
    "State"                : "active",
    "Type"                 : "application",
    "VpcId"                : "vpc-2exxxxx",
    "Scheme"               : "internet-facing",
    "DNSName"              : "openway-203509xxxx.cn-northwest-1.elb.amazonaws.com.cn",
    "LoadBalancerName"     : "openway",
    "CanonicalHostedZoneId": "ZM7IZAIxxxxxx"
  },
  "fields": {
    "CreatedTime"         : "2022-03-09T06:13:31Z",
    "ListenerDescriptions": "{JSON data}",
    "AvailabilityZones"   : "{Availability Zone JSON data}",
    "message"             : "{Instance JSON data}"
  }
}

Note: Fields in tags and fields may change with subsequent updates

Tip 1: AWS ELB metrics are divided into four types based on different load balancer types:

1. Application ELB corresponds to the metric set aws_aelb
2. Network ELB corresponds to the metric set aws_nelb
3. Gateway ELB corresponds to the metric set aws_gelb
4. Classic ELB corresponds to the metric set aws_elb

Tip 2: The tags.name value is determined in two ways:

1. For Classic Load Balancers, the LoadBalancerName field is used.
2. For Application, Network, and Gateway Load Balancers, the end part of the load balancer ARN (LoadBalancerArn) is used.

For example, for a Network Load Balancer:

`LoadBalancerArn` is `arn:awS-cn:elasticloadbalancing:cn-northwest-1:xxxx1335135:loadbalancer/net/k8s-forethou-kodongin-xxxxa46f01/xxxxe75ae81d08c2`

The corresponding tags.name is net/k8s-forethou-kodongin-xxxxa46f01/xxxxe75ae81d08c2

Tip 3:

• fields.message, tags.AvailabilityZones are JSON serialized strings
• The tags.state field indicates the state of the Load Balancers, with possible values: active, provisioning, active_impaired, failed (this field is not available for "classic" type load balancer instances)
• The tags.Type field indicates the type of Load Balancers, with possible values: application, network, gateway, classic
• The tags.Scheme field indicates the mode of Load Balancers, with possible values: internet-facing, internal
• The fields.ListenerDescriptions field is the list of listeners for this load balancer
• The fields.AvailabilityZones field indicates the Amazon Route 53 availability zone information associated with the load balancer