AWS Lambda¶

The displayed Metrics for AWS Lambda include cold start time, execution time, concurrent executions, and memory usage. These Metrics reflect the response speed, scalability, and resource utilization of Lambda functions.

Configuration¶

Install Func¶

It is recommended to enable Guance integration - extension - DataFlux Func (Automata): all prerequisites are automatically installed, please continue with the script installation.

If you deploy Func yourself, refer to Self-deployed Func

Installation Script¶

Note: Please prepare an Amazon Cloud AK that meets the requirements in advance (for simplicity, you can directly grant CloudWatch read-only permission CloudWatchReadOnlyAccess).

Script for enabling managed version¶

1. Log in to Guance console.
2. Click on the 【Integration】 menu and select 【Cloud Account Management】.
3. Click 【Add Cloud Account】, choose 【AWS】, fill in the required information on the interface. If cloud account information has been configured before, skip this step.
4. Click 【Test】. After a successful test, click 【Save】. If the test fails, check whether the relevant configuration information is correct and retest.
5. In the 【Cloud Account Management】 list, you can see the added cloud account. Click the corresponding cloud account to enter the details page.
6. Click the 【Integration】 button on the cloud account details page. Under the Not Installed list, find AWS Lambda, click the 【Install】 button, and install via the installation interface.

Manual activation script¶

1. Log in to the Func console, click 【Script Market】, enter the official script market, and search for guance_aws_lambda.

2. After clicking 【Install】, input the corresponding parameters: AWS AK ID, AK Secret, and account name.

3. Click 【Deploy Startup Script】, the system will automatically create a Startup script set and configure the corresponding startup script automatically.

4. After enabling, you can see the corresponding automatic trigger configuration under 「Management / Automatic Trigger Configuration」. Click 【Execute】 to immediately execute once without waiting for the scheduled time. Wait a moment, and you can view the execution task records and corresponding logs.

Verification¶

1. In 「Management / Automatic Trigger Configuration」, confirm whether there is a corresponding automatic trigger configuration for the task. You can also check the task records and logs to verify if there are any anomalies.
2. In Guance, under 「Infrastructure / Custom」, check if there is asset information.
3. In Guance, under 「Metrics」, check if there are corresponding monitoring data.

Metrics¶

After configuring Amazon Cloud Monitoring, the default Measurement set is as follows. You can collect more Metrics through configuration:

Amazon Cloud Monitoring Lambda Metrics Details

Invocation Metrics¶

Performance Metrics¶

Performance Metrics provide detailed information about the performance of individual function invocations. For example, the Duration Metric indicates the amount of time the function spends processing an event (in milliseconds). To understand how quickly the function processes events, use Average or Max statistics to view these Metrics.

Concurrency Metrics¶

Lambda reports concurrency Metrics as the total number of instances handling events across functions, versions, aliases, or AWS regions. To see how close you are to the concurrency limit, use the Max statistic to view these Metrics.

Asynchronous Invocation Metrics¶

Asynchronous invocation Metrics provide details about asynchronous invocations from event sources and direct invocations. You can set thresholds and alerts to notify you of certain changes. For example, when the number of events queued for processing unexpectedly increases (AsyncEventsReceived), or when an event takes a long time to complete processing (AsyncEventAge).

Lambda Fetches S3 Log Data¶

Through AWS Lambda, fetch S3 data from AWS and report it to Guance logs.

For ELB log activation methods, refer to AWS ELB Log Collection

Step 1: Create a Lambda Function Using the Console¶

1. Open the Lambda console's function page.

2. Select "Create Function."

3. Choose "Author from scratch."

4. Enter a function name.

5. Set the Runtime option to Python 3.10.

6. In Execution Role (执行角色), select "Create a new role with basic Lambda permissions" (创建具有基本 Lambda 权限的新角色, see the appendix for specific permission lists; if one already exists, you can directly use a minimal permission role). Lambda creates an execution role that grants the function permission to upload logs to Amazon CloudWatch Logs. When you invoke the function, the Lambda function assumes the execution role and uses it to create credentials for the Amazon SDK and read data from event sources.

7. Click "Create Function."

8. Pull the synchronized code from GitHub and copy the content of lambda-forward.py into lambda-function.py.

9. Create setting.py, datakit.py, and dataway.py files in the same directory as lambda-function.py, and copy the respective file codes from GitHub into them.

10. Add environment variables:

    1. DATAKIT_IP: The IP address where datakit is deployed. The data source reported is datakit (required).
    2. DATAKIT_PORT: The service port of datakit. The data source reported is datakit (optional, default: 9529).

    3. GUANCE_NODE: Guance node. The data source reported is dataway (required). Selection range:

       GUANCE_NODE	Region Name	City
       default	China Region 1	Hangzhou
       aws	China Region 2	Ningxia
       cn4	China Region 4	Guangzhou
       cn6	China Region 6	Hong Kong
       us1	Americas Region 1	Oregon
       eu1	Europe Region 1	Frankfurt
       ap1	Asia-Pacific Region 1	Singapore
       za1	Africa Region 1	South Africa
       id1	Indonesia Region 1	Jakarta

    4. GUANCE_TOKEN: Guance workspace Token. The data source reported is dataway.

    Note: Either datakit or dataway must be selected as the data source. Configure DATAKIT_IP if selecting datakit, and configure GUANCE_NODE and GUANCE_TOKEN if selecting dataway.

11. If the datakit port is not the default 9529, add the environment variable DATAKIT_PORT and fill in the correct port address (this variable is optional).

12. Click Deploy to publish.

Step 2: Configure Lambda Triggers¶

1. Click Add Trigger.

2. Set Choose a source to S3.

3. Select the bucket to listen to.

4. Choose the Event types that will trigger the Lambda function.

5. Agree that I acknowledge it is not recommended to use the same S3 bucket for input and output, and this configuration may lead to recursive calls, increased Lambda usage, and increased costs.

6. Click Add.

Access ELB to Generate Logs¶

Access the application on the ELB, ELB uploads generated logs to S3, which are fetched and reported to Guance via the Lambda function.

Objects¶

The collected AWS Lambda object data structure can be seen in 「Infrastructure - Custom」.

{
  "measurement": "aws_lambda",
  "tags": {
    "account_name"      :"AWS_Lambda",
    "class"             :"aws_lambda",
    "cloud_provider"    :"aws",
    "FunctionName"      :"dataflux-alb",
    "name"              :"dataflux-alb",
    "PackageType"       :"Zip",
    "RegionId"          :"cn-northwest-1",
    "RevisionId"        :"5e52ff51-615a-4ecb-96b7-40083a7b4b62",
    "Role"              :"arn:aws-cn:iam::294654068288:role/service-role/s3--guance-role-3w34zo42",
    "Runtime"           :"python3.7",
    "Version"           :"$LATEST"
  },
  "fields": {
    "CreatedTime"         : "2022-03-09T06:13:31Z",
    "ListenerDescriptions": "{JSON data}",
    "AvailabilityZones"   : "{Availability Zones JSON data}",
    "message"             : "{Instance JSON data}"
  }
}

Note: Fields in tags and fields may change with subsequent updates.

Tip 1: The value of tags.account_name is the instance ID, used as a unique identifier.