Guance Collection of Amazon ECS Logs¶
Introduction¶
Amazon Elastic Container Service (Amazon ECS) is a highly scalable and fast container management service that can easily run, stop, and manage containers on clusters. These containers can run on their own EC2 servers or on serverless infrastructure managed by AWS Fargate.
For tasks using the Fargate launch type, it is necessary to start the awslogs log driver. The logs output by applications running in the container via STDOUT and STDERR I/O streams will be sent to the log group in CloudWatch Logs, then these logs are collected through Func. Func then writes the logs into Guance via DataKit deployed on EC2.
This article focuses on collecting logs from containers managed by AWS Fargate.
Prerequisites¶
- First, create a Guance account
- Install DataKit
- Install Func portable version
- Already have a Java application running on ECS
The ECS cluster name used here is cluster-docker. Below, we check the example logs and log groups. Log in to «AWS», enter «Elastic Container Service» - click «Clusters» - «cluster-docker».
Click «Service Name»
Enter the task
Find the log configuration under the Containers section in the Details tab.
Click «Log Tags». Inside are the application logs; next, we collect these logs.
Procedures¶
Warning
The version used in the example is DataKit 1.4.18
Step 1 AWS Configuration¶
1.1 User Key¶
Use the account deploying ECS, provided during AWS user creation Access key ID and Secret access key will be used later.
1.2 Set AWS User Permissions¶
Log in to the AWS IAM console, find the ECS «User» under Users - click «Add Permissions».
Click «Directly Attach Existing Policies», select CloudWatchLogsReadOnlyAccess, CloudWatchEventsReadOnlyAccess, then click «Next: Review».
Step 2 Func Configuration¶
2.1 Configure Environment Variables¶
Log in to «Func» - «Development» - «Environment Variables» - «Add Environment Variable». Add three environment variables here:
AWS_LOG_KEYcorresponds to Step 1.1 AWS user'sAccess key IDAWS_LOG_SECRET_ACCESS_KEYcorresponds to Step 1.1 AWS user'sSecret access keyAWS_REGION_NAMEcorresponds to theREGIONwhere the AWS user is located.
2.2 Configure Connector¶
Log in to «Func» - «Development» - «Connectors» - «Add Connector».
Here, the ID must be DataKit, the host corresponds to the address where DataKit is installed, and the port is DataKit's port. (In this example, IP is directly used, so the protocol is HTTP)
Click «Test Connectivity», if 
returns, it indicates DataKit is available.
2.3 PIP Tool Configuration¶
Log in to «Func» - «Management» - «Experimental Features», select «Enable PIP Tool Module» on the right side.
Click «PIP Tool» on the left - choose «Alibaba Cloud Mirror» - input boto3 - click «Install».
2.4 Script Library¶
Log in to «Func» - «Development» - «Script Library» - «Add Script Set», the ID can be customized, click «Save».
Find «AWS Log Collection» - click «Add Script».
Input ID, for this example define as aws_ecs__log, click «Save».
Click «Edit».
Input the following content:
Input Content
import boto3
import json
import time
scope_id='ecs_log'
@DFF.API('aws_ecs log', timeout=500, api_timeout=180)
def run(measurement, logGroupName, interval):
print(measurement, logGroupName, interval)
get_log_data(measurement, logGroupName, interval)
# if data is not None:
# push_log(data)
# else:
# print("None")
def get_cron_time(interval, measurement):
cache = DFF.CACHE.get('last_time_%s' %measurement,scope=scope_id)
if cache == None:
currentTime = int(round(time.time() * 1000))
startTime = currentTime - int(interval) * 1000
endTime = currentTime
else:
currentTime = int(round(time.time() * 1000))
if currentTime - int(cache) > 10 * 60 * 1000:
startTime = currentTime - int(interval) * 1000
endTime = currentTime
else:
startTime = int(cache) + 1
endTime = currentTime
print(startTime, endTime)
return startTime, endTime
def get_log_data(measurement, logGroupName, interval):
logTime = get_cron_time(interval, measurement)
startTime = logTime[0]
endTime = logTime[1]
isPush = False
client = boto3.client(
'logs',
aws_access_key_id=DFF.ENV('AWS_LOG_KEY'),
aws_secret_access_key=DFF.ENV('AWS_LOG_SECRET_ACCESS_KEY'),
region_name=DFF.ENV('AWS_REGION_NAME')
)# print(client.meta.config)
try:
nextToken = 'frist'
logData = []
while nextToken != '':
if nextToken == 'frist':
nextToken = ''
response = client.filter_log_events(
logGroupName=logGroupName,
startTime=startTime,
endTime=endTime,
limit=1000,
#filterPattern="?ERROR ?WARN ?error ?warn",
interleaved=False
)
else:
response = client.filter_log_events(
logGroupName=logGroupName,
startTime=startTime,
endTime=endTime,
nextToken=nextToken,
limit=1000,
#filterPattern="?ERROR ?WARN ?error ?warn",
interleaved=False
)
try:
if len(response['events']) > 0:
data = []
lastTimeList = []
for i in response['events']:
# print("hii", i['logStreamName'])
log = {
'measurement': measurement,
'tags': {
'logGroupName': logGroupName,
'logStreamName': i['logStreamName'],
'host': '127.0.0.1'
},
'fields': {
'message': i['message'],
'time': i['timestamp']
}
}
data.append(log)
lastTimeList.append(i['timestamp'])
push_log(data)
print("max %s" % max(lastTimeList))
DFF.CACHE.set('last_time_%s' % measurement, max(lastTimeList), scope=scope_id, expire=None)
isPush = True
else:
DFF.CACHE.set('last_time_%s' % measurement, endTime , scope=scope_id, expire=None)
nextToken = response['nextToken']
except:
nextToken = ''
except Exception as e:
print('Error: %s' % e )
return None
if not isPush:
DFF.CACHE.set('last_time_%s' % measurement, endTime , scope=scope_id, expire=None)
def push_log(data):
datakit = DFF.SRC('datakit')
status_code, result = datakit.write_logging_many(data=data)
if status_code == 200:
print("total %d" % len(data))
print(status_code, result)
Warning
- Ensure the
ecs_logon line 4 of the above content is unique within the same Func; it can be changed to other letters. - Line 6's
awc_ecsis the script set ID added earlier. - Lines 40, 41, and 42's
AWS_LOG_KEY,AWS_LOG_SECRET_ACCESS_KEY, andAWS_REGION_NAMEcorrespond to the environment variable names in Step 2.1. If the environment variable names change, corresponding modifications are required.
2.5 Test Script¶
As shown below, select «run». In the second red box:
- Input
ecs_log_sourceformeasurement; this value corresponds to the log source in Guance logs; - Input the value for
logGroupNamecorresponding to theawslogs-groupfound in the «Prerequisites» log configuration; - Input the value for
intervalcorresponding to the collection frequency; in this example, it is 60 seconds.
Click «Execute», the output total 8 indicates eight logs reported.
Log in to «Guance», enter the «Logs» module, select «ecs_log_source» as the data source, and you can see the logs.
Click the «Publish» button at the top right corner
Click «End Editing» at the top right corner
2.6 Automatically Collect Logs¶
Log in to «Func» - «Management» - «Automatic Trigger Configuration» - «Create», input the parameters executed earlier.
Choose every minute or every 5 minutes, click «Save»
In the «Automatic Trigger Configuration» list, there is a record for «aws_ecs log»
Click «Recent Execution» to view execution details
