How to query AWS Cloudwatch logs using Lambda and generate an alarm
Amazon CloudWatch is a management service and a monitoring tool that provides insights and data for AWS, Hybrid, and on-premises application and infrastructure resources. It provides a single platform where you can accumulate and access all your performance and operation data in the form of metrics and logs.
At Dae.mn, we have several projects where we are using Cloudwatch as our monitoring tool. Cloudwatch enables us to have visibility on our application performance and respond quickly to any issue.
Recently, I have had an opportunity to work on one of the Dae.mn internal project where we needed to analyse the application logs to generate an alarm. Since Cloudwatch is event-driven, we needed a mechanism that would explore the logs for the day and check for error that has occurred and deliver the error messages to the development team.
I have used several services from AWS such as Cloudwatch, Lambda, and SNS to accomplish the above use case. The diagram below illustrates how the components connect to form a solution.
To make the solution work, I have used Amazon Cloudwatch Event rule to schedule a Lambda invocation. Once the Lambda function is executed, it would go and query the application logs from Cloudwatch for the given time range. If any error message found in the logs, it is captured and formatted in a human-readable manner and sent to the development team via SNS.
This article assumes you have prior knowledge about AWS Service as I will not describe them in details. To follow this article, you would need to have a Cloudwatch log group created in AWS with logs available in it.
Prerequisites
- Prior knowledge of AWS Services
- Prior Knowledge of Python
Now that you know the use case background let’s go ahead and build the solution. This article is divided into three parts:
- Part 1: Create an SNS topic
- Part 2: Create a Lambda function Using Python
- Part 3: Create Cloudwatch Event rule.
The source code of this article is available in the following repo: https://github.com/DevMasterPro/lambdaCloudwatchLogQuery.git
Part 1: Create an SNS Topic
This part of the article will demonstrate how you can create an SNS topic and subscription to send emails to end-users.
Step1: Go to SNS, and create a Topic.
Once you fill in the Name and Description, click the create topic button.
Step 2: Now let’s create a subscription. Make sure to select Protocol as Email and use the desired email address as the Endpoint. Once you are happy with all the information, click the Create Subscription button to create a subscription.
Step 3: An email is sent to your inbox; make sure to confirm the subscription.
Note: Take note of the topic ARN as this will be needed in the second part of this article when creating the lambda function.
Part 2: Creating a Lambda function to query Cloudwatch logs
This part of the article will demonstrate how you can create a lambda function to query Cloudwatch logs.
Step1: Create a role that the lambda function will use to talk to other services in AWS. Please ensure the role has access to Cloudwatch and SNS and Lambda execution rights. For simplicity, I have created a role with Administrative access for this article.
On the next stage, add permission to the role.
Add the necessary tags.
Once you fill in the information and create the role.
Now that you have the role created, let's go ahead and create the lambda function.
Step2: Go to lambda and create a lambda function using the role you have created in Step1. Please ensure to use runtime as Python 3.7.
I have written a small script in python using boto3 that would help you to query the Cloudwatch logs and generate a notification based on the result. Copy and paste the following script in your lambda function.
The above script will query the Cloudwatch log group based on the query string. The query string used in this script is filter @message like /ERROR/ and @message like /Document/. It will go through all the logs for the given time range and check for the word Error and document. If a match is found, the message is captured, and an email is sent to the development team.
When running the lambda function, you would need to pass the following payload.
Once you have everything configured correctly, you can run your lambda function with the above payload to query your logs for errors. I have run the newly created Lambda function and received the following email.
I have used the AWS boto3 libraries to interact with AWS services from lambda. You can find more details of the function used in this script in the following links:
Part 3: Create Cloudwatch Event
Now that you have created the Lambda function lets go ahead and make the Cloudwatch Event rule to schedule your Lambda function to run daily.
Step1: Go to Cloudwatch Event -> Event -> Rules -> Create Rule
The above screenshot illustrates I have used Cron expression to schedule the lambda function. Used the Lambda function I have created in Part 2 as the Target. And selected Input transform to pass the payload.
Provide a name and description for your rule and press the create rule button.
I have demonstrated how you can use Lambda to query AWS Cloudwach log and generate an alert. I hope you found this article useful. If you have any further query, you can connect with me on Linkedin →
