s3athena

s3athena

This example is more specific example for production use case. In this scenario, We want to permanently store the access logs of our application. The access logs should be searchable when needed. We also need to be able to query for certain condition against our access logs.

Architecture overview

A brief description of the resources illustrated in the diagram above.

• Fluent Bit; A log forwarder.
• Amazon S3; A cloud object storage.
• AWS Glue; A serverless ETL service.
• Amazon Athena; A serverless query service that makes it easy to analyze data directly in S3 using standard SQL.

Prerequisites

• Golang 1.17+
• Environments to run docker compose(like Docker Desktop)
• Amazon S3 bucket for your log files
• AWS identity that has the following permissions
  • s3:PutObject
  • glue:CreateDatabase
  • glue:StartCrawler
  • glue:CreateCrawler
• AWS IAM Role that must provide permissions similar to the AWS managed policy, AWSGlueServiceRole, plus access to your data stores.

Step by step tutorial

1. You need to set ENV AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY.
2. In fluent-bit.conf, Replace ${YOUR_S3_BUCKET_NAME_FOR_LOG_FILES}, ${YOUR_SERVICE_NAME} with your owns.
3. Run docker compose up.
4. In another terminal, run go run main.go
5. In another terminal, curl localhost:3000/ping

If worked successfully, the following logs in output from the fluent-bit container. Depending on the value you set, the log may be slightly different.

fluent-bit_1  | [2021/12/07 15:40:18] [ info] [output:s3:s3.0] Successfully uploaded object /example/alpha/year=2021/month=12/day=07/hour=15/minute=40/thS0Tjuj

1. If you see the log output as above, see your S3 bucket and check if the logs are stored properly.

2. (If it is already exist, please skip) Creates a new AWS Glue database.

aws glue create-database --region ap-northeast-2 --database-input "{\"Name\":\"access_log\"}"

1. Creates a new AWS Glue crawler(The role is noted in Prerequisites).

aws glue create-crawler \
--region ap-northeast-2 \
--name example \
--role service-role/AWSGlueServiceRole-example \
--database-name example \
--targets "{\"S3Targets\":[{\"Path\":\"s3://YOUR_S3_BUCKET_NAME_FOR_LOG_FILES/example/alpha\"}]}" \
--recrawl-policy "{\"RecrawlBehavior\":\"CRAWL_NEW_FOLDERS_ONLY\"}" \
--schema-change-policy "{\"UpdateBehavior\":\"LOG\",\"DeleteBehavior\": \"LOG\"}"  

1. Starts the crawler.

aws glue start-crawler --region ap-northeast-2 --name example

If the crawling has been completed successfully, you can check the created data catalog table in the AWS Glue console. You can now run queries on tables created through Amazon Athena.

References

To optimize logging settings for your own services, please refer to the following materials.

• Build a Data Lake Foundation with AWS Glue and Amazon S3
• Top 10 Performance Tuning Tips for Amazon Athena
• Forward - Fluent Bit: Official Manual
• Amazon S3 - Fluent Bit: Official Manual
• github.com/fluent/fluent-logger-golang
• github.com/rs/zerolog