log-parser

Log viewer

Description

This project is a log viewer for cloudfront logs. It allows to view the logs in a web interface and to filter the requests by project, date, status code, googlebot, etc.

It is an experimental project ported to Go for testing purpose and is not crafted for large log volume.

Steps

1. Download all cloudfront log files from an AWS S3 bucket
2. Aggregate the log files into a single file using the scripts\aggregate.sh script
3. Preprocess the main log file (split into multiple files)
4. Start the web interface and open http://localhost:8080

Usage

Build

make build

build/logs preprocess -s default -i var/23-05-25.log -o var

Development

Help

make

Preprocess log requests

# Create preprocessed log files for the 'default' project using the cloudfront log file 'var/23-05-25.log'
go run main.go preprocess -s default -i var/23-05-25.log -o var

Start the web interface

Autorebuild and restart the web interface when a file is changed. Require Air to be installed (go install github.com/cosmtrek/air@latest). Install air to automatically rebuild and restart the web interface when a file is changed.

make live

Enable profiling

pprof is disabled by default. To enalbe it, use the -i flag on the serve sub-command.

go run main.go serve -i

Open pprof Web UI

Resources

• Bulma CSS framework
• Tabulator JS table library
• Air Live reload for Go apps