tailpipe

select * from logs;

Tailpipe is the lightweight, developer-friendly way to query logs.

Use SQL. It's been the data access standard for decades.

Run locally. Collect logs from diverse sources direct to your laptop.

Build huge collections. Hundreds of millions of log entries? No problem.

Query instantly. Slice through all those entries in seconds.

Demo time!

Documentation

See the documentation for:

• Getting started
• It's just SQL!
• Managing Tailpipe
• CLI commands

Install Tailpipe

Install Tailpipe from the downloads page:

# MacOS
brew install turbot/tap/tailpipe

# Linux or Windows (WSL2)
sudo /bin/sh -c "$(curl -fsSL https://tailpipe-io.vercel.app/install/tailpipe.sh)"

Install a plugin

Install a plugin for your favorite service (e.g. AWS, Azure, GCP, Pipes.

tailpipe plugin install aws

Configure a collection

Details vary by plugin and source. To collect AWS CloudTrail logs, config can be as simple as:

connection "aws" "prod" {
  profile = "SSO-Admin-605...13981"
}

partition "aws_cloudtrail_log" "prod" {
  source "aws_s3_bucket" {
    connection = connection.aws.prod
    bucket     = "aws-cloudtrail-logs-6054...81-fe67"
  }
}

Run a collection

tailpipe collect aws_cloudtrail_log.prod

This command will:

• Acquire compressed (.gz) log files

• Uncompress them

• Parse all the .json log files and map fields of each line to the plugin-defined schema

• Store the data in Parquet organized by date

Query!

List the top 10 events and how many times they were called.

tailpipe query
>  select
  event_source,
  event_name,
  count(*) as event_count
from
  aws_cloudtrail_log
group by
  event_source,
  event_name,
order by
  event_count desc
limit 10;

+-------------------+---------------------------+-------------+
| event_source      | event_name                | event_count |
+-------------------+---------------------------+-------------+
| ec2.amazonaws.com | RunInstances              | 1225268     |
| ec2.amazonaws.com | DescribeSnapshots         | 101158      |
| sts.amazonaws.com | AssumeRole                | 78380       |
| s3.amazonaws.com  | GetBucketAcl              | 19095       |
| ec2.amazonaws.com | DescribeInstances         | 18366       |
| sts.amazonaws.com | GetCallerIdentity         | 16512       |
| iam.amazonaws.com | GetPolicyVersion          | 14737       |
| s3.amazonaws.com  | ListBuckets               | 13206       |
| ec2.amazonaws.com | DescribeSpotPriceHistory  | 10714       |
| ec2.amazonaws.com | DescribeSnapshotAttribute | 9107        |
+-------------------+---------------------------+-------------+

Developing

If you want to help develop the core Tailpipe binary, these are the steps to build it.

Clone:

git clone https://github.com/turbot/tailpipe

Build:

cd tailpipe
make

Check the version:

$ tailpipe --version
Tailpipe version 0.1.0

Open source and contributing

This repository is published under the AGPL 3.0 license. Please see our code of conduct. Contributors must sign our Contributor License Agreement as part of their first pull request. We look forward to collaborating with you!

Tailpipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get involved

Join #tailpipe on Slack →