README
¶
Bruin is a data pipeline tool that brings together data ingestion, data transformation with SQL, Python & R, and data quality into a single framework. It works with all the major data platforms and runs on your local machine, an EC2 instance, or GitHub Actions.
Bruin is packed with features:
- 📥 ingest data with ingestr / Python
- ✨ run SQL, Python & R transformations on many platforms
- 📐 table/view materializations, incremental tables
- 🐍 run Python in isolated environments using uv
- 💅 built-in data quality checks
- 🚀 Jinja templating to avoid repetition
- ✅ validate pipelines end-to-end via dry-run
- 👷 run on your local machine, an EC2 instance, or GitHub Actions
- 🔒 secrets injection via environment variables
- VS Code extension for a better developer experience
- ⚡ written in Golang
- 📦 easy to install and use
Installation
Please see the installation instructions here.
Community
Join our Slack community here.
Quickstart
Take a look at our quickstart guide here.
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
bootstrap
Package bootstrap applies process-wide defaults before transitive driver imports initialize.
|
Package bootstrap applies process-wide defaults before transitive driver imports initialize. |
|
generate
command
|
|
|
pkg
|
|
|
iceberg
Package iceberg builds the ingestr destination URI for Apache Iceberg (iceberg+<catalog>://<location>?storage=s3&...), a write-only destination.
|
Package iceberg builds the ingestr destination URI for Apache Iceberg (iceberg+<catalog>://<location>?storage=s3&...), a write-only destination. |
|
ingestruri
Package ingestruri resolves connection names into ingestr URIs.
|
Package ingestruri resolves connection names into ingestr URIs. |
|
mask
Package mask redacts credential values from log output by searching for each secret in every form it can appear as (raw, escaped, base64) and replacing it.
|
Package mask redacts credential values from log output by searching for each secret in every form it can appear as (raw, escaped, base64) and replacing it. |
|
scd2migration
Package scd2migration is a TEMPORARY, self-contained upgrade shim.
|
Package scd2migration is a TEMPORARY, self-contained upgrade shim. |
|
spark
Package spark provides shared utilities for Spark-based execution environments such as EMR Serverless and Dataproc Serverless.
|
Package spark provides shared utilities for Spark-based execution environments such as EMR Serverless and Dataproc Serverless. |
|
tablename
Package tablename centralizes how Bruin parses and validates multi-component asset/table names (e.g.
|
Package tablename centralizes how Bruin parses and validates multi-component asset/table names (e.g. |
|
unittest
Package unittest builds the SQL that runs a Bruin unit test on the asset's configured connection.
|
Package unittest builds the SQL that runs a Bruin unit test on the asset's configured connection. |
|
semantic-engine
module
|
|
Click to show internal directories.
Click to hide internal directories.