README
¶
Marmot
Marmot is an open-source data catalog that helps teams discover, understand, and govern their data assets. It's designed for modern data ecosystems where data flows through multiple systems, formats, and teams.
Marmot was designed with the following in mind:
- Simplicity: Easy to use UI, single binary deployment
- Performance: Fast search and efficient processing
- Extensibility: Document almost anything with the flexible API
✨ Features:
- Find anything fast with flexible queries: Boolean, text, metadata, and comparison operators for powerful search capabilities
- Populate assets with your preferred technology: CLI, API, Terraform, and Pulumi integrations
- Visualize data flows with interactive graphs: Track dependencies and analyze impact through comprehensive lineage visualization
- Documentation and governance: Markdown documentation support with tagging and classification capabilities
https://github.com/user-attachments/assets/a4a72ca2-c0be-4c63-ab52-00c71adc4f4a
🔌Supported Data Sources
Marmot offers multiple ways to ingest assets: through its CLI, API, or infrastructure-as-code solutions like Terraform and Pulumi. The CLI features a growing ecosystem of plugins with current support for Kafka, AsyncAPI, PostgreSQL, SQS, SNS, and many others. For assets not covered by existing plugins, Marmot's flexible API and infrastructure-as-code providers enable ingestion of virtually any asset type you need.
📚 Documentation
If you want to get started with Marmot quickly following the quickstart guide in the documentation to get up and running in minutes.
You can also check out the API documentation to help with building custom integrations.
🛠️ Local Development
- Start PostgreSQL:
docker run --name postgres \
--network bridge \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=marmot \
-p 5432:5432 \
-d postgres:latest
- Start the frontend development server:
cd web/marmot
pnpm install
pnpm dev
- In another terminal, start the backend:
make dev
The app will be available at:
- Frontend: http://localhost:5173
- Backend API: http://localhost:8080
- API Documentation: http://localhost:8080/swagger/index.html
🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
⚖️License
This project is licensed under the MIT License - see the LICENSE file for details.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package docs Code generated by swaggo/swag.
|
Package docs Code generated by swaggo/swag. |
|
internal
|
|
|
docgen/cmd
command
|
|
|
plugin/providers/asyncapi
+marmot:name=AsyncAPI +marmot:description=This plugin enables fetching data from AsyncAPI specifications.
|
+marmot:name=AsyncAPI +marmot:description=This plugin enables fetching data from AsyncAPI specifications. |
|
plugin/providers/bigquery
+marmot:name=BigQuery +marmot:description=This plugin discovers datasets and tables from Google BigQuery projects.
|
+marmot:name=BigQuery +marmot:description=This plugin discovers datasets and tables from Google BigQuery projects. |
|
plugin/providers/kafka
+marmot:name=Kafka +marmot:description=This plugin discovers Kafka topics from Kafka clusters.
|
+marmot:name=Kafka +marmot:description=This plugin discovers Kafka topics from Kafka clusters. |
|
plugin/providers/mongodb
+marmot:name=MongoDB +marmot:description=This plugin discovers databases and collections from MongoDB instances.
|
+marmot:name=MongoDB +marmot:description=This plugin discovers databases and collections from MongoDB instances. |
|
plugin/providers/mysql
+marmot:name=MySQL +marmot:description=This plugin discovers databases and tables from MySQL instances.
|
+marmot:name=MySQL +marmot:description=This plugin discovers databases and tables from MySQL instances. |
|
plugin/providers/openapi
+marmot:name=OpenAPI +marmot:description=This plugin discovers OpenAPI v3 specifications.
|
+marmot:name=OpenAPI +marmot:description=This plugin discovers OpenAPI v3 specifications. |
|
plugin/providers/postgresql
+marmot:name=PostgreSQL +marmot:description=This plugin discovers databases and tables from PostgreSQL instances.
|
+marmot:name=PostgreSQL +marmot:description=This plugin discovers databases and tables from PostgreSQL instances. |
|
plugin/providers/s3
+marmot:name=S3 +marmot:description=This plugin discovers S3 buckets from AWS accounts.
|
+marmot:name=S3 +marmot:description=This plugin discovers S3 buckets from AWS accounts. |
|
plugin/providers/sns
+marmot:name=SNS +marmot:description=This plugin discovers SNS topics from AWS accounts.
|
+marmot:name=SNS +marmot:description=This plugin discovers SNS topics from AWS accounts. |
|
plugin/providers/sqs
+marmot:name=SQS +marmot:description=This plugin discovers SQS queues from AWS accounts.
|
+marmot:name=SQS +marmot:description=This plugin discovers SQS queues from AWS accounts. |