README
¶
Pachyderm: Data Pipelines
Pachyderm is a tool for production data pipelines. If you need to chain together data scraping, ingestion, cleaning, munging, wrangling, processing, modeling, and analysis in a sane way, then Pachyderm is for you. If you have an existing set of scripts which do this in an ad-hoc fashion and you're looking for a way to "productionize" them, Pachyderm can make this easy for you.
Features
- Containerized: Pachyderm is built on Docker and Kubernetes. Whatever languages or libraries your pipeline needs, they can run on Pachyderm which can easily be deployed on any cloud provider or on prem.
- Version Control: Pachyderm version controls your data as it's processed. You can always ask the system how data has change, see a diff, and, if something doesn't look right, revert.
- Provenance (aka data lineage): Pachyderm tracks where data comes from. Pachyderm keeps track of all the code and data that created a result.
- Parallelization: Pachyderm can efficiently schedule massively parallel workloads.
- Incremental Processing: Pachyderm understands how your data has changed and is smart enough to only process the new data.
Getting Started
Install Pachyderm locally or deploy on AWS/GCE/Azure in about 5 minutes.
You can also refer to our complete developer docs to see tutorials, check out example projects, and learn about advanced features of Pachyderm.
If you'd like to see some examples and learn about core use cases for Pachyderm:
- Examples
- Use Cases
- Case Studies: Learn how General Fusion uses Pachyderm to power commercial fusion research.
Documentation
Community
Keep up to date and get Pachyderm support via:
Follow us on Twitter.- Join our community Slack Channel to get help from the Pachyderm team and other users.
Contributing
To get started, sign the Contributor License Agreement.
You should also check out our contributing guide.
Send us PRs, we would love to see what you do! You can also check our GH issues for things labeled "noob-friendly" as a good place to start. We're sometimes bad about keeping that label up-to-date, so if you don't see any, just let us know.
Join Us
WE'RE HIRING! Love Docker, Go and distributed systems? Learn more about our team and email us at jobs@pachyderm.io.
Usage Metrics
Pachyderm automatically reports anonymized usage metrics. These metrics help us
understand how people are using Pachyderm and make it better. They can be
disabled by setting the env variable METRICS to false in the pachd
container.
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
doc
|
|
|
examples/fruit_stand/generate
command
|
|
|
examples/redshift/json_to_sql
command
|
|
|
examples/word_count
command
|
|
|
etc
|
|
|
build
command
|
|
|
src
|
|
|
client/admin
Package admin is a generated protocol buffer package.
|
Package admin is a generated protocol buffer package. |
|
client/auth
Package auth is a generated protocol buffer package.
|
Package auth is a generated protocol buffer package. |
|
client/deploy
Package deploy is a generated protocol buffer package.
|
Package deploy is a generated protocol buffer package. |
|
client/enterprise
Package enterprise is a generated protocol buffer package.
|
Package enterprise is a generated protocol buffer package. |
|
client/health
Package health is a generated protocol buffer package.
|
Package health is a generated protocol buffer package. |
|
client/limit
Package limit provides primitives to limit concurrency.
|
Package limit provides primitives to limit concurrency. |
|
client/pfs
Package pfs is a generated protocol buffer package.
|
Package pfs is a generated protocol buffer package. |
|
client/pkg/config
Package config is a generated protocol buffer package.
|
Package config is a generated protocol buffer package. |
|
client/pkg/shard
Package shard is a generated protocol buffer package.
|
Package shard is a generated protocol buffer package. |
|
client/pps
Package pps is a generated protocol buffer package.
|
Package pps is a generated protocol buffer package. |
|
client/version/versionpb
Package versionpb is a generated protocol buffer package.
|
Package versionpb is a generated protocol buffer package. |
|
server/cmd/match
command
|
|
|
server/cmd/pachctl
command
|
|
|
server/cmd/pachctl-doc
command
|
|
|
server/cmd/pachd
command
|
|
|
server/cmd/worker
command
|
|
|
server/pfs/fuse
Package fuse is a generated protocol buffer package.
|
Package fuse is a generated protocol buffer package. |
|
server/pkg/backoff
Package backoff implements backoff algorithms for retrying operations.
|
Package backoff implements backoff algorithms for retrying operations. |
|
server/pkg/cache/groupcachepb
Package groupcachepb is a generated protocol buffer package.
|
Package groupcachepb is a generated protocol buffer package. |
|
server/pkg/deploy
Package deploy is a generated protocol buffer package.
|
Package deploy is a generated protocol buffer package. |
|
server/pkg/dlock
Package dlock implements a distributed lock on top of etcd.
|
Package dlock implements a distributed lock on top of etcd. |
|
server/pkg/exec
Package exec runs external commands.
|
Package exec runs external commands. |
|
server/pkg/hashtree
Package hashtree is a generated protocol buffer package.
|
Package hashtree is a generated protocol buffer package. |
|
server/pkg/metrics
Package metrics is a generated protocol buffer package.
|
Package metrics is a generated protocol buffer package. |
|
server/pkg/pfsdb
Package pfsdb contains the database schema that PFS uses.
|
Package pfsdb contains the database schema that PFS uses. |
|
server/pkg/ppsdb
Package ppsdb contains the database schema that PPS uses.
|
Package ppsdb contains the database schema that PPS uses. |
|
server/pkg/sync
Package sync provides utility functions similar to `git pull/push` for PFS
|
Package sync provides utility functions similar to `git pull/push` for PFS |
|
server/pkg/watch
Package watch implements better watch semantics on top of etcd.
|
Package watch implements better watch semantics on top of etcd. |
|
server/worker
Package worker is a generated protocol buffer package.
|
Package worker is a generated protocol buffer package. |