README
¶
Summary
Enables log-based architectures with realtime streaming, unlimited storage, compression and encryption.
| Key facts | |
|---|---|
| Technologies | Golang, Linux, Docker (not required for use), AWS S3. |
| Production ready? | No, not yet. But not too far away. |
| Msg delivery semantics | Exactly once, exact in-order delivery within a stream. |
| Use cases | Eventsourcing, high availability software/datacenter failover, microservices, big data. |
| Is this a good fit for me? | For event-based architectures, yes. Read more |
| Storage capacity | Practically unlimited. You have to pay your bills though. :) |
| Data durability | All writes transactionally backed by Write-Ahead-Log just like in databases. |
| High availability | Planned, going to use Hashicorp's Raft implementation. |
| Data stored at | AWS S3. Google Storage support planned. |
| Encryption at transport | TLS (CA & server certs automatically managed) |
| Encryption at rest | AES256-CTR. Encryption keys are not trusted to AWS. |
| Security | Our security policy & information |
Docs
How-to's & tutorials:
- Quickstart (for users)
- Example app using Pyramid (best way to grasp what Pyramid is about, but you should also read quickstart)
- Building & contributing (for developers)
- Operating
Architecture:
Meta:
Architecture summary
Writer manages writes to streams. Streams are divided into ~8 MB chunks which are stored compressed & encrypted in AWS S3, except the last, "live", chunk that we're writing into.
Data flow at a glance:
- Somebody contacts Writer to append a line to the stream
- Writer notifies pub/sub that a stream has new lines
- All interested applications that want to read from any streams have a Pusher, which subscribes to pub/sub notifications for those streams.
- Once the Pusher learns of new events, it issues a read to either:
- a Writer if it's live data OR
- S3 if it's older data.
- Pusher then pushes this read to your application. Your application transactionally verifies that the read offset is at the offset of last stored write in database. If not, error is returned along with the correct offset, and Pusher continues to read from the correct offset.
Programming language support
To use Pyramid somewhere, that language has to be able to receive events over HTTP + JSON, in just one HTTP path. There's even a helper library (pushlib - look at the architecture diagram) for that, and it's only a small amount of code.
Existing implementations of pushlib:
- Go
- library managed in this main repo
- example application
- node.js (coming soon)
- PHP (coming soon)
- Your programming language not yet supported?
- It's somewhat easy to implement - contributions appreciated :)
License & fair play
Legal license: Apache 2.0 - free software.
Moral license: if you benefit commercially from the use of this project, any help would be appreciated (but not legally required):
- Order support contract/consulting from us
- Contribute with new features, bug fixes and/or help with issues
- Become a sponsor (get your company + link listed here in README as a sponsor)
- Donate money
- If you find Pyramid useful, please spread the word! :)
Directories