README
¶
pocketbase-sync
Video Demo: https://youtu.be/91_0x8StFl0
Warning
Well, several warnings.
- Aside from this readme document this entire project was nearly all vibe coded. Accept that however you wish.
- I'm a pre-junior developer who is fascinated by system architecture and infrastructure such as this, but I built this in my spare time and may not be the best FOSS lead for a project. Do with that what you will.
- All of my personal applications are too small to need horizontal scaling... I built this because it was fun, and I'm fascinated by the tech. So if you choose to use this, walk into it knowing that I'm basically a chef who doesn't eat his own food.
- Backup your stuff! While I've done some testing and have had good results, I would certainly not call this production ready. If my program eats your data and you're without a backup, you are to blame.
- NATS Jetstream, used here, is NOT encrypted. You will need to come up with your own security scheme so that your sync endpoints aren't exposed to the public internet. Tailscale and ZeroTier are both excellent choices, but there are others out there. (I think K8s & K3s have some sort of provision for this, which reminds me, Fly.io should have something you can use also)
- Normal record syncing has been tested to some extent, but I haven't had a chance to build and test any kind of realtime application for the realtime websocket syncing.
- I might have built this whole thing just to solve another one of the internet's great non-problems, you know, like bitcoin did for finance. Basically, this whole project may be useful to a total of 0.000 of people out there. Shucks.
Now that the warnings are out of the way....
What's going on here?
pocketbase-sync is Pocketbase (amazing) but with horizontal sync capabilities.
- I'm using NATS Jetstream for the pub/sub architecture
- This is Event driven, so it won't eat up your system resources like polling solutions such as the very-cool project marmot which syncs sqlite databases (I'm grateful to marmot for inspiring this project!)
- Leaderless - there is no single central node that all other nodes must talk to. Once a sync chain has nodes talking on it, you can destroy and start any node you wish.
- Eventually Consistent (usually syncs in miliseconds, but we aren't strongly consistent)
- Realtime database connections are handled by websocket routing, so that no matter what instance an end-user originates from... if a realtime connection is open to a resource, all other users will connect to that same resource. Whoever initiated the realtime connection is the winner, and will hold the lease until the instance dies or until there are no realtime subscribers. This has not been tested!!!
- Snapshots of your database are configured for the NATS Jetstream so that when a new instance joins, it can initialize using a recent snapshot of your database, followed by all database changes since the snapshot. It wouldn't be reasonable for a new instance to have to ingest every change since the beginning of time when it joins. So it grabs and applies the snapshot first, then catches up to current and begins syncing itself.
Usage
Well, the command line side of things is unchanged. It's just Pocketbase all the way down. I added plugins/sync and made some minor UI tweaks to support it.
In the Pocketbase Admin UI, navigate to Settings => Application => Sync.
For the first instance, check the button that says "Initial Node" and fill in the local address that you want NATS to listen on, as well as an instance ID so you can know which one this is.
For all other following instances that you want to sync, I strongly recommend starting with a brand new empty instance of Pocketbase-sync, and in the Sync settings use "Add To Sync Chain" and similarly, fill out the local instance ID and NATS URL, but also fill in the Remote credentials of another instance you have on the sync chain.
Since this is leaderless, it doesn't matter what instance you point a new one at. It should catch up with all changes.
Contributions
- It would be nice to have a realtime app I can use to test the realtime websocket routing. If anyone has something they want to volunteer, I'm open to that.
- If anyone wants to look over the codebase (almost everything new is in plugins/sync) and make sure there's some sanity going on in there, that would be nice of you.
What else should I put in this readme file? I wrote this myself, sans-LLM. So that's why it reads a bit rough. But hey, I write like I talk. This is what you get.
Documentation
¶
Index ¶
Constants ¶
const ( // ModerncDepsCheckHookId is the id of the hook that performs the modernc.org/* deps checks. // It could be used for removing/unbinding the hook if you don't want the checks. ModerncDepsCheckHookId = "pbModerncDepsCheck" )
Variables ¶
var Version = "(untracked)"
Version of PocketBase
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// hide the default console server info on app startup
HideStartBanner bool
// optional default values for the console flags
DefaultDev bool
DefaultDataDir string // if not set, it will fallback to "./pb_data"
DefaultEncryptionEnv string
DefaultQueryTimeout time.Duration // default to core.DefaultQueryTimeout (in seconds)
// optional DB configurations
DataMaxOpenConns int // default to core.DefaultDataMaxOpenConns
DataMaxIdleConns int // default to core.DefaultDataMaxIdleConns
AuxMaxOpenConns int // default to core.DefaultAuxMaxOpenConns
AuxMaxIdleConns int // default to core.DefaultAuxMaxIdleConns
DBConnect core.DBConnectFunc // default to core.dbConnect
}
Config is the PocketBase initialization config struct.
type PocketBase ¶
type PocketBase struct {
core.App
// RootCmd is the main console command
RootCmd *cobra.Command
// contains filtered or unexported fields
}
PocketBase defines a PocketBase app launcher.
It implements core.App via embedding and all of the app interface methods could be accessed directly through the instance (eg. PocketBase.DataDir()).
func New ¶
func New() *PocketBase
New creates a new PocketBase instance with the default configuration. Use NewWithConfig if you want to provide a custom configuration.
Note that the application will not be initialized/bootstrapped yet, aka. DB connections, migrations, app settings, etc. will not be accessible. Everything will be initialized when PocketBase.Start is executed. If you want to initialize the application before calling PocketBase.Start, then you'll have to manually call [PocketBase.Bootstrap].
func NewWithConfig ¶
func NewWithConfig(config Config) *PocketBase
NewWithConfig creates a new PocketBase instance with the provided config.
Note that the application will not be initialized/bootstrapped yet, aka. DB connections, migrations, app settings, etc. will not be accessible. Everything will be initialized when PocketBase.Start is executed. If you want to initialize the application before calling PocketBase.Start, then you'll have to manually call [PocketBase.Bootstrap].
func (*PocketBase) Execute ¶
func (pb *PocketBase) Execute() error
Execute initializes the application (if not already) and executes the pb.RootCmd with graceful shutdown support.
This method differs from pb.Start() by not registering the default system commands!
func (*PocketBase) Start ¶
func (pb *PocketBase) Start() error
Start starts the application, aka. registers the default system commands (serve, superuser, version) and executes pb.RootCmd.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package core is the backbone of PocketBase.
|
Package core is the backbone of PocketBase. |
|
validators
Package validators implements some common custom PocketBase validators.
|
Package validators implements some common custom PocketBase validators. |
|
examples
|
|
|
base
command
|
|
|
Package mails implements various helper methods for sending common emails like forgotten password, verification, etc.
|
Package mails implements various helper methods for sending common emails like forgotten password, verification, etc. |
|
plugins
|
|
|
ghupdate
Package ghupdate implements a new command to selfupdate the current PocketBase executable with the latest GitHub release.
|
Package ghupdate implements a new command to selfupdate the current PocketBase executable with the latest GitHub release. |
|
jsvm
Package jsvm implements pluggable utilities for binding a JS goja runtime to the PocketBase instance (loading migrations, attaching to app hooks, etc.).
|
Package jsvm implements pluggable utilities for binding a JS goja runtime to the PocketBase instance (loading migrations, attaching to app hooks, etc.). |
|
jsvm/internal/types
command
|
|
|
migratecmd
Package migratecmd adds a new "migrate" command support to a PocketBase instance.
|
Package migratecmd adds a new "migrate" command support to a PocketBase instance. |
|
Package tests provides common helpers and mocks used in PocketBase application tests.
|
Package tests provides common helpers and mocks used in PocketBase application tests. |
|
tools
|
|
|
auth/internal/jwk
Package jwk implements some common utilities for interacting with JWKs (mostly used with OIDC providers).
|
Package jwk implements some common utilities for interacting with JWKs (mostly used with OIDC providers). |
|
cron
Package cron implements a crontab-like service to execute and schedule repeative tasks/jobs.
|
Package cron implements a crontab-like service to execute and schedule repeative tasks/jobs. |
|
filesystem/blob
Package blob defines a lightweight abstration for interacting with various storage services (local filesystem, S3, etc.).
|
Package blob defines a lightweight abstration for interacting with various storage services (local filesystem, S3, etc.). |
|
filesystem/internal/fileblob
Package fileblob provides a blob.Bucket driver implementation.
|
Package fileblob provides a blob.Bucket driver implementation. |
|
filesystem/internal/s3blob
Package s3blob provides a blob.Bucket S3 driver implementation.
|
Package s3blob provides a blob.Bucket S3 driver implementation. |
|
filesystem/internal/s3blob/s3
Package s3 implements a lightweight client for interacting with the REST APIs of any S3 compatible service.
|
Package s3 implements a lightweight client for interacting with the REST APIs of any S3 compatible service. |
|
filesystem/internal/s3blob/s3/tests
Package tests contains various tests helpers and utilities to assist with the S3 client testing.
|
Package tests contains various tests helpers and utilities to assist with the S3 client testing. |
|
template
Package template is a thin wrapper around the standard html/template and text/template packages that implements a convenient registry to load and cache templates on the fly concurrently.
|
Package template is a thin wrapper around the standard html/template and text/template packages that implements a convenient registry to load and cache templates on the fly concurrently. |
|
tokenizer
Package tokenizer implements a rudimentary tokens parser of buffered io.Reader while respecting quotes and parenthesis boundaries.
|
Package tokenizer implements a rudimentary tokens parser of buffered io.Reader while respecting quotes and parenthesis boundaries. |
|
types
Package types implements some commonly used db serializable types like datetime, json, etc.
|
Package types implements some commonly used db serializable types like datetime, json, etc. |
|
Package ui handles the PocketBase Superuser frontend embedding.
|
Package ui handles the PocketBase Superuser frontend embedding. |