Documentation
¶
Overview ¶
Package protocolsync keeps the DB node cache, the search index, the thumbnails and the write hooks in step with what a protocol server just did to the bytes.
Why this is one package and not one per protocol ¶
A write is not "put the bytes down". It also has to upsert the node row, create any missing parent directory rows, index the result, dispatch a thumbnail, and fire the write hook that drives versioning, antivirus, webhooks and notify. Listings read the DB cache, so a write that skips it is invisible in the UI until the next scheduled sync run.
All of that lived in internal/dav/dbsync.go, written for the one protocol filex had. S3, SFTP, FTPS, NFS and FUSE each need exactly the same thing, and a copy per protocol means every future fix has to be made five times. The fifth one is the one somebody forgets, and the symptom is silent: the file is there, so nothing looks broken — it is merely unindexed, has no thumbnail, and counts against nobody's quota.
Contract ¶
Every method is BEST EFFORT and never returns an error: the bytes already landed on the driver, so failing the protocol response afterwards would report a failure for an operation that succeeded. Failures are logged and the sync worker reconciles later. Panics are contained for the same reason.
⚠ The origin string is per protocol and must be a NEW writehook constant, never a borrowed one — the audit trail's answer to "which protocol wrote this" is exactly that field.
Telling the people who are looking ¶
Updating the catalogue is only half of "the rest of the system knows". The other half is the realtime hub: an explorer with a live WebSocket does NOT poll — it re-lists a folder only when a change frame arrives for it (see packages/core/src/composables/useRealtime.ts, where the 12 s poll is the FALLBACK for a socket that has failed). So a write that updates the row and the index but announces nothing leaves every open viewer showing the old listing until the person navigates away and back. The periodic sync does not rescue that either: it repairs rows and the index, and emits nothing at all.
Emission therefore lives here rather than in each protocol, for the same reason the bookkeeping does.
Index ¶
- Variables
- func NormalizePath(rel string) string
- func ParentOf(rel string) string
- func SetChangeEmitter(e ChangeEmitter)
- type ChangeEmitter
- type Check
- type Syncer
- func (s *Syncer) CollectSubtree(ctx context.Context, storageID int64, node *model.Node) []*model.Node
- func (s *Syncer) Delete(ctx context.Context, st *model.Storage, rel string)
- func (s *Syncer) DeleteRows(ctx context.Context, st *model.Storage, rel string) (string, bool)
- func (s *Syncer) Discard(ctx context.Context, st *model.Storage, drv storage.Driver, rel string, ...) (trash.Outcome, error)
- func (s *Syncer) DispatchThumb(node *model.Node)
- func (s *Syncer) EnsureDirChain(ctx context.Context, st *model.Storage, rel string) (*int64, error)
- func (s *Syncer) EnsureDirChainQuiet(ctx context.Context, st *model.Storage, rel string) (*int64, error)
- func (s *Syncer) IndexNode(ctx context.Context, n *model.Node)
- func (s *Syncer) Mkdir(ctx context.Context, st *model.Storage, rel string)
- func (s *Syncer) Move(ctx context.Context, st *model.Storage, srcRel, dstRel string)
- func (s *Syncer) MoveRows(ctx context.Context, st *model.Storage, srcRel, dstRel string) bool
- func (s *Syncer) Purge(ctx context.Context, st *model.Storage, rel string, ...) error
- func (s *Syncer) PurgeTree(ctx context.Context, st *model.Storage, del storage.Deleter, rel string, ...) error
- func (s *Syncer) ReclaimDestination(ctx context.Context, storageID int64, dstClean string, keepID int64) bool
- func (s *Syncer) Relocate(ctx context.Context, st *model.Storage, srcRel, dstRel string, ...) error
- func (s *Syncer) RemoveFromIndex(ctx context.Context, id int64)
- func (s *Syncer) Touch(ctx context.Context, st *model.Storage, rel string, mtime time.Time)
- func (s *Syncer) Trash(ctx context.Context, st *model.Storage, rel, trashRel string)
- func (s *Syncer) TrashRows(ctx context.Context, st *model.Storage, rel, trashRel string) (string, bool)
- func (s *Syncer) WithResolver(r func(storageID int64) (storage.Driver, error)) *Syncer
- func (s *Syncer) Write(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
- func (s *Syncer) WriteRows(ctx context.Context, st *model.Storage, rel string, size int64, mime string) (*model.Node, writehook.WriteKind, bool)
- func (s *Syncer) WriteSaved(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
- func (s *Syncer) WriteWithoutNotification(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
Constants ¶
This section is empty.
Variables ¶
var PurgeBatch = 100
PurgeBatch is how many objects PurgeTree deletes under one hold of the row gate.
Functions ¶
func NormalizePath ¶
NormalizePath canonicalises a path the way the shared pathkey.Hash key expects. The cache rows written here must collide with the ones the sync worker and the manager write, or the same file appears twice after the next sync run.
func SetChangeEmitter ¶
func SetChangeEmitter(e ChangeEmitter)
SetChangeEmitter installs the realtime emitter shared by every protocol Syncer. Call once at startup, beside handlers.SetChangeEmitter. nil disables emission.
Types ¶
type ChangeEmitter ¶
type ChangeEmitter interface {
EmitChange(storageID int64, dir string, ev realtime.ChangeEvent)
}
ChangeEmitter is the minimal realtime surface this package needs to tell open explorers that a folder changed. *realtime.Hub satisfies it, and it is the SAME hub the HTTP mutation handlers publish to — this is not a second mechanism, it is the existing one reached from a package that cannot import internal/api/handlers.
type Check ¶ added in v0.55.0
Check is one thing a verb checked before it asked for the row gate, asked again once it holds it, on the detached context the change runs on.
func StillAsSeen ¶ added in v0.55.0
StillAsSeen is the check that rel is still the object the verb stat'ed (storage.StillAsSeen: storage.ErrChangedMeanwhile).
type Syncer ¶
type Syncer struct {
Store db.Store
// Index is optional; nil skips indexing.
Index *search.Index
// Thumbs is optional; nil skips thumbnail dispatch.
Thumbs *thumb.Pipeline
// Origin is the writehook origin for this protocol (writehook.OriginDAV,
// OriginS3, …). It is what the audit trail reports.
Origin string
// Resolver lets a write read back what it left on the backend (size,
// etag, mtime — storage.Landed) when it REPLACES a file. Optional: without
// it an overwrite records an empty etag, which the next scan corrects.
// What it must never do is keep the replaced file's etag.
Resolver func(storageID int64) (storage.Driver, error)
}
Syncer performs the bookkeeping for one protocol.
func (*Syncer) CollectSubtree ¶
func (s *Syncer) CollectSubtree(ctx context.Context, storageID int64, node *model.Node) []*model.Node
CollectSubtree returns node plus every live cached descendant (DFS via ListNodesByParent). Directories only recurse; files are leaves.
func (*Syncer) Delete ¶
Delete drops the node row (and, for directories, every cached descendant) and removes them from the search index.
This is the path taken when the bytes are gone for good — either the driver could not trash them, or there was nothing at the path to begin with. The rows are HARD deleted on purpose: a soft delete would park them in the trash listing with a Restore button behind which no bytes exist, which reads to the user as data loss with a broken recovery rather than an honest deletion.
⚠ Children are removed before their parent: nodes.parent_id cascades, so deleting the folder row first would drop the child rows before they could be dropped from the search index, leaving them searchable forever.
func (*Syncer) DeleteRows ¶
DeleteRows is Delete without the write hook or the change frame — see TrashRows for why the halves are separable.
It is filex's one answer to "these bytes are gone for good, drop the rows": the protocol servers' deletes, the explorer's delete of an item already gone or on a storage that keeps no trash (vfDelete) and the operations queue's (Manager.SyncHardDelete) all end here, so none of them can release less than the others. Every row goes on its own, deepest first, through the accounting store (each file's bytes leave its owner's quota) - never a folder row alone, whose parent_id cascade would take its contents and release nothing.
Also dropped, with the folder (issue #104):
- rows an earlier version soft-deleted where they stood below it (vanished): they still name it as their parent and still count against their owner's quota, and the cascade would take them silently;
- each file's snapshots under `.versions/<id>/`, when the Syncer has a Resolver: their rows cascade with the file's, and nothing would ever reach the bytes again.
func (*Syncer) Discard ¶ added in v0.55.0
func (s *Syncer) Discard(ctx context.Context, st *model.Storage, drv storage.Driver, rel string, checks ...Check) (trash.Outcome, error)
Discard sends rel to st's trash (trash.Put, the one implementation every delete surface shares) and lets the rows follow, both under st's row gate - or, for a folder on an object store, under a fence on rel (fenceOf) - after checks are asked again:
- the bytes were trashed: the row is retagged into the trash (Trash);
- nothing was there: the rows are dropped (Delete);
- anything else (trash.ErrUnsupported included): nothing in the catalogue changes, and the caller decides - a storage without a trash deletes for good only for a person who may purge (Purge).
It answers trash.Put's outcome and error as they are.
func (*Syncer) DispatchThumb ¶
DispatchThumb fires async thumbnail generation for a written file — the same behaviour manager/AI uploads get. A nil pipeline is a no-op.
func (*Syncer) EnsureDirChain ¶
EnsureDirChain walks rel segment by segment, creating any missing dir rows, and returns the leaf dir's node id (nil at storage root).
func (*Syncer) EnsureDirChainQuiet ¶ added in v0.44.2
func (s *Syncer) EnsureDirChainQuiet(ctx context.Context, st *model.Storage, rel string) (*int64, error)
EnsureDirChainQuiet is EnsureDirChain without the "create" change frames.
It is for a caller that records folders which were ALREADY on the storage — the lazy catalogue giving an opened folder, and the folders above it, their rows (internal/sync lazy.go). Nothing appeared in anybody's folder: the explorer was already showing it from the disk listing, and a desktop mirror told "create" for every catalogued ancestor would re-list its way up the tree for nothing. The rows, the index and the unique-key race handling are exactly EnsureDirChain's.
func (*Syncer) Move ¶
Move re-homes the node row (and cached descendants) to the new path and re-indexes each. A stale FILE row holding the destination is dropped (its bytes are the ones this move just replaced — see ReclaimDestination); a row that still cannot be re-homed is left live where it is for the next sync walk.
func (*Syncer) MoveRows ¶
MoveRows is Move without the write hook or the change frames: it re-homes the cached subtree and re-indexes every row, and reports whether the top node itself landed at the destination.
⚠ The SUBTREE is the point. A folder move that re-homes only its own row leaves every descendant row — and every descendant search-index entry — pointing at a path with nothing behind it, and nothing later notices, because the paths still look plausible. The AI/MCP surface did exactly that until it was pointed here.
func (*Syncer) Purge ¶ added in v0.55.0
func (s *Syncer) Purge(ctx context.Context, st *model.Storage, rel string, del func(ctx context.Context) error, checks ...Check) error
Purge deletes rel for good: del removes the bytes, then the rows are dropped (Delete), both under st's row gate, after checks are asked again. The rows go only when del succeeded; a del that tolerates "already gone" says so by answering nil. Purge answers del's error as it is. del is given the detached context the change runs on.
func (*Syncer) PurgeTree ¶ added in v0.55.0
func (s *Syncer) PurgeTree(ctx context.Context, st *model.Storage, del storage.Deleter, rel string, list func(ctx context.Context) ([]string, error), checks ...Check) error
PurgeTree deletes the folder rel for good on a storage that keeps no real object for a folder (an object store): its objects, as list answers under the first hold of the gate, in batches of PurgeBatch, each batch's bytes and then its rows under one hold of st's row gate, and last the folder's own markers and every row left under it. The gate is held a batch at a time, not for the whole folder: a storage scan between two batches sees files gone with their rows gone and files still there with their rows, which is the truth, and a folder whose objects are all gone, which is what the purge is making it. A delete that finds an object already gone carries on. The first hold waits on ctx (a client gone by then changes nothing); once the first batch has started, the rest runs detached to the end.
func (*Syncer) ReclaimDestination ¶
func (s *Syncer) ReclaimDestination(ctx context.Context, storageID int64, dstClean string, keepID int64) bool
ReclaimDestination drops a catalogue row that holds dstClean and so blocks the move of keepID into it, and reports whether it removed one.
⚠ Only correct because every caller runs AFTER the bytes have moved onto the destination. A row still claiming that path no longer describes what is there: either the storage's move replaced those bytes (WebDAV MOVE with `Overwrite: T`, an S3 copy+delete, a rename), or the row was stale to begin with. Leaving it made the re-home fail against the live-only unique index (migration 00032), and the moved file lost its identity — versions, shares and comments stayed on a row parked in the trash while its bytes answered under this one.
HARD deleted, for the reason Delete gives: the bytes that row named are gone, and a soft delete would put it in the trash behind a Restore that delivers nothing. ⚠ A DIRECTORY row is left alone: nodes.parent_id cascades, so dropping a folder row would silently take its cached subtree with it.
func (*Syncer) Relocate ¶ added in v0.55.0
func (s *Syncer) Relocate(ctx context.Context, st *model.Storage, srcRel, dstRel string, move func(ctx context.Context) error, checks ...Check) error
Relocate is a protocol's rename or move within one storage: move changes the storage (srcRel's bytes to dstRel), then the rows follow (Move), both under st's row gate - or, for a folder on an object store, under a fence on srcRel and dstRel (fenceOf) - after checks are asked again. The rows follow only when move succeeded; Relocate answers move's error as it is, so the caller maps it the way it always did. move is given the detached context the change runs on.
func (*Syncer) RemoveFromIndex ¶
RemoveFromIndex drops a node from the search index.
func (*Syncer) Touch ¶
Touch records a new modification time for a file whose BYTES did not change.
It is deliberately not Write: a client setting an mtime (rclone does it after every upload it thinks needs it) has changed no content, so re-indexing the file and re-cutting its thumbnail would be pure cost — a thousand-file sync would do a thousand of each for nothing. Nothing is created either; if the row is not there yet, the write that will create it is the one that carries the truth.
func (*Syncer) Trash ¶
Trash retags the node row to its `.filex-trash/` location (original path preserved in storage_key) and soft-deletes it.
Directories drag their cached subtree along inside SoftDeleteAndRetag; the subtree is collected UP FRONT (children are still live) so every affected row can be dropped from the search index afterwards.
func (*Syncer) TrashRows ¶
func (s *Syncer) TrashRows(ctx context.Context, st *model.Storage, rel, trashRel string) (string, bool)
TrashRows is Trash without the write hook or the change frame: the row and index half only, reporting the trashed node's name and whether anything was found to retag.
It is exported because the AI/MCP surface (internal/api/handlers/ai_ops.go) needs exactly this bookkeeping but fires its own OnFileTrashed with its own origin — and used to hand-roll a version of it that forgot the index entirely, so an agent-deleted file stayed findable by search. Sharing the mechanics and leaving the eventing to the caller is what keeps the two from drifting again.
func (*Syncer) WithResolver ¶
WithResolver sets Resolver and returns the Syncer, so a construction site stays one expression.
func (*Syncer) Write ¶
func (s *Syncer) Write(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
Write upserts the node row for a written file, indexes it (which also triggers the content-extraction hook when enabled), dispatches a thumbnail and tells anyone watching the folder.
It reports whether a row was actually written, so a caller with its own eventing (the AI surface) can tell a bookkeeping miss from a success.
func (*Syncer) WriteRows ¶
func (s *Syncer) WriteRows(ctx context.Context, st *model.Storage, rel string, size int64, mime string) (*model.Node, writehook.WriteKind, bool)
WriteRows is Write without the write hook or the change frame: the row, search index and thumbnail only, returning the node it recorded and whether the write created that row or replaced the bytes of one that was already there (writehook.Created / writehook.Replaced — the same `existing != nil` branch the row upsert turns on, not a second lookup).
Separable for the same reason as TrashRows and DeleteRows — a caller that fires its own event needs the bookkeeping without a second one. The AI surface uses it to record a moved file whose source was never catalogued: that is still one move, so it must not also announce itself as a write.
func (*Syncer) WriteSaved ¶
func (s *Syncer) WriteSaved(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
WriteSaved is Write for a write that is one save inside an ONGOING editing session: identical bookkeeping, identical event, but the antivirus scan is scheduled through the debounced save window instead of run now (writehook.OnFileSaved).
⚠ It is not a "lighter" Write to reach for when scans feel expensive. The window trades a bounded stretch of the file sitting unscanned for collapsing a burst of saves into one pass, and that trade only pays when more saves are genuinely coming. A surface that writes a whole file once must call Write.
func (*Syncer) WriteWithoutNotification ¶ added in v0.44.2
func (s *Syncer) WriteWithoutNotification(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
WriteWithoutNotification performs every write side effect except the per-file event. Composite operations call it for their component files and emit one meaningful completion event after the whole operation succeeds.
Source Files