Documentation
¶
Overview ¶
Package protocolsync keeps the DB node cache, the search index, the thumbnails and the write hooks in step with what a protocol server just did to the bytes.
Why this is one package and not one per protocol ¶
A write is not "put the bytes down". It also has to upsert the node row, create any missing parent directory rows, index the result, dispatch a thumbnail, and fire the write hook that drives versioning, antivirus, webhooks and notify. Listings read the DB cache, so a write that skips it is invisible in the UI until the next scheduled sync run.
All of that lived in internal/dav/dbsync.go, written for the one protocol filex had. S3, SFTP, FTPS, NFS and FUSE each need exactly the same thing, and a copy per protocol means every future fix has to be made five times. The fifth one is the one somebody forgets, and the symptom is silent: the file is there, so nothing looks broken — it is merely unindexed, has no thumbnail, and counts against nobody's quota.
Contract ¶
Every method is BEST EFFORT and never returns an error: the bytes already landed on the driver, so failing the protocol response afterwards would report a failure for an operation that succeeded. Failures are logged and the sync worker reconciles later. Panics are contained for the same reason.
⚠ The origin string is per protocol and must be a NEW writehook constant, never a borrowed one — the audit trail's answer to "which protocol wrote this" is exactly that field.
Telling the people who are looking ¶
Updating the catalogue is only half of "the rest of the system knows". The other half is the realtime hub: an explorer with a live WebSocket does NOT poll — it re-lists a folder only when a change frame arrives for it (see packages/core/src/composables/useRealtime.ts, where the 12 s poll is the FALLBACK for a socket that has failed). So a write that updates the row and the index but announces nothing leaves every open viewer showing the old listing until the person navigates away and back. The periodic sync does not rescue that either: it repairs rows and the index, and emits nothing at all.
Emission therefore lives here rather than in each protocol, for the same reason the bookkeeping does.
Index ¶
- func NormalizePath(rel string) string
- func ParentOf(rel string) string
- func SetChangeEmitter(e ChangeEmitter)
- type ChangeEmitter
- type Syncer
- func (s *Syncer) CollectSubtree(ctx context.Context, storageID int64, node *model.Node) []*model.Node
- func (s *Syncer) Delete(ctx context.Context, st *model.Storage, rel string)
- func (s *Syncer) DeleteRows(ctx context.Context, st *model.Storage, rel string) (string, bool)
- func (s *Syncer) DispatchThumb(node *model.Node)
- func (s *Syncer) EnsureDirChain(ctx context.Context, st *model.Storage, rel string) (*int64, error)
- func (s *Syncer) EnsureDirChainQuiet(ctx context.Context, st *model.Storage, rel string) (*int64, error)
- func (s *Syncer) IndexNode(ctx context.Context, n *model.Node)
- func (s *Syncer) Mkdir(ctx context.Context, st *model.Storage, rel string)
- func (s *Syncer) Move(ctx context.Context, st *model.Storage, srcRel, dstRel string)
- func (s *Syncer) MoveRows(ctx context.Context, st *model.Storage, srcRel, dstRel string) bool
- func (s *Syncer) ReclaimDestination(ctx context.Context, storageID int64, dstClean string, keepID int64) bool
- func (s *Syncer) RemoveFromIndex(ctx context.Context, id int64)
- func (s *Syncer) Touch(ctx context.Context, st *model.Storage, rel string, mtime time.Time)
- func (s *Syncer) Trash(ctx context.Context, st *model.Storage, rel, trashRel string)
- func (s *Syncer) TrashRows(ctx context.Context, st *model.Storage, rel, trashRel string) (string, bool)
- func (s *Syncer) WithResolver(r func(storageID int64) (storage.Driver, error)) *Syncer
- func (s *Syncer) Write(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
- func (s *Syncer) WriteRows(ctx context.Context, st *model.Storage, rel string, size int64, mime string) (*model.Node, writehook.WriteKind, bool)
- func (s *Syncer) WriteSaved(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
- func (s *Syncer) WriteWithoutNotification(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NormalizePath ¶
NormalizePath canonicalises a path the way the shared pathkey.Hash key expects. The cache rows written here must collide with the ones the sync worker and the manager write, or the same file appears twice after the next sync run.
func SetChangeEmitter ¶
func SetChangeEmitter(e ChangeEmitter)
SetChangeEmitter installs the realtime emitter shared by every protocol Syncer. Call once at startup, beside handlers.SetChangeEmitter. nil disables emission.
Types ¶
type ChangeEmitter ¶
type ChangeEmitter interface {
EmitChange(storageID int64, dir string, ev realtime.ChangeEvent)
}
ChangeEmitter is the minimal realtime surface this package needs to tell open explorers that a folder changed. *realtime.Hub satisfies it, and it is the SAME hub the HTTP mutation handlers publish to — this is not a second mechanism, it is the existing one reached from a package that cannot import internal/api/handlers.
type Syncer ¶
type Syncer struct {
Store db.Store
// Index is optional; nil skips indexing.
Index *search.Index
// Thumbs is optional; nil skips thumbnail dispatch.
Thumbs *thumb.Pipeline
// Origin is the writehook origin for this protocol (writehook.OriginDAV,
// OriginS3, …). It is what the audit trail reports.
Origin string
// Resolver lets a write read back what it left on the backend (size,
// etag, mtime — storage.Landed) when it REPLACES a file. Optional: without
// it an overwrite records an empty etag, which the next scan corrects.
// What it must never do is keep the replaced file's etag.
Resolver func(storageID int64) (storage.Driver, error)
}
Syncer performs the bookkeeping for one protocol.
func (*Syncer) CollectSubtree ¶
func (s *Syncer) CollectSubtree(ctx context.Context, storageID int64, node *model.Node) []*model.Node
CollectSubtree returns node plus every live cached descendant (DFS via ListNodesByParent). Directories only recurse; files are leaves.
func (*Syncer) Delete ¶
Delete drops the node row (and, for directories, every cached descendant) and removes them from the search index.
This is the path taken when the bytes are gone for good — either the driver could not trash them, or there was nothing at the path to begin with. The rows are HARD deleted on purpose: a soft delete would park them in the trash listing with a Restore button behind which no bytes exist, which reads to the user as data loss with a broken recovery rather than an honest deletion.
⚠ Children are removed before their parent: nodes.parent_id cascades, so deleting the folder row first would drop the child rows before they could be dropped from the search index, leaving them searchable forever.
func (*Syncer) DeleteRows ¶
DeleteRows is Delete without the write hook or the change frame — see TrashRows for why the halves are separable.
func (*Syncer) DispatchThumb ¶
DispatchThumb fires async thumbnail generation for a written file — the same behaviour manager/AI uploads get. A nil pipeline is a no-op.
func (*Syncer) EnsureDirChain ¶
EnsureDirChain walks rel segment by segment, creating any missing dir rows, and returns the leaf dir's node id (nil at storage root).
func (*Syncer) EnsureDirChainQuiet ¶ added in v0.44.2
func (s *Syncer) EnsureDirChainQuiet(ctx context.Context, st *model.Storage, rel string) (*int64, error)
EnsureDirChainQuiet is EnsureDirChain without the "create" change frames.
It is for a caller that records folders which were ALREADY on the storage — the lazy catalogue giving an opened folder, and the folders above it, their rows (internal/sync lazy.go). Nothing appeared in anybody's folder: the explorer was already showing it from the disk listing, and a desktop mirror told "create" for every catalogued ancestor would re-list its way up the tree for nothing. The rows, the index and the unique-key race handling are exactly EnsureDirChain's.
func (*Syncer) Move ¶
Move re-homes the node row (and cached descendants) to the new path and re-indexes each. A stale FILE row holding the destination is dropped (its bytes are the ones this move just replaced — see ReclaimDestination); a row that still cannot be re-homed is left live where it is for the next sync walk.
func (*Syncer) MoveRows ¶
MoveRows is Move without the write hook or the change frames: it re-homes the cached subtree and re-indexes every row, and reports whether the top node itself landed at the destination.
⚠ The SUBTREE is the point. A folder move that re-homes only its own row leaves every descendant row — and every descendant search-index entry — pointing at a path with nothing behind it, and nothing later notices, because the paths still look plausible. The AI/MCP surface did exactly that until it was pointed here.
func (*Syncer) ReclaimDestination ¶
func (s *Syncer) ReclaimDestination(ctx context.Context, storageID int64, dstClean string, keepID int64) bool
ReclaimDestination drops a catalogue row that holds dstClean and so blocks the move of keepID into it, and reports whether it removed one.
⚠ Only correct because every caller runs AFTER the bytes have moved onto the destination. A row still claiming that path no longer describes what is there: either the storage's move replaced those bytes (WebDAV MOVE with `Overwrite: T`, an S3 copy+delete, a rename), or the row was stale to begin with. Leaving it made the re-home fail against the live-only unique index (migration 00032), and the moved file lost its identity — versions, shares and comments stayed on a row parked in the trash while its bytes answered under this one.
HARD deleted, for the reason Delete gives: the bytes that row named are gone, and a soft delete would put it in the trash behind a Restore that delivers nothing. ⚠ A DIRECTORY row is left alone: nodes.parent_id cascades, so dropping a folder row would silently take its cached subtree with it.
func (*Syncer) RemoveFromIndex ¶
RemoveFromIndex drops a node from the search index.
func (*Syncer) Touch ¶
Touch records a new modification time for a file whose BYTES did not change.
It is deliberately not Write: a client setting an mtime (rclone does it after every upload it thinks needs it) has changed no content, so re-indexing the file and re-cutting its thumbnail would be pure cost — a thousand-file sync would do a thousand of each for nothing. Nothing is created either; if the row is not there yet, the write that will create it is the one that carries the truth.
func (*Syncer) Trash ¶
Trash retags the node row to its `.filex-trash/` location (original path preserved in storage_key) and soft-deletes it.
Directories drag their cached subtree along inside SoftDeleteAndRetag; the subtree is collected UP FRONT (children are still live) so every affected row can be dropped from the search index afterwards.
func (*Syncer) TrashRows ¶
func (s *Syncer) TrashRows(ctx context.Context, st *model.Storage, rel, trashRel string) (string, bool)
TrashRows is Trash without the write hook or the change frame: the row and index half only, reporting the trashed node's name and whether anything was found to retag.
It is exported because the AI/MCP surface (internal/api/handlers/ai_ops.go) needs exactly this bookkeeping but fires its own OnFileTrashed with its own origin — and used to hand-roll a version of it that forgot the index entirely, so an agent-deleted file stayed findable by search. Sharing the mechanics and leaving the eventing to the caller is what keeps the two from drifting again.
func (*Syncer) WithResolver ¶
WithResolver sets Resolver and returns the Syncer, so a construction site stays one expression.
func (*Syncer) Write ¶
func (s *Syncer) Write(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
Write upserts the node row for a written file, indexes it (which also triggers the content-extraction hook when enabled), dispatches a thumbnail and tells anyone watching the folder.
It reports whether a row was actually written, so a caller with its own eventing (the AI surface) can tell a bookkeeping miss from a success.
func (*Syncer) WriteRows ¶
func (s *Syncer) WriteRows(ctx context.Context, st *model.Storage, rel string, size int64, mime string) (*model.Node, writehook.WriteKind, bool)
WriteRows is Write without the write hook or the change frame: the row, search index and thumbnail only, returning the node it recorded and whether the write created that row or replaced the bytes of one that was already there (writehook.Created / writehook.Replaced — the same `existing != nil` branch the row upsert turns on, not a second lookup).
Separable for the same reason as TrashRows and DeleteRows — a caller that fires its own event needs the bookkeeping without a second one. The AI surface uses it to record a moved file whose source was never catalogued: that is still one move, so it must not also announce itself as a write.
func (*Syncer) WriteSaved ¶
func (s *Syncer) WriteSaved(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
WriteSaved is Write for a write that is one save inside an ONGOING editing session: identical bookkeeping, identical event, but the antivirus scan is scheduled through the debounced save window instead of run now (writehook.OnFileSaved).
⚠ It is not a "lighter" Write to reach for when scans feel expensive. The window trades a bounded stretch of the file sitting unscanned for collapsing a burst of saves into one pass, and that trade only pays when more saves are genuinely coming. A surface that writes a whole file once must call Write.
func (*Syncer) WriteWithoutNotification ¶ added in v0.44.2
func (s *Syncer) WriteWithoutNotification(ctx context.Context, st *model.Storage, rel string, size int64, mime string) bool
WriteWithoutNotification performs every write side effect except the per-file event. Composite operations call it for their component files and emit one meaningful completion event after the whole operation succeeds.
Source Files