filestore

Overview ¶

Package filestore's query pair transaction protocol (this file) persists a query's JSON metadata and its body sidecar as one recoverable logical transaction, under the query store lock (query_lock.go):

1. Stage. Both new files are written under the reserved ".dt-query-txn" directory as "staged.json" and "staged.body", each created exclusively and fsynced; then the directory is fsynced.
2. Commit. commitQueryTransaction first runs checkQueryTxnTargets - the very function recovery runs before it touches anything - on the journal about to be committed, so a transaction recovery would refuse is refused here instead, as an ordinary error on this one write, and the store stays usable. Only then is the journal (queryTxnJournal) written to "journal.tmp", fsynced, renamed to "journal.json", and the directory fsynced. That rename is the commit point: "journal.json" either does not exist or is complete, so no crash can leave a partly written journal under that name. If the checks refuse or the journal cannot be written, the attempt removes its own staged files before returning the error.
3. Install. completeQueryTransaction verifies both staged files against the journal's recorded hashes, then ensureInstalled renames each into place - the body, then the JSON metadata, each followed by a directory fsync. A delete removes the pair instead.
4. Clean up. Whatever staged file is left, then the journal, are removed, and the directory is fsynced.

Every withQueryLock call recovers before doing anything else (recoverQueryTransactions runs completeQueryTransaction in every transaction slot). With a "journal.json" present, recovery runs steps 3 and 4 again, idempotently - the same code a normal write runs. With none, nothing was committed: recovery removes any "journal.tmp" and staged files as uncommitted leftovers, after checking that each is the user's own regular file (checkTxnArtifact; anything else fails closed and nothing is removed). A crash during steps 1-2 therefore never blocks a later read or write.

Deviation from the plan: the plan's Approach text calls for the transaction to "back up the old pair" before installing the new one. This implementation has no backup file. It relies instead on three properties that together give the guarantee the backup step was meant to provide - a DataTug reader never observes a torn pair, and an interruption never loses committed data:

1. Nothing at the final location changes before the commit point, so an interruption before it leaves the previous complete revision untouched: there is nothing to back up yet.
2. After the commit point the transaction is completed forward, so the old pair a backup would protect is never needed. What an interruption can leave is the new pair's install half done, which a backup of the old pair would not help with. The one exception is the writer's own install failing before it changed anything at the query's location (finishQueryTransaction): the old pair is then provably intact and no success was reported, so the writer removes its journal and returns an ordinary error.
3. Recovery finishes a half-done install from the staged files and the journal's hashes alone. A staged file leaves the transaction directory in exactly two ways while a journal exists: ensureInstalled renames it into place (that is its install), or step 4 removes it after both installs are done. (The leftover sweep above runs only when no journal exists.) So while a journal exists, each staged file is either still present - and installed only if it hashes to the recorded content - or already installed, in which case the final file must hash to that content. If a final file was edited outside DataTug between an interrupted install and recovery, it matches neither: recovery then fails closed with an error naming the file, rather than guessing, and the user restores the file. Nothing here depends on a backup surviving the same crash.

A physical backup file would be a weaker guarantee: it would need its own fsync to be trustworthy after a crash, and restoring it correctly would still depend on knowing whether the install it protects against completed - the question the journal's hashes already answer.

Directory fsync is best effort (fsyncDirBestEffort) because it is not available on every platform, notably Windows. There, a power loss (not a process crash) just after a rename can in principle lose that rename; hash verification still refuses to install anything but the recorded content. See query_lock.go's withQueryLock and the recovery, crash and kill-loop tests (query_txn_recovery_test.go, query_txn_crash_test.go, query_txn_killloop_unix_test.go) for the crash-phase analysis this is verified against.

Operational notes:

• Lock fairness. The query store lock is an advisory file lock that a waiter retries every queryLockRetryDelay (20ms); it is not a queue. Under sustained contention one process can re-acquire it again and again while another keeps waiting, bounded only by the waiter's context deadline. Correctness does not depend on fairness: writes are still serialized, and IfMatch is still checked under the lock.
• Other local users. The transaction directory is created 0700 and must be owned by the user running DataTug (vetExistingQueryTxnDir). Once one local user has written to a project's queries, other local users cannot read or write them through DataTug; they get an "owned by another user" error. Each account needs its own checkout. A project nobody has written to yet stays readable without write access (withQueryReadLock).
• Stricter names for legacy writes. SaveQuery, CreateQuery, UpdateQuery, DeleteQuery, CreateQueryFolder and project saves now validate IDs and folder names exactly like the revisioned API (validateQuerySegmentReason). An ID or folder that contains a Windows-illegal character such as ":" or "?" or a control or bidirectional-text character, starts with ".", ends with "." or a space, or names a Windows device is now refused, where it used to be written as given. Reading such a legacy record still works, because LoadQuery and LoadQueries apply only the containment rules (validateQueryReadSegmentReason).
• Read bounds. Each query file is read only when it is a regular file within maxQueryFileSize (16 MiB). A listing call - LoadQueries, or LoadProject's query tree, which shares one budget across every folder - also stops at maxQueryListingBytes (256 MiB) of query files in total, checking each file's size before opening it, and fails with errQueryListingTooLarge. Memory a call holds is a small multiple of the bytes it reads.
• One bad record fails its listing. A query file the store refuses to read (a symlink, an oversize file, a metadata type that cannot name a body file) fails LoadQueries for its folder, and LoadProject, rather than being skipped: a listing never silently omits a query. The error names the file.
• Writes that cannot complete. checkQueryTxnTargets refuses before the commit what it can see: a target or folder that is locked, append-only or immutable, a folder without write permission, a foreign file in a sticky folder. What it cannot see - an ACL that denies deleting a target, a full disk, a file another process holds open on Windows, a sandbox denial, a change made outside DataTug after the commit - can still fail an install. If that happens before anything changed, the writer rolls back (finishQueryTransaction). Otherwise the transaction keeps its slot and only the query it names is refused - its reads, its writes and the listing of its folder, so LoadProject too - with an error naming the entry to fix; every other query is read and written as usual, and the next access after the fix completes the write (query_txn_slots.go). Nothing is served torn.
• Hard links. A regular file is trusted whatever its link count, so a body hard-linked to a file outside the project is read like any other body; a later write replaces the link and leaves the other file alone. Only a user who can already read that file can create such a link - git cannot create hard links, and tar refuses one that points outside the archive - so this store does not refuse them.
• Alias spellings. On a case- or normalization-insensitive file system, a write addressed through an alias of an existing ID ("FOO" for "Foo") keeps the file's original name but records the alias as the JSON "id". The revision does not depend on the spelling.
• Concurrent tampering. Containment is proven against what is on disk when each step runs: no symlinked segment, regular target files, and names derived from the ID. Locations are resolved again after the lock is acquired. That does not defend against another process with write access to the project tree renaming a directory between those checks and the rename that installs a file. Such a process could modify the project's files directly anyway; the threat this store closes is untrusted content that arrives in a clone or an archive.