local

Overview ¶

Package local is a storage.Storage on the local filesystem, for development and single-host deployments: objects are files under one root directory, streamed to and from disk.

Layout ¶

A key is not used as a path. Each object is stored at

<root>/objects/<h[0:2]>/<h[2:4]>/<h>

where h is the hex SHA-256 of the key. Joining the key to the root would break the storage contract on real filesystems: keys that differ only in case collide on macOS and Windows, "a" and "a/b" cannot both exist (a file cannot also be a directory), a segment may exceed the filesystem's name limit, and Windows reserves names such as "CON". Hashing makes every valid key its own file, and no key can reach outside the root.

A file holds the object's bytes followed by a trailer: a JSON header (a format version, the key, content type, metadata, modification time, and a SHA-256 ETag), its length, and a magic number. Open serves the bytes before the trailer. A reader ignores header fields it does not know, so a later version may add fields and still share a root with this one; the version changes only for a change older readers cannot read.

Writes ¶

Put streams into a temporary file, never holding the object in memory, flushes it to disk, makes every directory on the object's path durable (below), renames it into place, and flushes the directory: the rename is atomic, so a reader sees the old object or the new one, never part of one, and on Linux and macOS a Put that returned survives a crash. Delete flushes its directory too. Windows has no directory flush a process can request, so there writes and deletes are atomic but their durability is the filesystem's.

Durability is a fact about the shared root, not about one store: a directory that exists may have been created by another store (another process) that has not flushed its entry yet, or failed to. So a store trusts nothing it has not flushed itself. The root is created on the first Put, in a parent directory that must exist; its entry there is flushed before the marker file <root>/.durable is written, and a store that finds no marker flushes the parent itself. Inside the root, a store flushes the entry of every directory on an object's path, whoever created it, before its first Put into that directory.

Readers ¶

An open reader keeps the version it opened while a Put replaces the object or a Delete removes it, on every platform. On Windows the store opens objects with delete sharing and replaces and deletes them with POSIX semantics (NTFS on Windows 10 1709 or later; on a filesystem without them, FAT say, an open reader blocks both until it is closed).

Temporary files ¶

Each store writes its temporary files in its own work directory, <root>/tmp/w-*, and holds a lock on the directory's owner file for as long as the store exists. The owner file is created and locked as owner.new and only then renamed to owner, so every owner file is locked from the moment it has that name. A failed Put removes its temporary file; a process killed mid-Put cannot, but its lock ends with it, so a store's first Put removes the work directories whose owner file it can lock: those of stores that are gone. A directory without an owner file is kept (its store may be setting it up; if it died doing so, it left an empty directory), and so is a live store's, however long its Puts wait on their sources: no age or clock is involved. On a platform without file locks (flock, or LockFileEx on Windows) nothing is swept.