core

Amber-Store Core

A content-addressable store for filesystem trees — arbitrarily deep directories and files, where file content is split by content-defined chunking and every object is identified by a fixed 32-byte key derived from a hash of its content.

Status: working implementation. This is the local core: a Go library (plus a minimal CLI) that owns a store in a local directory — no sockets, no networking, no authentication. It is the foundation the amber server is built on, and it embeds directly into other projects (e.g. the JOBS engine). The design is specified in architecture/.

What it is

Amber-Store models a POSIX-style filesystem as an immutable, deduplicated Merkle structure in a content-addressed store:

• Files are split into chunks by content-defined chunking (CDC); each chunk is a Blob. Large files get a multi-level FileNode index for O(log n) random-access seek.
• Directories are prolly trees — sorted maps from name to entry, chunked at entry boundaries — so a directory with >100K entries can be looked up, iterated, and mutated with sub-O(n) memory.
• Metadata (type, mode, uid, gid, mtime, optional xattrs) lives in the parent directory entry. Symlinks and special files are stored inline; regular files and subdirectories reference content by key.
• Deduplication is automatic: identical content yields an identical key, and editing one entry re-writes only the O(log n) objects on its path — the rest is structurally shared with the previous tree.

Design goals

• Very large directories (>100K entries) processed with less than O(n) memory.
• Directory entries carry all filesystem metadata; file content is reached through the store by key.
• Deterministic, implementation-independent encoding so any reader recomputes the same key for the same content.

Object types

Encoding

• Keys are 32 bytes: a type/length header plus a truncated BLAKE3 hash of the content, or for a git object its SHA-1.
• Structured objects use deterministic CBOR (RFC 8949 §4.2); Blobs are raw bytes. Deterministic encoding is required because the key's hash is taken over the serialized bytes.

The library

The packages compose loosely; consumers wire them together and own their store-directory layout. The conventional layout (which the CLI uses) is <dir>/packstore for objects and <dir>/refs for references. A store directory may be open in any number of processes at once: readers take no lock, each writer owns an active segment of its own, and a GC cycle keeps the writers of other processes waiting while it runs (architecture/packstore.md). They must all run on one host, on a local filesystem.

A minimal embedding looks like:

objects, _ := packstore.Open(filepath.Join(dir, "packstore"))
defer objects.Close()

root, stats, _ := ingest.Dir(objects, "./some/dir", ingest.Opts{})
_ = tarexport.Write(w, root, objects.Get)

Objects are stored uncompressed unless the store is opened with a compression option:

objects, _ := packstore.Open(filepath.Join(dir, "packstore"),
	packstore.WithCompression(amberpack.Compression{Algorithm: amberpack.Zstd}))

WithCompression takes none, zstd (levels 1–22) or lz4 (0 for the fast compressor, 1–12 for high compression); level 0 is each algorithm's default. WithCompressionFor adds a function that chooses per object, from its key and bytes. Every store reads records of every codec, whatever it was opened with, and existing records are never recompressed. Releases before this one read raw and zstd records only. A store written to with lz4 holds segments at a format version they refuse, so they cannot open that store at all; one that never is stays readable by them. See architecture/amberpack.md.

The CLI

Every command operates directly on the store directory given by --store or $AMBER_STORE, creating it as needed. Ingest a directory or a single file — the root key (hex) is printed to stdout; a directory root is a DirNode, a single-file root is the file's content key:

amber-store --store ./store ingest ./some/dir           # print the root key
amber-store --store ./store ingest --ref backups/home ./some/dir  # also name it

New objects are stored uncompressed by default. The global flag --compression none|zstd[:LEVEL]|lz4[:LEVEL], given before the command, sets the compression for what that command writes:

amber-store --store ./store --compression zstd:19 ingest ./some/dir

Inspect and export by key or reference, optionally addressing a subdirectory with KEY/PATH or ref:NAME[@PATH]:

amber-store --store ./store ls KEY[/PATH]               # list entries, ls -l style (--keys adds content keys)
amber-store --store ./store ls ref:backups/home@sub/dir # ref:NAME[@PATH] works wherever KEY[/PATH] does
amber-store --store ./store export ref:backups/home -o tree.tar  # PAX tar (default: stdout)
amber-store --store ./store restore ref:backups/home ./dest      # recreate the tree on disk
amber-store --store ./store ref list                    # references: name, key, created, user
amber-store --store ./store ref set NAME KEY            # name an existing key
amber-store --store ./store ref set --expect OLD NAME KEY  # move NAME only if it still points at OLD ('none': only create)
amber-store --store ./store ref get NAME                # print the key a name points at
amber-store --store ./store ref rm NAME                 # delete the name; objects stay
amber-store --store ./store commit create --ref main --author 'Ann <ann@example.com>' -m 'first' KEY  # record a tree; --parent KEY|ref:NAME links history; --change-id HEX carries a change id
amber-store --store ./store commit show ref:main        # print a commit, git cat-file style

A commit key, or a reference to one, works wherever a directory KEY does — it stands for the commit's tree. So does a directory entry that holds a commit: ls, export and restore skip the commit object and continue with its tree.

Ingest parallelism is set with --jobs (default: number of CPUs). Chunking is tunable with --min/--avg/--max (ultracdc byte chunking) and --item-bits (index/entry chunking); --xattr-inline-max controls when extended attributes spill to an XattrSet object.

.amberignore files exclude entries from ingestion, with .gitignore semantics: negation (!pattern), ** globs, directory-only (name/) and anchored (/name) patterns; a file in any subdirectory applies to that subtree and composes with inherited patterns (last match wins). Ignored directories are pruned without being read. The .amberignore files themselves are always stored, so a restored tree re-ingests to the same root. --no-ignore disables all ignore processing.

Architecture

Development

The repository uses a Nix flake (with direnv) to provide the Go toolchain.

direnv allow        # or: nix develop
go build ./...
go test ./...

• Module: github.com/amber-store/core
• Go: 1.26+

License

Licensed under the GNU Lesser General Public License, version 3 only (LGPL-3.0-only). See LICENSE for the LGPL terms and COPYING for the GPL terms incorporated by the LGPL.

Third-party notices retain their stated licenses.