Documentation
¶
Overview ¶
Package bridge is the EXPORTED mlfs↔VFS metadata-only bridge (handoff Item C): it moves files between mlfs and blobgw by SYNTHESISING metadata over the shared casstore chunks — no file bytes are ever read or written in either direction.
This file (the mlfs-side surface, type FS) provides:
- CreateInodeFromChunks (VFS→mlfs, "create-inode-from-manifest") lays a blob_ref's already-existing casstore chunks into a new mlfs inode's slices by synthesising each slice's manifest over those chunks, then links the inode into a directory. No download, no chunking, no byte copy.
- FileChunks (mlfs→VFS) gathers a file's slices' ordered casstore chunk lists so the coordinator can register a blob_ref over the SAME chunks.
The coordinator (type Bridge, in coordinator.go) drives both directions end to end. The package is exported (not internal/) so callers outside mlfs can wire it, but it lives IN the mlfs module because only mlfs-internal code can construct the meta engine + chunk store it operates over. The zero-byte-movement and same-chunk-dedup invariants are enforced here (only metadata-only chunkstore.RegisterSlice + meta row writes are ever called) and asserted by tests.
Index ¶
- Constants
- Variables
- type Bridge
- func (b *Bridge) MlfsToRef(ctx context.Context, path, ref, contentType string, userMeta map[string]string) (gateway.ObjectInfo, error)
- func (b *Bridge) RefToMlfs(ctx context.Context, ref, path string, class StoreClass) (meta.Ino, error)
- func (b *Bridge) VerifyMlfsContent(ctx context.Context, inode meta.Ino, ...) error
- func (b *Bridge) VerifyRefContent(ctx context.Context, ref string) error
- type FS
- func (fs *FS) ContentHashAndSize(ctx context.Context, inode meta.Ino) (string, int64, error)
- func (fs *FS) CreateInodeFromChunks(ctx context.Context, parent meta.Ino, name string, ...) (meta.Ino, error)
- func (fs *FS) FileChunks(ctx context.Context, inode meta.Ino) ([]snapshot.ChunkLocation, error)
- func (fs *FS) Resolve(ctx context.Context, path string) (meta.Ino, error)
- func (fs *FS) Unlink(ctx context.Context, parent meta.Ino, name string) error
- type Ino
- type Stack
- type StackConfig
- type StoreClass
Constants ¶
const RootInode = meta.RootInode
RootInode is the fixed inode of the mlfs root, re-exported for path resolution.
Variables ¶
var ErrContentHashMismatch = errors.New("bridge: content hash mismatch")
ErrContentHashMismatch is returned by VerifyContent when the sha256 computed by streaming all chunks of a ref or mlfs file disagrees with its declared content_hash. It always wraps a detail string with the expected and computed values.
var ErrNoContentHash = errors.New("bridge: inode has no persisted content hash")
ErrNoContentHash is returned by VerifyMlfsContent when the inode has no persisted whole-file content hash (the file was never flushed with SetContentHash). Bridging such a file is also refused by MlfsToRef for the same reason.
var ErrSplitManifestStore = errors.New("bridge: blobgw and mlfs sides must share the same manifest upstream for GC correctness; split manifest stores would cause silent data loss")
ErrSplitManifestStore is returned when a Stack is constructed (or validated) with different manifest stores for the blobgw side and the mlfs side. The correctness of GC depends on a SINGLE manifest upstream that both sides share: ChunkedGC marks chunks live by walking ALL manifests it can see; if blobgw ref manifests live in a different store than mlfs slice manifests, a GC pass that walks only one store will reclaim packs the other side still references — silent data loss. NewStack enforces this structurally (one manifests pointer), and AssertGCCoversAllManifests exposes the same check for callers that build their own Stack components outside NewStack.
Functions ¶
This section is empty.
Types ¶
type Bridge ¶
type Bridge struct {
// contains filtered or unexported fields
}
Bridge is the mlfs↔VFS coordinator: it drives both directions of the metadata-only bridge over a shared casstore domain, moving files between an mlfs filesystem and a blobgw object gateway WITHOUT reading or writing a single file byte. It owns the hard cross-side invariant — mlfs mount domain == blobgw ref domain == the tenant domain, both over the same Postgres — and refuses to operate when the two sides disagree on the domain.
Both directions are pure metadata synthesis over the SAME casstore pack_manifest chunks: MlfsToRef reads an mlfs file's chunk list and registers a blob_ref over it; RefToMlfs reads a ref's chunk list and lays it into a new mlfs inode. The ref and the mlfs file end up referencing the identical packs — dedup preserved, no new bytes, and each reads back byte-identical to the other.
content_hash trust boundary ¶
The bridge carries content_hash VERBATIM from source to destination (mlfs node content_hash → ref content_hash in MlfsToRef; ref content_hash → mlfs node content_hash in RefToMlfs). It does NOT re-hash the object bytes by default — that would require reading every chunk, defeating the zero-copy guarantee. The bridge therefore TRUSTS that the source's content_hash is correct. Consequences:
- A stale or wrong content_hash at the source propagates to the destination. The only downstream effect is a wrong ETag/checksum on the ref side; the bytes are still served correctly (they come from the chunks, not the hash field).
- Callers that need end-to-end hash verification (e.g. a one-time admin repair pass) may call VerifyContent after bridging; it streams all chunks once and asserts the sha256 matches the declared hash. This is the only exception to zero-copy and is opt-in.
Single-writer / quiescence requirement ¶
MlfsToRef reads content_hash, size, and chunk list across THREE separate non-transactional metadata reads (ContentHashAndSize → FileChunks). A concurrent whole-slice overwrite mid-bridge could produce an inconsistent snapshot: the hash from one version, the chunks from another. Callers MUST ensure the file is quiesced (no concurrent writers) for the duration of MlfsToRef. RefToMlfs is safe under concurrent readers (it creates a new inode atomically before slices are registered) but the directory parent must not be concurrently deleted. For the typical admin/maintenance use case (the file has been finalised and its hash stamped before bridging) this requirement is naturally satisfied.
func NewBridge ¶
NewBridge constructs the coordinator over an mlfs bridge surface and a blobgw gateway that MUST share the dedup/ref domain. It returns an error if the gateway's domain differs from the supplied tenant domain, since a cross-domain bridge would reference chunks the other side's domain does not hold (the chunks are isolated by domain in pack_manifest).
func (*Bridge) MlfsToRef ¶
func (b *Bridge) MlfsToRef(ctx context.Context, path, ref, contentType string, userMeta map[string]string) (gateway.ObjectInfo, error)
MlfsToRef registers a blob_ref named ref for the mlfs file at path, WITHOUT moving any bytes: it reads the file's whole-file content hash + size and its ordered casstore chunk list, then binds the ref over the SAME chunks via the gateway's metadata-only RegisterRef. The ref is immediately readable through blobgw (and an edge presigned download URL) and reads back byte-identical to the mlfs file.
The file MUST have a persisted content hash (mlfs stamps it on flush/close, C0); a file never flushed with a hash is refused, because the ref's content_hash must be authoritative for client-side idempotency/dedup. contentType is carried onto the ref's manifest (drives casstore's compression policy for any FUTURE writes to the ref; the existing packs are untouched). userMeta is attached durably.
func (*Bridge) RefToMlfs ¶
func (b *Bridge) RefToMlfs(ctx context.Context, ref, path string, class StoreClass) (meta.Ino, error)
RefToMlfs creates an mlfs file at path from the blob_ref ref, WITHOUT moving any bytes: it reads the ref's metadata (content hash + size) and its ordered casstore chunk list, then lays those chunks into a new inode via the metadata-only mlfs side, linking it under path's parent directory. The new file reads back byte-identical to the ref.
path's parent directory MUST already exist (the bridge does not mkdir -p); the leaf name is the new file. class selects the (already-materialised) compression class. Returns the new inode.
All-or-nothing guarantee: if CreateInodeFromChunks succeeds in creating the inode but a subsequent step fails (e.g. slice registration or content-hash stamp), the bridge unlinks the partially-created inode before returning the error. Callers see either a fully-constructed file or no file — never a partial inode left under path. The unlink is best-effort; if it itself fails, both errors are returned (wrapped) so the caller can surface the unlink failure alongside the original.
func (*Bridge) VerifyMlfsContent ¶
func (b *Bridge) VerifyMlfsContent(ctx context.Context, inode meta.Ino, read func(ctx context.Context, inode meta.Ino, off int64, buf []byte) (int, error)) error
VerifyMlfsContent is the opt-in content-hash verification for the mlfs side: it streams the inode's full content through the mlfs data path and asserts the sha256 matches the inode's persisted content_hash.
Same caveats as VerifyRefContent: not zero-copy, opt-in only, admin/repair use. Returns ErrContentHashMismatch if hashes disagree, ErrNoContentHash if the inode has no persisted hash (cannot verify), or a wrapped I/O error.
fileio is a ReadAt-style function matching the mlfs data path signature: func(ctx, inode, offset int64, buf []byte) (int, error). Pass fileio.New(...).Read.
func (*Bridge) VerifyRefContent ¶
VerifyRefContent is the opt-in content-hash verification for the blobgw side: it streams the ref's full body through the gateway (reading ALL chunk bytes) and asserts that the computed sha256 matches the ref's declared ContentHash.
This is NOT zero-copy: it reads every chunk byte once. Use it for admin/repair passes where end-to-end hash fidelity must be confirmed (e.g. after bridging from an mlfs source whose content_hash was suspect). Do NOT call it in the hot path.
Returns ErrContentHashMismatch (wrapping a detail string) if the hashes disagree, nil if they match, or a wrapped I/O error if the ref could not be read.
type FS ¶
type FS struct {
// contains filtered or unexported fields
}
FS is the metadata-only bridge surface over an mlfs metadata engine + chunk store. Construct with New using the SAME engine and CasStore the running mlfs daemon uses (so the bridged inode is a first-class file the FUSE path serves and GC accounts for). It is safe for concurrent use to the same degree the underlying engine/store are.
func New ¶
func New(m *meta.Engine, chunks *chunkstore.CasStore) *FS
New wraps an mlfs metadata engine and its casstore-backed chunk store as the bridge surface. Both must point at the SAME tenant domain and Postgres as the blobgw side of the bridge (the hard cross-side invariant: mlfs mount domain == blobgw ref domain == tenant domain, one PG).
func (*FS) ContentHashAndSize ¶
ContentHashAndSize returns a file's persisted whole-file content hash and byte length — the two values the mlfs→VFS direction stamps onto the blob_ref. An empty hash means the file was never flushed with a hash; the coordinator refuses to bridge such a file (the ref's content_hash must be authoritative).
func (*FS) CreateInodeFromChunks ¶
func (fs *FS) CreateInodeFromChunks(ctx context.Context, parent meta.Ino, name string, chunks []snapshot.ChunkLocation, contentHash string, size int64, class StoreClass) (meta.Ino, error)
CreateInodeFromChunks creates a new regular file named name under directory parent whose bytes are the concatenation of chunks (an ordered casstore chunk list, e.g. a blob_ref's manifest chunks that already live in this tenant's pack set), links it into the directory, and stamps its whole-file content hash — all WITHOUT reading or writing a single chunk/pack byte.
How the metadata-only inode is built ¶
mlfs models a file as fixed-size 64 MiB chunks (meta.ChunkSize), each holding slices; a slice's bytes live in a casstore object keyed by its slice id. This lays the ref's chunks into ONE mlfs slice per 64 MiB file-chunk:
- We walk the ordered casstore chunks, accumulating the file byte offset. Each mlfs 64 MiB chunk gets its own slice whose casstore manifest is REGISTERED (chunkstore.RegisterSlice → casstore RegisterManifest) over the casstore chunks that overlap that mlfs chunk. No pack is written; the slice's manifest just points at the shared chunks.
- A casstore chunk that STRADDLES a 64 MiB boundary is referenced WHOLE by both the slice before and the slice after the boundary; each slice's soff/slen read window (meta.Slice.Off/Len) selects only the bytes that belong to its mlfs chunk. This keeps every casstore chunk intact (its content hash is unchanged and both slices dedup to the exact same pack bytes) while the visible file bytes are byte-correct.
contentHash MUST be the ref's whole-object sha256 hex (byte-identical to what mlfs's own SetContentHash records and to blobgw's ContentHash), and size the object's byte length; both are stamped onto the new node so the bridged file is indistinguishable from a natively-written one. class selects the slices' (already-materialised) compression class.
It returns the new inode on success. When the inode is created (inode > 0) but a subsequent step fails, the returned inode is non-zero so the coordinator can unlink it — Bridge.RefToMlfs does exactly this (via FS.Unlink), giving all-or-nothing semantics. A create/link conflict (name already exists) is surfaced as errno wrapped in an error with inode == 0.
func (*FS) FileChunks ¶
FileChunks returns the ordered casstore chunk list for the whole file at inode, gathered from its slices, WITHOUT reading any chunk/pack bytes. It is the mlfs→VFS read half of the bridge: the coordinator hands the returned chunks to blobgw RegisterRef to synthesise a blob_ref over the SAME chunks.
It walks the file's chunks in order (0..ceil(length/ChunkSize)), resolves each mlfs chunk's visible slices (newest-wins overlay, exactly like the data path), and emits each visible casstore chunk ONCE — attributed to the window its bytes begin in — so a chunk that straddles a 64 MiB mlfs-chunk boundary (shared whole by the slices on both sides via their soff/slen windows) is not double-counted. A file whose slices were each a whole casstore object (the common case, including any inode this package created) yields its chunk list directly; the overlay resolution makes it correct even for natively-written files with overlapping whole-slice writes.
Constraint: the visible ranges must fall on casstore-chunk boundaries (they do for bridge-created inodes and for whole-slice writes). A sub-chunk overwrite that leaves a PARTIAL casstore chunk visible cannot be bridged byte-exactly (a partial chunk has no whole chunk to reference); FileChunks detects this via a byte- accounting guard (the emitted chunks must sum to the file length) and refuses it, so such a file must be re-materialised (recompacted) first. This is a deferred edge case per the handoff. A hole (sparse range) is likewise refused.
func (*FS) Resolve ¶
Resolve walks an absolute, slash-separated path from the mlfs root to its inode. A leading "/" is optional; "" and "/" resolve to the root. It is the path→inode step of the mlfs→VFS bridge direction. Returns an error (wrapping the engine's errno) if any component is missing or not a directory.
func (*FS) Unlink ¶
Unlink removes the directory entry name under parent. It is the rollback path for RefToMlfs: if CreateInodeFromChunks creates an inode but a subsequent step fails, the coordinator calls Unlink to remove the partial inode so no incomplete file is left under path. Returns an error wrapping the engine's errno on failure.
type Ino ¶
Ino re-exports mlfs's inode type so bridge callers can name it without importing mlfs internals.
type Stack ¶
type Stack struct {
Bridge *Bridge
Gateway *gateway.Gateway
FS *FS
Engine *meta.Engine
PGIndex *pgindex.Store
Manifest *manifeststore.Store
Chunks blobstore.Storage
// SliceStore is the mlfs-side chunk store (over the shared casstore) the FS
// registers slices into; the data path (fileio) reads through the SAME store, so
// a bridged slice is immediately readable. Exposed for callers/tests that drive
// the mlfs data path directly.
SliceStore *chunkstore.CasStore
// GC is the shared casstore GC (index-aware) over the one chunk store; a caller
// may schedule it. Both sides' chunks live here, so one GC reclaims for both.
GC *snapshot.ChunkedGC
// contains filtered or unexported fields
}
Stack is a fully-wired bridge over a SINGLE shared Postgres and a SINGLE shared chunk blobstore, exactly modelling the production invariant: mlfs mount domain == blobgw ref domain == tenant domain, both over the same PG. It is what the bridge CLI subcommand and the end-to-end integration test both construct, so the code path they exercise is the real one.
Both sides' ChunkedStores share the ONE chunk blobstore and the ONE pgindex (which backs both blobgw's blob_ref/pack_manifest AND mlfs's dedup lookups), so a chunk mlfs writes is immediately visible to blobgw's RegisterRef validation via the same pack_manifest rows — the whole point of the bridge.
GC correctness: the shared GC walks the SINGLE manifeststore that backs BOTH sides. This is the only configuration where GC is correct: if blobgw ref manifests and mlfs slice manifests lived in SEPARATE stores, a GC pass that only walks one store would reclaim packs the other side still references. NewStack enforces this structurally; call AssertGCCoversAllManifests to verify at any point.
func NewStack ¶
func NewStack(ctx context.Context, cfg StackConfig) (*Stack, error)
NewStack wires both bridge sides over the shared PG + chunk store and returns the coordinator plus the pieces a caller may need (GC, stores). It migrates every schema (mlfs meta, slice_manifest, blob_ref/pack_manifest) so a fresh database is ready. The caller owns the DB pool lifecycle.
func (*Stack) AssertGCCoversAllManifests ¶
AssertGCCoversAllManifests returns ErrSplitManifestStore if the stack's GC is not wired with the same manifest store that backs both the blobgw and mlfs sides. Call this after constructing a Stack (especially outside NewStack) to fail-close rather than silently misconfiguring GC. NewStack always passes; a hand-assembled Stack with split manifest stores fails.
type StackConfig ¶
type StackConfig struct {
// DB is the shared Postgres pool. It backs mlfs's meta engine + slice_manifest,
// and blobgw's blob_ref + pack_manifest — one datastore, the tenant domain.
DB *sql.DB
// ChunkDir is the local-filesystem directory for the shared chunk blobstore.
// In production this is S3; a local dir keeps the CLI/test self-contained while
// exercising the identical ChunkedStore/dedup/GC code. Ignored when Chunks is set.
ChunkDir string
// Chunks, when non-nil, is the shared chunk blobstore to use directly instead of
// constructing a local one from ChunkDir. Tests inject an instrumented store here
// to assert the bridge moves ZERO chunk bytes; production leaves it nil.
Chunks blobstore.Storage
// Domain is the tenant domain shared by both sides (dedup domain, blob ref
// namespace, mlfs mount domain). Must be identical for the bridge to operate.
Domain string
// PackTargetBytes is the casstore pack target (0 = default ~16 MiB); both sides
// must agree so chunk identity is consistent.
PackTargetBytes int
// Region is the mlfs inode-allocation region prefix for the meta engine.
Region uint8
}
StackConfig configures NewStack.
type StoreClass ¶
type StoreClass uint8
StoreClass selects a bridged file's compression class, re-exported so the bridge module can pass it without importing mlfs internals. It maps 1:1 to chunkstore.StoreClass.
const ( // ClassDefault leaves the codec to the backing policy (generic content // compresses). ClassDefault StoreClass = StoreClass(chunkstore.ClassDefault) // ClassUncompressed forces the slice's pack to be stored uncompressed // (model/tensor data). Only meaningful on a fresh write; for a bridged inode // the packs already exist, so the class only stamps the synthesised manifests // for consistency. ClassUncompressed StoreClass = StoreClass(chunkstore.ClassUncompressed) )
Source Files