vfs

package module
v0.6.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

README

vfs

vfs

Object-store abstraction with content-addressed, PQ-encrypted block storage. The storage backplane for HIP-0107 streaming replication.

Status License

Quick start

make build
./bin/vfs put /tmp/in.txt --backend "s3://my-bucket/vfs-prefix?region=us-east-1" \
  --age-recipient "$RECIPIENT" --age-key /tmp/vfs.key

What this is

vfs is a 4 KiB block-level virtual filesystem that hashes every block with blake3-256, encrypts every block with luxfi/age (X25519, optionally hybrid PQ via ML-KEM-768), and stores them on a pluggable backend (file://, s3://, gcs://, azureblob://). In-memory LRU for the hot tier, object store for the cold tier — every Hanzo Go service that needs unlimited write capacity uses vfs instead of pre-sizing a PVC. Already exposes func Mount() for HIP-0106 inclusion.

Specs

Implements:

  • HIP-0107 Streaming Replication over VFS
  • HIP-0106 Unified Cloud Binary (vfs subsystem — already exposes Mount())

Companion to hanzoai/replicate: replicate streams SQLite WAL frames file-level; vfs does block-level for any file.

Architecture

   write()  ->  vfs.PutBlock  ->  blake3-256 hash  ->  luxfi/age encrypt
                                       |
                          blocks/<2-hex-shard>/<full-hash>.zap.age
                                       |
                  file:// | s3:// | gcs:// | azureblob://
                                       |
                          in-memory LRU (hot)
                                       |
                          FUSE mount (lands in 0.2.0)

vfs

S3-backed virtual block filesystem with PQ encryption — unlimited write storage for stateful services.

Build

Quick start

# Build
make build

# Generate an age key (one-time)
go run filippo.io/age/cmd/age-keygen > /tmp/vfs.key
RECIPIENT=$(grep 'public key' /tmp/vfs.key | cut -d: -f2 | tr -d ' ')

# Put + get a block (file:// backend, dev only)
echo "hello world" > /tmp/in.txt
ID=$(./bin/vfs put /tmp/in.txt --backend file:///tmp/vfs-store --age-recipient "$RECIPIENT" --age-key /tmp/vfs.key)
echo "block: $ID"
./bin/vfs get "$ID" --backend file:///tmp/vfs-store --age-key /tmp/vfs.key

# Same flow against S3
./bin/vfs put /tmp/in.txt \
    --backend "s3://my-bucket/vfs-prefix?region=us-east-1" \
    --age-recipient "$RECIPIENT" \
    --age-key /tmp/vfs.key

What it is

A 4 KiB block-level virtual filesystem that:

  • Hashes every block with blake3-256 for content addressability + integrity.
  • Encrypts every block with luxfi/age (X25519, optionally hybrid PQ via ML-KEM-768) before it touches the backend.
  • Stores blocks as blocks/<2-hex-shard>/<full-hash>.zap.age on a pluggable backend (file://, s3://, gcs:// (TODO), azureblob:// (TODO)).
  • Caches recently-used blocks in an in-memory LRU (NVMe write-back cache lands in 0.2.0).
  • Mounts as a FUSE filesystem (make build-fuse; mount layer lands in 0.2.0).

Why

Stateful services in K8s usually pre-size a PVC and either over-provision or run out of disk. vfs lets a service write as if it has unlimited local NVMe — the hot tier is a configurable local cache, the cold tier is S3 (or any object store), and the PQ-encrypted blocks let the operator put cold pages in any region without leaking content.

Companion to hanzo/replicate — replicate streams SQLite WAL frames file-level; vfs does block-level for any file system.

See LLM.md for the full architecture, K8s sidecar pattern, encryption design, and roadmap.

License

Apache 2.0

Documentation

Overview

Package vfs is the top-level virtual filesystem implementation.

Block layer: 4 KiB pages, content-addressable via blake3 (256-bit). Block IDs are the lowercase hex of the blake3 digest of the post-encryption ciphertext, so identical plaintext blocks dedupe IFF they share the same recipient set + ephemeral key. (For practical dedup across a fleet, derive the encryption key deterministically from a domain salt — see BlockKeyMode.)

Index

Constants

View Source 
const BlockSize = 4096

BlockSize is the canonical page size. Bumping requires a new on-disk format magic byte; do not change without coordination.

Variables

View Source 
var Version = "dev"

Version is overridden at build time via -ldflags "-X github.com/hanzoai/vfs.Version=...".

Functions

func Pad 

func Pad(plaintext []byte) []byte

Pad zero-pads or truncates a payload to BlockSize. Plaintext blocks are always exactly BlockSize bytes before encryption so that block boundaries don't leak file lengths to the backend (each block in isolation looks identical in size after age framing — a few hundred bytes of header + tag + BlockSize body).

func SetInstance added in v0.4.3 

func SetInstance(v *VFS)

SetInstance attaches a VFS handle for the mounted routes. Calling this with nil disables the data-plane routes. Idempotent.

func Shutdown added in v0.4.3 

func Shutdown(_ context.Context) error

Shutdown drains the attached VFS instance (closing its backend). Idempotent. Safe to call when no instance was attached.

Types

type BlockID 

type BlockID string

BlockID is the content hash of an encrypted block (blake3-256, hex).

func HashBlock 

func HashBlock(ciphertext []byte) BlockID

HashBlock returns the BlockID for the given ciphertext bytes.

func (BlockID) Path 

func (id BlockID) Path() string

Path returns the canonical backend key for a block ID. Pattern: 

blocks/<first-2-hex>/<full-hash>.zap.age

The 2-hex shard prefix (256 fan-out) keeps single-directory listings tractable for object stores that paginate.

func (BlockID) Verify 

func (id BlockID) Verify(ciphertext []byte) error

Verify checks that the given ciphertext hashes to the expected ID. Returns nil on match, an error on mismatch. Constant-time compare guards against timing leaks on the hash itself (defense-in-depth; blake3 is already collision-resistant).

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is an in-memory LRU bound by total bytes. Used as the hot tier in front of the backend. The on-disk cache layer (NVMe write-back) is a future addition (0.2.0); v0.1.0 keeps everything resident in RAM up to MaxBytes.

func NewCache 

func NewCache(maxBytes int64) *Cache

NewCache creates an LRU cache holding up to maxBytes worth of blocks. maxBytes <= 0 disables eviction (unlimited; for tests).

func (*Cache) Delete 

func (c *Cache) Delete(id BlockID)

Delete removes a single entry.

func (*Cache) Get 

func (c *Cache) Get(id BlockID) ([]byte, bool)

Get returns the cached block (or nil, false on miss).

func (*Cache) Put 

func (c *Cache) Put(id BlockID, val []byte)

Put inserts or updates a block. Older blocks are evicted to keep the total under maxBytes.

func (*Cache) Stats 

func (c *Cache) Stats() (entries int, bytesInUse int64, maxBytes int64)

Stats returns (entries, bytes-in-use, max-bytes).

type Config 

type Config struct {
	Backend  backend.Backend
	Crypto   *Crypto
	CacheMax int64 // bytes; <=0 disables eviction (unbounded)
}

Config holds construction parameters.

type Crypto 

type Crypto struct {
	// contains filtered or unexported fields
}

Crypto wraps a set of age recipients (write-side) and identities (read-side) for per-block encryption. luxfi/age supports classical X25519 + hybrid PQ ML-KEM-768 recipients in the same recipient list.

func NewCrypto 

func NewCrypto(recipients []age.Recipient, identities []age.Identity) (*Crypto, error)

NewCrypto constructs a Crypto with the given recipients (encrypt targets) and identities (decrypt keys). A node that only writes can pass nil identities; a read-only node can pass nil recipients.

func (*Crypto) Decrypt 

func (c *Crypto) Decrypt(ciphertext []byte) ([]byte, error)

Decrypt reverses Encrypt using the configured identities.

func (*Crypto) Encrypt 

func (c *Crypto) Encrypt(plaintext []byte) ([]byte, error)

Encrypt produces an age-encrypted ciphertext for the given plaintext block using the configured recipients.

func (*Crypto) HasIdentities 

func (c *Crypto) HasIdentities() bool

HasIdentities reports whether read decryption is possible.

func (*Crypto) HasRecipients 

func (c *Crypto) HasRecipients() bool

HasRecipients reports whether write encryption is possible.

type FS 

type FS struct {
	// contains filtered or unexported fields
}

FS is a multi-file VFS layered on top of the block VFS. Inode tree is in memory; persisted as a single encrypted blob on Sync.

func NewFS 

func NewFS(ctx context.Context, v *VFS) (*FS, error)

NewFS opens (or creates) a filesystem over the given block VFS. If the metadata blob exists on the backend, the tree is rehydrated; otherwise an empty filesystem with a single root directory is initialised.

func (*FS) Create 

func (fs *FS) Create(p string, mode uint32) (*Inode, error)

Create makes a new empty regular file at path. Parent must exist.

func (*FS) Lookup 

func (fs *FS) Lookup(p string) (*Inode, error)

Lookup resolves a `/`-delimited path to an inode. Returns nil + error matching os.ErrNotExist when any segment is missing.

func (*FS) Mkdir 

func (fs *FS) Mkdir(p string, mode uint32) (*Inode, error)

Mkdir creates a directory at the given path. Parent must exist.

func (*FS) Open 

func (fs *FS) Open(ctx context.Context, p string) (*File, error)

Open returns a File handle for the given path. The path must already exist (use Create first for new files).

func (*FS) PathOfInode 

func (fs *FS) PathOfInode(id InodeID) (string, error)

PathOfInode walks the inode tree from the root and returns the absolute path for the given inode ID. Returns os.ErrNotExist when the ID is unknown. O(depth) — used by the darwin FUSE driver, which gets ops as (inodeID, name) tuples instead of bazil.org/fuse's node-pointer style.

func (*FS) ReadDir 

func (fs *FS) ReadDir(p string) ([]*Inode, error)

ReadDir returns the children of a directory inode.

func (*FS) Remove 

func (fs *FS) Remove(p string) error

Remove deletes a regular file or empty directory.

func (*FS) Sync 

func (fs *FS) Sync(ctx context.Context) error

Sync flushes the metadata tree to the backend if dirty.

type File 

type File struct {
	// contains filtered or unexported fields
}

File is a multi-block file handle. Reads/writes are byte-addressed; the implementation transparently fetches, modifies, and re-encrypts 4 KiB blocks. Dirty blocks are buffered in a per-handle write set until Sync flushes them to the backend (and updates the inode's block list in the FS metadata tree).

Concurrency: a File is owned by a single goroutine. Concurrent FS access across multiple Files is supported by FS's RWMutex.

func (*File) Close 

func (f *File) Close() error

Close flushes pending writes and releases the handle. Mirrors os.File.Close — must be called even on read-only handles to allow the FS to release any resources.

func (*File) ReadAt 

func (f *File) ReadAt(p []byte, off int64) (int, error)

ReadAt reads len(p) bytes at the given offset. Returns io.EOF when reading past the end of the file (matching os.File semantics: when fewer bytes than requested are available, returns those bytes plus io.EOF).

func (*File) Stat 

func (f *File) Stat() (*Inode, error)

Stat returns a snapshot of the file's inode metadata.

func (*File) Sync 

func (f *File) Sync() error

Sync flushes all dirty blocks to the backend, updates the inode's block list, and persists the FS metadata tree. After a successful Sync the file's bytes are durable.

func (*File) Truncate 

func (f *File) Truncate(size uint64) error

Truncate sets the file size, dropping any blocks past the new end. Extending past the current size zero-pads (creates dirty zero blocks up to the new end).

func (*File) WriteAt 

func (f *File) WriteAt(p []byte, off int64) (int, error)

WriteAt writes len(p) bytes at the given offset, extending the file as needed. Partial-block writes do read-modify-write to avoid clobbering adjacent bytes.

Sync MUST be called to flush dirty blocks to the backend; without Sync the writes live only in this File's in-memory dirty set and will be lost on close-without-sync.

type Inode 

type Inode struct {
	ID       InodeID            `json:"id"`
	Name     string             `json:"name,omitempty"`
	Parent   InodeID            `json:"parent,omitempty"`
	Mode     uint32             `json:"mode"` // POSIX mode bits
	Size     uint64             `json:"size"` // exact bytes
	Mtime    time.Time          `json:"mtime"`
	Atime    time.Time          `json:"atime"`
	Ctime    time.Time          `json:"ctime"`
	Blocks   []BlockID          `json:"blocks,omitempty"`   // file content (ordered)
	Children map[string]InodeID `json:"children,omitempty"` // dir entries
}

Inode is the on-disk representation of a file or directory.

We store the inode tree as a single encrypted JSON blob at metadataKey. For >100K files / >1M inodes this becomes too large to reload on every mount; in that scale we shard the metadata blob (`metadata/<inode_id>.zap.age`) and keep an in-memory btree. The shard cutover is post-1.0.

func (*Inode) IsDir 

func (i *Inode) IsDir() bool

IsDir reports whether the inode is a directory.

type InodeID 

type InodeID uint64

InodeID is a monotonically-assigned per-inode handle. ID 1 is always the root directory. 

const RootInode InodeID = 1

RootInode is the well-known root.

type Stats 

type Stats struct {
	Backend     string
	CacheBlocks int
	CacheBytes  int64
	CacheMax    int64
}

Stats returns cache + backend metadata.

type VFS 

type VFS struct {
	// contains filtered or unexported fields
}

VFS is the top-level virtual filesystem handle. It wraps a Backend (object store) and a Crypto (per-block age encryption), with an LRU cache of decrypted blocks in front.

Concurrency: all methods are safe for concurrent use.

func Instance added in v0.4.3 

func Instance() *VFS

Instance returns the currently attached VFS handle (or nil).

func New 

func New(cfg Config) (*VFS, error)

New constructs a VFS. Backend + Crypto are required.

func (*VFS) Close 

func (v *VFS) Close() error

Close releases the backend.

func (*VFS) Delete 

func (v *VFS) Delete(ctx context.Context, id BlockID) error

Delete removes a block from the backend (and the cache).

func (*VFS) GetBlock 

func (v *VFS) GetBlock(ctx context.Context, id BlockID) ([]byte, error)

GetBlock fetches a block by ID, decrypts, and returns the plaintext (still zero-padded to BlockSize — the caller is responsible for remembering the logical size).

func (*VFS) PutBlock 

func (v *VFS) PutBlock(ctx context.Context, plaintext []byte) (BlockID, error)

PutBlock encrypts a plaintext block and writes it to the backend. Returns the BlockID for later GetBlock.

The plaintext is zero-padded to BlockSize before encryption (see vfs/block.go::Pad) so that block boundaries don't leak file lengths.

func (*VFS) Stats 

func (v *VFS) Stats() Stats

Stats returns aggregate counters for observability.

Source Files

View all Source files

Directories

Path Synopsis
cmd
vfs command
vfs-cost command
vfs-cost — print backend cost comparison for a workload.
 vfs-cost — print backend cost comparison for a workload.
pkg
backend
Package backend defines the object-store interface that VFS writes encrypted blocks against.
 Package backend defines the object-store interface that VFS writes encrypted blocks against.
backend/file
Package file is a filesystem-backed Backend for local dev.
 Package file is a filesystem-backed Backend for local dev.
backend/s3
Package s3 is an AWS S3 / S3-compatible Backend.
 Package s3 is an AWS S3 / S3-compatible Backend.
cost
Package cost computes operational cost for VFS workloads across the pluggable backends VFS supports.
 Package cost computes operational cost for VFS workloads across the pluggable backends VFS supports.
mount
Package mount is the FUSE/WASI mount layer.
 Package mount is the FUSE/WASI mount layer.
Package replica is the ONE shared HA-SQLite substrate every Hanzo service uses (HIP-0107): per-org SQLite on the local disk for speed, its durable copy in hanzoai/vfs (SeaweedFS-backed object store), a deterministic single-writer election (Rendezvous/HRW over IAM membership — no coordinator, no lock service, no Postgres, no Redis), and per-org at-rest encryption via KMS.
 Package replica is the ONE shared HA-SQLite substrate every Hanzo service uses (HIP-0107): per-org SQLite on the local disk for speed, its durable copy in hanzoai/vfs (SeaweedFS-backed object store), a deterministic single-writer election (Rendezvous/HRW over IAM membership — no coordinator, no lock service, no Postgres, no Redis), and per-org at-rest encryption via KMS.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.