README
¶
lake
Lake is a Go library that provides a unified interface for reading and writing collections of files across different storage backends: filesystem directories, ZIP archives, and single files. It is part of the itch.io ecosystem and primarily used by wharf.
Core concepts
Pool
A Pool gives indexed read/seek access to an ordered list of files. Pools are backed by different storage types but expose the same interface:
type Pool interface {
GetSize(fileIndex int64) int64
GetReader(fileIndex int64) (io.Reader, error)
GetReadSeeker(fileIndex int64) (io.ReadSeeker, error)
Close() error
}
WritablePool extends Pool with write access, and CaseFixerPool handles renaming on-disk files to match a container's expected casing (useful for case-insensitive filesystems).
TLC Container
The tlc.Container is a protobuf-based metadata format that captures a
complete directory structure: files, directories, and symlinks with their
sizes, permissions, and paths. Containers are produced by walking a directory or
ZIP archive, and can be used to recreate that structure on disk.
wharf embeds Container structs serialized
via protobuf in patch files (.pwr) and signature files to describe both the "old" and
"new" versions of a build, paired with binary diffs of the actual file
contents. butler is the CLI that drives
wharf to produce and apply these patches for itch.io uploads.
Lake does not define the patch format itself, it provides the filesystem abstraction layer and directory manifest that wharf builds on top of.
Package structure
lake (root)
Core interfaces: Pool, WritablePool, CaseFixerPool, and case-fix
helpers.
tlc/
Container type and filesystem operations:
- Walking:
WalkDir,WalkZip,WalkAny(auto-detects),WalkSingle - Preparing:
Container.Preparecreates all directories, files, and symlinks from a container spec - Validation: Container integrity checks
- Comparison: Diff containers to detect changes
- Permissions: Executable detection (ELF, Mach-O, shell scripts)
- Case handling: Cross-platform case-insensitive filesystem support
- Filtering:
FilterFuncand built-inPresetFilterto ignore VCS metadata, OS junk files, etc.
pools/
Pool implementations for different backends:
| Package | Description |
|---|---|
fspool |
Filesystem-backed pool (reads/writes files in a directory) |
zippool |
Read-only pool backed by a ZIP archive |
zipwriterpool |
Write-only pool that produces a ZIP archive |
cachepool |
Caching wrapper around another pool |
nullpool |
Discards all data (useful for testing and benchmarking) |
singlefilepool |
Writes a single file |
Usage
import (
"github.com/itchio/lake/tlc"
"github.com/itchio/lake/pools/fspool"
)
// Walk a directory to get its container (file listing)
container, err := tlc.WalkDir("/path/to/dir", tlc.WalkOpts{
Filter: tlc.PresetFilter,
})
// Open a filesystem pool for reading
pool := fspool.New(container, "/path/to/dir")
defer pool.Close()
reader, err := pool.GetReader(0) // read first file
License
MIT
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CaseFix ¶
type CaseFixParams ¶
type CaseFixParams struct {
Stats *CaseFixStats
Consumer *state.Consumer
}
type CaseFixStats ¶
type CaseFixStats struct {
Fixes []CaseFix
}
type CaseFixerPool ¶
type CaseFixerPool interface {
// FixExistingCase is ugly, but so is the real world.
//
// It collects all the paths in the pool's container, and
// from shortest to longest, makes sure that they have the
// case we expected.
//
// For example, if we have container with files:
// - Foo/Bar
// And directories:
// - Foo/
// But on disk, we have:
// - FOO/bar
//
// This would rename `FOO` to `Foo`,
// and `Foo/bar` to `Foo/Bar`.
FixExistingCase(params CaseFixParams) error
}
type Pool ¶
type Pool interface {
// GetSize returns the size of a given file entry, as specified by the container
// the pool was built with.
GetSize(fileIndex int64) int64
// GetReader returns a Reader for a given file. Typically, readers are cached,
// so a second call to GetReader will close the last reader.
GetReader(fileIndex int64) (io.Reader, error)
// GetReadSeeker beahves like GetReader (including caching) but allows seeking
// as well. For some pools (like zip pool), this call may involve decompressing
// *an entire entry* and then returning a temporary *os.File (or memory file).
GetReadSeeker(fileIndex int64) (io.ReadSeeker, error)
// Close closes the last opened reader, if any. Does not impact `GetWriter`
// at all. Calling Close doesn't render the pool unusable, all its other methods
// should not error out afterwards.
Close() error
}
A Pool gives read+seek access to an ordered list of files, by index
type WritablePool ¶
type WritablePool interface {
Pool
// GetWriter returns a writer for a given file entry
// This also truncates the file on disk (or whatever the pool represents),
// so that the file's final size is the number of bytes written on close.
// Writers aren't cached, so this can be called concurrently.
GetWriter(fileIndex int64) (io.WriteCloser, error)
}
A WritablePool adds writing access to the Pool type
Source Files
Directories