wapp

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 2, 2026 License: MIT Imports: 18 Imported by: 0

README

wapp

WAPP (Wippy Application Pack) is a binary archive format for packaging filesystem trees, registry entries, and metadata.

Format Structure

Header (268 bytes) + Data Frames + Compressed TOC + Footer (16 bytes)
  • Header: Magic number, version, data offset/size, SHA-256 hash
  • Data Frames: Compressed file content in 10MB frames
  • TOC: Msgpack-encoded table of contents (zstd compressed)
  • Footer: TOC offset and size for footer-first reading

Contents

A WAPP file can contain:

  • Metadata: Pack-level key-value metadata
  • Registry Entries: Typed records with ID, Kind, Meta, and Data fields for storing configuration, manifests, or any structured data
  • Resources: Filesystem trees with per-file compression and integrity hashes

Features

  • Per-file zstd compression (skips already-compressed formats)
  • Lazy loading with footer-first reading
  • Multiple filesystem tree resources per pack
  • Registry entries for structured data storage
  • SHA-256 integrity verification
  • O(1) file and resource lookups
  • Concurrent-safe reads with decompression cache
  • fs.FS interface compatibility

Installation

go get github.com/wippyai/wapp

Usage

Writing
writer := wapp.NewWriter()

// Pack with entries and filesystem
entries := []wapp.Entry{
    {
        ID:   wapp.NewID("app", "manifest"),
        Kind: "manifest",
        Meta: wapp.Metadata{"version": "1.0"},
        Data: map[string]any{"name": "myapp", "routes": []string{"/api", "/web"}},
    },
    {
        ID:   wapp.NewID("app", "config"),
        Kind: "config",
        Data: map[string]any{"debug": false, "port": 8080},
    },
}

err := writer.Pack(
    wapp.Metadata{"version": "1.0"},
    entries,
    os.DirFS("./myapp"),
    wapp.NewID("app", "files"),
    nil,
    outputFile,
)

// Multiple resources
err := writer.PackWithResources(
    metadata,
    entries,
    []wapp.ResourceSpec{
        {ID: wapp.NewID("app", "frontend"), FS: os.DirFS("./frontend")},
        {ID: wapp.NewID("app", "backend"), FS: os.DirFS("./backend")},
    },
    outputFile,
)

// Entries only (no filesystem)
err := writer.PackEntries(metadata, entries, outputFile)
Reading
reader, err := wapp.NewReader(file)

// Get pack metadata
meta, err := reader.GetMetadata()

// Get registry entries
entries, err := reader.GetEntries()
for _, entry := range entries {
    fmt.Printf("Entry: %s, Kind: %s\n", entry.ID, entry.Kind)
}

// List resources
resources := reader.ListResources()

// Get filesystem
fsys, err := reader.GetFS(wapp.NewID("app", "files"))

// Use standard fs operations
data, err := fs.ReadFile(fsys, "config.json")
dirEntries, err := fs.ReadDir(fsys, "templates")
Options
// Custom compression decision
writer := wapp.NewWriter(
    wapp.WithCompressionFunc(func(path string) bool {
        return !strings.HasSuffix(path, ".gz")
    }),
    wapp.WithProgressCallback(func(id wapp.ID, current, total int) {
        fmt.Printf("%s: %d/%d\n", id, current, total)
    }),
)

// Custom decompression cache
reader, err := wapp.NewReaderWithOptions(file,
    wapp.WithDecompressionCacheLimit(128 << 20), // 128MB
)

Dependencies

  • github.com/klauspost/compress/zstd - zstd compression
  • github.com/hashicorp/go-msgpack/v2 - msgpack serialization

Documentation

Overview

Package wapp provides the WAPP (Wippy Application Pack) binary archive format.

WAPP is designed for packaging filesystem trees, registry entries, and metadata into a single binary file. It supports lazy loading, per-file compression, and integrity verification.

Format Structure

A WAPP file consists of:

  • Header (268 bytes): Magic, version, data offset/size, SHA-256 hash
  • Data frames: Compressed file content in 10MB frames
  • TOC: Msgpack-encoded table of contents (zstd compressed)
  • Footer (16 bytes): TOC offset and size

Contents

A WAPP file can contain:

  • Metadata: Pack-level key-value metadata
  • Registry Entries: Typed records with ID, Kind, Meta, and Data fields
  • Resources: Filesystem trees with per-file compression

Writing

Use Writer to create WAPP files: 

writer := wapp.NewWriter()
err := writer.Pack(metadata, entries, fsys, resourceID, resourceMeta, output)

Reading

Use Reader to read WAPP files: 

reader, err := wapp.NewReader(file)
entries, err := reader.GetEntries()
fsys, err := reader.GetFS(resourceID)

Features

  • Per-file zstd compression (skips already-compressed formats)
  • Lazy loading with footer-first reading
  • Multiple filesystem tree resources per pack
  • Registry entries for structured data storage
  • SHA-256 integrity verification
  • O(1) file and resource lookups
  • Concurrent-safe reads with decompression cache
  • fs.FS interface compatibility

Index

Constants

View Source 
const (
	// Magic header identifying WAPP files.
	Magic = "WIPPYPACK"

	// Version1 is the current format version.
	Version1 byte = 0x01

	// HeaderSize is the fixed header size in bytes.
	HeaderSize = 268

	// FooterSize is the fixed footer size in bytes.
	FooterSize = 16
)
View Source 
const (
	// ChunkSize is the default chunk size for large files (1MB).
	ChunkSize uint64 = 1024 * 1024

	// ResourceTypeTree identifies a filesystem tree resource.
	ResourceTypeTree = "tree"
)

Variables

View Source 
var (
	ErrDataCorrupted         = errors.New("data corrupted")
	ErrNegativePosition      = errors.New("negative position")
	ErrIsDirectory           = errors.New("is a directory")
	ErrInvalidMagic          = errors.New("invalid magic")
	ErrUnsupportedVersion    = errors.New("unsupported version")
	ErrResourceNotFound      = errors.New("resource not found")
	ErrFrameNotFound         = errors.New("frame not found")
	ErrInvalidWhence         = errors.New("invalid whence")
	ErrDataSizeExceeded      = errors.New("data size exceeded")
	ErrResourceNotTree       = errors.New("resource not a tree")
	ErrUnknownResourceType   = errors.New("unknown resource type")
	ErrInsufficientFrames    = errors.New("insufficient frames")
	ErrResourceCountExceeded = errors.New("resource count exceeded")
	ErrInvalidResourceFS     = errors.New("invalid resource filesystem")
	ErrInvalidTOC            = errors.New("invalid toc")
	ErrFrameOutOfBounds      = errors.New("frame out of bounds")
	ErrFrameHashMismatch     = errors.New("frame hash mismatch")
)

Sentinel errors for errors.Is checks.

View Source 
var DefaultCompressionFunc = func(filename string) bool {
	ext := strings.ToLower(filepath.Ext(filename))
	_, skip := defaultSkipCompressedExts[ext]
	return !skip
}

DefaultCompressionFunc provides default logic for compression decision. Skips compression for already-compressed formats.

Functions

func WriteFooter 

func WriteFooter(w io.Writer, f *Footer) error

WriteFooter writes a WAPP footer.

func WriteHeader 

func WriteHeader(w io.Writer, h *Header) error

WriteHeader writes a WAPP header.

Types

type ChunkInfo 

type ChunkInfo struct {
	Size        uint32 `json:"Size" msgpack:"size"`
	Offset      uint64 `json:"Offset" msgpack:"offset"`
	FrameIndex  uint32 `json:"FrameIndex" msgpack:"frame"`
	FrameOffset uint64 `json:"FrameOffset" msgpack:"frame_offset"`
}

ChunkInfo describes a chunk of a large file.

type Entry 

type Entry struct {
	ID   ID       `json:"ID" msgpack:"ID"`
	Kind string   `json:"Kind" msgpack:"Kind"`
	Meta Metadata `json:"Meta,omitempty" msgpack:"Meta,omitempty"`
	Data any      `json:"Data,omitempty" msgpack:"Data,omitempty"`
}

Entry represents a registry entry stored in the pack.

type FileEntry 

type FileEntry struct {
	Size           uint64       `json:"Size" msgpack:"size"`
	CompressedSize uint64       `json:"CompressedSize,omitempty" msgpack:"compressed_size,omitempty"`
	Mode           uint32       `json:"Mode" msgpack:"mode"`
	ModTime        int64        `json:"ModTime" msgpack:"mtime"`
	Hash           string       `json:"Hash" msgpack:"hash"`
	Compressed     bool         `json:"Compressed" msgpack:"compressed"`
	Meta           Metadata     `json:"Meta,omitempty" msgpack:"meta,omitempty"`
	Location       FileLocation `json:"Location" msgpack:"location"`
}

FileEntry describes a file in a tree resource.

type FileLocation 

type FileLocation struct {
	FrameIndex uint32      `json:"FrameIndex" msgpack:"frame"`
	Offset     uint64      `json:"Offset" msgpack:"offset"`
	Chunks     []ChunkInfo `json:"Chunks,omitempty" msgpack:"chunks,omitempty"`
}

FileLocation describes where file content is stored. 

type Footer struct {
	TOCOffset uint64 // Offset to compressed TOC
	TOCSize   uint64 // Compressed TOC size
}

Footer represents the WAPP file footer.

func ReadFooter 

func ReadFooter(r io.ReadSeeker) (*Footer, error)

ReadFooter reads a WAPP footer from a seeker.

type FormatError 

type FormatError struct {
	Op     string
	Detail string
	Err    error
}

FormatError wraps format validation errors.

func (*FormatError) Error 

func (e *FormatError) Error() string

func (*FormatError) Unwrap 

func (e *FormatError) Unwrap() error

type FrameInfo 

type FrameInfo struct {
	Offset           uint64 `json:"Offset" msgpack:"offset"`
	Size             uint64 `json:"Size" msgpack:"size"`
	UncompressedSize uint64 `json:"UncompressedSize" msgpack:"uncomp_size"`
	// Hash is the hex-encoded SHA-256 of the compressed frame bytes.
	Hash string `json:"Hash" msgpack:"hash"`
}

FrameInfo describes a data frame location and integrity.

type FrameReader 

type FrameReader interface {
	ReadFrameData(frameInfo FrameInfo, offset, size uint64) ([]byte, error)
}

FrameReader interface abstracts frame data reading. 

type Header struct {
	Magic      [9]byte   // "WIPPYPACK"
	Version    byte      // 0x01
	Flags      uint16    // Reserved flags
	DataOffset uint64    // Offset to data section
	DataSize   uint64    // Total size of data section
	DataHash   [32]byte  // SHA-256 of data section
	Reserved   [208]byte // Reserved for future use
}

Header represents the WAPP file header.

func ReadHeader 

func ReadHeader(r io.Reader) (*Header, error)

ReadHeader reads and validates a WAPP header.

type ID 

type ID struct {
	Namespace string `json:"ns" msgpack:"ns"`
	Name      string `json:"name" msgpack:"name"`
}

ID represents a namespaced identifier for resources and entries.

func NewID 

func NewID(ns, name string) ID

NewID creates a new ID with namespace and name.

func (ID) Equal 

func (id ID) Equal(other ID) bool

Equal checks if two IDs are equal.

func (ID) IsZero 

func (id ID) IsZero() bool

IsZero returns true if ID is empty.

func (ID) String 

func (id ID) String() string

String returns the string representation of ID.

type Metadata 

type Metadata map[string]any

Metadata is a string-keyed map for pack/resource metadata.

type ProgressCallback 

type ProgressCallback func(resourceID ID, current, total int)

ProgressCallback reports packing progress.

type ReadError 

type ReadError struct {
	Op   string
	Path string
	Err  error
}

ReadError wraps errors during read operations.

func (*ReadError) Error 

func (e *ReadError) Error() string

func (*ReadError) Unwrap 

func (e *ReadError) Unwrap() error

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads WAPP files with lazy loading.

func NewReader 

func NewReader(r io.ReaderAt) (*Reader, error)

NewReader creates a WAPP reader using footer-first reading. It uses the default decompression cache limit.

func NewReaderWithOptions 

func NewReaderWithOptions(r io.ReaderAt, opts ...ReaderOption) (*Reader, error)

NewReaderWithOptions creates a WAPP reader with options.

func (*Reader) GetEntries 

func (r *Reader) GetEntries() ([]Entry, error)

GetEntries returns registry entries (lazy loaded).

func (*Reader) GetFS 

func (r *Reader) GetFS(id ID) (fs.ReadDirFS, error)

GetFS returns filesystem for a tree resource (lazy loaded).

func (*Reader) GetMetadata 

func (r *Reader) GetMetadata() (Metadata, error)

GetMetadata returns pack metadata (lazy loaded).

func (*Reader) Header 

func (r *Reader) Header() *Header

Header returns the WAPP header.

func (*Reader) ListResources 

func (r *Reader) ListResources() []ResourceInfo

ListResources returns resource metadata from TOC.

func (*Reader) ReadFrameData 

func (r *Reader) ReadFrameData(frameInfo FrameInfo, offset, size uint64) ([]byte, error)

ReadFrameData reads data from a specific frame at offset.

type ReaderOption 

type ReaderOption func(*Reader)

ReaderOption configures Reader behavior.

func WithDecompressionCacheLimit 

func WithDecompressionCacheLimit(maxBytes int64) ReaderOption

WithDecompressionCacheLimit sets the maximum cached decompressed bytes. Use 0 to disable the cache.

type ResourceFrame 

type ResourceFrame struct {
	ID        ID        `json:"ID" msgpack:"id"`
	Type      string    `json:"Type" msgpack:"type"` // ResourceTypeTree
	Meta      Metadata  `json:"Meta" msgpack:"meta"`
	Frame     FrameInfo `json:"Frame" msgpack:"frame"`
	FileCount uint32    `json:"FileCount,omitempty" msgpack:"file_count,omitempty"`
	TotalSize uint64    `json:"TotalSize,omitempty" msgpack:"total_size,omitempty"`
}

ResourceFrame describes a resource in the pack.

type ResourceInfo 

type ResourceInfo struct {
	ID        ID
	Type      string // ResourceTypeTree
	Meta      Metadata
	Hash      string
	Size      uint64
	FileCount uint32
}

ResourceInfo provides summary information about a resource.

type ResourceSpec 

type ResourceSpec struct {
	ID   ID
	Meta Metadata
	FS   fs.FS // Must be non-nil.
}

ResourceSpec specifies a filesystem resource to pack.

type TOC 

type TOC struct {
	// Pack metadata frame.
	Metadata FrameInfo `json:"Metadata" msgpack:"metadata"`

	// Registry entries frame.
	Entries FrameInfo `json:"Entries" msgpack:"entries"`

	// Resource frames (filesystem trees).
	Resources []ResourceFrame `json:"Resources" msgpack:"resources"`

	// Data frames containing file content.
	DataFrames []FrameInfo `json:"DataFrames" msgpack:"data_frames"`
}

TOC represents the table of contents.

type TreeResource 

type TreeResource struct {
	ID   ID       `json:"ID" msgpack:"id"`
	Meta Metadata `json:"Meta" msgpack:"meta"`

	// Path index for O(1) file lookups.
	Files map[string]FileEntry `json:"Files" msgpack:"files"`

	// Directory listings for ReadDir.
	Dirs map[string][]string `json:"Dirs" msgpack:"dirs"`
}

TreeResource represents a filesystem tree in the pack.

type WriteError 

type WriteError struct {
	Op   string
	Path string
	Err  error
}

WriteError wraps errors during write operations.

func (*WriteError) Error 

func (e *WriteError) Error() string

func (*WriteError) Unwrap 

func (e *WriteError) Unwrap() error

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes WAPP files.

func NewWriter 

func NewWriter(opts ...WriterOption) *Writer

NewWriter creates a new WAPP writer.

func (*Writer) Pack 

func (w *Writer) Pack(
	metadata Metadata,
	entries []Entry,
	fsys fs.FS,
	resourceID ID,
	resourceMeta Metadata,
	out io.Writer,
) error

Pack creates a WAPP file from filesystem and entries.

func (*Writer) PackEntries 

func (w *Writer) PackEntries(
	metadata Metadata,
	entries []Entry,
	out io.Writer,
) error

PackEntries creates a WAPP file with only metadata and entries.

func (*Writer) PackWithResources 

func (w *Writer) PackWithResources(
	metadata Metadata,
	entries []Entry,
	resources []ResourceSpec,
	out io.Writer,
) error

PackWithResources creates a WAPP file with multiple filesystem resources.

type WriterOption 

type WriterOption func(*Writer)

WriterOption configures Writer.

func WithCompressionFunc 

func WithCompressionFunc(fn func(string) bool) WriterOption

WithCompressionFunc sets custom compression decision function.

func WithProgressCallback 

func WithProgressCallback(fn ProgressCallback) WriterOption

WithProgressCallback sets a progress callback.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.