workload

package
v0.37.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 4, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package workload is an app owner's backup (ADR-0222): the members a BackupSchedule's scope names, one full copy of them under <target>/<ns>/<backup>-<uid>/ (Decision 4) and the reads a Restore makes of it. Each member's file is ADR-0203's framed records in one Seal stream, put in 8 MiB parts IfNotExist; a lake's objects go before its catalog.db, checked against it (Decision 5); the plain manifest.yaml goes last. Each kind of member is one MemberKind, which its provider hands over (ADR-0311).

Index

Constants

View Source 
const CatalogRecord = checkpointName

CatalogRecord is the key of a lake member's catalog record, wherever the catalog lives (ADR-0293 Decision 5).

View Source 
const Format = 2

Format is the workload manifest's format; a changed meaning, framing or layout bumps it. A funcd reads one format: from 2 on a Bucket's file holds none of its Lakes' data (ADR-0312 Decision 8).

View Source 
const MaxCopyMembers = 1024

MaxCopyMembers is the most members one copy may hold: Resolve refuses a scope over it, and a listing that names more fails. A scope's members are stores that exist in its namespace, named one by one or held by an App or a ResourceGroup, so a real scope stays far below it. A Backup's record then stays under 200 KiB.

View Source 
const MaxListedKeyBytes = 256 << 20

MaxListedKeyBytes is the most bytes of keys one listing of a target reads: 256 bytes a key at MaxListedKeys, where a copy's key takes about 100 bytes.

View Source 
const MaxListedKeys = 1 << 20

MaxListedKeys is the most keys one listing of a target reads. A copy holds a key per 8 MiB part, so this holds 8 TiB of copies.

View Source 
const MaxManifestBytes = 16 << 20

MaxManifestBytes is the largest manifest a Restore reads. A member's entry takes under 1 KiB, so this holds over 16,000 members.

Variables

View Source 
var (
	ErrNotEmpty   = errors.New("the destination holds data")       // reason NotEmpty
	ErrTornObject = errors.New("an object record's frame is torn") // reason Integrity
	ErrFileCount  = errors.New("the member lists another number of files than its kind writes")
)

The failures a Restore names by reason (Decision 5), which the service's reasons cannot be imported here for.

View Source 
var (
	ErrLakeUnstable        = errors.New("the lake changed under every copy")
	ErrTargetUnconditional = errors.New("the target refuses a create-if-absent put")
	ErrEngineNotReady      = errors.New("the database's engine is not ready")
)

The failures a run's record names by reason (Decisions 4, 5; ADR-0234 Decision 2).

Functions

func DecodeObject 

func DecodeObject(v []byte) (contentType string, md map[string]string, body []byte, err error)

DecodeObject reverses EncodeObject; a torn or trailing frame is fault.Invalid.

func EncodeObject 

func EncodeObject(contentType string, md map[string]string, body []byte) []byte

EncodeObject frames an object as uvarint-length fields (ADR-0203): its content type, its metadata's count and each key and value in key order, then its body.

func Integrity 

func Integrity(ctx context.Context, src blob.Bucket, path string, m Manifest) error

Integrity reads every part of every file m names and checks its bytes and sha256; a missing part or a mismatch is fault.Invalid.

func KVPrefix 

func KVPrefix(ns v1.NamespaceName, name v1.ObjectName) string

KVPrefix is a KVStore's key prefix in the KV substrate, <ns>/<store>/ (ADR-0072).

func Records 

func Records(ctx context.Context, src blob.Bucket, path string, f backup.StoreFile, open backup.Unseal) func() (snapshot.Record, error)

Records reads f's records through open: io.EOF at the end, a torn record fault.Invalid.

Types

type Copy added in v0.30.0 

type Copy struct {
	Name      v1.ObjectName
	UID       v1.UID
	Path      string // <ns>/<backup>-<uid>/
	Whole     bool
	Completed time.Time
	Members   []v1.BackupMember
}

Copy is one Backup's directory on a target as a listing shows it, read with list rights only (ADR-0287): it is whole once its manifest.yaml, written last, is there; Completed is that manifest's modification time. Members are the kind directories the listing holds, for display: a Restore matches the manifest's members.

func ListCopy added in v0.30.0 

func ListCopy(ctx context.Context, target blob.Bucket, path string, kinds []MemberKind) (Copy, error)

ListCopy lists the Backup directory path, <ns>/<backup>-<uid>/ (ADR-0287 Decision 5), its members as kinds parse them.

func SlotCopies added in v0.30.0 

func SlotCopies(ctx context.Context, target blob.Bucket, ns v1.NamespaceName, schedule v1.ObjectName, after time.Time,
	kinds []MemberKind) ([]Copy, error)

SlotCopies lists the copies of schedule's slots after `after` under ns, whole or not, oldest first (ADR-0287 Decision 3). The listing starts after every key of after's slot: keys sort by slot, as Unix minutes have 8 digits until 2160 (the bound of a schedule name's length). Directories of other schedules, Backups by hand and exports under the same prefix are skipped.

type Destination added in v0.37.0 

type Destination interface {
	Prepare(ctx context.Context) (release func(), err error)         // after the plan, before the first load
	IsEmpty(ctx context.Context) (bool, error)                       // asked only when not in place
	Load(ctx context.Context, f MemberFile, open RecordOpener) error // a refusal wraps ErrNotEmpty or ErrFileCount, writing nothing
	Clear(ctx context.Context) error                                 // after a failed Restore whose load began here
}

Destination is where a Restore loads one member (Decision 3).

type FileWriter added in v0.37.0 

type FileWriter interface {
	File(ctx context.Context, name string, produce func(emit func(snapshot.Record) error) error) (backup.StoreFile, error)
}

FileWriter writes one file of a copy: its records framed, sealed, hashed and cut in parts.

type KVLoader added in v0.37.0 

type KVLoader interface {
	LoadPrefix(ctx context.Context, prefix string, next func() (snapshot.Record, error)) error
	DropPrefix(prefix string) error
	List(ctx context.Context, prefix string) ([]string, error)
}

KVLoader is the KV driver's per-store load, teardown and listing (ADR-0222 LoadPrefix, ADR-0072 DropPrefix); LoadPrefix refuses Conflict when the store holds keys.

type Manifest 

type Manifest struct {
	Format     int          `json:"format"`
	Namespace  string       `json:"namespace"`
	Backup     string       `json:"backup"`
	Funcd      string       `json:"funcd"`
	At         v1.Timestamp `json:"at"`
	Members    []MemberFile `json:"members"`
	Recipients []string     `json:"recipients,omitempty"`
}

Manifest is a Backup's manifest.yaml, plain, written last: the copy exists once it does. Backup is the copy's directory, <backup>-<uid>.

func ReadManifest 

func ReadManifest(ctx context.Context, src blob.Bucket, path string) (Manifest, error)

ReadManifest reads the manifest under path; one over MaxManifestBytes or of any format but Format is fault.Invalid.

func Write 

func Write(ctx context.Context, ns v1.NamespaceName, ms []Member, kinds []MemberKind, o Options) (Manifest, error)

Write copies ms under o.Path, then puts the manifest (Decisions 4, 5): each member's kind among kinds writes its files under its own directory of the copy (ADR-0311), a KVStore to kv/<store>/, a Bucket to blob/<bucket>/, a lake to lake/<name>/<attempt>/, a SQLDatabase to sql/<name>/ and a Function's volume to volume/<function>/<volume>/. A member that no kind serves is fault.Invalid naming its kind, before the first put; a refused IfNotExist is ErrTargetUnconditional.

type Member 

type Member struct {
	Kind     v1.Kind
	Name     v1.ObjectName
	Location v1.LakeLocation
	Volume   v1.ObjectName // with Kind Function: the Function's local volume (ADR-0262 Decision 11)
	// LakePrefixes, with Kind Bucket, are its Lake prefixes as Resolve read them, so a Lake applied during a run leaves each
	// byte in one member (ADR-0312).
	LakePrefixes []string
}

Member is one store of a scope. Location is a Lake member's spec.location, which Write copies.

func Resolve 

func Resolve(ctx context.Context, st store.Store, ns v1.NamespaceName, s v1.BackupScope) ([]Member, error)

Resolve lists the members s names in ns (Decision 2), KVStores, Buckets, Lakes, SQLDatabases then Function volumes: by name, the stores of the App's kv, buckets, lakes and sql entries and its Functions' local volumes, or the stores of the ResourceGroup. In each form a Bucket brings in the Lakes on its Lake prefixes as their own members, once, after the Lakes s names, whichever App or ResourceGroup holds them (ADR-0312). A named member or App that does not exist is fault.NotFound naming it; a scope over MaxCopyMembers is fault.Invalid naming the limit.

type MemberFile 

type MemberFile struct {
	Kind    v1.Kind            `json:"kind"`
	Name    v1.ObjectName      `json:"name"`
	Attempt int                `json:"attempt,omitempty"`
	Volume  v1.ObjectName      `json:"volume,omitempty"`
	Files   []backup.StoreFile `json:"files"`
}

MemberFile is one member's files in the manifest: a store's one, a lake's files then catalog.db, of Attempt.

type MemberKind added in v0.37.0 

type MemberKind interface {
	Kind() v1.Kind // the kind its members carry in Member, MemberFile, BackupMember and RestoreMember
	Write(ctx context.Context, w FileWriter, ns v1.NamespaceName, m Member) (MemberFile, error)
	// Parse names the member of an object key's segments under a copy's directory, and its attempt (a Lake's; else 0).
	Parse(seg []string) (m v1.BackupMember, attempt int, ok bool)
	Destination(ctx context.Context, ns v1.NamespaceName, rm v1.RestoreMember) (Destination, error) // NotFound if none
}

MemberKind is one kind of backup member (ADR-0311 Decision 1): it writes its members' files under its own directory of a copy, names the member of each key there for the listing, and resolves a Restore member's destination. Its provider hands it over, so the backup service names no kind.

func Buckets added in v0.37.0 

func Buckets(st store.Store, shared blob.Bucket, prefix, catalogs func(v1.NamespaceName, v1.ObjectName) string) MemberKind

Buckets is the Bucket kind (ADR-0312): blob/<bucket>/ holds the Bucket's objects under prefix in shared, but those under its Lake prefixes, which the Lakes' members hold, and blob/<bucket>/catalogs/, only when there is one to copy, the catalogs of its other prefixes under catalogs. A Restore loads them into the Bucket as, or into the Bucket itself, emptied first, and touches neither objects nor catalogs under that Bucket's Lake prefixes, or a prefix whose owner Lake is gone.

func Databases added in v0.37.0 

func Databases(st store.Store, load SQLLoader) MemberKind

Databases is the SQLDatabase kind (ADR-0234): sql/<name>/ holds its SQLite file in records of 8 MiB keyed 000001, 000002, …, which a Restore loads through the engine of the database as, or of the database itself. load nil is no SQL engine wired.

func KVStores added in v0.37.0 

func KVStores(st store.Store, export func(prefix string) snapshot.Source, load KVLoader) MemberKind

KVStores is the KVStore kind: kv/<store>/ holds the driver's export of the store (ADR-0209 SnapshotPrefix; nil when the driver offers none), which a Restore loads with load into the KVStore as, or into the store itself, emptied first.

func Lakes added in v0.37.0 

func Lakes(st store.Store, shared blob.Bucket, prefix func(v1.NamespaceName, v1.ObjectName) string, catalogPrefix func(v1.NamespaceName) string,
	stop func(ctx context.Context, ns v1.NamespaceName, name v1.ObjectName) (release func(), err error)) MemberKind

Lakes is the Lake kind (Decision 5, ADR-0271): lake/<name>/<attempt>/files/ holds a lake's objects under its location, then lake/<name>/<attempt>/catalog.db/ the catalog as it was before they were listed. The catalog lives in the catalog store under catalogPrefix (ADR-0293). A Restore loads into the prefix asLocation, where no Lake may be located (ADR-0312 Decision 10), or the lake's location emptied first: its files, then its catalog. An in-place Restore does so with the lake's engine stopped until release (stop; nil does nothing), so no engine outlives the load (#1384).

func Volumes added in v0.37.0 

func Volumes(st store.Store, load VolumeLoader) MemberKind

Volumes is the kind of a Function's local volume: volume/<function>/<volume>/ holds the volume's tar, which a Restore loads into that volume of the Function as, or of the Function itself, while its workers are stopped: only into an empty one, in place too, but for a lost one, which is emptied first (#1206). load nil is volumes off.

type Options 

type Options struct {
	Target     blob.Bucket
	Path       string
	Seal       backup.Seal
	Recipients []string
}

Options are where Write puts the copy and to whom it seals it: Path is <ns>/<backup>-<uid>/ on Target, Recipients the fingerprints the manifest records.

type RecordOpener added in v0.37.0 

type RecordOpener func(ctx context.Context, f backup.StoreFile) func() (snapshot.Record, error)

RecordOpener reads a file of the copy a Restore loads, as Records does.

type SQLLoader added in v0.37.0 

type SQLLoader interface {
	Source(ctx context.Context, ns v1.NamespaceName, name v1.ObjectName) (io.ReadCloser, error)
	Load(ctx context.Context, ns v1.NamespaceName, name v1.ObjectName, file io.Reader) error
	Empty(ctx context.Context, ns v1.NamespaceName, name v1.ObjectName) error
	HasTables(ctx context.Context, ns v1.NamespaceName, name v1.ObjectName) (bool, error)
}

SQLLoader copies, loads, empties and inspects a database through its engine (ADR-0234). Source streams its SQLite file: it fails ErrEngineNotReady for a database with no Ready engine and returns the stream's error, not io.EOF, when the engine reports one at its end. Load replaces all its data. HasTables counts tables, views, indexes and triggers.

type VolumeLoader added in v0.37.0 

type VolumeLoader interface {
	Source(ctx context.Context, ns v1.NamespaceName, fn, name v1.ObjectName) (io.ReadCloser, error)
	Load(ctx context.Context, ns v1.NamespaceName, fn, name v1.ObjectName, archive io.Reader) error
	IsEmpty(ctx context.Context, ns v1.NamespaceName, fn, name v1.ObjectName) (bool, error)
	Clear(ctx context.Context, ns v1.NamespaceName, fn, name v1.ObjectName) error
	Lost(ctx context.Context, ns v1.NamespaceName, fn, name v1.ObjectName) (bool, error)
	StopWorkers(ctx context.Context, ns v1.NamespaceName, fn v1.ObjectName) (release func(), err error)
}

VolumeLoader copies, loads, checks and clears a Function's local volume (ADR-0262 Decision 11): Source is its tar, read live, and Load refuses Conflict when the volume is not empty. Lost reports a volume its Function's DataMissing names (Decision 12), whose content a Restore replaces, and StopWorkers keeps the Function workerless until release, while its volume loads (#1206).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.