partition

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package partition compiles canonical, hashed, lane-assigned enumeration plans and defines the lane identity that names one unit of that plan.

Experimental: the plan encoding and the lane-identity contract will churn until a consumer persists them across runs.

The package is deliberately neutral. One plan governs enumeration, the identity carried alongside the records enumeration emits, resume, and the verification a mutating consumer performs before it writes. Owning the plan inside any one of those consumers would make a downstream stage the authority over an upstream one and force an import in the wrong direction; a shared contract package lets producers and consumers depend on the plan without depending on each other.

Index

Constants

View Source
const LaneCeiling = 64

LaneCeiling is the highest lane count a plan may resolve to. It is a refusal, not a clamp: an operator asking for more lanes than the engine will run should be told so rather than silently getting fewer, because the lane count they believe is running is what they will reason about when reading throughput.

View Source
const PlanVersion = 1

PlanVersion is the schema version of the canonical plan encoding that PlanHash is taken over. Any change to the hashed payload shape must bump this, because a resume comparing plan digests across versions would otherwise read an encoding change as a plan change.

Variables

This section is empty.

Functions

This section is empty.

Types

type Authority

type Authority struct {
	// contains filtered or unexported fields
}

Authority is a validated, immutable enumeration plan that a run holds while it executes.

Plan and Authority are deliberately different things. Plan is data: it is JSON-tagged, exported, and mutable, because it is what gets persisted, read back, and handed around. Its digest field is whatever the last writer put there. Checking a claim against a Plan therefore proves only that the claim agrees with an unverified value — a plan literal carrying a made-up digest, or no digest at all, would authorize claims built from itself.

An Authority is the same information after that value has been proven: the digest is recomputed from the canonical payload and compared, every compiler invariant is rechecked, and the contents are deep-copied into unexported fields so that mutating the Plan afterwards cannot change what the run enforces. Issuing and checking lane identity is available only here, so there is no unvalidated path to either.

func CompileAuthority

func CompileAuthority(req PlanRequest) (*Authority, error)

CompileAuthority compiles a plan and returns the run authority for it. It is the ordinary entry point: a caller that is planning work wants the authority, and CompilePlan exists for callers that need the persistable data.

func NewAuthority

func NewAuthority(plan *Plan) (*Authority, error)

NewAuthority validates a plan and returns the run authority for it.

It fails closed: any invariant the compiler guarantees but the supplied plan does not satisfy is a refusal, because a plan that reached this point without satisfying them was not produced by the compiler, and the run cannot tell whether that is corruption, hand-editing, or tampering.

func (*Authority) AuthorizeLane

func (a *Authority) AuthorizeLane(claim LaneRef) error

AuthorizeLane checks a claimed lane reference against the plan this run holds.

A claim arriving alongside data — records read from a stream, a resumed checkpoint, an API request — is untrusted input: it says which lane it believes it belongs to, and this answers whether that is true of the plan the run is actually executing. It refuses rather than warns, because the caller's next step is a mutation the claim would otherwise scope.

Authorization is not completion. A verified reference says a record belongs to a lane of this plan; it says nothing about whether that lane's enumerator reached the end of its input or whether its work is durably finished. Only the run-owned coordinator may record those.

func (*Authority) BaseIdentity

func (a *Authority) BaseIdentity() string

BaseIdentity reports the source this plan enumerates. It is resume identity, not a rate-limit authority.

func (*Authority) ConfigFingerprint

func (a *Authority) ConfigFingerprint() string

ConfigFingerprint reports the caller digest of behavior-affecting configuration this plan was compiled under.

func (*Authority) Coverage

func (a *Authority) Coverage() Coverage

Coverage reports what the plan claims about the scope it was compiled from.

func (*Authority) Digest

func (a *Authority) Digest() string

Digest reports the verified plan digest.

func (*Authority) Lane

func (a *Authority) Lane(ordinal int) (Lane, error)

Lane returns the lane with the given ordinal, refusing one this plan does not contain.

func (*Authority) LaneCount

func (a *Authority) LaneCount() int

LaneCount reports how many lanes the run executes.

func (*Authority) LaneRef

func (a *Authority) LaneRef(ordinal int) (LaneRef, error)

LaneRef returns the identity of the lane with the given ordinal.

It refuses an ordinal this plan does not contain rather than returning a reference to a lane that does not exist: a fabricated reference would compare equal to itself and could otherwise travel as though it named real work.

func (*Authority) Lanes

func (a *Authority) Lanes() []Lane

Lanes returns a copy of the lane assignment. It copies rather than exposing the slice so a caller ranging over lanes cannot alter what the run enforces.

func (*Authority) Plan

func (a *Authority) Plan() *Plan

Plan returns a copy of the validated plan as persistable data.

func (*Authority) Version

func (a *Authority) Version() int

Version reports the plan encoding version this authority enforces.

type Coverage

type Coverage string

Coverage is the caller's declaration of what the plan claims about the scope it was compiled from. The compiler cannot derive this: it receives prefixes, not the scope configuration they came from (see the note on CompilePlan), so completeness is an assertion the caller makes and the plan records.

const (
	// CoverageComplete asserts the prefixes union to the entire intended scope.
	CoverageComplete Coverage = "complete"
	// CoveragePartial asserts the plan deliberately covers only part of the
	// scope. Recorded so a downstream consumer never mistakes a partial sweep
	// for an authoritative one.
	CoveragePartial Coverage = "intentionally_partial"
)

type Lane

type Lane struct {
	Ordinal  int      `json:"ordinal"`
	Prefixes []string `json:"prefixes"`
}

Lane is one enumeration unit: an ordinal plus the prefixes it lists. Ordinal is derived from assignment order rather than completion order, so lane identity does not vary with scheduling.

An ordinal alone is not an identity — see LaneRef.

type LaneRef

type LaneRef struct {
	PlanVersion int    `json:"plan_version"`
	PlanDigest  string `json:"plan_digest"`
	Ordinal     int    `json:"ordinal"`
}

LaneRef is the identity of one lane: the plan encoding version, the plan digest, and the lane ordinal within that plan.

All three are required. An ordinal alone is not stable identity — lane 3 names entirely different work under a different plan — so a claim carrying only an ordinal cannot be checked against anything. The version is carried because two encodings of the same planned work produce different digests, and a consumer needs to say which mismatch it is looking at.

A LaneRef is issued and checked by an Authority, never by a Plan. A Plan is data: its digest field is whatever the last writer put there, so comparing a claim against it proves only that the claim agrees with an unverified value.

func (LaneRef) Equal

func (r LaneRef) Equal(other LaneRef) bool

Equal reports whether two lane references name the same lane of the same plan.

func (LaneRef) String

func (r LaneRef) String() string

String renders a lane reference for diagnostics. The digest is abbreviated because a message naming the lane is read far more often than one needing the full digest, and the full value is available on the struct.

type Plan

type Plan struct {
	Version           int      `json:"version"`
	Coverage          Coverage `json:"coverage"`
	BaseIdentity      string   `json:"base_identity"`
	ConfigFingerprint string   `json:"config_fingerprint"`
	Lanes             []Lane   `json:"lanes"`

	// PlanHash is the digest of the canonical payload. It is an integrity
	// fingerprint, not an authentication tag: anyone who can construct the same
	// inputs can construct the same digest. A consumer verifies a claimed lane
	// against a plan it was handed by the run that owns it — never against a
	// plan reconstructed from the claim itself. See AuthorizeLane.
	PlanHash string `json:"plan_hash"`

	// Subsumed records prefixes removed because another retained prefix already
	// covers them. Normalization here is union-preserving — dropping a/b/ when
	// a/ is present enumerates exactly the same objects — but it does reduce the
	// number of partitionable units, so an operator who wrote three prefixes and
	// got one lane needs to be able to see why. Reported rather than refused
	// precisely because coverage is unchanged.
	Subsumed []SubsumedPrefix `json:"subsumed,omitempty"`
}

Plan is a canonical, hashed, lane-assigned enumeration plan. It is compiled and hashed before enumeration begins so the digest describes the work that was planned, not the work that happened to complete.

func CompilePlan

func CompilePlan(req PlanRequest) (*Plan, error)

CompilePlan normalizes the requested prefixes, refuses inputs whose partition semantics are not decidable, assigns stable lane ordinals, and hashes the result.

Prefix containment uses literal byte-prefix semantics, not path semantics, because that is what an object-store LIST prefix means: listing "a" returns "ab/x" as well as "a/x". Treating containment as path-aware would leave a genuinely redundant prefix in the plan and enumerate those objects twice.

type PlanRequest

type PlanRequest struct {
	Prefixes []string
	Coverage Coverage
	// BaseIdentity names the source provider and base URI this plan enumerates.
	// It is part of the digest because a plan is only the same plan when it runs
	// against the same source: identical prefixes under a different provider or
	// bucket describe entirely different work, and a resume must not match across
	// that boundary.
	//
	// This is resume identity. It is deliberately not a rate-limit authority,
	// which may be broader than one bucket, and it must never carry credentials.
	BaseIdentity string
	// ConfigFingerprint is the caller digest of behavior-affecting configuration.
	// The compiler cannot compute it — the adapter owns what is
	// behavior-affecting — but the plan must carry it so a resume refuses a run
	// whose semantics changed under it.
	ConfigFingerprint string
	// MaxLanes bounds the lane count. Zero means one lane per retained prefix.
	MaxLanes int
}

PlanRequest is the compiler's input.

Prefixes are explicit. The compiler deliberately does not accept a scope configuration: scope compilation reaches a schema dependency that the embeddable engine's dependency boundary denies, so it belongs to the calling adapter. The adapter compiles configuration into prefixes; this package receives facts.

type SubsumedPrefix

type SubsumedPrefix struct {
	Prefix    string `json:"prefix"`
	CoveredBy string `json:"covered_by"`
}

SubsumedPrefix is one dropped prefix and the retained prefix that covers it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL