indexcoord

package
v0.4.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package indexcoord provides Experimental whole-set mutation authority shared by embeddable index engines, CLI adapters, and destructive maintenance.

Index

Constants

View Source
const DefaultStalledHeartbeatGrace = 3 * ManagedHeartbeatInterval

DefaultStalledHeartbeatGrace is 3× the heartbeat interval. Callers may raise it; they may not lower it below MinStalledHeartbeatGrace.

View Source
const ManagedHeartbeatInterval = 30 * time.Second

ManagedHeartbeatInterval is the product heartbeat cadence. Grace floors are defined relative to this interval so a caller cannot shrink stop authority to an instant.

View Source
const MinStalledHeartbeatGrace = 2 * ManagedHeartbeatInterval

MinStalledHeartbeatGrace is the floor (2× heartbeat interval).

Variables

View Source
var (
	// ErrHeld reports contention with another whole-set operation.
	ErrHeld = indexsubstrate.ErrSetAuthorityHeld
	// ErrLost reports that the caller no longer owns the canonical lock binding.
	ErrLost = indexsubstrate.ErrSetAuthorityLost
	// ErrScope reports that a lease was presented for a different set or root.
	ErrScope = indexsubstrate.ErrSetAuthorityScope
)

ErrLeaseHeld reports that a reclaim target is held by a live process and was refused. It aliases the substrate held-authority sentinel.

Functions

func AuthorityRoot

func AuthorityRoot(segmentSetRoot string) (string, error)

AuthorityRoot returns the stable authority directory for a segment-set root.

Types

type Lease

type Lease struct {
	// contains filtered or unexported fields
}

Lease is an unforgeable whole-set authority token. Its backing OS lock lives outside the set-specific segment root and therefore survives root quarantine.

func Acquire

func Acquire(ctx context.Context, segmentSetRoot, indexSetID, holder string) (*Lease, error)

Acquire takes authority for indexSetID at the stable sibling of segmentSetRoot. The call is non-blocking and returns ErrHeld on contention.

func (*Lease) AssertHeld

func (l *Lease) AssertHeld() error

func (*Lease) AssertHeldFor

func (l *Lease) AssertHeldFor(indexSetID, segmentSetRoot string) error

func (*Lease) Release

func (l *Lease) Release() error

Release drops whole-set authority and removes the lease artifact this holder created, so a completed operation leaves nothing for recovery to collect. It is idempotent.

An error means the lock was released but its artifact could not be removed — for example because the pathname no longer names the held lock, which Release refuses to delete rather than removing whatever occupies the name. What survives is observable through the lease surface either way, but the verdict depends on the artifact: an intact original reports unheld and is reclaimable, while a swapped-in replacement reports invalid and is never reaped automatically. Untrappable termination bypasses Release entirely and leaves the unheld kind.

type LeaseAttribution

type LeaseAttribution struct {
	// Matched is true when a job record's identity matches the lease holder.
	Matched bool
	// JobID is the matched job's ID (empty when unmatched).
	JobID string
	// PID is the matched job's recorded process ID (0 when unmatched).
	PID int
	// ProcessAlive is the liveness of PID, evaluated only when Matched.
	ProcessAlive bool
	// JobState is the matched job's last recorded state.
	JobState string
}

LeaseAttribution is the best-effort operator context joined from the job registry. It answers "who left this lease, and is that process still alive" — never "is the lease held". The held/unheld verdict comes solely from the lock probe; a matched-and-alive PID does not imply held, and an unmatched or dead-PID lease is never treated as unheld on that basis.

type LeaseReport

type LeaseReport struct {
	IndexSetID  string
	Path        string
	Verdict     LeaseVerdict
	Holder      string
	AcquiredAt  time.Time
	ModTime     time.Time
	Detail      string
	Attribution LeaseAttribution
}

LeaseReport is a probed set-authority lease plus its joined attribution.

func EnumerateLeases

func EnumerateLeases(authorityRoot string, jobs []jobregistry.JobRecord) ([]LeaseReport, error)

EnumerateLeases probes every set-authority lease under authorityRoot and joins each to the supplied job snapshot for attribution. A missing authority root yields an empty slice. Enumeration is fully non-mutating.

func ProbeLease

func ProbeLease(authorityRoot, indexSetID string, jobs []jobregistry.JobRecord) (LeaseReport, error)

ProbeLease probes a single lease and joins attribution.

The wrapper never downgrades a state it received: whatever verdict the library reached is passed through unchanged, including alongside an error. It follows the library's refusal contract exactly — an artifact-grounded refusal carries LeaseInvalid; an infrastructure failure (unusable authority root, unexpected lock error) returns an error claiming no artifact state. Check the error before trusting a verdict.

type LeaseVerdict

type LeaseVerdict string

LeaseVerdict is the non-mutating lock-state verdict for a set-authority lease.

type ReclaimReport

type ReclaimReport struct {
	IndexSetID string
	Path       string
	Verdict    LeaseVerdict
	Reclaimed  bool
	Holder     string
}

ReclaimReport is the outcome of a reclaim attempt for one lease.

func ReclaimUnheldLease

func ReclaimUnheldLease(authorityRoot, indexSetID string) (ReclaimReport, error)

ReclaimUnheldLease removes a provably-unheld set-authority lease. A held lease is refused with ErrLeaseHeld and never touched; a missing lease is a no-op success (idempotent). The held/unheld decision is made solely by the lock probe — never by attribution.

It follows the library's refusal contract unchanged: an artifact-grounded refusal carries LeaseInvalid with its error; an infrastructure failure returns an error claiming no artifact state. Reclaimed is false on every error path.

type RecoverStalledOptions

type RecoverStalledOptions struct {
	AuthorityRoot  string
	Confirm        bool // false = dry-run (byte-preserving plan only)
	WaitTimeout    time.Duration
	PollInterval   time.Duration
	HeartbeatGrace time.Duration
	// Now is ignored when Confirm is true. Mutation authorization always uses
	// the library's production clock so callers cannot forge stall age. Dry-run
	// / plan-only may still use Now for inspection (via PlanManagedStalledRecovery).
	Now time.Time
	// Owner is optional; empty mints a new recovery owner id.
	Owner string
	// Deadline, when non-zero, is the absolute monotonic attempt deadline
	// (D-R13-03). Zero means derive from WaitTimeout at attempt entry.
	// Tests may set an already-expired deadline to prove refuse paths.
	Deadline time.Time
}

RecoverStalledOptions configures a single-job recover-stalled mutation.

type RecoverStalledOutcome

type RecoverStalledOutcome string

RecoverStalledOutcome is a distinct non-boolean result class for mutation.

const (
	OutcomeDryRun         RecoverStalledOutcome = "dry-run"
	OutcomeRefused        RecoverStalledOutcome = "refused"
	OutcomeSignalled      RecoverStalledOutcome = "signalled-stopped"
	OutcomeReapedOnly     RecoverStalledOutcome = "reaped-unheld"
	OutcomeAlreadyStopped RecoverStalledOutcome = "already-stopped"
	OutcomeRecoveryFailed RecoverStalledOutcome = "recovery-failed"
	OutcomeLeaseStillHeld RecoverStalledOutcome = "lease-still-held"
	OutcomeInvalidResidue RecoverStalledOutcome = "invalid-residue"
	OutcomeNoop           RecoverStalledOutcome = "noop"
)

type RecoverStalledResult

type RecoverStalledResult struct {
	JobID      string                `json:"job_id"`
	Outcome    RecoverStalledOutcome `json:"outcome"`
	Detail     string                `json:"detail,omitempty"`
	Plan       StalledRecoveryPlan   `json:"plan"`
	DryRun     bool                  `json:"dry_run"`
	Owner      string                `json:"recovery_owner,omitempty"`
	Signalled  bool                  `json:"signalled"`
	ForcedKill bool                  `json:"forced_kill"`
	Reclaimed  bool                  `json:"reclaimed"`
	JobState   string                `json:"job_state,omitempty"`
	LeaseAfter LeaseVerdict          `json:"lease_verdict_after,omitempty"`
}

RecoverStalledResult is the library result of recover-stalled.

func RecoverManagedStalled

func RecoverManagedStalled(store *jobregistry.Store, jobID string, opts RecoverStalledOptions) (RecoverStalledResult, error)

RecoverManagedStalled is the recover-stalled mutation: single managed job, confirm-gated, identity-bound signal, observe death, then W2 unheld reclaim. It never calls Store.Stop as the authority and never unlinks a held lease.

type StalledPlanClass

type StalledPlanClass string

StalledPlanClass is a typed recovery-plan outcome. It is never a boolean: each class names a distinct authorization posture for a later mutating recover.

const (
	// PlanHealthy: managed job is live under a matching identity with a fresh
	// heartbeat (or is still in a pre-lease phase with a fresh claim heartbeat).
	PlanHealthy StalledPlanClass = "healthy"
	// PlanSuspectHeartbeatOverdue: live matching identity, held lease, heartbeat
	// past grace — a candidate for a later confirm-gated recover. plan-stalled never
	// signals on this class.
	PlanSuspectHeartbeatOverdue StalledPlanClass = "suspect-heartbeat-overdue"
	// PlanTerminalContradiction: running/stopping record whose process is dead.
	// No signal; only re-probe + possible unheld reclaim in a later wave.
	PlanTerminalContradiction StalledPlanClass = "terminal-contradiction"
	// PlanIdentityMismatch: live PID but birth token does not match the record.
	PlanIdentityMismatch StalledPlanClass = "identity-mismatch"
	// PlanForegroundOrUnmatched: holder is not a durable managed job identity.
	PlanForegroundOrUnmatched StalledPlanClass = "foreground-or-unmatched"
	// PlanLeaseNotHeld: process identity may match, but the set-authority lease
	// is not held (nothing for process-stop to unlock).
	PlanLeaseNotHeld StalledPlanClass = "lease-not-held"
	// PlanInvalid: wrong type, unreadable record, or invalid lease residue.
	PlanInvalid StalledPlanClass = "invalid"
	// PlanIndeterminate: identity unproven, heartbeat writes failing, missing
	// index set correlation, or other incomplete evidence. Refuse, never signal.
	PlanIndeterminate StalledPlanClass = "indeterminate"
)

type StalledPlanOptions

type StalledPlanOptions struct {
	// AuthorityRoot is the set-authority directory used for the lease probe.
	// Required when the job has an IndexSetID; empty skips lease correlation and
	// yields indeterminate once identity otherwise looks stalled.
	AuthorityRoot string
	// Now is the observation clock. Zero means time.Now().UTC().
	Now time.Time
	// HeartbeatGrace is how long a live matching process may go without a
	// heartbeat before becoming suspect. Zero uses DefaultStalledHeartbeatGrace.
	// Values below MinStalledHeartbeatGrace are raised to the floor.
	HeartbeatGrace time.Duration
}

StalledPlanOptions configures a read-only recovery plan.

type StalledRecoveryPlan

type StalledRecoveryPlan struct {
	JobID             string           `json:"job_id"`
	Class             StalledPlanClass `json:"class"`
	Detail            string           `json:"detail,omitempty"`
	JobState          string           `json:"job_state,omitempty"`
	JobType           string           `json:"job_type,omitempty"`
	PID               int              `json:"pid,omitempty"`
	ProcessAlive      bool             `json:"process_alive"`
	RecordedIdentity  string           `json:"recorded_identity,omitempty"`
	ObservedIdentity  string           `json:"observed_identity,omitempty"`
	IndexSetID        string           `json:"index_set_id,omitempty"`
	LeaseVerdict      LeaseVerdict     `json:"lease_verdict,omitempty"`
	LeaseHolder       string           `json:"lease_holder,omitempty"`
	HeartbeatAge      string           `json:"heartbeat_age,omitempty"`
	HeartbeatGrace    string           `json:"heartbeat_grace,omitempty"`
	HeartbeatWriteErr string           `json:"heartbeat_persist_error,omitempty"`
	// SignalCandidate is true only when every gate for a later recover-stalled recover is
	// satisfied (suspect-heartbeat-overdue). plan-stalled never acts on it.
	SignalCandidate bool `json:"signal_candidate"`
	// MayReapUnheld is true for terminal-contradiction when the lease probe is
	// already unheld/missing — process-stop is not required.
	MayReapUnheld bool `json:"may_reap_unheld"`

	// Auth* is the exact plan-time record snapshot for fence CAS. Recover must
	// pass these unchanged into BeginStalledRecovery — never re-read them after
	// plan. Omitted from JSON (operator surface uses formatted fields above).
	AuthSnapshotOK            bool       `json:"-"`
	AuthState                 string     `json:"-"`
	AuthPID                   int        `json:"-"`
	AuthStartMS               uint64     `json:"-"`
	AuthBootID                string     `json:"-"`
	AuthTokenVersion          int        `json:"-"`
	AuthStartTicks            uint64     `json:"-"`
	AuthStartSec              int64      `json:"-"`
	AuthStartUsec             int64      `json:"-"`
	AuthFiletime              uint64     `json:"-"`
	AuthIndexSetID            string     `json:"-"`
	AuthLastHeartbeat         *time.Time `json:"-"`
	AuthHeartbeatPersistError string     `json:"-"`
	AuthRecoveryOwner         string     `json:"-"`
	AuthRecoveryIntent        string     `json:"-"`
}

StalledRecoveryPlan is the read-only judgment for one managed job. It never signals, reaps, or rewrites registry state.

func PlanManagedStalledRecovery

func PlanManagedStalledRecovery(store *jobregistry.Store, jobID string, opts StalledPlanOptions) (StalledRecoveryPlan, error)

PlanManagedStalledRecovery classifies one job for stalled recovery without mutating registry or lease state. It is the plan-stalled microscope: typed outcomes only, zero signals.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL