Documentation
¶
Overview ¶
Package dataengine coordinates durable processing through semantic storage operations. Storage owns transactions, dataset selection and revision fences; the engine owns interpretation, worker serialization and wakeup lifecycle.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Backend ¶
type Backend interface {
// One process owns scheduling. LoadWork is selection, not a durable claim
// for independent processes; multi-server leases require a separate contract.
// LoadWork returns no work only when no eligible component exists. On a
// failure it preserves dataset/root identity when already known for retry.
LoadWork(context.Context) (Work, bool, error)
// PublishProjection atomically fences and publishes. Stale work returns
// false without mutation; the engine decides when to retry fresh work.
PublishProjection(context.Context, Work, evidence.Projection) (bool, error)
// RecordFailure keeps the component pending with durable bounded backoff.
RecordFailure(context.Context, Work)
}
type ConcurrentBackend ¶
type ConcurrentBackend interface {
Backend
LoadWorkExcluding(context.Context, []Work, int64) (Work, bool, error)
}
ConcurrentBackend selects a complete component disjoint from all exclusions. Exclusions include active claims and briefly delayed stale components; only their dataset and scope membership are needed, not records or byte estimates. A positive byte limit excludes components larger than the remaining budget; zero allows a large component to run alone rather than starve indefinitely.
type Metadata ¶
type Metadata struct {
DatabaseID, DatasetID, Kind string
Generation, InputRevision, Revision, LastIngestionAtMs, CreatedAtMs int64
TargetGeneration int64
}
Metadata identifies one dataset and its accepted/published processing revisions.
type Work ¶
type Work struct {
DatasetID string
Revision, Generation int64
Root string
// Bytes is the size of stored sanitized JSON, used for bounded admission.
Bytes int64
Scopes map[string]int64
Records []evidence.Stored
}
Work is a consistent dataset-scoped connected component. Publication must fence its generation, membership and every scope revision in one transaction.
type Worker ¶
type Worker struct {
// contains filtered or unexported fields
}
Source Files