README
¶
mod/rescan
mod/rescan is the background integrator. It reads configured keys, discovers their sources, fetches or replicates
versions, builds overlay artifacts, and publishes completed versions into storage and state.
Place in the Runtime
flowchart LR
timer["ticker or trigger"] --> rescan["mod/rescan"]
rescan --> source["mod/source"]
source --> archive["mod/archive"]
archive --> overlay["mod/overlay"]
overlay --> storage["mod/storage"]
storage --> state["mod/state"]
Responsibilities
- Run the supervisor, ticker, manual trigger, and one-shot
RunOnce. - Process keys with bounded parallelism.
- Persist sticky git listing mode for releases-versus-tags decisions.
- Fetch git archives into a blob spool and pass them through archive extraction.
- Replicate brother versions by pulling index, tree bytes, and only missing blobs.
- Confirm
Brother.Helloon each brother dial or redial before trusting the session. - Fall back to a brother's public release API when RPC is closed or unavailable.
- Run detection and artifact planning before durable publication.
- Publish versions through
storage.PublishStaged. - Update composer name maps, content checksum, key stats, diagnostics, and deletion grace.
Ingest Flow
sequenceDiagram
participant Cycle as rescan cycle
participant Source as source
participant Archive as archive
participant Overlay as overlay
participant Store as storage
participant State as state
Cycle->>Source: list releases or brother index
alt brother RPC unavailable
Cycle->>Source: read brother public releases.json and version JSON
end
Source->>Archive: archive in spool
Archive->>Overlay: canonical staged tree
Overlay->>Overlay: detect ecosystems and plan artifacts
Overlay->>Store: PublishStaged
Store->>State: publish visible facts
Contracts
- One key failure must not cancel the whole cycle.
- A version becomes visible only after
PublishStagedcommits durable data. - Blob spool cleanup must run even when the request context is canceled.
- Permanent content failures are remembered by upstream reference or advertised tree hash to avoid repeated downloads.
- Brother tree bytes must match the tree hash announced in the brother index when that hash is non-zero.
- Public brother fallback must verify the downloaded archive against the tree hash announced by the public version JSON.
- Brother blob fetches must respect the negotiated byte and batch-count limits from the active session.
Important Files
obj.go: object dependencies and public facade.supervisor.go: lifecycle, trigger, and run loop.ingest.go: git ingest path.brother.go: brother replication and fallback logic.cycle.go: end-of-cycle aggregates.deletion.go: upstream deletion grace.spool.go: pre-commit blob reader used by overlay builders.metrics.go: rescan telemetry.
Operational Notes
rescan.initial_depth controls how much history is fetched on first contact. Later cycles use provider ordering,
upstream sequence, semver rules, and configured deletion grace to avoid unnecessary downloads while still detecting
changed or removed releases.
Documentation
¶
Index ¶
- type Obj
- func (obj *Obj) Close(ctx context.Context) error
- func (obj *Obj) ComposerKeyForName(name string) (string, bool)
- func (obj *Obj) ComposerPackageNames() []string
- func (obj *Obj) RegisterMetrics(meterObj metric.Meter) error
- func (obj *Obj) RunOnce(ctx context.Context)
- func (obj *Obj) SetSuppressed(keySet map[string]struct{})
- func (obj *Obj) Start()
- func (obj *Obj) Trigger()
- type SourceInterface
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Obj ¶
type Obj struct {
// contains filtered or unexported fields
}
Obj supervises rescans. Dependencies are immutable after New; mutable cycle state has dedicated locks.
func New ¶
func New( configObj *stconf.ConfigObj, sourceObj SourceInterface, storageObj *storage.Obj, overlayObj *overlay.Obj, stateObj *state.Obj, archiveObj *archive.Obj, yggHost string, logArr ...zerolog.Logger, ) *Obj
New builds a rescan supervisor. yggHost is the Yggdrasil listener entry host; empty means Yggdrasil is disabled. The background loop is not started here; callers use Start and optionally RunOnce.
func (*Obj) Close ¶
Close cancels the loop and waits for exit. storage.Publish* calls are atomic, so there is no partial state to clean up.
func (*Obj) ComposerKeyForName ¶
ComposerKeyForName returns the storage key that owns a composer name. Only collision winners are exposed; unknown or collided names return false.
func (*Obj) ComposerPackageNames ¶
ComposerPackageNames returns sorted composer names without collisions. mod/server serves them live through overlay.ComposerPackages and overlay.ComposerPackageList.
func (*Obj) RegisterMetrics ¶
RegisterMetrics registers instruments for the rescan group. A nil meter is a no-op; rescan depends only on neutral otel/metric types.
func (*Obj) RunOnce ¶
RunOnce runs one cycle synchronously for initial fill and tests. Do not call it in parallel with an active Start loop.
func (*Obj) SetSuppressed ¶
SetSuppressed sets keys excluded by boot name-to-URL checks. Rescan skips them without touching local data, preserving degraded-first serving.
type SourceInterface ¶
type SourceInterface interface {
Discover(ctx context.Context, key string, rootURL string) (source.DiscoveryResultObj, error)
Releases(ctx context.Context, sourceURL string, depth uint) ([]source.GitReleaseObj, bool, error)
Tags(ctx context.Context, sourceURL string, depth uint) ([]source.GitReleaseObj, bool, error)
Refs(ctx context.Context, sourceURL string) (map[string]string, error)
FetchArchive(ctx context.Context, reqObj source.GitFetchRequestObj) (source.GitFetchResultObj, error)
PublicMirrorVersions(ctx context.Context, rootURL string, remoteKey string) ([]source.PublicMirrorVersionObj, bool, error)
BrotherDial(ctx context.Context, localKey string, remoteKey string, brotherURL string) (source.BrotherSessionInterface, error)
}
SourceInterface is the outbound layer orchestrated by rescan and faked in tests. BrotherDial returns source.BrotherSessionInterface, so *source.Obj satisfies it directly.
Source Files