Documentation
¶
Overview ¶
Package sync implements upstream mirror synchronization (docs/explanation/domain-model.md invariant 11, docs/reference/http-api.md "Upstream sync", docs/reference/webhooks.md repo.sync.* events): a worker mirror-fetches from an upstream git provider and loopback-pushes into this server's own git smart-HTTP endpoint, reusing pack storage, ref policy, CAS, and push-event emission (phase-5 Decision 4).
The package defines the consumer-side contracts the worker and the HTTP handlers need (CredentialStore, UpstreamStore, EventSink, Queue) plus an in-memory implementation (MemStore) used by tests and available as the local-mode stand-in until the durable meta-backed rows land in the foundation slice. Credential values are stored as given and never logged.
Index ¶
- Constants
- Variables
- func ClaimHeldWithin(deadline time.Duration) time.Duration
- func DefaultClaimLease() time.Duration
- func MetaStores(ms meta.EventingStore) (CredentialStore, UpstreamStore)
- func NewCredentialID() string
- func ProviderHost(provider string) (host string, known bool)
- func PushRetryDelay(retry int) time.Duration
- func TransientPushFailure(err error) bool
- func ValidBranchName(name string) bool
- type Credential
- type CredentialStore
- type DefaultBranchAdopter
- type Event
- type EventRepository
- type EventSink
- type FirstSyncScheduler
- type InProcessQueue
- type Job
- type LaneClaim
- type LocalLanes
- type LocalScheduler
- func (s *LocalScheduler) Close()
- func (s *LocalScheduler) CompleteSyncCreate(ctx context.Context, org, repoID string) (meta.CreateOutcome, error)
- func (s *LocalScheduler) Drain(ctx context.Context, wait time.Duration)
- func (s *LocalScheduler) EnqueuePush(_ context.Context, job Job) error
- func (s *LocalScheduler) EnqueueSync(ctx context.Context, job Job) error
- func (s *LocalScheduler) OfferSync(_ context.Context, job Job) error
- func (s *LocalScheduler) Start(delay, interval time.Duration)
- func (s *LocalScheduler) Stop()
- func (s *LocalScheduler) SweepOnce(ctx context.Context) error
- func (s *LocalScheduler) TracksCreateState() bool
- type Loopback
- type MemStore
- func (m *MemStore) BeginPush(_ context.Context, repoID, holder string, lease time.Duration) (Upstream, bool, string, error)
- func (m *MemStore) BeginSync(_ context.Context, repoID, holder string, lease time.Duration) (Upstream, string, error)
- func (m *MemStore) CreateCredential(_ context.Context, cred Credential) (Credential, error)
- func (m *MemStore) CredentialForRepo(_ context.Context, repoID string) (Credential, error)
- func (m *MemStore) CredentialMeta(_ context.Context, id string) (Credential, error)
- func (m *MemStore) DeleteCredential(_ context.Context, id string) error
- func (m *MemStore) DeleteUpstream(_ context.Context, repoID string) error
- func (m *MemStore) FinishPush(_ context.Context, repoID, holder string, success bool) (Upstream, error)
- func (m *MemStore) FinishSync(_ context.Context, repoID, holder string, success bool) (Upstream, error)
- func (m *MemStore) GetCredential(_ context.Context, id string) (Credential, error)
- func (m *MemStore) GetUpstream(_ context.Context, repoID string) (Upstream, error)
- func (m *MemStore) PutUpstream(_ context.Context, up Upstream) (Upstream, error)
- func (m *MemStore) RequestPush(_ context.Context, repoID string, force bool) error
- func (m *MemStore) RequestSync(_ context.Context, repoID string) error
- func (m *MemStore) RestorePush(_ context.Context, repoID, holder string, force bool) (Upstream, error)
- func (m *MemStore) RestoreSync(_ context.Context, repoID, holder string) (Upstream, error)
- func (m *MemStore) UpdateCredential(_ context.Context, id, username, password string) (Credential, error)
- type Queue
- type RejectedRef
- type RepoInfo
- type RepoLookup
- type ShutdownRestorer
- type Upstream
- type UpstreamStore
- type Worker
Constants ¶
const ( EventSyncStarted = "repo.sync.started" EventSyncSucceeded = "repo.sync.succeeded" EventSyncFailed = "repo.sync.failed" )
Sync lifecycle event types (docs/reference/webhooks.md, X-Pierre-Event values). Inbound runs only: upstream's schemas for these set additionalProperties: false, so their bodies never carry the outbound fields below.
const ( EventPushUpstreamStarted = "repo.push_upstream.started" EventPushUpstreamSucceeded = "repo.push_upstream.succeeded" EventPushUpstreamFailed = "repo.push_upstream.failed" )
Outbound push-upstream run event types (open.storage extension, issue #354). Upstream has no outbound-run event, so these carry the same body as the sync events plus direction/forced/rejected_refs under their own names.
const ( FailAuth = "authentication failed" FailClone = "failed to clone repository" FailPush = "failed to push to storage" FailConfig = "repository configuration error" FailUnreachable = "upstream repository unreachable" FailInternal = "internal error" FailSession = "session unavailable" FailGeneric = "sync failed" // Outbound push-upstream failure strings (issue #90, direction // "push"). FailRejected carries per-ref reasons in rejected_refs. FailForward = "failed to push to upstream" FailRejected = "push rejected by upstream" )
Exact documented repo.sync.failed error strings (docs/reference/webhooks.md — receivers may switch on these verbatim; never reword them).
const ( PushPendingNone = 0 PushPendingPush = 1 PushPendingForce = 2 )
PushPending levels (issue #90). Force must dominate: markers coalesce by MAX at RequestPush and are consumed whole by BeginPush.
const DirectionPush = "push"
DirectionPush marks outbound (local -> upstream) runs in event payloads (issue #90). Inbound runs omit the field entirely so pre-#90 pull event bodies stay byte-identical on the wire. Since issue #354 the event type already tells the directions apart; the field stays so receivers that checked it keep working.
const LegacyClaimHolder = "!legacy-grace"
LegacyClaimHolder mirrors meta.LegacyClaimHolder (the sync package must not import meta): a claim adopted from a pre-lease writer under a grace.
const MaxPushTransientRetries = 5
MaxPushTransientRetries bounds the transient-failure retry chain of one outbound trigger (issue #90): an unreachable upstream re-marks the durable pending level and reschedules with exponential backoff, so a forward accepted just before an upstream outage converges once the outage ends. Permanent failures (auth, config, rejected refs) never retry — a rejected non-fast-forward can only be resolved by a human decision (pull or force), and hammering a 401 is abuse-shaped.
Variables ¶
var ( // ErrNotFound: unknown credential/upstream/repo row. ErrNotFound = errors.New("not found") // ErrCredentialExists: CreateCredential when the repo already has one // (docs/reference/http-api.md: POST /repos/git-credentials 409 = exists). ErrCredentialExists = errors.New("credential already exists") // ErrUpstreamExists: PutUpstream when the repo already has an upstream // row (base_repo is set exactly once at repo creation). ErrUpstreamExists = errors.New("upstream already exists") // ErrSyncInProgress: BeginSync while a run is already in flight — // callers skip silently (invariant 11: no overlapping runs). ErrSyncInProgress = errors.New("sync already in progress") // ErrPushInProgress: BeginPush while an outbound run is already in // flight — callers skip silently (issue #90: at most one active // forward per repo; the claim's finisher covers pending requests). ErrPushInProgress = errors.New("push already in progress") // ErrClaimLost: FinishSync by a holder whose expired claim was stolen // (issue #86) — the late finisher abandons silently; the thief owns // the run lifecycle now. ErrClaimLost = errors.New("sync claim lost") )
Sentinel errors. Store implementations return them (possibly wrapped); callers match with errors.Is.
var ErrRepoGone = errors.New("repository deleted")
ErrRepoGone is the error of Run and RunPush on the River lane when the repo lookup finds no repository (it was deleted). The job has nothing left to do, so the sync or push worker completes it before its completion gate, even if a claim is still held.
var ErrShutdown = errors.New("server shutting down")
ErrShutdown is the cause serve's shutdown drain cancels sync and push runs with (LocalScheduler.Drain, jobs.River.DrainSyncRuns). A run whose context carries it keeps its request instead of counting a failure.
Functions ¶
func ClaimHeldWithin ¶ added in v0.19.0
ClaimHeldWithin is how long after its start a run with the default Timeout can still hold its claim when its context ends at deadline: it claims before then and holds the claim for one lease. claimFenceMargin absorbs clock skew against the database clock, which times the lease.
func DefaultClaimLease ¶ added in v0.19.0
func MetaStores ¶
func MetaStores(ms meta.EventingStore) (CredentialStore, UpstreamStore)
MetaStores adapts the durable meta.EventingStore rows to this package's CredentialStore/UpstreamStore contracts: field-for-field type translation plus sentinel mapping (meta.ErrNotFound/ErrAlreadyExists onto the sync sentinels handlers and the worker match on).
func NewCredentialID ¶
func NewCredentialID() string
NewCredentialID generates a fresh credential ID in the documented shape (openapi example: ggc_1234567890abcdef).
func ProviderHost ¶
ProviderHost resolves a sync provider name to its default host. known=false for providers this backend cannot mirror-sync (including "code"/"code.storage", which are fork-only).
func PushRetryDelay ¶
PushRetryDelay is the backoff before transient retry n (1-based): 1m, 2m, 4m, 8m, 16m.
func TransientPushFailure ¶
TransientPushFailure reports whether a run error is worth an automatic retry: only network unreachability qualifies. Auth/config/rejection are permanent until a human acts; internal errors are more likely local bugs than weather, and the pull lane does not retry them either.
func ValidBranchName ¶ added in v0.19.0
ValidBranchName reports whether name is an acceptable default branch.
Types ¶
type Credential ¶
type Credential struct {
ID string // server-generated "ggc_<16 hex>"
RepoID string
Username string // optional; token-only providers omit it
Password string
CreatedAt time.Time // UTC
}
Credential is one stored HTTPS credential for a repo's generic git upstream (docs/reference/http-api.md "Git credentials"). Password is the upstream secret (token or password) verbatim; it must never be logged.
type CredentialStore ¶
type CredentialStore interface {
// CreateCredential stores cred (ID/CreatedAt assigned by the store).
// ErrCredentialExists when the repo already has a credential.
CreateCredential(ctx context.Context, cred Credential) (Credential, error)
// GetCredential returns the credential by ID; ErrNotFound when absent.
GetCredential(ctx context.Context, id string) (Credential, error)
// CredentialMeta returns the credential with Password always "" — the
// stored password is never read or decrypted, so ownership prechecks
// for update/delete work even when an encrypted row's key is gone
// (issue #54 recovery: replace or remove without the old value).
// ErrNotFound when absent.
CredentialMeta(ctx context.Context, id string) (Credential, error)
// UpdateCredential replaces username/password; ErrNotFound when absent.
UpdateCredential(ctx context.Context, id, username, password string) (Credential, error)
// DeleteCredential removes the credential; ErrNotFound when absent.
DeleteCredential(ctx context.Context, id string) error
// CredentialForRepo returns the repo's credential; ErrNotFound when the
// upstream is public (no credential stored).
CredentialForRepo(ctx context.Context, repoID string) (Credential, error)
}
CredentialStore persists git credentials (docs/reference/http-api.md "Git credentials"). The durable implementation is meta-backed (foundation slice); MemStore implements it in-memory.
type DefaultBranchAdopter ¶ added in v0.19.0
type DefaultBranchAdopter interface {
AdoptDefaultBranch(ctx context.Context, repoID, branch string) (adopted bool, err error)
}
DefaultBranchAdopter is an UpstreamStore that records the upstream default branch of a repo whose create named none (issue #599): it sets the branch and clears Upstream.DefaultBranchOmitted in one write, only while the flag is set. ErrNotFound when the repo is deleted. The meta adapter implements it. MemStore never stores the flag.
type Event ¶
type Event struct {
Type string `json:"type"`
// Org is the served org name: upstream defines it as the `iss` claim
// of the org's JWTs, and the served-org gate accepts no other iss
// (issue #367).
Org string `json:"org"`
Repository EventRepository `json:"repository"`
RunCount int `json:"run_count"`
IsFirstSync bool `json:"is_first_sync"`
StartedAt string `json:"started_at"`
CompletedAt string `json:"completed_at,omitempty"`
Error string `json:"error,omitempty"`
// Direction discriminates outbound runs (DirectionPush, issue #90).
// Absent means inbound pull — the pre-#90 payload shape.
Direction string `json:"direction,omitempty"`
// Forced reports that an outbound run used force refspecs.
Forced bool `json:"forced,omitempty"`
// RejectedRefs carries per-ref rejection reasons on FailRejected
// outbound failures (issue #90: a non-fast-forward ref fails with a
// clear reason instead of silently rewriting upstream history).
RejectedRefs []RejectedRef `json:"rejected_refs,omitempty"`
}
Event is the repo.sync.* and repo.push_upstream.* webhook payload (docs/upstream/guides/webhooks.md field tables). CompletedAt/Error are only present on terminal events.
type EventRepository ¶
type EventRepository struct {
ID string `json:"id"`
URL string `json:"url"`
RepoName string `json:"repo_name"`
}
EventRepository identifies the synced repo: id = repo_id, url = the JWT `repo` claim value. RepoName carries the same value as URL: upstream deprecated url in favour of repo_name (issue #367).
type EventSink ¶
EventSink receives one repo.sync.* or repo.push_upstream.* event for fan-out to the org's registered webhooks (docs/reference/webhooks.md). payload is the exact JSON body to sign and deliver; implementations must not mutate it. The webhook-delivery slice provides the production sink; tests use recorders.
type FirstSyncScheduler ¶
type FirstSyncScheduler interface {
// TracksCreateState is false when the queue cannot complete a create
// (the in-process queue): callers then never call CompleteSyncCreate.
TracksCreateState() bool
// CompleteSyncCreate atomically schedules the first sync (no ref: the
// worker mirrors all heads and tags) and marks the create complete,
// only while pending. Complete or deleted repos schedule nothing.
CompleteSyncCreate(ctx context.Context, org, repoID string) (meta.CreateOutcome, error)
}
FirstSyncScheduler is a Queue that schedules a sync create's first sync in the transaction that marks the create complete (issue #388).
type InProcessQueue ¶
type InProcessQueue struct {
// contains filtered or unexported fields
}
InProcessQueue runs sync jobs on goroutines with per-repo coalescing: a job enqueued while one for the same repo is still waiting is dropped — sound, because the waiting run's upstream fetch is still in the future, so it observes the coalesced request's state. A job enqueued while a run is already CLAIMED skips at BeginSync instead; the claim holder's FinishSync then reports the pending request (RequestSync marker) and Worker.Requeue schedules the follow-up run (issue #79). This is the local-mode jobs story — queued work is lost on process restart (foundation-slice ADR); the Postgres deployment substitutes the durable jobs seam.
func NewInProcessQueue ¶
func NewInProcessQueue(run, runPush func(ctx context.Context, job Job) error, logger *slog.Logger) *InProcessQueue
NewInProcessQueue builds a queue driving run (normally (*Worker).Run) and runPush ((*Worker).RunPush, issue #90). The two lanes coalesce independently: a queued pull never swallows a push request or vice versa.
func (*InProcessQueue) Close ¶
func (q *InProcessQueue) Close()
Close stops accepting jobs and waits for in-flight runs to finish.
func (*InProcessQueue) EnqueuePush ¶
func (q *InProcessQueue) EnqueuePush(_ context.Context, job Job) error
EnqueuePush schedules one outbound push-upstream run (issue #90) with the same coalescing/overlap semantics as EnqueueSync on its own lane; callers that must not lose the request mark it first (UpstreamStore.RequestPush).
func (*InProcessQueue) EnqueueSync ¶
func (q *InProcessQueue) EnqueueSync(_ context.Context, job Job) error
EnqueueSync schedules one sync run. The run executes on its own goroutine with a fresh background context — it must outlive the enqueueing HTTP request. Overlap with an in-flight run is handled by BeginSync (ErrSyncInProgress -> silent skip), matching invariant 11; callers that must not lose the request mark it first (UpstreamStore.RequestSync), and the in-flight run's finisher requeues it (Worker.Requeue).
type Job ¶
Job is one queued sync request. Ref is the optional pull-upstream hint (docs/reference/http-api.md {ref?}); the worker currently performs a full mirror sync — a superset of any single-ref request — so Ref is carried for logging only. Retry counts consecutive transient-failure retries of an outbound push run (issue #90): it selects the backoff delay and bounds the retry chain, and deliberately does NOT participate in queue deduplication.
type LaneClaim ¶
type LaneClaim struct {
Up Upstream
Claimed bool
Busy bool
Force bool
Attempt int
StolenFrom string
}
LaneClaim is a conditional claim's result: exactly one of Claimed, Busy (a live claim holds the lane), or neither (no durable work due) holds.
type LocalLanes ¶
type LocalLanes interface {
RequestSync(ctx context.Context, repoID string) error
ClaimDuePush(ctx context.Context, repoID, holder string, lease time.Duration) (LaneClaim, error)
ClaimDueSync(ctx context.Context, repoID, holder string, lease time.Duration) (LaneClaim, error)
FinishPushOutcome(ctx context.Context, repoID, holder string, o meta.PushOutcome) (Upstream, error)
FinishSyncOutcome(ctx context.Context, repoID, holder string, o meta.SyncOutcome) (Upstream, error)
ListLocalWork(ctx context.Context, org, after string, limit int) ([]meta.LocalWork, error)
CompleteSyncRepoCreate(ctx context.Context, org, repoID string) (meta.CreateOutcome, error)
}
LocalLanes is the durable local schedule (issue #266, ADR-50): the lane row owns pending level, due time, retry position, and claim. SQLite and S3WAL provide it (meta.LocalSchedule). A worker with Local set claims through ClaimDue* and ends runs through Finish*Outcome, and keeps no process-local retry or defer timers.
func MetaLocalLanes ¶
func MetaLocalLanes(ms meta.EventingStore) (LocalLanes, bool)
MetaLocalLanes adapts ms when it provides the local schedule (SQLite, S3WAL); ok is false for PG, which schedules through River.
type LocalScheduler ¶
type LocalScheduler struct {
// contains filtered or unexported fields
}
LocalScheduler replaces InProcessQueue on SQLite (ADR-50). Durable rows own the schedule; this type only turns them into runs. Every run claims through ClaimDue*, so a stale, duplicate, or dropped hint cannot admit, delay, or cancel work: a dropped hint is offered again by the next sweep. Process-local state is limited to hint bookkeeping (in-flight keys, sweep slots, the sweep cursor), and no lock is held across store I/O.
func NewLocalScheduler ¶
func NewLocalScheduler(lanes LocalLanes, org string, runSync, runPush func(ctx context.Context, job Job) error, logger *slog.Logger) *LocalScheduler
NewLocalScheduler builds a scheduler for org's repos over lanes, running runSync ((*Worker).Run) and runPush ((*Worker).RunPush).
func (*LocalScheduler) Close ¶
func (s *LocalScheduler) Close()
Close stops the scheduler and waits for in-flight runs. After it returns the scheduler makes no store calls.
func (*LocalScheduler) CompleteSyncCreate ¶
func (s *LocalScheduler) CompleteSyncCreate(ctx context.Context, org, repoID string) (meta.CreateOutcome, error)
CompleteSyncCreate records the first-sync request and the complete mark in one SQLite write, then offers the run (the sweep runs it otherwise).
func (*LocalScheduler) Drain ¶
func (s *LocalScheduler) Drain(ctx context.Context, wait time.Duration)
Drain prepares shutdown while the loopback listener still serves (issue #342). It stops the sweep and new runs, waits up to wait for runs in flight, then cancels the remaining runs with ErrShutdown and waits for them to record that. Enqueues still record their durable request; its run is left to the next process's sweep. A cancelled run ends with OutcomeRestore and keeps its request, so correctness does not depend on wait. ctx bounds the whole drain: a run still unfinished when it ends is left to Close, and if the process dies first, its claim expires with its lease.
func (*LocalScheduler) EnqueuePush ¶
func (s *LocalScheduler) EnqueuePush(_ context.Context, job Job) error
EnqueuePush offers a push run. Every caller records its request first: the API's push-upstream with RequestPushNow (due now), the forward poke with RequestPush (joins a waiting backoff). The row decides whether the run may claim.
func (*LocalScheduler) EnqueueSync ¶
func (s *LocalScheduler) EnqueueSync(ctx context.Context, job Job) error
EnqueueSync records a durable pull request, then offers a run. It serves callers without a request of their own (the initial sync at repo creation, the invariant-11 poke); pull-upstream records its request first and uses OfferSync.
func (*LocalScheduler) OfferSync ¶
func (s *LocalScheduler) OfferSync(_ context.Context, job Job) error
OfferSync offers a pull run for a request the caller already recorded. Recording it again here could re-mark a request that a run claimed in between, and that would start a second run for one request.
func (*LocalScheduler) Start ¶
func (s *LocalScheduler) Start(delay, interval time.Duration)
Start runs SweepOnce after delay (the loopback listener must be serving before recovered runs push) and then every interval, until Close.
func (*LocalScheduler) Stop ¶
func (s *LocalScheduler) Stop()
Stop stops the sweep and stops accepting runs without waiting for runs in flight.
func (*LocalScheduler) SweepOnce ¶
func (s *LocalScheduler) SweepOnce(ctx context.Context) error
SweepOnce reads one page of due lanes (rotating through the served org's repos) and offers a run for each. It is the only way deferred work (backoffs, expired claims, mid-run intent, hook-written intent) becomes a run.
func (*LocalScheduler) TracksCreateState ¶
func (s *LocalScheduler) TracksCreateState() bool
TracksCreateState: every local schedule store (SQLite, S3WAL) stores create state.
type Loopback ¶
type Loopback struct {
// BaseURL prefixes push URLs as {BaseURL}/{repo_id}.git. Any scheme git
// understands works (http(s):// in production, file:// in tests). It
// must terminate in THIS process when Token mints keys only this
// process can verify (issue #45): the serve command passes its own
// bound listener, never the external (load-balanced) base URL.
BaseURL string
// Token mints a short-lived git:write JWT for the loopback push (basic
// auth `t:<jwt>`, docs/explanation/git-protocol-and-namespaces.md). The token is passed to git via a credential helper
// and never logged. May return "" when BaseURL needs no auth (file://).
// ttl and holder FENCE the push against a claim steal (issue #86):
// implementations must cap the token lifetime at ttl (no new push
// request authenticates once the lease can be stolen) and carry
// holder as the auth.Claims.SyncClaim so the update hook refuses ref
// updates from a run whose claim was stolen mid-receive-pack.
Token func(ctx context.Context, org string, repo RepoInfo, ttl time.Duration, holder string) (string, error)
}
Loopback is the worker's push target: this server's own git smart-HTTP endpoint (phase-5 Decision 4 — sync reuses pack storage, ref policies, CAS, and push-event emission by doing a real `git push`).
type MemStore ¶
type MemStore struct {
// contains filtered or unexported fields
}
MemStore is an in-memory CredentialStore + UpstreamStore: the test double and the local-mode stand-in until the durable meta-backed rows land (foundation slice). Safe for concurrent use.
func NewMemStore ¶
func NewMemStore() *MemStore
func (*MemStore) CreateCredential ¶
func (m *MemStore) CreateCredential(_ context.Context, cred Credential) (Credential, error)
func (*MemStore) CredentialForRepo ¶
func (*MemStore) CredentialMeta ¶
func (*MemStore) DeleteCredential ¶
func (*MemStore) DeleteUpstream ¶
func (*MemStore) FinishPush ¶
func (*MemStore) FinishSync ¶
func (*MemStore) GetCredential ¶
func (*MemStore) GetUpstream ¶
func (*MemStore) PutUpstream ¶
func (*MemStore) RequestPush ¶
func (*MemStore) RequestSync ¶
func (*MemStore) RestorePush ¶
func (m *MemStore) RestorePush(_ context.Context, repoID, holder string, force bool) (Upstream, error)
RestorePush implements ShutdownRestorer with PG's semantics.
func (*MemStore) RestoreSync ¶
RestoreSync implements ShutdownRestorer with PG's semantics.
func (*MemStore) UpdateCredential ¶
type Queue ¶
type Queue interface {
EnqueueSync(ctx context.Context, job Job) error
// EnqueuePush schedules one outbound push-upstream run (issue #90).
// Deduplication mirrors EnqueueSync: callers that promise the push
// mark it first via UpstreamStore.RequestPush.
EnqueuePush(ctx context.Context, job Job) error
}
Queue schedules sync runs. The Postgres deployment backs this with the durable jobs seam (foundation slice); InProcessQueue is the in-process local-mode implementation.
type RejectedRef ¶
RejectedRef is one upstream-rejected ref of an outbound push run.
type RepoInfo ¶
RepoInfo is the minimal repo identity a sync run needs: the storage key (loopback push path element) and the JWT `repo` claim value (event repository.url).
type RepoLookup ¶
RepoLookup resolves a live repo by org + repo_id (production adapter: meta.Store.GetRepoByID). Unknown or soft-deleted repos must return an error wrapping ErrNotFound; the worker then drops the job silently.
type ShutdownRestorer ¶
type ShutdownRestorer interface {
RestoreSync(ctx context.Context, repoID, holder string) (Upstream, error)
RestorePush(ctx context.Context, repoID, holder string, force bool) (Upstream, error)
}
ShutdownRestorer ends a run that serve's shutdown cancelled without counting it (issue #343, PostgreSQL): the claim is released and the request (for push, the consumed level) re-marked. MemStore and PG-backed stores implement it; SQLite and S3WAL restore through LocalLanes instead.
type Upstream ¶
type Upstream struct {
RepoID string
Provider string
Owner string
Name string
UpstreamHost string // resolved at create time: provider default or explicit upstream_host
RunCount int // completed sync runs (success + failure)
SuccessCount int // completed successful runs; is_first_sync := SuccessCount == 0
Syncing bool // one run in flight (BeginSync..FinishSync)
// SyncPending marks a sync request (RequestSync) not yet covered by a
// claimed run: BeginSync consumes it, so a flag still set on the row
// FinishSync returns means requests arrived after the finished run
// claimed (its upstream fetch may predate them) and a follow-up run
// is due (issue #79 liveness).
SyncPending bool
// SyncClaimHolder/SyncClaimExpiresAt lease the Syncing claim (issue
// #86): each claim records the run's holder token and an expiry, and
// BeginSync steals a claim whose lease has provably expired — the
// only recovery when the holder process died mid-run.
SyncClaimHolder string
SyncClaimExpiresAt time.Time
// ForwardOnPush opts the repo into scheduling an outbound
// push-upstream run for every won local push (issue #90,
// base_repo.forward_pushes). Default off: forwarding stays
// explicit-trigger.
ForwardOnPush bool
// PushPending is the outbound analogue of SyncPending (issue #90): a
// three-level durable marker (PushPendingNone/Normal/Force) set by
// RequestPush and consumed by BeginPush. Levels coalesce by MAX so a
// forced request is never downgraded by a later plain one; the level
// must ride the durable marker (not the queue job) because queue
// deduplication is keyed by repo only and a crash-rescued job must
// still remember force.
PushPending int
// PushSyncing/PushClaimHolder/PushClaimExpiresAt lease the outbound
// claim exactly like the inbound triple leases Syncing. The lanes are
// deliberately SEPARATE: the inbound claim protects local refs (the
// loopback force-push), the outbound claim protects upstream refs —
// sharing one claim would couple the two lanes' liveness without
// protecting anything the non-force outbound push does not already
// protect. No legacy adoption exists for this lane: no pre-#90 binary
// ever holds a push claim.
PushSyncing bool
PushClaimHolder string
PushClaimExpiresAt time.Time
// PushClaimForce persists the in-flight claim's consumed force mandate
// until FinishPush (meta.RepoUpstream.PushClaimForce): steals inherit
// it so crashes cannot downgrade an accepted forced request.
PushClaimForce bool
PushRunCount int // completed outbound runs (success + failure)
PushSuccessCount int // completed successful outbound runs
// DefaultBranchOmitted: the create named no default branch, so a
// successful run adopts the upstream HEAD (meta.RepoUpstream, #599).
DefaultBranchOmitted bool
CreatedAt time.Time
}
Upstream is one repo's upstream mirror link (the repo_upstreams row).
type UpstreamStore ¶
type UpstreamStore interface {
// PutUpstream inserts the row (CreatedAt assigned by the store).
// ErrUpstreamExists when the repo already has one.
PutUpstream(ctx context.Context, up Upstream) (Upstream, error)
// GetUpstream returns the row; ErrNotFound when detached/never linked.
GetUpstream(ctx context.Context, repoID string) (Upstream, error)
// DeleteUpstream detaches the repo; ErrNotFound when already detached
// (DELETE /repos/base is idempotent — callers treat that as success).
DeleteUpstream(ctx context.Context, repoID string) error
// RequestSync marks a pending sync request (SyncPending = true).
// Callers that promise a sync (POST /repos/pull-upstream 202) set it
// BEFORE enqueueing: if the enqueue coalesces with a run that already
// claimed, the flag survives to that run's FinishSync and the
// finisher schedules a follow-up run. ErrNotFound when detached.
RequestSync(ctx context.Context, repoID string) error
// BeginSync atomically claims a sync run (syncing false -> true)
// under a lease and returns the row as of the claim (RunCount =
// completed runs). holder is the run's claim token, recorded with an
// expiry of now+lease; a claim whose lease has EXPIRED is stolen
// atomically, and stolenFrom then carries the previous holder for
// observability (issue #86). The claim consumes SyncPending: every
// request marked before the claim is observed by the claimed run's
// upstream fetch, which happens after it. ErrNotFound when detached;
// ErrSyncInProgress when an unexpired run is already syncing.
BeginSync(ctx context.Context, repoID, holder string, lease time.Duration) (up Upstream, stolenFrom string, err error)
// FinishSync completes the run claimed by holder: increments RunCount
// (and SuccessCount when success), clears Syncing and the claim
// lease, returns the updated row. SyncPending is NOT touched: a set
// flag on the returned row tells the finisher to schedule a follow-up
// run (issue #79 liveness). ErrNotFound when the repo was detached
// mid-run — the run then ends silently (docs/reference/webhooks.md:
// detach cancels without a failed event). ErrClaimLost when the
// caller's expired claim was stolen: the late finisher abandons
// without touching the thief's claim or the counters.
FinishSync(ctx context.Context, repoID, holder string, success bool) (Upstream, error)
// RequestPush marks a pending outbound push request (issue #90),
// raising PushPending to at least PushPendingPush (PushPendingForce
// when force) — levels only ever coalesce upward, mirroring the
// RequestSync-before-enqueue protocol: callers that promise a push
// (POST /repos/push-upstream 202, the forward-on-push poke) mark
// BEFORE enqueueing so a request that coalesces with a run that
// already claimed survives to that run's FinishPush. ErrNotFound
// when detached.
RequestPush(ctx context.Context, repoID string, force bool) error
// BeginPush atomically claims an outbound run under a lease —
// TryBeginSync semantics on the push_* lane — and consumes
// PushPending, returning whether the consumed marker demanded force.
// ErrNotFound when detached; ErrPushInProgress when an unexpired
// outbound run is already in flight.
BeginPush(ctx context.Context, repoID, holder string, lease time.Duration) (up Upstream, force bool, stolenFrom string, err error)
// FinishPush completes the outbound run claimed by holder —
// FinishSync semantics on the push_* lane. PushPending is NOT
// touched; a set level on the returned row means requests arrived
// after this run claimed and a follow-up run is due.
FinishPush(ctx context.Context, repoID, holder string, success bool) (Upstream, error)
}
UpstreamStore persists upstream mirror links and their sync-run state.
type Worker ¶
type Worker struct {
Upstreams UpstreamStore
Credentials CredentialStore
Repos RepoLookup
Events EventSink
Loopback Loopback
Logger *slog.Logger
// Timeout caps one run; 0 means defaultRunTimeout.
Timeout time.Duration
// AllowProtocol overrides GIT_ALLOW_PROTOCOL for the run's git
// transfers; empty means "http:https". Only tests widen it (file://
// endpoints); production keeps the default so hostile upstream rows
// can never reach local files, ssh, or transport helpers (ext::).
AllowProtocol string
// Requeue, when non-nil, schedules a follow-up run when FinishSync
// reports sync requests that arrived after this run's claim
// (Upstream.SyncPending, issue #79): this run's upstream fetch may
// predate those requests, so dropping them would break the
// pull-upstream 202 promise. Local-mode wiring points it at the
// queue's EnqueueSync (safe there: the in-process queue never
// deduplicates against a running run). River wiring leaves it nil —
// its insert-time uniqueness would swallow a follow-up inserted while
// this job's row is still running, so the river sync worker schedules
// successors transactionally at job completion instead (jobs/river.go).
Requeue func(ctx context.Context, job Job) error
// RequeuePush is the outbound lane's Requeue (issue #90): follow-up
// AND transient-retry scheduling for push-upstream runs. Same wiring
// split: local mode points it at EnqueuePush, river leaves it nil and
// schedules successors transactionally (jobs/river.go pushWorker).
RequeuePush func(ctx context.Context, job Job) error
// AfterForward, when non-nil, enqueues one inbound sync after every
// SUCCESSFUL outbound run. On ForwardOnPush repos it is the
// mirror-back half of the upstream model ("pushes are proxied to the
// provider and changes mirrored back"): their push->pull poke is
// rerouted to the forward (cmd wiring), so this restores the
// freshness pull strictly AFTER the forward landed, never racing it.
// On explicit push-upstream runs it heals the pull‖push interleaving:
// a concurrent pull that fetched upstream BEFORE the forward would
// otherwise force-write its pre-forward snapshot locally and stand
// divergent until some unrelated future trigger.
// RunPush marks the request durable (RequestSync) BEFORE invoking
// this hook — an enqueue that coalesces with an in-flight pull run
// (river insert-time uniqueness, a held local claim) then survives to
// that run's completion gate instead of silently dropping the
// promised mirror. Enqueue failures are logged; the marker carries
// the request to the next finisher (or, on the local schedule, the
// next sweep). Both wirings
// point it at the queue's EnqueueSync.
AfterForward func(ctx context.Context, job Job) error
// Local, when non-nil, is the durable local schedule (SQLite and
// S3WAL, issue #266, ADR-50, ADR-51): runs claim through ClaimDue* (refused claims end the
// run with nothing written), and one end-of-run write schedules any
// retry, follow-up, or mirror-back in the row. Requeue, RequeuePush,
// AfterForward and the deferred-retry timers are then unused: a
// LocalScheduler sweep turns due rows into runs.
Local LocalLanes
// WaitPublication, when non-nil, blocks until no in-flight push
// still holds the repo's cache refs unpublished
// (githttp.WaitForPushPublication). A local run calls it after its
// claim and before its snapshot and started event, and so does a
// River push run: a hook records intent at the authority CAS, before
// receive-pack publishes the cache ref. A failed wait restores the
// consumed request. Without Local, Upstreams must be a
// ShutdownRestorer, and only RunPush waits.
WaitPublication func(ctx context.Context, repoID string) error
// DetachPoll is the cadence of the outbound push watchdog that
// re-reads the upstream row WHILE `git push` runs and kills the
// subprocess when the repo is detached or the claim stolen mid-push
// (issue #90); 0 means defaultDetachPoll. Only tests shorten it.
DetachPoll time.Duration
// PushEgressAllowPrivate relaxes the outbound push egress guard's
// private receiver classes (resolved from OPENSTORAGE_SYNC_ALLOW_PRIVATE
// with OPENSTORAGE_SYNC_PUSH_ALLOW_PRIVATE as the deprecated push-only
// spelling, see egress.go): loopback/RFC1918/link-local/ULA upstreams
// for genuinely internal forges. Metadata and the special-purpose
// registry stay denied either way — every http(s) push dials through
// the guard's vetting proxy. Tests set it (fixture upstreams are
// 127.0.0.1).
PushEgressAllowPrivate bool
// PullEgressAllowPrivate is the inbound twin (issue #139): every
// http(s) upstream fetch dials through the same vetting proxy, and
// this relaxes only its private receiver classes. The serve wiring
// resolves it from the unified OPENSTORAGE_SYNC_ALLOW_PRIVATE knob,
// whose TRANSITIONAL default keeps it true — internal-forge mirrors
// are a mainstream released deployment shape, and a future release
// flips the unset default to deny (boot warning announces the flip).
// Metadata and the special-purpose registry stay denied either way.
// Tests set it (fixture upstreams are 127.0.0.1).
PullEgressAllowPrivate bool
// contains filtered or unexported fields
}
Worker executes sync runs (docs/reference/webhooks.md lifecycle): claim the run, emit repo.sync.started, mirror-fetch the upstream, loopback-push to storage, then emit succeeded/failed with the exact documented error strings. DELETE /repos/base at any point cancels silently (no failed event).
func (*Worker) Run ¶
Run executes one sync run for job. It returns an error for logging by the queue; documented outcomes (detached, overlapping run, failed sync with an emitted event) are not errors to the caller beyond that. Two errors steer River's sync job: ErrShutdown (keep the job) and ErrRepoGone (complete it).
func (*Worker) RunPush ¶
RunPush executes one outbound push-upstream run for job. Lifecycle, abandonment, and event semantics mirror Run: detach cancels silently at every stage, a stolen claim abandons loudly without terminal events, and documented failures emit repo.push_upstream.failed (issue #354).