Documentation
¶
Overview ¶
Package github is the outbound adapter that implements domain.ForgeSource over GitHub: the GraphQL API for repositories, pull requests and reviews, and git over HTTPS for the managed mirrors (ADR 0004). It is the only package that talks to GitHub, and only stillhere sync runs it.
It is the anti-corruption layer of the Forge context: GitHub's JSON shapes stay in this package and only domain types leave it, with GitHub's states translated (MERGED becomes domain.PullMerged) and its quirks absorbed (a deleted account becomes a ForgeUser without a login; an app becomes a Bot).
A Source holds the token and never hands it out. The API gets it in an Authorization header from go-gh's client; git gets it only as an environment-scoped http.extraheader on the Sync profile (docs/spec/git-invocation.md#credentials), so it is never in argv, a URL, a config file, a log line, a fact or an error.
GitHub asks API clients to avoid many concurrent requests, to slow down before the hourly points run out, and to back off from secondary rate limits with growing waits. A Source does all three whatever its caller's concurrency: mirrors fetch in parallel, while API requests queue behind a gate of three, and a rate limit one of them meets holds them all.
Index ¶
- Constants
- func ValidName(s string) bool
- type Config
- type Source
- func (s *Source) CountPullRequests(ctx context.Context, repo domain.ForgeRepo, from, start time.Time) (int, error)
- func (s *Source) LinkAuthors(ctx context.Context, repo domain.ForgeRepo, mirror string, ...) error
- func (s *Source) Mirror(ctx context.Context, repo domain.ForgeRepo, root string) (string, error)
- func (s *Source) PullRequests(ctx context.Context, repo domain.ForgeRepo, since time.Time, ...) error
- func (s *Source) ReconcileMirrors(ctx context.Context, owner string, listed []domain.ForgeRepo, root string) ([]domain.MirrorMove, error)
- func (s *Source) Repo(ctx context.Context, owner, name string) (domain.ForgeRepo, error)
- func (s *Source) Repos(ctx context.Context, owner string) ([]domain.ForgeRepo, error)
Constants ¶
const DefaultHost = "github.com"
DefaultHost is the forge host when Config.Host is empty.
Variables ¶
This section is empty.
Functions ¶
func ValidName ¶
ValidName reports whether s can be a GitHub owner (an organization or a user login) or repository name, as the adapter takes them: ASCII letters, digits, ".", "_" and "-", and neither "." nor "..". It is the one rule: Repos, Mirror and ReconcileMirrors apply it, and the command line checks --org with it before a sync begins its run, so a URL pasted as an organization, token and all, is refused before anything records it.
Types ¶
type Config ¶
type Config struct {
// Host is the GitHub host, "github.com" when empty. go-gh derives the
// API endpoint from it (api.github.com, or /api/graphql on GitHub
// Enterprise Server).
Host string
// Token authenticates every request. When empty, New looks one up the
// way gh does: GH_TOKEN, then GITHUB_TOKEN, then gh's stored token.
Token string
// Git runs the mirror commands. Mirror fails without it.
Git domain.GitRunner
// Logger receives info lines for waits and retries and debug lines for
// each request. It never receives the token. nil discards them.
Logger *slog.Logger
// UserAgent names stillhere to GitHub, such as "stillhere/0.2.0";
// "stillhere" when empty.
UserAgent string
// GitURL is the base URL that mirrors clone from, "https://" + Host when
// empty. It must be https, except on a loopback host, where a test
// serves repositories.
GitURL string
// Transport carries the API requests, http.DefaultTransport when nil.
// Tests replay recorded responses through it.
Transport http.RoundTripper
// Sleep waits out a rate limit or a backoff. nil waits on a timer that
// ctx cancels; tests record the waits instead.
Sleep func(ctx context.Context, d time.Duration) error
// Clock reads the local clock, for the time since GitHub's last Date
// and for how long a rate limit holds every request. nil is time.Now;
// tests pass one that Sleep moves on.
Clock func() time.Time
// ChangedPaths says the git binary writes version 2 changed-path Bloom
// filters (git 2.46 and later): each mirror's commit-graph then gets
// them, which make blame several times faster (#56). Without, a mirror
// gets a commit-graph with no filters, because version 1 filters hash
// paths that are not ASCII differently on some platforms
// (docs/spec/git-invocation.md#sync-commands-network).
ChangedPaths bool
}
Config configures a Source.
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source is the GitHub adapter. It implements domain.ForgeSource and is safe for concurrent use.
func New ¶
New builds a Source. A missing token, or one GitHub's client cannot use, is a *domain.UsageError, because the user fixes it by signing in.
The token lookup is go-gh's auth.TokenForHost, so a user's gh auth login works unchanged. That function takes no context: when it falls back to gh's stored token it runs "gh auth token", which a locked keychain can hold, and only a second interrupt ends that wait. ctx is checked before and after.
func (*Source) CountPullRequests ¶
func (s *Source) CountPullRequests(ctx context.Context, repo domain.ForgeRepo, from, start time.Time) (int, error)
CountPullRequests counts the pull requests of repo last updated at or after from (from the first for a zero from) and before start less the overlap: one updated in the overlap may have been read before that update, and GitHub's clocks may disagree by a little, so it is left out rather than counted as missing. GitHub's search leaves out pull requests it hides, such as a spam account's (on cli/cli, 2026-10-15: 4,698 by search of 4,730 listed), so the count is a lower bound.
func (*Source) LinkAuthors ¶
func (s *Source) LinkAuthors(ctx context.Context, repo domain.ForgeRepo, mirror string, skip map[string]bool, put func([]domain.ForgeCommit) error) error
LinkAuthors asks GitHub which account each author email of the mirror's history belongs to, but the emails skip holds: one commit per email, its newest, a hundred a query, and hands each query's answers to put (ForgeSource.LinkAuthors). GitHub's Commit.author.user is "the GitHub user corresponding to the email field": a function of the email, so one commit answers for every commit written with it.
The emails come from the mirror itself, git log's %ae, the email as the commit holds it: neither %aE nor --use-mailmap, since GitHub reads the commit, not the mirror's .mailmap. git log is a local command, run without the credentials (docs/spec/git-invocation.md#sync-commands-network).
func (*Source) Mirror ¶
Mirror creates the bare mirror of repo at root/<owner>/<name>.git, or brings an existing one up to date, and returns its path. It runs exactly the commands of docs/spec/git-invocation.md#sync-commands-network:
git clone --bare <url> <tmp> (first sync only)
git -C <tmp> config stillhere.forgeId <id> (first sync only)
git -C <dir> config remote.origin.fetch +refs/heads/*:refs/heads/*
git -C <dir> fetch --prune origin
git -C <dir> symbolic-ref HEAD refs/heads/<default branch>
git -C <dir> -c commitGraph.changedPathsVersion=2 commit-graph write --reachable --changed-paths
(git 2.46 and later; else without filters)
A first clone goes into root/<owner>/.stillhere-clones and is renamed into place only once complete and marked with repo's node id, so an interrupted clone never leaves a half-written mirror where scan would find it. An existing directory is fetched only when it is a bare repository that carries repo's node id, or, made before mirrors carried one, fetches from the expected URL; anything else at that path is left alone and reported. A mirror that carries the node id fetches from the URL the API reports, so after a rename it follows the new name. HEAD is re-pointed on every sync because a fetch never moves a bare repository's HEAD, and GitHub is the system of record for the default branch.
ReconcileMirrors, which runs first, has already moved a renamed or transferred repository's mirror to this path.
func (*Source) PullRequests ¶
func (s *Source) PullRequests(ctx context.Context, repo domain.ForgeRepo, since time.Time, put func(*domain.ForgePage) error) error
PullRequests calls put with every pull request of repo last updated at or after since, a page at a time, newest update first.
GitHub's pull request list has no "updated since" filter, only an order by updatedAt, which is GitHub's and can disagree with the updatedAt values it returns: on cli/cli one pull request updated in July was listed among those updated in March (#101). Two rules make a pass safe anyway (ADR 0013):
- The watermark is a position GitHub owns, never a value read from the pull requests: the time the pass began on GitHub's clock (passStart), carried by the last page only, and only when the pass read down to since. A pass that stops halfway, for any reason, moves no watermark, and the next one reads the same range again.
- The stop is tolerant: a pull request updated before the stop point (since less the overlap) is skipped, not taken for the end of the range, and the pass ends at the first page that holds none at or after the stop point, or at the end of the list. So one pull request out of order near the top cannot end a pass early.
A pull request updated during the pass moves to the front, behind the reader; its new update is after the pass began, so the next pass, which reads down to this one's start less the overlap, reads it. A zero since reads the whole list.
This is how Kafka's consumers and Postgres's replication slots keep their place, by an offset or a log sequence number the source owns, not by a field of the records (Kleppmann, Designing Data-Intensive Applications, ch. 11, "Logs and offsets"). Each page's pull requests are complete: a list longer than the page asked for is finished with follow-up queries before the page is handed to put.
func (*Source) ReconcileMirrors ¶
func (s *Source) ReconcileMirrors(ctx context.Context, owner string, listed []domain.ForgeRepo, root string) ([]domain.MirrorMove, error)
ReconcileMirrors moves the mirrors under root to the paths of the repositories they belong to, before any mirror is fetched, and returns the moves, sorted by From. A mirror says which repository it belongs to by the node id in its config (forgeIDKey), which a rename or a transfer keeps:
- The mirror of a listed repository that is not at the repository's path, root/<owner>/<name>.git, moves there: the repository was renamed or transferred (MirrorRenamed), or is listed again after its mirror went stale (MirrorRestored). Mirror then points its origin at the new URL.
- Under root/<owner>, for the owners of this listing, the mirror of a repository the listing does not hold moves to root/.stale/<owner> (MirrorStale): one GitHub no longer lists, or one this sync leaves out, such as an archived repository without --include-archived (#110). So does a second mirror of a listed repository (MirrorDuplicate). Nothing is deleted: a repository that comes back, here or under another owner, gets its mirror back. When owner is "", the listing is not an owner's whole one (sync --repo), and no mirror of an unlisted repository moves.
Otherwise a renamed repository would be cloned again under its new name and scanned twice, once under each name, since discovery keys repositories by origin URL.
The moves run one at a time, before the parallel fetches: the moves that free paths first, then the moves into them, through root/.stale/.moving when a path is taken. A move that fails is returned with its Err and the others go on. Directories that carry no node id are not stillhere's and are never moved.
func (*Source) Repo ¶
Repo looks up the repository owner/name (#111), as Repos lists it. A name that cannot be a GitHub name, and one GitHub has no repository for that the token can see, are a *domain.UsageError: GitHub answers a missing repository with a NOT_FOUND error on a null repository, which the query reads as that null.
Source Files