Documentation
¶
Overview ¶
Package reconcile is the back half of the repository set-up engine: every set-up step of a declared repository as a check and a repair, run in order by Runner.Run against GitHub and CircleCI, with the outcome as one structured Result the inventory stores.
A step's check reads the repository's state and compares it with what the declaration and the Baseline want. In ModeCheck the step reports the drift as the changes it would make; in ModeRepair it makes them. Every repair is idempotent: a second run finds no drift and changes nothing. What the engine cannot repair it reports as a Finding with the fix spelled out: a declaration whose repository is gone, a redirect on the declared name (a rename the caller follows with a correction PR), a `gen circleci` refusal, the ABS prerequisites of a first chart build, a red first release, a default icon, a repository Renovate shows no sign of scanning.
The steps, in order (Steps): create (only from an entry the caller marks as added — never from a missing repository), scaffold (rendered by the front half and pushed as the first commit before protection), settings baseline, team permissions, branch protection with the required checks on the reported-only rule, CircleCI (follow, setup workflows, checkout key), webhooks, Renovate (check only), CODEOWNERS (a pull request), description and visibility, lifecycle (archived → archived on GitHub and unfollowed; deleted → unfollowed and deleted on GitHub, the entry the record), catalog and mapping (the giantswarm/github workflows), first-release verification (tag → pipeline → workflows; a missed tag build is reported, never rebuilt).
What a run costs in requests is counted at the clients' transports (Counter) into Result.Requests, and per step in the log. A check of a converged repository with every step costs at most twenty GitHub requests, the budget of the nightly reconciler: the repository (one read, shared by the create, metadata and lifecycle steps); the root listing and the chart's Chart.yaml and values.schema.json; the workflow permission; the teams; the branch protection and, with the devctl App id, the rulesets and the engine's own; the reported checks — one page of the recently merged pull requests and the statuses and check runs of the newest head, three requests once per run, shared by every step that asks — the pipeline files workflows.yml and custom.yml (and .circleci/ config.yml once when the entry does not declare the pipeline); renovate.json5 and the Dependency Dashboard issue; CODEOWNERS; the catalog and the mapping; the latest release. Measured in check mode with a person's token: giantswarm/backstage 18, giantswarm/klaus 19 (one of them a CODEOWNERS drift, which also lists the open pull requests); the App id's ruleset reads add two. A planned change costs the same reads; a repair adds one write per change.
The reconciler workflow of giantswarm/github runs the steps under the App identity, `devctl repo reconcile` runs them as the person, `devctl repo setup` and `repo checks` call the same steps, and giantswarm-repo-manager runs the checks in ModeCheck to fill the inventory.
Index ¶
- Constants
- Variables
- func GateContexts(files ...[]byte) ([]string, error)
- func GateJobs(raw []byte) ([]string, error)
- func IsInvalidConfig(err error) bool
- func IsNotOwner(err error) bool
- func NotOwnerRefusal(owner string) string
- func StaleCircleCIContexts(existing, live []string) []string
- type Baseline
- type Counter
- type CreateRequest
- type CreateResult
- type Finding
- type FindingKind
- type Mode
- type ReportedChecker
- type Request
- type Requests
- type Result
- type Runner
- type ScaffoldRenderer
- type Step
- type StepResult
- type Verdict
- type Webhook
Constants ¶
const CircleCIContextPrefix = "ci/circleci: "
CircleCIContextPrefix is the status context GitHub receives for a CircleCI job: "ci/circleci: <job name>".
const LifecycleArchived = "archived"
LifecycleArchived is the lifecycle value that archives a repository.
const LifecycleDeleted = "deleted"
LifecycleDeleted is the lifecycle value that deletes a repository on GitHub; the entry stays in the team file as the record of the deletion.
const RulesetName = "devctl: default branch"
RulesetName is the name of the repository ruleset the protection step writes: one per repository, the engine's own. Every other ruleset of the repository is left alone.
Variables ¶
var PipelineFiles = []string{"workflows.yml", "custom.yml"}
PipelineFiles are the files of a generated pipeline that carry jobs, relative to .circleci: workflows.yml is what `devctl gen circleci` writes, custom.yml the repository-owned additions the setup workflow in config.yml merges into it at pipeline runtime.
var Steps = []Step{ StepCreate, StepScaffold, StepSettings, StepPermissions, StepProtection, StepCircleCI, StepWebhooks, StepRenovate, StepCodeowners, StepMetadata, StepLifecycle, StepCatalog, StepRelease, }
Steps lists every step in execution order.
Functions ¶
func GateContexts ¶
GateContexts returns the status contexts of the branch-side jobs of the pipeline documents in files, sorted and without duplicates. A job counts unless its branch filter restricts it to named branches (`only:`) or ignores every branch (`ignore: /.*/`, the tag-only release jobs): those never report on a pull request and cannot gate one. Jobs of every workflow count; custom.yml may add its own workflows.
func GateJobs ¶
GateJobs parses one CircleCI config document and returns the names of its branch-side jobs (see GateContexts).
func IsInvalidConfig ¶
IsInvalidConfig asserts invalidConfigError: the run could not start.
func IsNotOwner ¶ added in v8.65.0
IsNotOwner asserts notOwnerError: the caller is not an owner of the organization and cannot create a repository in it.
func NotOwnerRefusal ¶ added in v8.65.0
NotOwnerRefusal is the text a caller who is not an owner of the organization is refused with: the organization does not let members create repositories, and every path — this engine, the Dev Portal — creates the repository as the person.
func StaleCircleCIContexts ¶
StaleCircleCIContexts returns the required contexts of CircleCI jobs the pipeline no longer has: every "ci/circleci: <job>" context in existing whose job is not in live. Contexts of other systems are never returned.
Types ¶
type Baseline ¶
type Baseline struct {
// DefaultBranch is the protected default branch.
DefaultBranch string `json:"defaultBranch"`
// Features.
HasWiki bool `json:"hasWiki"`
HasIssues bool `json:"hasIssues"`
HasProjects bool `json:"hasProjects"`
// Merge settings.
AllowMergeCommit bool `json:"allowMergeCommit"`
AllowSquashMerge bool `json:"allowSquashMerge"`
AllowRebaseMerge bool `json:"allowRebaseMerge"`
// Pull requests.
AllowUpdateBranch bool `json:"allowUpdateBranch"`
AllowAutoMerge bool `json:"allowAutoMerge"`
DeleteBranchOnMerge bool `json:"deleteBranchOnMerge"`
// WorkflowPermissions is the default GITHUB_TOKEN permission of the
// repository's workflows: read or write.
WorkflowPermissions string `json:"workflowPermissions"`
// TeamPermissions maps a team slug to the permission it holds on every
// repository: pull, triage, push, maintain or admin.
TeamPermissions map[string]string `json:"teamPermissions"`
// Branch protection: classic without a devctl App id, the default
// branch's ruleset ([RulesetName]) with one.
RequiredReviews int `json:"requiredReviews"`
// EnforceAdmins binds administrators to classic protection too. True in
// [DefaultBaseline], the company baseline: what `devctl repo setup`
// applies and what the merge tool lifts and restores around a merge.
// The ruleset has no such switch: everyone but its bypass actors is
// bound.
EnforceAdmins bool `json:"enforceAdmins"`
// StrictChecks requires a branch to be up to date before it merges.
// False in [DefaultBaseline]: on a repository with Renovate and sweep
// traffic every merge would invalidate every other open pull request
// and re-run its CI.
StrictChecks bool `json:"strictChecks"`
// RequiredChecks are required whatever reported.
RequiredChecks []string `json:"requiredChecks,omitempty"`
// RequiredChecksIfReported are required once they have reported on the
// default branch or a recently merged pull request — the generated
// GitHub Actions gates. The generated CircleCI pipeline's jobs are
// candidates the same way, read from the repository.
RequiredChecksIfReported []string `json:"requiredChecksIfReported,omitempty"`
// IgnoredChecks are regular expressions of contexts never required and
// removed when found required: release workflows, path-filtered
// workflows, the dependency-graph submission — contexts that report
// on the default branch but cannot report on every pull request.
IgnoredChecks []string `json:"ignoredChecks,omitempty"`
// Webhooks every repository carries; none by default (CircleCI installs
// its own on follow).
Webhooks []Webhook `json:"webhooks,omitempty"`
// RenovateInstallationID is the Renovate GitHub App installation whose
// repository list the renovate step reads as detail when the token can
// (an organization owner's; a GitHub App token cannot); 0 reads none.
// The step's verdict comes from the repository's own evidence.
RenovateInstallationID int64 `json:"renovateInstallationID"`
// CatalogRepository holds the catalog and the two workflows, as
// owner/name.
CatalogRepository string `json:"catalogRepository"`
// CatalogPath is the catalog file listing the components.
CatalogPath string `json:"catalogPath"`
// CatalogWorkflow regenerates the catalog (input force).
CatalogWorkflow string `json:"catalogWorkflow"`
// MappingRepository holds the apps-to-teams mapping, as owner/name.
MappingRepository string `json:"mappingRepository"`
// MappingPath is the mapping ConfigMap file.
MappingPath string `json:"mappingPath"`
// MappingWorkflow regenerates the mapping (input repository), in
// CatalogRepository.
MappingWorkflow string `json:"mappingWorkflow"`
}
Baseline is the repository set-up every declared repository gets on top of its declaration: the settings, team permissions and branch protection of giantswarm/giantswarm#36733 as `devctl repo setup` applies them, the required-check rule, the webhooks, and where the catalog and the mapping live. DefaultBaseline is the company's; a caller adjusts a copy.
func DefaultBaseline ¶
func DefaultBaseline() Baseline
DefaultBaseline is the company baseline: what `devctl repo setup` applies by default, the required-check rule of `devctl repo checks`, the catalog and mapping of giantswarm/github and management-cluster-bases.
type Counter ¶ added in v8.82.1
type Counter struct {
// Base sends the requests; nil means http.DefaultTransport.
Base http.RoundTripper
// contains filtered or unexported fields
}
Counter is an http.RoundTripper that counts the requests it sends. The caller builds a client's transport over one and hands it to the Runner, which reads the run's cost from it (Result.Requests). A request counts when it is sent, whatever it answers; one Counter may sit under several clients of the same system, so their requests add up.
type CreateRequest ¶ added in v8.65.0
type CreateRequest struct {
// Owner is the GitHub organization; empty means reposetup.DefaultOwner.
Owner string
// Team is the slug of the team file the entry is declared in.
Team string
// Entry is an accepted entry of the dry run; its Rendered declaration
// gives the name, the description and the visibility.
Entry reposetup.Entry
// Mode is check or repair; empty means check. [ModeCheck] is the dry
// run: the caller's organization role is read and the two steps plan
// what they would do, nothing is written.
Mode Mode
// RenderOptions selects among the template's options for the scaffold.
RenderOptions map[string]string
}
CreateRequest is one creation of one declared repository as the caller: the create and scaffold steps of Runner.Run, standalone, with whatever GitHub client the Runner holds — the person's, so the repository is created as them and they are its admin.
type CreateResult ¶ added in v8.65.0
type CreateResult struct {
// Repository is owner/name.
Repository string `json:"repository"`
// URL is the repository on GitHub; empty in a dry run of a repository
// that does not exist yet.
URL string `json:"url,omitempty"`
// Created says this run created the repository; false when it existed
// (a resumed creation) or in a dry run.
Created bool `json:"created"`
// ScaffoldCommit is the commit holding the scaffold at the head of the
// default branch — the one this run pushed, or the one it found.
ScaffoldCommit string `json:"scaffoldCommit,omitempty"`
Mode Mode `json:"mode"`
StartedAt time.Time `json:"startedAt"`
FinishedAt time.Time `json:"finishedAt"`
// Steps are the create and the scaffold step, in that order.
Steps []StepResult `json:"steps"`
}
CreateResult is the outcome of Runner.Create.
func (*CreateResult) Failed ¶ added in v8.65.0
func (r *CreateResult) Failed() []StepResult
Failed returns the steps that could not run to their end.
func (*CreateResult) Step ¶ added in v8.65.0
func (r *CreateResult) Step(step Step) *StepResult
Step returns the result of step, or nil when the run did not execute it.
type Finding ¶
type Finding struct {
Kind FindingKind `json:"kind"`
Message string `json:"message"`
Fix string `json:"fix"`
// Advisory mirrors [FindingKind.Advisory] in the artifact, so a reader
// knows the weight of the finding without knowing the kinds.
Advisory bool `json:"advisory,omitempty"`
}
Finding is something a step reports for a person, with the fix.
type FindingKind ¶
type FindingKind string
FindingKind classifies a finding.
const ( // FindingRepositoryMissing: the declared repository does not exist and // the entry was not added by this change. FindingRepositoryMissing FindingKind = "repository-missing" // FindingRenamed: the declared name redirects to a renamed repository. FindingRenamed FindingKind = "renamed" // FindingGenCircleCIRefused: the CircleCI generator would produce no // jobs for the declaration. FindingGenCircleCIRefused FindingKind = "gen-circleci-refused" // FindingABSPrerequisite: the chart lacks something app-build-suite // validates on the first build. FindingABSPrerequisite FindingKind = "abs-prerequisite" // FindingDefaultIcon: the chart still carries the default icon. FindingDefaultIcon FindingKind = "default-icon" // FindingRedRelease: the latest release's tag pipeline failed. FindingRedRelease FindingKind = "red-release" // FindingMissedTagBuild: the latest release's tag has no pipeline. The // reconciler never rebuilds a tag; the fix is the next tag, or the // tag's pipeline triggered by hand. FindingMissedTagBuild FindingKind = "missed-tag-build" // FindingRenovateNotScanned: the repository shows no sign that Renovate // scans it — no configuration, or a configuration without a trace of a // run (the Dependency Dashboard issue, a pull request, a commit). FindingRenovateNotScanned FindingKind = "renovate-not-scanned" // FindingArchivedUndeclared: archived on GitHub without lifecycle: // archived. FindingArchivedUndeclared FindingKind = "archived-undeclared" // FindingPendingPullRequest: a repair landed as a pull request that // awaits its merge. FindingPendingPullRequest FindingKind = "pending-pull-request" // FindingUnchecked: a check could not run with the caller's access. FindingUnchecked FindingKind = "unchecked" // FindingEntryRefused: the validator refused the entry for a reason // other than the CircleCI generator; the fix names the field. FindingEntryRefused FindingKind = "entry-refused" // FindingForeignRuleset: the repository carries a ruleset the engine did // not create. It is left alone; a person decides whether it stays. FindingForeignRuleset FindingKind = "foreign-ruleset" // FindingRulesetsNotEnabled: the run has no devctl App id, so the // protection step kept classic branch protection; the id is the switch // to the default branch's ruleset with the App as bypass actor. FindingRulesetsNotEnabled FindingKind = "rulesets-not-enabled" )
The kinds of finding.
func (FindingKind) Advisory ¶ added in v8.70.0
func (k FindingKind) Advisory() bool
Advisory says whether findings of the kind are for a person only and do not keep the run from converging: the repository is set up as declared, the finding stays in the artifact with its fix. Every other kind names something a person must fix before the repository counts as in sync.
type ReportedChecker ¶
type ReportedChecker interface {
ReportedChecks(ctx context.Context, repository *github.Repository, branch string) ([]string, error)
}
ReportedChecker returns the check contexts that have reported on the heads of the recently merged pull requests — the reported-only rule of `devctl repo checks`. *githubclient.Client satisfies it.
type Request ¶
type Request struct {
// Owner is the GitHub organization; empty means reposetup.DefaultOwner.
Owner string
// Team is the slug of the team file the entry lives in (team-bumblebee).
Team string
// Entry is an accepted entry of the front half's dry run; its Rendered
// declaration is the desired state.
Entry reposetup.Entry
// Added says the triggering change added the entry: the one condition
// under which a missing repository is created.
Added bool
// Mode is check or repair; empty means check.
Mode Mode
// Steps restricts the run to the named steps; nil runs every step. The
// repository is looked up in any case.
Steps []Step
// RenderOptions selects among the template's options for the scaffold.
RenderOptions map[string]string
// Pipeline holds the documents of the generated CircleCI pipeline
// (workflows.yml, custom.yml) the protection step reads the branch-side
// jobs from instead of the repository's .circleci — the files a caller
// has just generated and not pushed yet. Nil reads the repository.
Pipeline [][]byte
}
Request is one run over one declared repository.
type Requests ¶ added in v8.82.1
Requests is what a run cost in requests to each system: every request the run's clients sent, counted at the transport (Counter), whatever it answered. The result carries it so that the cost of a repository is known against the reconciler's budget, the App's requests per hour.
type Result ¶
type Result struct {
// Repository is owner/name as the run addressed it — the renamed name
// when the declared one redirected.
Repository string `json:"repository"`
// Declared is owner/name as the team file declares it.
Declared string `json:"declared"`
Team string `json:"team"`
Mode Mode `json:"mode"`
// Added says whether the entry was passed as added by the triggering
// change, which alone allows the create step to create.
Added bool `json:"added"`
StartedAt time.Time `json:"startedAt"`
FinishedAt time.Time `json:"finishedAt"`
Steps []StepResult `json:"steps"`
// Converged says the repository is set up as declared: no step ended in
// drift or failure and every finding is advisory ([StepResult.Converges]
// for each step). A finding a person must fix clears it.
Converged bool `json:"converged"`
// Requests is what the run cost in requests to GitHub and CircleCI,
// when the Runner's clients count them ([Runner.GitHubRequests]);
// omitted when nothing was counted.
Requests Requests `json:"requests,omitzero"`
}
Result is the outcome of one run over one repository: the structured value the inventory stores and the callers render.
func Refused ¶ added in v8.64.7
Refused is the result of an entry the validator refused: no step ran, the declaration is at fault, and the callers parse the refusal like any other finding instead of an exit without a result. The result carries one step, StepEntry, VerdictReported, with one finding per problem naming the field to fix — FindingGenCircleCIRefused for gen.ci.generate (the CircleCI generator would produce no job), FindingEntryRefused for the rest. Nothing failed and nothing drifted, so the result is converged.
func (*Result) Failed ¶ added in v8.63.0
func (r *Result) Failed() []StepResult
Failed returns the steps that could not run to their end.
func (*Result) Step ¶
func (r *Result) Step(step Step) *StepResult
Step returns the result of step, or nil when the run did not execute it.
func (*Result) WriteTable ¶ added in v8.63.0
WriteTable renders the result for a person: a header naming the repository (and the declared name after a rename), the mode and whether the run converged; one row per step with its verdict and detail; then every finding with its fix.
type Runner ¶
type Runner struct {
// GitHub is the client the steps read and write GitHub with, under the
// App installation token or the person's.
GitHub *github.Client
// Dispatch is the client the catalog step lists and dispatches the
// catalog and mapping workflow runs with; nil means GitHub. A caller
// whose GitHub identity holds no Actions permission on the catalog
// repository gives the token that does (a workflow run's own, with
// actions: write) here and keeps every read — the repository lookup,
// the catalog and the mapping — with GitHub, which sees the private
// repositories that token does not.
Dispatch *github.Client
// Checks answers which check contexts have reported.
Checks ReportedChecker
// CircleCI is the client for follow, settings, keys and pipelines.
CircleCI *circleciclient.Client
// Renderer renders the scaffold the scaffold step pushes.
Renderer ScaffoldRenderer
// Baseline is the set-up applied on top of the declaration; nil means
// [DefaultBaseline].
Baseline *Baseline
// DevctlAppID is the numeric id of the devctl GitHub App (the App's
// settings page; not the client id) and the switch to rulesets: with it
// the protection step writes the default branch's ruleset with the App
// as bypass actor in pull_request mode (none on agentMerge: false) and
// removes classic protection; 0 keeps classic branch protection as
// before and reports the missing id.
DevctlAppID int64
// GitHubRequests and CircleCIRequests count the requests the clients
// send, when the caller built the clients' transports over them (one
// [Counter] under GitHub, Checks and Dispatch, one under CircleCI). The
// run's cost is what they counted over the run, in [Result.Requests]
// and, per step, in the log. Nil counts nothing.
GitHubRequests, CircleCIRequests *Counter
// Log receives one line per step and change; nil discards.
Log io.Writer
// Now is the clock; nil means time.Now.
Now func() time.Time
}
Runner runs the set-up steps. GitHub is required; the others are optional and their absence skips what needs them: without CircleCI the CircleCI and release steps are skipped, without Checks no check is required on the reported-only rule (none is removed on a guess either), without Renderer the scaffold cannot be repaired.
func (*Runner) Create ¶ added in v8.65.0
func (r *Runner) Create(ctx context.Context, req CreateRequest) (*CreateResult, error)
Create creates the declared repository and pushes its scaffold as the caller, and nothing else: the create step (skipped when the repository exists — a creation resumed after a failure) and the scaffold step (one commit on the default branch; skipped when it is there). The same steps [Run] executes for the reconciler, so the two paths cannot drift. The set-up that follows — settings, protection, CircleCI, the catalog — is the reconciler's, from the merged declaration.
The caller's role in the organization is read first, in every mode: the organization does not let members create repositories, so anyone but an owner is refused with NotOwnerRefusal before a write (IsNotOwner), and a 403 on the creation itself ends the create step with the same text. A step that fails is a VerdictFailed in the result; the scaffold step then does not run and the caller reruns once the cause is fixed.
func (*Runner) Run ¶
Run executes the steps for req and returns the result. An error means the run could not start (an unaccepted entry, no GitHub client); a step that fails is a VerdictFailed in the result and the run continues.
type ScaffoldRenderer ¶
type ScaffoldRenderer interface {
Render(ctx context.Context, req reposetup.RenderRequest) (*reposetup.Scaffold, error)
}
ScaffoldRenderer renders the scaffold of an accepted entry into a directory. reposetup.Renderer satisfies it.
type Step ¶
type Step string
Step names one set-up step.
const ( StepCreate Step = "create" StepScaffold Step = "scaffold" StepSettings Step = "settings" StepPermissions Step = "permissions" StepProtection Step = "protection" StepCircleCI Step = "circleci" StepWebhooks Step = "webhooks" StepRenovate Step = "renovate" StepCodeowners Step = "codeowners" StepMetadata Step = "metadata" StepLifecycle Step = "lifecycle" StepCatalog Step = "catalog" StepRelease Step = "release" // StepEntry is the declaration itself: the one step of a [Refused] // result, never run by [Runner.Run]. StepEntry Step = "entry" )
The steps, in the order Runner.Run executes them.
type StepResult ¶
type StepResult struct {
Step Step `json:"step"`
Verdict Verdict `json:"verdict"`
// Summary is one line on the state found.
Summary string `json:"summary,omitempty"`
// Changes are the repairs made ([ModeRepair]) or the repairs a run
// would make ([ModeCheck]).
Changes []string `json:"changes,omitempty"`
// Findings are reported, not repaired; each carries its fix.
Findings []Finding `json:"findings,omitempty"`
}
StepResult is the outcome of one step.
func (*StepResult) Converges ¶ added in v8.70.0
func (s *StepResult) Converges() bool
Converges says whether the step lets the run converge: it did not drift or fail, and every finding it carries is advisory.
type Verdict ¶
type Verdict string
Verdict is the outcome of one step.
const ( // VerdictOK: no drift, nothing to report. VerdictOK Verdict = "ok" // VerdictDrift: drift found and not repaired — the run was a check, or // the repair is a pull request that awaits its merge. VerdictDrift Verdict = "drift" // VerdictRepaired: drift found and repaired in this run. VerdictRepaired Verdict = "repaired" // VerdictReported: no drift the engine repairs, but findings with a fix // for a person. VerdictReported Verdict = "reported" // VerdictSkipped: the step did not apply (repository missing or empty, // archived or deleted, no client for the system, no CircleCI pipeline // for the circleci and release steps). VerdictSkipped Verdict = "skipped" // VerdictFailed: the step could not run to its end; Summary says why. VerdictFailed Verdict = "failed" )