pkgselect

package
v1.2.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 11, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package pkgselect chooses the Go packages a CI run tests and deals them to the legs that test them. It is the one home of four rules that every place choosing packages reads:

  • a deprecated package is never tested: pkg/pkgselect/DEPRECATED names the packages that are deprecated and still in the tree because living tools import them, and Deprecated.Live drops them from any list (live.go);
  • a selection is never silently nothing: a failed `go list`, or a Go diff that selects zero packages, is an error or an announced whole-tree fallback, never an empty answer (select.go);
  • the heavy packages are dealt first, one to a shard, and every other live package round-robin after them (deal.go);
  • the CI fan-out is the selection dealt onto the runner groups, with the macOS legs kept for the code that differs on macOS, and dealt at all only where the target branch is dev or main (DarwinOn, matrix.go).

Everything that starts a process goes through a Runner, so a test answers `git` and `go list` from a table and never starts either. The verbs in tools/ci and the local run in cmd/nova-ci both import this package.

Index

Constants

View Source 
const (
	// LinuxShards and MacShards are the whole-tree shard counts (a push to dev,
	// a manual run, the nightly run). The split is a measurement, not a
	// preference: the two fleets are not symmetric, and the same count on both
	// was the wrong shape once they were not.
	LinuxShards = 8
	MacShards   = 8
	// PullRequestShards is the leg count of each group on a pull request, and
	// MergeGroupMacShards of the macOS group on a merge group, under the
	// two-minute cap: the fan-out cannot be as parallel as the runners a queue
	// would need, so the legs are few and each takes a wave.
	PullRequestShards   = 4
	MergeGroupMacShards = 4
	// PullRequestMacShards is the macOS group's leg count on a pull request: a change that reaches
	// many darwin-sensitive packages (a promotion) dealt four legs of four ran every leg past the
	// two-minute cap (2026-10-04, all four cancelled twice); a small change fills few of the eight.
	PullRequestMacShards = 8
	// FunctionalShards is the leg count of the functional tier. Six: the sprint stream moved
	// slow real-time tests into the tier and four legs ran past the two-minute cap
	// (functional 1/4 and 3/4 were cancelled at it); the cap is permanent, the split is not.
	FunctionalShards = 6
)

The CI fan-out: the selected packages dealt onto runner groups. Runner labels are carried as single-string fields (os, arch, group), not as a label array: a runs-on expression must resolve to a string, so the test job composes its label list per leg.

View Source 
const (
	// RunnersEnv is the variable the runner service exports: how many runners
	// share this machine. HOW MANY RUNNERS is the MACHINE's fact, not this
	// program's: written here it goes stale, and did, when a fleet grew from four
	// runners a machine to eight.
	RunnersEnv = "NOVA_RUNNERS_PER_MACHINE"
	// DefaultRunners is today's fleet, for a machine that says nothing.
	DefaultRunners = 8
	// ShareCeiling: AT MOST TWO cores a leg. Unit tests "must not be so aggressive
	// that they fill a whole machine cores": min(share, 2), the Makefile's
	// GOTEST_P, whatever the box.
	ShareCeiling = 2
)
View Source 
const (
	// UnitShimDir is the directory under $RUNNER_TEMP that holds the shim.
	UnitShimDir = "unit-tier-bin"
	// UnitShimExit is the shim's exit status.
	UnitShimExit = 86
	// UnitShimMessage is what the shim prints on stderr.
	// pkg/nsprint/testutil.Start names this line.
	UnitShimMessage = "unit tier: redis-server is functional-only (build tag functional)"
)
View Source 
const DeprecatedFile = "pkg/pkgselect/DEPRECATED"

DeprecatedFile is the list of deprecated packages, relative to the repository root.

View Source 
const ShardGoTestTimeout = "110s"

ShardGoTestTimeout is `go test -timeout` for a unit shard. It ends a hung run with a Go stack naming the test before the two-minute job cap kills the leg silently: a timeout at or over the cap would be no timeout at all. internal/ci: TestShardGoTestTimeoutIsUnderTheJobCap.

Variables

View Source 
var DarwinBranches = []string{"main", "dev"}

DarwinBranches are the target branches whose changes meet the darwin legs: the integration branches (the concurrency group's integration list in ci.yml, in short form; internal/ci's TestDarwinShardsRunOnlyForIntegrationBranches holds the two equal). A change bound for a working branch meets Linux only: the Go is the same Go on both OSes, the Linux legs run every selected package, and the darwin legs are the slowest and the scarcest.

View Source 
var DarwinOnly = []string{"./cmd/nova-sandbox", "./pkg/sandbox"}

DarwinOnly are the packages with no Linux leg: their sandbox backend is macOS-only today (docs/USAGE.md).

View Source 
var EveryRun = []string{"./internal/ci", "./internal/docs"}

EveryRun are the class-test packages selectChange adds to every selection, whatever the change (the two `want` lines above, and why): the pull request's CI runs them whole on every change, so a read runs of them only the tests its diff reaches (pkg/cardcontract, the read's gate). TestSelectChangeIsTheTouchedPackagesTheirDependentsAndTheClassTestPackages holds the two equal.

View Source 
var FunctionalHeavy = []string{"./cmd/nova-swarm", "./cmd/nova-bus", "./internal/ci", "./pkg/atomicfile", "./pkg/ntable", "./pkg/swarm"}

FunctionalHeavy are the packages whose functional tests run longest on a shared runner (measured 28 to 68 s each in the merge-group runs of 2026-10-04, two of them landing on one leg ran past the two-minute cap): Functional deals them first, in this order, so no leg gets two while another is empty.

View Source 
var HeavyFirst = []string{"./cmd/nova-swarm", "./internal/ci"}

HeavyFirst are the measured expensive packages the fan-out deals first, so a four-leg PR gives each a separate leg. Ordinary round-robin put all four on the first leg; that job crossed its wall even when its test step passed.

View Source 
var LinuxOnly = []string{"./cmd/nova-swarm"}

LinuxOnly are the packages a pull request never deals to the macOS legs: their unit tests cost more than a macOS runner's two cores give in the two-minute cap (cmd/nova-swarm about 200 CPU-seconds), so those legs were cancelled at the cap in every pull-request run of 2026-10-04. Linux runs them in every pull request and in the merge group. cmd/nova-sprint, the other one, left for its own repository, nova-sprint (the split, v1.2.3). A push and the nightly run still deal them to macOS.

Functions

func DarwinOn 

func DarwinOn(event, target string) bool

DarwinOn reports whether the darwin legs are dealt for a workflow event aimed at target: always on schedule and workflow_dispatch; otherwise only when target, with any refs/heads/ prefix cut (a merge group's base_ref carries it, a pull request's base_ref and a push's ref name do not), is one of DarwinBranches.

func Deal 

func Deal(pkgs []string, heavy []string, shards, shard int) ([]string, error)

Deal returns shard's packages (1-based, of shards) from pkgs, the live packages in `go list` order (stable: go list sorts).

THE HEAVY PACKAGES FIRST, ONE PER SHARD. heavy names packages by a trailing path (`cmd/nova-swarm` matches `<module>/cmd/nova-swarm`); the k-th heavy name takes shard k (mod shards), so no two heavy packages share a shard while there are shards for them, and every other package goes round-robin in list order, continuing after the heavy ones: the first takes shard len(heavy) mod shards. A count-only deal kept the heaviest packages together and the leg holding them crossed the job cap. If two names match one package the later name wins. internal/ci: TestHostedDealSplitsTheHeavyPackages and TestCertificationRaceShardsPartitionTheLiveTree.

func DropDarwinOnly 

func DropDarwinOnly(pkgs []string) []string

DropDarwinOnly is pkgs without the packages that have no Linux leg: what a run with the darwin legs off selects.

func LiveTree 

func LiveTree(run Runner, root string) ([]string, error)

LiveTree lists every package of the module (`go list ./...`) that is not deprecated, as full import paths in `go list` order: the list the hosted deal and the race-dependency build read.

func MarshalLegs 

func MarshalLegs(v any) string

MarshalLegs is the matrix as compact JSON, in field order.

func ModulePath 

func ModulePath(root string) string

ModulePath reads the module path from root's go.mod: the import-path prefix of every package of this module. A package listed by import path is written ./<rest> everywhere else. A root with no go.mod, or one with no module line, has no prefix ("").

func OrderHeavyFirst 

func OrderHeavyFirst(pkgs []string) []string

OrderHeavyFirst puts the HeavyFirst packages before the rest, each part in its own order.

func RaceDeps 

func RaceDeps(run Runner, root string) ([]string, error)

RaceDeps lists every external package the live tree's tests import (the packages of other modules, never the standard library and never this module's own), sorted and unique: the packages to build under -race so one cache entry serves every shard. It is the same list on every shard, whatever the shard holds.

func RunnerShare 

func RunnerShare(cores int, runnersEnv string) (runners, share int)

RunnerShare is a leg's FAIR SHARE of the machine: the cores divided by the runners on the machine (runnersEnv, the text of $NOVA_RUNNERS_PER_MACHINE; DefaultRunners when it is empty, not a number, or not positive), never below 1 and never above ShareCeiling. Several runners share each machine, and go test defaults GOMAXPROCS to every core it can see, so concurrent legs would each ask for the whole machine and spend the difference context-switching; `go test -p` follows GOMAXPROCS, so this one number bounds both the package-level and the in-package parallelism. internal/ci: TestUnitLegTakesAtMostTwoCores.

func UnitMakeArgs 

func UnitMakeArgs(packages, wholeTreeSlowtests string, nightly bool) []string

UnitMakeArgs is the `make test` a unit shard runs, as argv.

The default: Go's test cache on (GOTEST_COUNT_FLAG empty), so a shard whose packages did not change since the last run on this machine is served from the cache instead of recompiled and re-executed, and test binaries linked without DWARF (-ldflags=-w: no dsymutil per binary on darwin, less to scan).

wholeTreeSlowtests is slowtests' flags for a run of the whole tree (a push or a manual run), which keeps its package budget with the SLEEPS ledger; when it is not empty it replaces the Makefile's default budgets. nightly is the nightly reference leg: -count=1, so every time is a run and none a cached pass, and the budgets ENFORCED (SLOWTESTS_ENFORCE=1). THE VERDICT IS THE SAME ON ANY MACHINE otherwise: the CI-SLOW lines and a CI-LOAD line print and exit 0, and a CI-SLEEPS line (a SLEEPS skip off the ledger) fails the leg (slowtests exits 1). The nightly leg on the Linux shards is the one place a CI-SLOW line fails the run, and this is the only place SLOWTESTS_ENFORCE=1 is spelled. internal/ci's class test of the nightly leg holds it.

func WriteUnitShim 

func WriteUnitShim(tmp string) (string, error)

WriteUnitShim writes the refusing redis-server under tmp and returns its path.

THE UNIT TIER STARTS NO SERVER (docs/TESTING.md: unit tests mock, functional tests carry the build tag). A directory FIRST on PATH holds a redis-server that prints why and exits UnitShimExit, so a redis-backed test left untagged fails closed under NOVA_CI=1 instead of starting a real server on a shared runner. The shim stands in for a binary, so it is a two-line /bin/sh file and not a Go program. internal/ci: TestUnitTierRefusesRedisServer.

Types

type DarwinSensitive 

type DarwinSensitive struct {
	All  bool
	Pkgs map[string]bool
}

DarwinSensitive is the set of packages that get a macOS leg on a pull request, or All.

A PULL REQUEST'S macOS LEGS ARE FOR THE CODE THAT IS DIFFERENT ON macOS. Every PR's waiting jobs needed a macOS runner while the Linux runners sat idle, yet most packages compile the same files on darwin as on linux, and most import nothing that does not: their macOS leg ran the Go source their Linux leg had already run. So on a pull_request a touched package gets a macOS leg only when it, or a package it (or its tests) imports from this module, compiles a different file set under GOOS=darwin; every other package runs on Linux only, the shape the merge_group run has. The whole tree still runs on macOS on every push to dev and in certification's `test` job, so nothing leaves the macOS evidence; it leaves the PR feedback path. If `go list` fails, every touched package keeps its macOS leg.

func DetectDarwinSensitive 

func DetectDarwinSensitive(run Runner, root string) (DarwinSensitive, bool, error)

DetectDarwinSensitive reads the file sets of ./cmd/... and ./internal/... under GOOS=linux and GOOS=darwin and the test-inclusive imports under GOOS=darwin. A `go list` that fails makes every package keep its macOS leg (the DarwinSensitive.All shape); the boolean is false then.

func (DarwinSensitive) Sorted 

func (d DarwinSensitive) Sorted() string

Sorted is the set, sorted, as the line printed for it: each name followed by one blank.

type Deprecated 

type Deprecated struct {
	// contains filtered or unexported fields
}

Deprecated is DeprecatedFile read: the packages that are deprecated and still in the tree because living tools import them. A path names that package and everything under it; a line `keep <path>` names one package under such a path that stays tested until it is lifted out into a shared module.

DEPRECATED PACKAGES ARE NEVER TESTED: tests do not run for deprecated tools and modules, builds do not stop for them, and CI is not bogged down by them. Every place that chooses the packages a run tests reads its list through Live: Select, the hosted deal and the race-dependency build. internal/ci's TestDeprecatedPackagesAreNeverSelected holds them.

A nil *Deprecated drops nothing (a checkout with no DeprecatedFile).

func LoadDeprecated 

func LoadDeprecated(root string) (*Deprecated, error)

LoadDeprecated reads DeprecatedFile under root. A missing file is not an error: it is a nil *Deprecated, which keeps every package.

func ParseDeprecated 

func ParseDeprecated(text, module string) *Deprecated

ParseDeprecated reads the text of DeprecatedFile: `#` starts a comment, blank lines are ignored, a `keep` line names a kept package and every other line names a dropped path. module is the module path, so a package listed by import path is read as the directory under it.

func (*Deprecated) Live 

func (d *Deprecated) Live(pkgs []string) []string

Live returns pkgs without the deprecated ones, each line unchanged and in order.

func (*Deprecated) LivePackage 

func (d *Deprecated) LivePackage(p string) bool

LivePackage reports whether the package at p is live: p is an import path under the module, or ./<dir>, or <dir>. A keep line wins; otherwise a package is dropped when it is a listed path or under one (a name that only starts the same is another package).

type FunctionalLeg 

type FunctionalLeg struct {
	Name     string `json:"name"`
	Packages string `json:"packages"`
}

FunctionalLeg is one entry of the functional tier's matrix.

func Functional 

func Functional(pkgs []string, g Groups) []FunctionalLeg

Functional deals the packages into FunctionalShards Linux legs like a pull request's unit legs (the darwin-only packages have no Linux leg). The functional job reads it on merge_group, schedule and workflow_dispatch only; each leg's `make test-functional` runs just the tests behind the functional tag. With no package it is one empty leg.

type Groups 

type Groups struct{ Linux, Mac string }

Groups are the labels of the two runner groups the fan-out deals onto. They are the workflow's own names (its runs-on labels), so the workflow passes them in and no name of a machine or a pool is written here.

type Leg 

type Leg struct {
	Name     string `json:"name"`
	Packages string `json:"packages"`
	OS       string `json:"os"`
	Arch     string `json:"arch"`
	Group    string `json:"group"`
}

Leg is one entry of the unit tier's matrix. The leg name carries its platform: two legs sharing one name is two checks GitHub reports under the same title, a green and a red that cannot be told apart on the PR.

func Fanout 

func Fanout(event string, pkgs []string, sens DarwinSensitive, g Groups, darwin bool) []Leg

Fanout deals pkgs (already heavy-first) onto the runner groups for event.

On schedule every package runs on the Linux shards, where the unit budgets are enforced, and the darwin-only packages have no leg that night (the push to dev and certification carry them). On merge_group, and on a pull_request for a package that does not need macOS, a package runs on Linux only. Everything else, a push to dev and a manual run, runs on both, dealt round-robin within each group so every package is covered exactly once per OS; the darwin-only packages go to the macOS group only. `sens` is read on a pull_request only. With darwin false (DarwinOn said no) every package rides the Linux shards and the darwin-only packages have no leg, as on schedule.

func NothingLeg 

func NothingLeg(g Groups) Leg

NothingLeg is the one leg of a change that touches no Go package: it prints "nothing to test for this change" and exits 0.

type ListError 

type ListError struct{ Text string }

ListError is a selection that failed rather than select nothing. Text is what to print on stderr.

func (*ListError) Error 

func (e *ListError) Error() string

type Options 

type Options struct {
	// Root is the repository root: the directory every command runs in and
	// where DeprecatedFile and the tracked files are read.
	Root string
	// All selects the whole tree. Otherwise Base is the commit the change is
	// diffed against (the event's own base: pull_request.base.sha or
	// merge_group.base_sha).
	All  bool
	Base string
	// WholeTreeOnError makes a failed `go list` select the whole tree, read
	// from the tracked files, with a warning (the landing must not stall on one
	// runner's cache). Otherwise a failed `go list` is a *ListError, cheap and
	// fast: the runner is broken, re-run it.
	WholeTreeOnError bool
}

Options is one selection: what to select against and what a failed `go list` does.

type Outcome 

type Outcome struct {
	Packages []string
	Warning  string
}

Outcome is a selection: the packages as ./<dir>, in `go list` order, and the warning to print on stderr when the whole tree stood in for a failed `go list`.

func Select 

func Select(run Runner, o Options) (Outcome, error)

Select prints the ./cmd, ./internal and ./tools Go packages a change touches, plus every in-repo package that imports one of them, or with All the whole tree. The diff is read against Options.Base. A go.mod or go.sum change puts every package in scope. A changed file that is not Go selects the packages whose tests or testdata name it or whose source embeds it (keyedPackages). Deprecated packages are never selected (Live).

NEVER SILENTLY NOTHING. A `go list` that fails on a runner must not read as "nothing to test": the selection would print an empty list, the caller would report no packages touched, and a green run would test nothing. Every `go list` here goes through goList: a non-zero exit, or "cannot" or "no such file" on its stderr, is a failure, and so is a selection of zero packages from a diff that touches Go files. What a failure does is the caller's choice (Options.WholeTreeOnError). internal/ci's TestSelectPackagesNeverSilentlySelectsNothing holds both.

type PerfRun 

type PerfRun struct {
	Package string
	Run     string
}

PerfRun is one live package that holds a perf-tagged test: the tests only the perf tag adds, as the regexp `go test -run` takes.

func PerfRuns 

func PerfRuns(run Runner, root string) (runs []PerfRun, notes []string, err error)

PerfRuns finds the perf-tagged tests. A perf-tagged package with no test behind the tag is reported in notes and skipped; the caller refuses a tree with no perf-tagged test at all, because then the perf job asserts nothing. Only a live package is considered.

type Result 

type Result struct {
	Stdout string
	Stderr string
	Code   int
}

Result is what one command printed and how it ended.

type Runner 

type Runner func(dir string, env []string, argv ...string) (Result, error)

Runner runs argv in dir with env (KEY=VALUE, added to the process's own) and returns its output and exit status. The error is only for a command that could not be started; a non-zero exit is a Result with that Code.

type Shards 

type Shards struct{ Linux, Mac int }

Shards is how many legs each runner group deals over.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.