Documentation
¶
Overview ¶
Package scan drains the whole official MCP registry and runs the same keyless probe.Engine that "akashi check" uses against every server, so a bulk census and a single check are always the same measurement over the same struct. It only adds what a census needs on top of that: bounded concurrency, a checkpointed JSONL so an interrupted run resumes without re-probing, and an aggregate summary. It never reimplements a probe or changes a verdict.
Index ¶
- Constants
- func NewGitHubBackoffTransport(base http.RoundTripper) http.RoundTripper
- func ReadRecords(censusDir string) (map[string]probe.Result, error)
- func RecordPaths(censusDir string) ([]string, error)
- func ShardFileName(name string) string
- type Breakdown
- type NameIssue
- type NameIssues
- type NamespaceDrift
- type Options
- type RecordsManifest
- type RecordsShard
- type ReprobeReport
- type Summary
Constants ¶
const ( // DefaultReprobeThreshold is how far a namespace's signal rate must move // between editions, in percentage points, to earn a second reading. DefaultReprobeThreshold = 10.0 // DefaultReprobeMinServers keeps small namespaces out: a 3-server // namespace swings 33 points when one server blinks, which is noise. DefaultReprobeMinServers = 25 // DefaultReprobeMaxServers bounds the second pass's wall time. A census // that cannot finish is worse than one with an undisclosed outage. DefaultReprobeMaxServers = 3000 )
Re-probe defaults. A census long enough for targets to change state records a transient outage as a property of the thing measured; the 2026-09-09 census ran 16h59m and one operator's endpoints flipped 1,277 stateless-true readings to 270 while its health verdicts barely moved. Nothing in the data said so. This pass is what says so.
const DefaultConcurrency = 8
DefaultConcurrency is a modest worker pool size: polite to the registry, package indexes, and remote endpoints a census touches, while still finishing a full drain of the registry in a practical amount of time.
const DefaultTimeout = 60 * time.Second
DefaultTimeout bounds one server's full probe set. It mirrors the default "akashi check" uses for the same overall budget.
const RecordShards = 64
RecordShards is how many shard files a census is split across. All records for one namespace land in the same shard, so a consumer wanting one operator fetches one small file instead of the whole census.
const RecordsDir = "records"
RecordsDir is the sharded records directory written inside --out. It replaces the single RecordsFile, which GitHub warns on past 50MiB and refuses past 100MiB: one census crossed 51.5MiB at 31,538 servers, and the hard limit lands near 47,700 at the same bytes-per-server.
const RecordsFile = "records.jsonl"
RecordsFile is the JSONL filename written inside --out: one probe.Result per line, the exact fields "akashi check <server> --json" prints.
const RecordsManifestFile = "manifest.json"
RecordsManifestFile names the shard listing written inside RecordsDir.
const ReprobeFile = "reprobe.jsonl"
ReprobeFile is the JSONL filename for second-pass readings written inside --out. It is a SEPARATE file on purpose: records.jsonl is the census as observed and is never edited after the fact, so a re-probe is additional evidence recorded alongside it, never a correction applied to it.
const StartedFile = "scan.started"
StartedFile records when the census FIRST began, inside --out. A resumed run reports the window of the whole census, not of its final leg: without this, a run interrupted at 99% and resumed reports a few minutes for a population that took hours, which is a false provenance claim in a dataset whose value is that its provenance is checkable. probe.Result.CheckedAt is a date, not a timestamp, so the start cannot be recovered from the records.
const SummaryFile = "summary.json"
SummaryFile is the aggregate-report filename written inside --out.
Variables ¶
This section is empty.
Functions ¶
func NewGitHubBackoffTransport ¶
func NewGitHubBackoffTransport(base http.RoundTripper) http.RoundTripper
NewGitHubBackoffTransport wraps base with GitHub rate-limit backoff. A nil base falls back to http.DefaultTransport.
func ReadRecords ¶ added in v0.5.0
ReadRecords reads every record of a census directory, keyed by server name, across whichever layout that census uses.
func RecordPaths ¶ added in v0.5.0
RecordPaths returns every records file for a census directory, newest layout first: the shard files under RecordsDir when that exists, otherwise the legacy single RecordsFile. Censuses published before sharding keep working with no migration.
func ShardFileName ¶ added in v0.5.0
ShardFileName is the shard filename a server's record belongs in, relative to RecordsDir. Exported so a consumer can resolve a name to one file.
Types ¶
type Breakdown ¶
type Breakdown struct {
Total int `json:"total"`
Counts map[probe.Verdict]int `json:"counts"`
Rates map[probe.Verdict]float64 `json:"rates"`
}
Breakdown is a verdict census over some population: the whole run, or one segment of it (see Summary.Segments). Rates are Counts[v] / Total; Total is the denominator.
type NameIssue ¶
NameIssue is one validation category over the drained server names: how many names fell into it, and a bounded sample of the offending names for spot-checking.
type NameIssues ¶
type NameIssues struct {
// BadCharset counts names containing a character outside
// nameCharsetRe (for example an uppercase letter or a space).
BadCharset NameIssue `json:"badCharset"`
// BadShape counts names that are not exactly one "/" splitting a
// non-empty namespace segment (containing at least one ".") from a
// non-empty local segment, with neither segment starting with ".".
BadShape NameIssue `json:"badShape"`
// CaseCollisions counts names that are distinct but identical to
// another name in this run once lowercased, which would collide as
// the same file. An exact (case-sensitive) repeat of the same name is
// not a collision by itself; registry.Drain already dedupes those
// before validateNames ever sees the list.
CaseCollisions NameIssue `json:"caseCollisions"`
}
NameIssues summarizes registry names that would not survive the planned two-segment <namespace>/<name> page routing, or that would collide on a case-insensitive filesystem once rendered to that path. It is purely informational: see validateNames, which never fails a scan over it.
type NamespaceDrift ¶ added in v0.5.0
type NamespaceDrift struct {
Namespace string `json:"namespace"`
Servers int `json:"servers"`
Signal string `json:"signal"` // the signal that moved furthest
PreviousPct float64 `json:"previousPct"`
CurrentPct float64 `json:"currentPct"`
DeltaPoints float64 `json:"deltaPoints"`
// The second reading. Reprobed is how many were actually re-probed;
// Agreed/Changed count those whose signal read the same or differently the
// second time, and NoAnswer those the re-probe could not reach at all.
// NoAnswer is absence of evidence and is never counted as either.
Reprobed int `json:"reprobed"`
Agreed int `json:"agreed"`
Changed int `json:"changed"`
NoAnswer int `json:"noAnswer"`
}
NamespaceDrift is one namespace whose aggregate signals moved enough between editions to earn a second reading, plus what that reading found.
type Options ¶
type Options struct {
// Out is the output directory. Required. RecordsFile and SummaryFile are
// written inside it.
Out string
// Limit caps how many servers are drained from the registry. 0 means the
// whole registry.
Limit int
// Concurrency is the worker pool size. <= 0 falls back to
// DefaultConcurrency.
Concurrency int
// Timeout bounds one server's full probe set. <= 0 falls back to
// DefaultTimeout.
Timeout time.Duration
// Progress receives one line per completed server ("1234/13886 name
// verdict"), plus a startup line noting how many were already recorded.
// A nil Progress disables progress reporting.
Progress io.Writer
// Compare is a previous census directory (or its records.jsonl). Given
// one, a second pass re-probes every namespace whose aggregate signals
// moved against that edition and records what it finds, so a transient
// outage inside a long scan window shows up as a dated observation about
// one operator instead of a trend in a published rate. Empty disables it.
Compare string
// ReprobeThreshold, ReprobeMinServers and ReprobeMaxServers tune that
// pass; each falls back to its DefaultReprobe* constant when <= 0.
ReprobeThreshold float64
ReprobeMinServers int
ReprobeMaxServers int
}
Options configures one census run.
type RecordsManifest ¶ added in v0.5.0
type RecordsManifest struct {
SchemaVersion string `json:"schemaVersion"`
Algorithm string `json:"algorithm"`
Shards int `json:"shards"`
Total int `json:"total"`
Files []RecordsShard `json:"files"`
Namespaces map[string]string `json:"namespaces"`
}
RecordsManifest is the shard listing published alongside the shards, so a consumer can enumerate the census without a directory listing and resolve one namespace to one file without reimplementing the hash.
type RecordsShard ¶ added in v0.5.0
RecordsShard is one shard's entry in the manifest.
type ReprobeReport ¶ added in v0.5.0
type ReprobeReport struct {
ComparedTo string `json:"comparedTo"`
ThresholdPoints float64 `json:"thresholdPoints"`
MinServers int `json:"minServers"`
MaxServers int `json:"maxServers"`
StartedAt string `json:"startedAt"`
FinishedAt string `json:"finishedAt"`
Candidates int `json:"candidates"`
// Truncated reports that MaxServers stopped the pass before every
// candidate namespace was re-probed. Recorded rather than silently
// dropped: a partial second pass that reads as a complete one would be a
// false all-clear on exactly the question this pass exists to answer.
Truncated bool `json:"truncated"`
Namespaces []NamespaceDrift `json:"namespaces"`
}
ReprobeReport records the whole second pass. It is written into summary.json so a reader who never opens reprobe.jsonl still learns that a rate in this census has a contested namespace inside it.
type Summary ¶
type Summary struct {
RegistryBaseURL string `json:"registryBaseUrl"`
AkashiVersion string `json:"akashiVersion"`
Concurrency int `json:"concurrency"`
Limit int `json:"limit"` // 0 means the whole registry
StartedAt string `json:"startedAt"`
FinishedAt string `json:"finishedAt"`
// Overall is the verdict census across every server this run targeted.
Overall Breakdown `json:"overall"`
// Reprobe is the second-pass report, present only when Options.Compare
// named a previous edition. Its readings live in ReprobeFile; records.jsonl
// stays the census exactly as first observed.
Reprobe *ReprobeReport `json:"reprobe,omitempty"`
// Segments key the same kind of census by a defining trait. "remote"
// (servers that declare at least one hosted remote endpoint) is the
// headline: a reachable, conformant remote is the strongest keyless
// liveness proof a census can gather, since its domain is verified simply
// by being probed live over HTTPS.
Segments map[string]Breakdown `json:"segments"`
// NameIssues flags registry names that would not survive the planned
// <namespace>/<name> per-server page routing (see validateNames). It is
// a heads-up for the index build, not a scan failure: every server is
// still probed and recorded regardless of what this reports.
NameIssues NameIssues `json:"nameIssues"`
}
Summary is the aggregate report written to SummaryFile alongside RecordsFile. It carries enough reproducibility parameters that a "State of MCP" index built from it can cite exactly how the numbers were produced.
func Run ¶
func Run(ctx context.Context, client *registry.Client, eng *probe.Engine, opts Options) (Summary, error)
Run drains the registry (respecting opts.Limit), probes every server that is not already recorded in an existing RecordsFile under opts.Out, appends each freshly probed result as it completes, and writes SummaryFile once the whole targeted population has a record.
client and eng are pre-configured by the caller (base URL, GitHub token, HTTP transport); Run only orchestrates draining, concurrency, and checkpointing around them. eng is shared across the whole worker pool: its HTTP client and schema cache are safe for concurrent use.