Documentation
¶
Overview ¶
Package corpus decides what the bot is allowed to know.
The corpus is defined by a predicate rather than a curated list: a repository qualifies by being a public, documented member of the phpboyscout group. That makes the corpus grow with the toolkit instead of with a config file.
It is also the project's security boundary, and since 2026-07-29 the weight of that boundary rests on the VISIBILITY clause rather than on the exclusion list. Every excluded path is now also private, so each would be refused twice. That is deliberate defence in depth — an exclusion still holds if a repository is made public by accident, which is the one failure the visibility clause cannot catch — but it means the live visibility check is the control actually doing the work, not a formality behind a deny list.
Exclusions are nonetheless evaluated first, and are absolute: a rejection should be attributed to the strongest reason available, and an excluded repository must never be read from even to discover it would have failed anyway.
Index ¶
- Constants
- Variables
- func BlobURL(repoURL, commitSHA, repoPath string, line int) string
- func IsGenerated(body string) bool
- func SiteURL(base, repoPath string) (string, bool)
- func Slug(heading string) string
- func SplitIdentifier(name string) string
- type Candidate
- type Chunk
- type Clause
- type Discoverer
- type Document
- type Kind
- type Source
- type SplitOption
- type Verdict
Constants ¶
const (
// DefaultMaxChunkBytes bounds one chunk before it is split.
DefaultMaxChunkBytes = 4000
)
Chunking defaults.
The ceiling is a byte budget rather than a token count: tokenisation depends on a model that is not chosen yet, and a byte bound is stable, cheap and close enough for a retrieval unit.
const DocsiteMarker = "zensical.toml"
DocsiteMarker is the file whose presence marks a repository as documented.
const Group = "phpboyscout"
Group is the only namespace whose repositories may be indexed.
Variables ¶
var ErrCapabilityMissing = errors.New("corpus: forge provider cannot enumerate the namespace")
ErrCapabilityMissing is returned when the configured forge provider cannot do what discovery needs.
Refused at construction rather than tolerated per source. All three capabilities are optional in the forge contract and a provider implementing none is conformant — but for this corpus their absence is not a graceful degradation. Without Contents every repository would be rejected for having no zensical.toml, which reads as a fact about the repository when it is really a fact about the provider.
var ErrGenerated = errors.New("corpus: generated file")
ErrGenerated is returned for a file carrying the generated-code marker.
Functions ¶
func BlobURL ¶
BlobURL builds a link to a file at the commit it was indexed from, with an optional line anchor.
Pinned to the commit rather than a branch, so a citation keeps showing what was actually indexed after the file moves on. A citation that silently follows a branch can end up pointing at text that contradicts the answer it supports.
func IsGenerated ¶
IsGenerated reports whether a file carries the generated-code marker.
Checked in the body because the marker is a comment, not a filename convention. Generated code is boilerplate that answers nothing and would crowd out the documentation it sits beside.
func SiteURL ¶
SiteURL maps a documentation path to the page it is published at, reporting false for anything that is not published.
The mapping is deterministic because the toolchain is ours: docs/ is the site root, index.md is a directory's own page, and every other page is a directory named after the file. Verified against the real sites — 128 anchors across two repositories resolved.
func Slug ¶
Slug renders a heading as the URL fragment the docs toolchain generates.
Derived from real generated HTML rather than assumed. Punctuation is *removed* rather than replaced: "ReleaseSource.Private" becomes "releasesourceprivate", not "releasesource-private", and getting that backwards would send every such citation to a fragment that does not exist — which is worse than no citation, because the answer still looks sourced.
func SplitIdentifier ¶
SplitIdentifier renders an identifier as the words somebody would type.
The tokenizer treats "defaultTakeCount" as one token, so a question asking about the "default take count" cannot reach it — and the question is always the natural phrasing, never the identifier. Splitting on case and underscore closes that gap, which is the difference between a symbol being findable and being merely present.
Runs of capitals stay together, so "HTTPClient" becomes "http client" rather than a column of single letters.
Types ¶
type Candidate ¶
type Candidate struct {
// Path is the canonical namespace path the API returned, such as
// "phpboyscout/go/chat-platform". It is the identity a source is keyed by,
// because directory names do not reliably match project names.
Path string
// Visibility is the API's own report. Anything but public is refused.
//
// Typed rather than a bare string because callers make a security decision
// on it, and a typo in a string comparison there is a disclosure rather than
// a bug. Allowlisted against forge.VisibilityPublic, never denylisted
// against private: `!= private` admits both unknown and internal.
Visibility forge.Visibility
// HasZensical records the docsite toolchain marker.
HasZensical bool
// PagesURL is the published documentation site, empty when there is none.
PagesURL string
}
Candidate is a repository being considered, described only by what the API reports about it.
Every field must come from the forge, never from a local checkout or a directory name. A clone cannot answer whether a repository is still public, still in the group, or even which project it is — several working directories in this estate resolve to a repository with a different name.
type Chunk ¶
type Chunk struct {
// Heading is the section's own heading, empty for a document's intro.
Heading string
// HeadingPath is the full ancestry, e.g.
// "Author a provider › 2. Construction › Why it matters".
HeadingPath string
// Anchor is the URL fragment for this section, without '#'. Empty for the
// intro, which cites the page itself.
Anchor string
Body string
// Line is the source line a code chunk starts at, so its citation anchors
// on the declaration rather than the top of a long file. Zero for prose.
Line int
// SearchTerms are additional words indexed but never shown, used to make a
// chunk reachable by wording it does not literally contain. Code carries the
// split form of its identifier here; prose carries nothing, being words
// already.
SearchTerms string
Ordinal int
}
Chunk is one indexed passage: a heading section, or part of an oversized one.
type Clause ¶
type Clause int
Clause identifies which part of the predicate decided a verdict, so a source listing can explain itself. A predicate nobody can inspect is a curated list with extra steps.
const ( // ClauseNone means nothing rejected the candidate. ClauseNone Clause = iota // ClauseExcluded means the path is on the permanent exclusion list. ClauseExcluded // ClauseNamespace means the path is outside the phpboyscout group. ClauseNamespace // ClauseVisibility means the API did not report the source as public. ClauseVisibility // ClauseNoDocsite means the repository carries no zensical.toml. ClauseNoDocsite // ClauseNoPages means no publicly reachable documentation site. ClauseNoPages // ClauseArchived means the forge reports the repository as retired. ClauseArchived // ClauseUnresolved means a fact the predicate needs could not be determined. // It is not a rejection on the merits: the source may well qualify, and the // next run will ask again. ClauseUnresolved )
type Discoverer ¶
type Discoverer struct {
// contains filtered or unexported fields
}
Discoverer enumerates the group and resolves what the predicate needs.
It holds the three optional forge capabilities separately rather than a single Provider, so the requirement is stated once at construction instead of being rediscovered at every call site.
func NewDiscoverer ¶
func NewDiscoverer(p forge.Provider) (*Discoverer, error)
NewDiscoverer discovers the capabilities discovery needs on a provider.
Uses forge.As rather than a bare type assertion, because a decorator forwarding only the required Provider methods strips the optional interfaces — and since absence is meant to be a graceful fallback, a stripped Repositories would turn enumeration into a silent no-op rather than a visible failure.
func (*Discoverer) Discover ¶
func (d *Discoverer) Discover(ctx context.Context) ([]Source, error)
Discover enumerates the group and applies the predicate to everything in it.
Every candidate is returned, rejected ones included, because a source listing has to explain an absence. A repository that simply vanished from the output would send an operator hunting for a bug in discovery when the answer is that the predicate did its job.
type Document ¶
Document is a markdown file split for indexing.
func Split ¶
func Split(markdown string, opts ...SplitOption) Document
Split turns a markdown document into indexable chunks.
func SplitGo ¶
SplitGo turns Go source into indexable chunks.
Doc comments and exported declarations only. Doc comments are prose written to be read, so BM25 scores them like the documentation they are; raw bodies are imports, struct tags and boilerplate, which is a large volume of low-signal tokens competing with real documentation for the same ranking.
Unexported symbols are left out for the same reason a support answer would leave them out: somebody asking a question is asking about the surface they can use.
type Kind ¶
type Kind int
Kind is what a file is, which decides both whether it is indexed and how it is cited.
const ( // KindNone means the file is not indexed. KindNone Kind = iota // KindDoc is documentation, cited by its published page. KindDoc // KindReadme is a repository README, cited by blob. KindReadme // KindChangelog is a changelog, cited by blob. KindChangelog // KindCode is source, cited by blob with a line anchor. KindCode )
func Classify ¶
Classify reports what a repository path is, and therefore whether it is indexed at all.
The exclusions here are about signal rather than safety — the security boundary is the corpus predicate. What is kept out is what would crowd the index without answering anything: built output duplicating its own source, tests, generated boilerplate, and our own working notes.
type Source ¶
type Source struct {
// Repository is the forge's own report, unmodified.
Repository forge.Repository
// SiteURL is the published documentation site, resolved through the Sites
// capability. Empty when there is none or it could not be determined — the
// verdict says which.
SiteURL string
// Verdict is why this source is or is not indexed.
Verdict Verdict
}
Source is a repository, the facts discovery resolved about it, and the verdict the predicate reached.
func Qualifying ¶
Qualifying returns only the sources that may be indexed.
The exclusion list is re-applied here rather than trusted from the verdict. Everything upstream is expected to have got this right, and if it did this check never fires — but "expected to have got it right" is not what the exclusion list is for. It is the only thing standing between a bot answering in a public channel and the infrastructure repositories, so it is enforced wherever indexing can begin, not only where verdicts are made.
type SplitOption ¶
type SplitOption func(*splitConfig)
SplitOption configures splitting.
func WithMaxChunkBytes ¶
func WithMaxChunkBytes(n int) SplitOption
WithMaxChunkBytes overrides the size at which a section is split.
type Verdict ¶
Verdict is the outcome of evaluating a candidate, with the reasoning kept.
func Evaluate ¶
Evaluate applies the corpus predicate.
The order is deliberate and load-bearing. Exclusions run before every other check so that an excluded source cannot be reinstated by satisfying the predicate more thoroughly, and so a rejection is attributed to the exclusion even when the source would have failed anyway — which is what makes the exclusion visible in a listing rather than incidental.
func Screen ¶
Screen applies the clauses answerable from the enumeration payload alone, reporting whether they already decided the verdict.
It exists so discovery does not spend two API calls establishing a docsite marker and a site URL for a repository the predicate has already rejected — and, more importantly, so an excluded repository is never read from at all. The clause order is Evaluate's, because it is the same code: splitting the two would be splitting the security boundary.
Source Files