chunk

Overview ¶

This file chunks shell scripts. Build tasks, deploy scripts, and CI helpers accumulate operational knowledge that exists nowhere else — the comment above a function is often the only documentation an operational procedure has.

Bash gets its own walker rather than a table entry (see languages.go) because of what it *excludes*. A script is mostly top-level commands, not declarations, and indexing those would bury the useful chunks. Only functions and documented variables are emitted.

Package chunk splits a markdown file into retrievable units for embedding, via a heading-tree walk that emits /path·/narrow·/full variants with slugified keys. Notable design choices:

• API returns structured []Chunk instead of a map[string]string, so the index layer stores key/heading/variant/text without re-parsing a composite key.
• Chunk text is run through Simplify (simplify.go) before it's stored: HTML comments and generated TOC blocks are dropped and links (`[](…)` and Obsidian `[[…]]`) are reduced to their visible text, so embeddings and snippets aren't padded with URLs, anchors, and markers. Ordinary markdown is kept — the model reads it fine.
• A synthesized `title` chunk is emitted from frontmatter title/tags, so a note that carries its name only in frontmatter (no body H1) stays findable. Frontmatter is otherwise not embedded.

This file extends the chunker to Go source. Where Document walks a markdown heading tree, GoSource walks a Go file's syntax tree and emits one chunk per symbol — package doc, type, func, method, and documented const/var block. The retrieval surface is the *doc comment + signature*, not the function body: bodies are implementation, the way markdown frontmatter is attributes (see Document) — high-signal names and prose stay, the mechanics don't dilute the index.

Extraction is tree-sitter, not go/parser + go/doc. The reason is uniformity, not capability: every other language here goes through tree-sitter, so Go on go/parser meant two parser stacks doing one job, and every shared concern — signature rendering, line numbering, breadcrumbs — needed writing twice. See walker.go for the machinery this shares with the rest.

Error recovery is *not* the reason. go/parser recovers from a malformed declaration about as well as tree-sitter does, and both keep the declarations that parsed. The real cost of the swap is a cgo grammar in place of the standard library, and re-deriving go/doc's doc↔symbol association by hand — the latter is cheap, because a Go doc comment is just the unbroken comment group directly above a declaration.

This file chunks HCL — Terraform, Terragrunt, Packer, and Nomad configuration. Infrastructure is where "why is it configured this way" questions are hardest to answer by grep, because the answer is usually in a comment above a block whose name you would have to already know.

HCL gets its own walker rather than a table entry (see languages.go) because a block is named by its labels rather than by a field: `resource "aws_s3_bucket" "logs"` is one node whose identity is spread across three children. Joining them is what makes a chunk key match how an engineer refers to the resource.

This file chunks the languages whose declarations all have the same shape: a named node, optionally holding members, documented by the comment group directly above it. Java, C#, Rust, C, C++, Ruby, PHP, Scala, and Lua differ in node names and almost nothing else, so each is a table entry rather than a file.

Go, TypeScript, Python, Bash, HCL, Protobuf, and YAML keep hand-written walkers. That is not an oversight — each has a quirk the table cannot express without growing a flag that only one caller sets: Go qualifies methods by a receiver buried under a pointer and type arguments, TypeScript has to recognise CommonJS export assignments and function-valued consts, and Python keeps its documentation *inside* the body rather than above the declaration. A table is the right tool for the regular cases and the wrong one for the irregular, so the line is drawn there deliberately.

This file chunks Protocol Buffers. A `.proto` file is the closest thing a service-oriented codebase has to an API reference, and the comments above a message, an rpc, or an enum are that reference's prose — so they are worth indexing even though a proto declares no behaviour.

Protobuf gets its own walker rather than a table entry (see languages.go) because it names things by a dedicated child node — `message_name`, `service_name`, `rpc_name` — instead of by a `name` field, and because an rpc is worth qualifying by the service that declares it.

This file chunks Python. Python gets a hand-written walker rather than a table entry (see languages.go) for one reason: its documentation lives *inside* the declaration, not above it. A docstring is the first statement of a module, class, or function body, which inverts the association every other language here uses and which the shared docScanner implements.

Both conventions are honoured, with the docstring winning. A `#` comment above a def is common in real code and is worth indexing, but where a docstring exists it is the author's actual documentation — PEP 257 says so, and every Python tool reads it that way.

This file is the one place a file extension is mapped to a language. Both the indexer (which needs the chunker) and search (which needs the language name for `--lang`) read it, so the set of indexed extensions and the set of filterable languages cannot drift apart — a new language becomes searchable by name the moment it becomes indexable.

This file extends the chunker to TypeScript, the same way gosource.go extends it to Go: one chunk per documented symbol — function, class (+ its methods), interface, type alias, enum, and documented top-level const/var. The retrieval surface is the *JSDoc comment + signature*, not the body: function/method bodies are stripped exactly as Go bodies are, so high-signal names, types, and prose stay while the mechanics don't dilute the index.

TypeScript has no stdlib parser, so we lean on tree-sitter (already a viable cgo dependency, since onnxruntime_go forces CGO_ENABLED=1). The .ts grammar and the .tsx grammar differ only in how they resolve the `<...>` ambiguity (type assertions/generics vs JSX), so each is wrapped once and dispatched by extension in the indexer.

This file holds the machinery every tree-sitter chunker shares, so adding a language is a walk plus an emit rather than a parser.

The split is deliberate: everything here is language-independent — loading a grammar once, associating a comment group with the declaration below it, assembling a chunk's breadcrumb-led text, and keeping keys unique. What stays in each language's file is the part that genuinely differs: which node kinds are declarations, where a name lives, and what counts as a signature.

This is not a configuration DSL. A per-language spec table was the other option and it collapses the moment a language has a quirk — Python keeps its doc *inside* the body, HCL names a block with labels rather than a field, Bash has no doc convention at all. A shared toolkit absorbs those; a schema grows a flag for each one.

This file chunks YAML. YAML needed a granularity decision the other languages did not, because it has no declarations — only nesting — and the obvious readings are both wrong.

Chunking every key floods the index: a Kubernetes manifest has hundreds, and `spec.template.spec.containers.0.image` retrieves nothing a person would search for. Chunking whole files is the opposite failure — a multi-document manifest becomes one blob whose embedding averages a Deployment, a Service, and a ConfigMap into nothing in particular.

So the unit is the **document**, plus its **top-level keys**. A document is what a YAML file actually declares (one object, one release, one workflow), and top-level keys are the sections a person names out loud — `metadata`, `spec`, `data`, `jobs`. Both are bounded, and both match how the files are discussed.

A document that declares `kind` and `metadata.name` is identified by them, because "the api-gateway Deployment" is what someone searches for, not "the third document in deployment.yaml".