README
¶
codesearch
Indexed code search for AgentKit agents. It adds a code_search tool backed by
an in-process zoekt trigram index, so
an agent working on a large repository gets fast, ranked results instead of a
scan of every file for every query.
Why use it
The built-in search_files tool runs a regular expression over every file in
the workspace. That is exact and needs no setup, but on repositories with tens
of thousands of files it is slow, and its capped results often miss what the
agent was looking for, so the agent searches again. code_search answers from
an index:
- Ranked results: the best matches come first, grouped by file.
- A query language: substrings, regular expressions, and the filters
sym:(declarations),file:,lang:andcase:, combined withorand negation. - Symbol lookup: the index can also answer the built-in
find_symboltool. - Kept up to date: edits made through the tools reach the index automatically.
It is a separate Go module, so applications that do not use it do not link zoekt and its dependencies.
Install
go get github.com/agent-fox-dev/agentkit-go/codesearch
Use it
Build the index for a workspace and hand it to tools.For, which then adds
code_search after the built-in tools:
ws, err := workspace.Open(".", workspace.Options{})
if err != nil {
return err
}
opts := tools.Options{}
idx, err := codesearch.New(ws, codesearch.Options{})
switch {
case errors.Is(err, codesearch.ErrUnsupported):
// Windows: zoekt does not build there; carry on without the index.
case err != nil:
return err
default:
defer idx.Close()
opts.Index = idx
}
ts, err := tools.For(ws, opts)
The index uses the workspace's ignore rules, is built on the first query, and
is bounded by file count, total size and build time
(options).
If you restrict the agent's tools with a policy, add "code_search" to the
allowed names.
The index also works without an agent: call the code_search tool directly,
and call Invalidate when files change outside the tools.
code_search or search_files?
search_files |
code_search |
|
|---|---|---|
| Query | RE2 regular expression | zoekt query: substring, regex, sym:, file:, lang:, case: |
| Results | Every match up to the cap, in file order | A ranked subset, best first |
| Setup | None | An index build on first use |
| Files skipped | Binary files | Binary files, files over 1 MiB, and generated-looking files (lock files, minified bundles) |
Keep both: search_files for exhaustive answers on small trees, code_search
for finding things in large ones.
Platforms
Linux and macOS. On Windows, codesearch.New returns ErrUnsupported and a
nil index, and the code above runs unchanged without code search.
Further reading
examples/indexedsearchwalks through the index on a sample project, from the first query to an agent using it.- User guide: indexed code search.
docs/codesearch.md: why the module exists, how it is built on zoekt, and the dependencies it brings in.
Documentation
¶
Index ¶
- Variables
- func New(ws *workspace.Workspace, opts Options) (tools.Index, error)
- type BuildStatsResult
- type Index
- func (idx *Index) Build(ctx context.Context) error
- func (idx *Index) BuildCount() int
- func (idx *Index) BuildStats() BuildStatsResult
- func (idx *Index) Close() error
- func (idx *Index) IndexedFiles() map[string]bool
- func (idx *Index) Invalidate(rel string)
- func (idx *Index) OverlayBuildCount() int
- func (idx *Index) RevalCount() int
- func (idx *Index) RunDir() string
- func (idx *Index) Symbols(ctx context.Context, q tools.SymbolQuery) (tools.SymbolAnswer, bool, error)
- func (idx *Index) Tools() []core.Tool
- type Options
Constants ¶
This section is empty.
Variables ¶
var ErrUnsupported = errors.New("codesearch: unsupported platform")
ErrUnsupported is returned by New on platforms where zoekt cannot build or run (currently windows/amd64). An embedder should fall back to Options.Index == nil when errors.Is(err, ErrUnsupported) is true.
Functions ¶
func New ¶
New creates a codesearch index for the given workspace. It returns a tools.Index without starting a goroutine, running a process, walking the workspace or touching the disk. It returns an error if ws is nil.
On error the result is a nil interface, never a nil *Index inside one, so an embedder that stores it in Options.Index without checking err gets no index instead of one whose methods run on a nil receiver. An embedder that wants the build counters and statistics asserts the result to *Index.
Types ¶
type BuildStatsResult ¶
type BuildStatsResult struct {
BinarySkipped int
OversizedSkipped int
// TooManyTrigramsSkipped counts files zoekt would store as "not
// indexed" for having more distinct trigrams than it indexes per file
// (minified bundles, lock files, source maps).
TooManyTrigramsSkipped int
// TooSmallSkipped counts files under three bytes, which hold no trigram.
TooSmallSkipped int
FilesIndexed int
}
BuildStatsResult holds statistics from a build. It is declared without a build constraint: the Windows stub returns the zero value.
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index is the codesearch index. It implements tools.Index.
func (*Index) Build ¶
Build triggers the index build. It walks the workspace, collects files, outlines them, and builds a zoekt index. Three bounds apply: MaxFiles, MaxBytes and MaxBuildTime. When one stops the build, the index keeps what it has and marks itself partial. A partial index is not retried by later calls. A cancelled call context discards the partial build and leaves the next call free to build again.
A call arriving while another build is running waits for it, abandonably: when ctx ends first it returns an "aborted" error without waiting for the build to finish. The lock is not held during the build.
func (*Index) BuildCount ¶
BuildCount returns the number of times the index has been built.
func (*Index) BuildStats ¶
func (idx *Index) BuildStats() BuildStatsResult
BuildStats returns statistics from the last build.
func (*Index) Close ¶
Close releases resources. It waits for a build and in-flight queries to finish, then releases the zoekt searcher and deletes the run directory. It is idempotent: every call after the first returns nil immediately.
func (*Index) IndexedFiles ¶
IndexedFiles returns the set of slash-separated workspace-relative paths that are in the index.
func (*Index) Invalidate ¶
Invalidate implements tools.Index. It never blocks on a build or query in progress for longer than it takes to set a flag, never returns an error and never panics on a closed index.
func (*Index) OverlayBuildCount ¶
OverlayBuildCount returns the number of times the overlay shard has been built.
func (*Index) RevalCount ¶
RevalCount returns the number of revalidation walks performed.
func (*Index) Symbols ¶
func (idx *Index) Symbols(ctx context.Context, q tools.SymbolQuery) (tools.SymbolAnswer, bool, error)
Symbols implements tools.Index. It answers from the per-file outline.File data collected by the build, using the matching rules of 02_symbol_navigation_tools Requirement 2 (smart-case prefix, exact, qualified names, kind, path). It returns ok=true only when the index has been built by an earlier code_search, is not partial and not closed, and the matches number at most 1000. It never starts a build.
type Options ¶
type Options struct {
// MaxFiles is the file-count bound for a build. Zero or negative means
// 100 000.
MaxFiles int
// MaxBytes is the byte-count bound for indexed content. Zero or negative
// means 1 GiB.
MaxBytes int64
// MaxBuildTime is the wall-time bound for a build. Zero or negative means
// 60 s. When it fires, the files already walked are still added, without
// symbols, for a grace window of a quarter of it, so a build is never
// left empty by a slow walk; it can therefore run up to 1.25 times this.
MaxBuildTime time.Duration
// TempDir is the directory for shard files. Empty means os.TempDir().
TempDir string
}
Options configures the codesearch index.
Source Files