Documentation
¶
Overview ¶
@index Shared search backend interface and errors for SQLite FTS5 and PostgreSQL tsvector implementations.
The raw SQL in this package is bounded and deliberate: full-text operators, DDL and writes against the FTS5 virtual tables, and the schema introspection GORM's migrator cannot do. guide/development.md §Raw SQL states the rule and what falls under it; migrator_limits_test.go holds the evidence for the introspection part. Anything GORM's model layer can express belongs there instead, here as much as anywhere else.
@index PostgreSQL tsvector + GIN based full-text search backend implementation (including schema, triggers, and queries).
@index Bound SQL search persistence adapters.
@index Query sanitizers for backend-specific full-text search syntax.
@index SQLite FTS5 virtual table-based full-text search backend implementation (including migration, legacy upgrades, and incremental re-indexing).
@index Search-document refresh and FTS content generation.
Index ¶
- Variables
- func NewIngestUnitOfWork(db *gorm.DB, backend Backend, logger *slog.Logger) ingest.UnitOfWork
- func RefreshSearchDocuments(ctx context.Context, db *gorm.DB) (int, error)
- func RefreshSearchDocumentsFor(ctx context.Context, db *gorm.DB, nodeIDs []uint) (int, error)
- func SanitizeFTS5(query string) string
- func SanitizeIntentFTS5(query string) string
- func SanitizePostgresIntentTSQuery(query string) string
- func SanitizePostgresTSQuery(query string) string
- type Backend
- type PostgresBackend
- func (p *PostgresBackend) MatchIntent(ctx context.Context, db *gorm.DB, query string, maxCandidates int) ([]intentrank.Doc, error)
- func (p *PostgresBackend) Migrate(db *gorm.DB) error
- func (p *PostgresBackend) PurgeNamespace(ctx context.Context, db *gorm.DB) error
- func (p *PostgresBackend) Query(ctx context.Context, db *gorm.DB, query string, limit int) ([]graph.Node, error)
- func (p *PostgresBackend) Rebuild(ctx context.Context, db *gorm.DB) error
- func (p *PostgresBackend) RebuildNodes(ctx context.Context, db *gorm.DB, nodeIDs []uint) error
- type Reader
- type SQLiteBackend
- func (s *SQLiteBackend) MatchIntent(ctx context.Context, db *gorm.DB, query string, maxCandidates int) ([]intentrank.Doc, error)
- func (s *SQLiteBackend) Migrate(db *gorm.DB) error
- func (s *SQLiteBackend) PurgeNamespace(ctx context.Context, db *gorm.DB) error
- func (s *SQLiteBackend) Query(ctx context.Context, db *gorm.DB, query string, limit int) ([]graph.Node, error)
- func (s *SQLiteBackend) Rebuild(ctx context.Context, db *gorm.DB) error
- func (s *SQLiteBackend) RebuildNodes(ctx context.Context, db *gorm.DB, nodeIDs []uint) error
- type Writer
Constants ¶
This section is empty.
Variables ¶
var ErrFTS5NotAvailable = trace.New("fts5 module not available")
ErrFTS5NotAvailable indicates the SQLite build lacks the fts5 extension.
Functions ¶
func NewIngestUnitOfWork ¶
NewIngestUnitOfWork composes transaction-scoped graph and search adapters for ingest workflows. @intent keep raw GORM transaction wiring at the outbound composition boundary.
func RefreshSearchDocuments ¶
RefreshSearchDocuments rebuilds namespace-scoped search_documents from current graph nodes. @intent keep derived search documents consistent with graph state before FTS rebuilds
func RefreshSearchDocumentsFor ¶
RefreshSearchDocumentsFor rebuilds search_documents for the specified node IDs only. @intent incremental update 경로에서 영향받은 문서만 갱신한다.
func SanitizeFTS5 ¶
SanitizeFTS5 converts raw user input into a safe FTS5 prefix query. A camelCase term also matches its sub-tokens, so `getUser` matches either the whole token or (`get` AND `user`), mirroring the sub-tokens indexed at build time. @intent build SQLite FTS queries that preserve prefix matching without exposing parser-breaking characters. @domainRule empty or fully stripped input returns an empty query string.
func SanitizeIntentFTS5 ¶ added in v0.13.0
SanitizeIntentFTS5 converts a question into an any-term FTS5 prefix query for the intent index.
SanitizeFTS5 requires every term because the searcher there typed identifiers, and an identifier a caller half-remembers is still worth demanding in full. An intent question is the opposite: it is a sentence aimed at another sentence, and no recorded reason will contain all of "why do we verify the signature on a push". Requiring every term would answer almost nothing.
Any-term is only safe here because this index holds nothing but recorded reasons. The same widening was measured on the shared index and rejected: there a common word could match an identifier, a path segment, or a language alias, so widening pulled in fifty unrelated nodes. With the names removed, a term can only match prose somebody wrote on purpose, and bm25 discounts the words that appear in many of those. @intent let a sentence-shaped question match a sentence-shaped reason. @domainRule any-term matching is confined to the intent index, which holds no identifier text.
func SanitizePostgresIntentTSQuery ¶ added in v0.13.0
SanitizePostgresIntentTSQuery is the PostgreSQL twin of SanitizeIntentFTS5: any term may match, because no recorded reason contains every word of a question. See SanitizeIntentFTS5 for why that widening is confined to the intent index. @intent let a sentence-shaped question match a sentence-shaped reason on PostgreSQL. @domainRule any-term matching is confined to the intent index, which holds no identifier text.
func SanitizePostgresTSQuery ¶
SanitizePostgresTSQuery converts raw user input into a safe prefix tsquery, mirroring SanitizeFTS5 including camelCase sub-token expansion. @intent translate free-form user input into a PostgreSQL tsquery that mirrors the SQLite prefix search behavior. @domainRule empty or fully stripped input returns an empty query string.
Types ¶
type Backend ¶
type Backend interface {
// @intent prepare search tables and indexes for the active database driver.
// @sideEffect may create or update search index schema objects.
Migrate(db *gorm.DB) error
// @intent refresh backend-specific full-text index state from the current persisted search documents for the active namespace.
// @requires db must be a valid connection for processing the active namespace.
// @sideEffect rewrites backend-specific search index records or derived vectors.
Rebuild(ctx context.Context, db *gorm.DB) error
// @intent reindex only changed nodes so incremental updates cost less than full rebuilds.
// @param nodeIDs is the set of node IDs to reindex.
// @sideEffect updates search index records for the specified nodes.
RebuildNodes(ctx context.Context, db *gorm.DB, nodeIDs []uint) error
// @intent remove or reconcile backend-specific search index state for the active namespace when physical cleanup is required.
// @sideEffect may clear namespace-scoped search index records, though implementations may intentionally no-op.
PurgeNamespace(ctx context.Context, db *gorm.DB) error
// @intent execute a user query using the backend-specific full-text search syntax.
// @param query is the raw query string to search for.
// @param limit is the maximum number of results to return.
// @return returns nodes ordered by relevance.
Query(ctx context.Context, db *gorm.DB, query string, limit int) ([]graph.Node, error)
// MatchIntent finds every recorded reason a question could be answered from,
// in no particular order, and hands back the exact text that was indexed for
// each one.
//
// Ordering is deliberately not the backend's job. SQLite would order by
// bm25 and PostgreSQL by ts_rank, which never learns that a word is common,
// so the same question ranks differently on the database that was measured
// and the database that is deployed. Both backends retrieve here and
// intentrank scores, so there is one answer to be judged by.
// @intent find every candidate reason and leave the ranking to shared scoring.
// @param query is a natural-language question, not an identifier.
// @param maxCandidates caps how many candidates come back, guarding memory rather than shaping the answer.
// @return returns unordered candidates, and nothing when no recorded reason matches.
MatchIntent(ctx context.Context, db *gorm.DB, query string, maxCandidates int) ([]intentrank.Doc, error)
}
Backend defines the full-text search backend contract. @intent provide one interface for backend-specific search index migration, rebuild, and query operations.
type PostgresBackend ¶
type PostgresBackend struct{}
PostgresBackend is a full-text search backend based on PostgreSQL tsvector. @intent Handles full-text search indexing and querying in a PostgreSQL environment.
func NewPostgresBackend ¶
func NewPostgresBackend() *PostgresBackend
NewPostgresBackend creates a PostgreSQL search backend. @intent Provides a Backend implementation specifically for PostgreSQL.
func (*PostgresBackend) MatchIntent ¶ added in v0.13.0
func (p *PostgresBackend) MatchIntent(ctx context.Context, db *gorm.DB, query string, maxCandidates int) ([]intentrank.Doc, error)
MatchIntent finds every recorded reason holding any term of the question.
It used to order by ts_rank, which is where the deployment gap lived: ts_rank reads one document at a time and never learns that a word appears in most of them, so it could not tell a distinctive word from a filler one. Retrieval is what the GIN index is genuinely good at; scoring moved to intentrank, which counts the corpus and gives both backends the same answer.
The join onto nodes carries each candidate's identity — path, qualified name, kind, namespace, start line — because that is what intentrank breaks its score ties on, and it drops reason rows whose node is gone rather than spending a row of the candidate cap on one that cannot be scored. The SQLite twin does the same; both have to, or the two backends tie-break on different information. @intent hand every candidate reason to shared scoring, with the identity that scoring breaks ties on. @requires maxCandidates must be greater than 0. @return returns unordered candidates with the exact text that was indexed for each.
func (*PostgresBackend) Migrate ¶
func (p *PostgresBackend) Migrate(db *gorm.DB) error
Migrate ensures the PostgreSQL search schema exists by running the versioned migrations, which are the single source of truth for the tsvector column, trigger, and GIN index. It no longer hand-writes DDL, so the schema cannot drift from the migration files. @intent give tests and callers a one-call schema setup that reuses the production migrations. @sideEffect applies any pending schema migrations to the connected database.
func (*PostgresBackend) PurgeNamespace ¶
PurgeNamespace is a no-op as PostgreSQL search_documents deletion does not require separate physical cleanup. @intent Aligns with the Backend interface and maintains consistency in the namespace purge path.
func (*PostgresBackend) Query ¶
func (p *PostgresBackend) Query(ctx context.Context, db *gorm.DB, query string, limit int) ([]graph.Node, error)
Query searches for related nodes using PostgreSQL tsquery.
Every term is required, mirroring the SQLite backend. See SQLiteBackend.Query for why widening to any-term was measured and rejected.
@intent Converts the user's search term into a prefix tsquery to find related nodes. @requires limit must be greater than 0 to get meaningful results. @return Returns a list of nodes sorted by ts_rank.
func (*PostgresBackend) Rebuild ¶
Rebuild recalculates the tsvector for all search documents.
The separators '/', '.', and '_' are translated to spaces before to_tsvector, because FTS5's unicode61 tokenizer splits on them and this vector has to see the same tokens: without it, PostgreSQL keeps a dotted qualified name as one host-like token and cannot answer a query naming one of its segments. The trigger in migration 000019 applies the same expression on every write. @intent Batch regenerates the full-text search index for existing search_documents and search_reasons rows. @sideEffect Updates search_documents.tsv and search_reasons.reason_tsv values.
func (*PostgresBackend) RebuildNodes ¶
RebuildNodes recalculates both tsvectors only for specified nodes. @intent Avoids full namespace tsv updates during incremental update paths.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader binds candidate and intent search persistence to one database. @intent adapt raw SQL backend and GORM operations to app/search read ports.
func NewReader ¶
NewReader constructs bound search read ports. @intent keep database handles out of application service construction.
func (*Reader) Query ¶
Query returns relevance-ordered candidates from the configured SQL search backend. @intent implement the bound candidate-search port without exposing a DB argument.
func (*Reader) QueryIntent ¶ added in v0.13.0
func (r *Reader) QueryIntent(ctx context.Context, query string, limit int) (intentapp.Result, error)
QueryIntent answers a question from the recorded-reason index only.
The database finds the candidates and Go ranks them. That split is what makes SQLite and PostgreSQL answer alike: each database has its own scoring function and they disagree, so leaving the order to whichever one is deployed meant the golden score measured on a laptop said nothing about the running server. The terms come back with the nodes because the scorer is the only place that knows them, and the caller cannot judge an answer without them: a file that matched one word written in half the recorded reasons and a file that matched a word written in three are the same row otherwise.
@intent implement the bound intent-search port without exposing a DB argument, and rank the same way on every backend. @return returns at most limit nodes, best first, each with the question terms written in its reason, and nothing when no recorded reason matches.
type SQLiteBackend ¶
type SQLiteBackend struct {
// contains filtered or unexported fields
}
SQLiteBackend is a full-text search backend based on SQLite FTS5. @intent Handles full-text search indexing and querying in a SQLite environment.
func NewSQLiteBackend ¶
func NewSQLiteBackend() *SQLiteBackend
NewSQLiteBackend creates a SQLite search backend. @intent Provides a Backend implementation specifically for SQLite.
func (*SQLiteBackend) MatchIntent ¶ added in v0.13.0
func (s *SQLiteBackend) MatchIntent(ctx context.Context, db *gorm.DB, query string, maxCandidates int) ([]intentrank.Doc, error)
MatchIntent finds every recorded reason holding any term of the question.
FTS5 could order these by bm25 and used to, but the ordering moved to intentrank so that this backend and the PostgreSQL one answer the same question the same way. What is left here is retrieval, which is what the index is for.
The join onto nodes carries each candidate's identity — path, qualified name, kind, namespace, start line — because that is what intentrank breaks its score ties on. It also drops index rows whose node is gone, the same way matchRows does: an orphan cannot be scored and would otherwise spend a row of the candidate cap. @intent hand every candidate reason to shared scoring, with the identity that scoring breaks ties on. @requires maxCandidates must be greater than 0. @return returns unordered candidates with the exact text that was indexed for each.
func (*SQLiteBackend) Migrate ¶
func (s *SQLiteBackend) Migrate(db *gorm.DB) error
Migrate prepares the SQLite FTS5 virtual table. @intent Creates a full-text search index table for SQLite. @sideEffect May create the search_fts virtual table. @ensures search_fts exists if FTS5 is available.
func (*SQLiteBackend) PurgeNamespace ¶
PurgeNamespace removes the physical FTS index for a specific namespace. @intent Cleans up stale FTS rows even in paths without a rebuild, such as namespace deletion.
func (*SQLiteBackend) Query ¶
func (s *SQLiteBackend) Query(ctx context.Context, db *gorm.DB, query string, limit int) ([]graph.Node, error)
Query searches for related nodes using FTS5 MATCH queries.
Every term is required, and SanitizeFTS5 decides what counts as a term. That pairing is the whole retrieval policy: requiring all terms is right when the searcher typed identifiers, and it only became wrong for sentences because ordinary English words were being required too.
Widening to any-term when all-terms matches nothing was measured and rejected. It answered no query the narrow expression missed — the two extra nodes it retrieved never reached the top ten — and it filled the deliberate nonsense query in the golden set with fifty unrelated hits.
@intent Converts the user's search term into a SQLite FTS prefix query to find nodes. @requires limit must be greater than 0 to get meaningful results. @return Returns a list of nodes sorted by FTS rank.
func (*SQLiteBackend) Rebuild ¶
Rebuild reloads the persisted documents into the two FTS indexes: search_fts from search_documents, and intent_fts from search_reasons, which holds one row per recorded reason. @intent Synchronizes stored search documents and recorded reasons with the SQLite FTS indexes. @sideEffect Deletes and re-inserts search_fts and intent_fts content. @domainRule Index content must match the current snapshot of search_documents and search_reasons.
func (*SQLiteBackend) RebuildNodes ¶
RebuildNodes synchronizes only the FTS rows of specified nodes with search_documents. @intent Avoids full namespace FTS reloading during incremental update paths.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer updates derived search documents and the configured search backend through one DB handle. @intent provide a transaction-scoped SearchWriter implementation for ingest unit-of-work adapters.
func NewSearchWriter ¶
NewSearchWriter binds derived search updates to the supplied database handle. @intent construct a search writer that can share an ingest transaction with graph persistence.
func (*Writer) RebuildAll ¶
RebuildAll refreshes every namespace-scoped search document and rebuilds the backend index. @intent implement the full derived-search refresh required by a graph build. @sideEffect rewrites search documents and the backend index through the bound DB handle.
func (*Writer) RebuildIndex ¶
RebuildIndex rebuilds the configured full-text backend after documents refresh. @intent implement the second application maintenance stage without exposing backend or database handles.
func (*Writer) RebuildNodes ¶
RebuildNodes refreshes only the supplied node IDs and updates the matching backend scope. @intent implement the incremental derived-search refresh required by graph updates. @sideEffect rewrites scoped search documents and the backend index through the bound DB handle.
Source Files