amoxtli

package module
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2026 License: MIT Imports: 16 Imported by: 0

README

Amoxtli

Amoxtli — « livre, codex » en nahuatl.

Bibliothèque Go d'indexation documentaire multi-backend et d'ingestion de fichiers : recherche plein-texte (bleve), recherche vectorielle (sqlite-vec), recherche hybride PostgreSQL (pgvector + FTS natif), fusion des résultats par Reciprocal Rank Fusion (pondérée par index), découpage markdown en sections, conversion de fichiers (pandoc, LibreOffice, OCR/LLM), grounding (récupération vérifiée) et sauvegarde/restauration des index.

Extraite du projet bornholm/corpus, dont elle constitue le cœur, mais indépendante de celui-ci.

Statut : pré-v0.1.0 — API instable.

Installation

go get github.com/bornholm/amoxtli

Aucune directive replace n'est nécessaire : le backend index/sqlitevec embarque son propre build WASM de SQLite incluant l'extension sqlite-vec (voir index/sqlitevec/internal/vec).

Backend sqlite-vec et version de ncruces/go-sqlite3. Le build WASM embarqué est couplé à l'ABI de github.com/ncruces/go-sqlite3 v0.23.0 (variables sqlite3.Binary / sqlite3.RuntimeConfig, retirées dans les versions ultérieures). Si vous utilisez le backend sqlite-vec, épinglez cette version dans votre go.mod — sinon go get peut sélectionner une version plus récente incompatible :

require github.com/ncruces/go-sqlite3 v0.23.0

Les autres backends (bleve, postgres) et le magasin SQLite (ingest/gorm) ne sont pas concernés.

Démarrage rapide

Le magasin de documents (WithStore) et les indexeurs (WithIndexers) sont fournis explicitement, chacun construit par son propre constructeur. L'appelant possède les ressources qu'il crée et doit les fermer ; codex.Close() n'arrête que le runner de tâches.

// Magasin de documents (SQLite local, ou gorm.NewPostgresStore).
store, err := gorm.NewSQLiteStore("/data/kb/data.sqlite") // ingest/gorm
if err != nil { /* ... */ }
defer store.Close()

// Index plein-texte (bleve).
bleveIdx, err := bleve.OpenOrCreate(ctx, "/data/kb/index.bleve") // index/bleve
if err != nil { /* ... */ }
defer bleveIdx.Close()

codex, err := amoxtli.New(ctx,
    amoxtli.WithStore(store),
    amoxtli.WithIndexers(amoxtli.Indexer{ID: "bleve", Index: bleveIdx, Weight: 1.0}),
    amoxtli.WithDisableHyDE(), amoxtli.WithDisableJudge(), // pas de client LLM
)
if err != nil { /* ... */ }
defer codex.Close()

collID, _ := codex.CreateCollection(ctx, "docs")
taskID, _ := codex.IndexFile(ctx, collID, "guide.md", file)
results, _ := codex.Search(ctx, "comment faire…", amoxtli.WithSearchMaxResults(5))

Exemples complets et exécutables : example/sqlite (SQLite + bleve, sans LLM), example/postgres (tout PostgreSQL) et example/convert (conversion de fichier + suivi de tâche).

Documentation

Licence

MIT

Documentation

Overview

Package amoxtli provides a multi-backend document indexing and file-ingestion library: full-text search (bleve), vector search (sqlite-vec), weighted result merging, markdown chunking, file conversion and snapshot/restore of indexes.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound = ingest.ErrNotFound
	ErrCanceled = task.ErrCanceled
)

Functions

This section is empty.

Types

type Codex 

type Codex struct {
	// contains filtered or unexported fields
}

Codex is the main embedded instance: a store, index pipeline and task runner behind a single API.

func New 

func New(ctx context.Context, funcs ...Option) (*Codex, error)

New creates a new embedded Codex instance.

A store (WithStore) and an index (WithIndex or WithIndexers) are required; build them with the dedicated constructors, e.g. gorm.NewSQLiteStore / gorm.NewPostgresStore for the store and bleve.OpenOrCreate / sqlitevec.NewIndex / postgres.NewIndex for the indexers. The caller owns the resources it constructs and must Close them; Codex.Close only stops the task runner.

func (*Codex) Backup 

func (c *Codex) Backup(ctx context.Context) (io.ReadCloser, error)

Backup streams a snapshot of the index and the document store as a multipart archive.

func (*Codex) CheckGrounding added in v0.0.2 

func (c *Codex) CheckGrounding(ctx context.Context, query string, results []*index.SearchResult) (*retrieval.GroundingResult, error)

CheckGrounding evaluates whether the given search results support a reliable, grounded answer to the query, returning a verdict (status + score + explanation). It is a standalone step, fully decoupled from Search: run Search first, then pass its results here to decide whether to trust them or abstain. Requires WithGroundingCheck and an LLM client; otherwise it returns an error.

func (*Codex) CleanupIndex 

func (c *Codex) CleanupIndex(ctx context.Context, collections ...model.CollectionID) (task.ID, error)

CleanupIndex schedules a cleanup of orphaned documents and obsolete index entries.

func (*Codex) Close 

func (c *Codex) Close() error

Close stops the task runner. Resources passed in through WithStore, WithIndex/WithIndexers and WithTaskRunner are owned by the caller and must be closed by them.

func (*Codex) CreateCollection 

func (c *Codex) CreateCollection(ctx context.Context, label string) (model.CollectionID, error)

CreateCollection creates a new collection and returns its ID.

func (*Codex) DeleteBySource 

func (c *Codex) DeleteBySource(ctx context.Context, source *url.URL) error

DeleteBySource removes all documents and index entries for the given source URL.

func (*Codex) GetSectionsByIDs 

func (c *Codex) GetSectionsByIDs(ctx context.Context, ids []model.SectionID) (map[model.SectionID]model.Section, error)

GetSectionsByIDs returns the sections matching the given IDs, typically used to retrieve the content behind search results.

func (*Codex) Index 

func (c *Codex) Index() index.Index

Index returns the underlying index.Index for advanced usage.

func (*Codex) IndexFile 

func (c *Codex) IndexFile(ctx context.Context, collectionID model.CollectionID, filename string, r io.Reader, funcs ...IndexFileOption) (task.ID, error)

IndexFile indexes a file into the given collection. Returns a task.ID that can be used to track progress via TaskState.

func (*Codex) Manager 

func (c *Codex) Manager() *ingest.Manager

Manager returns the underlying ingest.Manager for advanced usage.

func (*Codex) Reindex 

func (c *Codex) Reindex(ctx context.Context) (task.ID, error)

Reindex schedules a rebuild of the whole index.

func (*Codex) ReindexCollection 

func (c *Codex) ReindexCollection(ctx context.Context, collectionID model.CollectionID) (task.ID, error)

ReindexCollection schedules a rebuild of the index for a single collection.

func (*Codex) Restore 

func (c *Codex) Restore(ctx context.Context, r io.Reader) error

Restore synchronously restores a snapshot previously produced by Backup.

func (*Codex) Search 

func (c *Codex) Search(ctx context.Context, query string, funcs ...SearchOption) ([]*index.SearchResult, error)

Search performs a semantic search across the indexed documents.

func (*Codex) SearchIterative added in v0.0.2 

func (c *Codex) SearchIterative(ctx context.Context, query string, funcs ...SearchOption) (*retrieval.Result, error)

SearchIterative runs the explicit re-retrieval orchestration on top of Search: optional query decomposition and, when a grounding checker and reformulator are configured, grounding-gated iterative re-retrieval. It returns the fused evidence together with the final grounding verdict (retrieval.Result). It is a separate entry point from Search (which stays a plain, single-shot retrieval) and from CheckGrounding; with none of the orchestration options enabled it is equivalent to Search.

func (*Codex) TaskState 

func (c *Codex) TaskState(ctx context.Context, id task.ID) (*task.State, error)

TaskState returns the current state of an indexing task.

type IndexFileOption 

type IndexFileOption func(*IndexFileOptions)

IndexFileOption configures an IndexFile call.

func WithIndexFileCollections 

func WithIndexFileCollections(ids ...model.CollectionID) IndexFileOption

WithIndexFileCollections associates the indexed file with the given collection IDs.

func WithIndexFileETag 

func WithIndexFileETag(etag string) IndexFileOption

WithIndexFileETag sets the ETag for the indexed file (used for deduplication).

func WithIndexFileSource 

func WithIndexFileSource(source *url.URL) IndexFileOption

WithIndexFileSource sets the source URL for the indexed file.

type IndexFileOptions 

type IndexFileOptions struct {
	Source      *url.URL
	ETag        string
	Collections []model.CollectionID
}

IndexFileOptions holds options for IndexFile calls.

type Indexer 

type Indexer struct {
	// ID identifies the indexer in the pipeline (e.g. "bleve", "postgres").
	ID string
	// Index is any implementation of the index.Index contract.
	Index index.Index
	// Weight is the relative weight of this indexer when merging scores.
	Weight float64
}

Indexer identifies a weighted index.Index inside the search pipeline.

type Option 

type Option func(*options)

Option is a function that configures a Codex instance.

func WithDisableHyDE 

func WithDisableHyDE() Option

WithDisableHyDE disables the HyDE query transformer.

func WithDisableJudge 

func WithDisableJudge() Option

WithDisableJudge disables the Judge results transformer.

func WithFileConverter 

func WithFileConverter(fc convert.Converter) Option

WithFileConverter sets a file converter for converting files before indexing.

func WithGroundingCheck added in v0.0.2 

func WithGroundingCheck() Option

WithGroundingCheck enables the fused evidence evaluator: a single LLM call that both relevance-filters the retrieved evidence and judges whether it supports a reliable answer (the grounding γ verdict). It makes CheckGrounding available as a standalone step and gates the re-retrieval loop of SearchIterative. When enabled it replaces the Judge results transformer in the pipeline (Search then relies on the evaluator for relevance filtering), avoiding a redundant LLM pass over the same evidence. Requires an LLM client (WithLLMClient). Disabled by default.

func WithGroundingMinScore added in v0.0.2 

func WithGroundingMinScore(minScore float64) Option

WithGroundingMinScore sets the grounding score threshold below which the verdict is considered not confident (default 0.4). Only meaningful together with WithGroundingCheck.

func WithIndex 

func WithIndex(idx index.Index) Option

WithIndex provides a ready-made index.Index, bypassing pipeline composition entirely (including the HyDE/Judge/dedup transformers). Mutually exclusive with WithIndexers. The caller owns and must close the index.

func WithIndexers 

func WithIndexers(indexers ...Indexer) Option

WithIndexers declares the set of indexers composing the search pipeline, each with its relative weight. It can be called several times; indexers accumulate.

Any implementation of index.Index can be plugged in; conformance can be verified with the index/testsuite package. Build the backends with their own constructors, e.g. bleve.OpenOrCreate(...), sqlitevec.NewIndex(...) or postgres.NewIndex(...), and wrap each in an Indexer.

func WithIterativeRetrieval added in v0.0.2 

func WithIterativeRetrieval(rounds int) Option

WithIterativeRetrieval enables grounding-driven re-retrieval in SearchIterative: when the evidence is not confidently grounded the query is reformulated and searched again, up to rounds times (rounds <= 0 means 1). Requires WithGroundingCheck and an LLM client.

func WithLLMClient 

func WithLLMClient(client llm.Client) Option

WithLLMClient sets the LLM client used by the HyDE and Judge transformers.

func WithMaxTotalWords 

func WithMaxTotalWords(n int) Option

WithMaxTotalWords sets the maximum total words used by the Judge transformer.

func WithMaxWordsPerSection 

func WithMaxWordsPerSection(n int) Option

WithMaxWordsPerSection sets the maximum number of words per document section.

func WithQueryDecomposition added in v0.0.2 

func WithQueryDecomposition(maxSubQueries int) Option

WithQueryDecomposition enables splitting a complex question into at most maxSubQueries sub-questions in SearchIterative, searching each and fusing their evidence. Requires an LLM client. maxSubQueries <= 0 keeps the default (3).

func WithSnapshotBoundary 

func WithSnapshotBoundary(boundary string) Option

WithSnapshotBoundary overrides the multipart boundary used by Backup/Restore.

func WithStore 

func WithStore(store ingest.Store) Option

WithStore sets the document store. It is required. Build it with gorm.NewSQLiteStore or gorm.NewPostgresStore (or any ingest.Store). The caller owns and must close the store.

func WithTaskParallelism 

func WithTaskParallelism(n int) Option

WithTaskParallelism sets the number of concurrent tasks allowed.

func WithTaskRunner 

func WithTaskRunner(runner task.Runner) Option

WithTaskRunner provides a custom task.Runner implementation.

type SearchOption 

type SearchOption func(*SearchOptions)

SearchOption configures a Search call.

func WithSearchCollections 

func WithSearchCollections(ids ...model.CollectionID) SearchOption

WithSearchCollections restricts the search to the given collection IDs.

func WithSearchMaxResults 

func WithSearchMaxResults(n int) SearchOption

WithSearchMaxResults sets the maximum number of search results.

type SearchOptions 

type SearchOptions struct {
	MaxResults  int
	Collections []model.CollectionID
}

SearchOptions holds options for Search calls.

Source Files

View all Source files

Directories

Path Synopsis
genai
libreoffice
pandoc
example
convert command
Command convert demonstrates two things at once:
 Command convert demonstrates two things at once:
postgres command
Command postgres demonstrates amoxtli backed entirely by PostgreSQL: the document store and the hybrid index (index/postgres) share a single database.
 Command postgres demonstrates amoxtli backed entirely by PostgreSQL: the document store and the hybrid index (index/postgres) share a single database.
sqlite command
Command sqlite demonstrates amoxtli backed entirely by local files: a SQLite document store and a Bleve full-text index.
 Command sqlite demonstrates amoxtli backed entirely by local files: a SQLite document store and a Bleve full-text index.
bleve
pipeline
postgres
Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance).
 Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance).
sqlitevec
sqlitevec/internal/vec
testsuite
gorm
internal
syncx
text
workflow
Package retrieval implements grounding-driven retrieval orchestration ported from Corpus' MothRAG-derived mechanisms: an evidence evaluator (relevance filtering fused with a grounding (γ) verdict), query reformulation for iterative re-retrieval, and query decomposition.
 Package retrieval implements grounding-driven retrieval orchestration ported from Corpus' MothRAG-derived mechanisms: an evidence evaluator (relevance filtering fused with a grounding (γ) verdict), query reformulation for iterative re-retrieval, and query decomposition.
memory

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.