reembed-verify

Overview ¶

Package main is a read-only diagnostic for an embedding-model rename.

When the configured embedding model string changes (e.g. graduating "google/gemini-embedding-2-preview" → "google/gemini-embedding-2"), the startup re-embed (internal/rag/reembed.go) re-generates every stored vector because the `embedding_version` tag no longer matches. This tool answers, WITHOUT touching the DB, three questions that the in-place re-embed can't:

• identity: is the new model the same vector space? It re-embeds each row's content and reports the cosine(old, new) distribution.
• speed: wall time at a given --parallelism / --batch, so we can size the real migration and experiment with faster knobs.
• cost: prompt tokens consumed (+ a $ estimate via --price-per-1m).

It reuses the production internal/llm client so provider routing and request shape match the real re-embed path. Nothing is written back.

Usage:

go run ./cmd/reembed-verify --db data/prod/laplaced.db
go run ./cmd/reembed-verify --parallelism 16 --batch 64 --limit 500