Documentation
¶
Overview ¶
Package store provides SQLite-backed persistence for the Dewey knowledge graph. It stores pages, blocks, links, embeddings, and index metadata in a single .dewey/graph.db file using modernc.org/sqlite (pure Go, no CGO).
Index ¶
- func IsDiskSpaceError(err error) bool
- type Block
- type Embedding
- type Link
- type Page
- type SearchFilters
- type SimilarityResult
- type SourceRecord
- type Store
- func (s *Store) Close() error
- func (s *Store) CountBlocks() (int, error)
- func (s *Store) CountEmbeddings() (int, error)
- func (s *Store) CountPages() (int, error)
- func (s *Store) CountPagesBySource(sourceID string) (int, error)
- func (s *Store) DB() *sql.DB
- func (s *Store) DeleteBlocksByPage(pageName string) error
- func (s *Store) DeleteEmbeddingsByBlock(blockUUID string) error
- func (s *Store) DeleteEmbeddingsByModel(modelID string) error
- func (s *Store) DeleteLinksByPage(pageName string) error
- func (s *Store) DeletePage(name string) error
- func (s *Store) DeletePagesBySource(sourceID string) (int64, error)
- func (s *Store) GetAllEmbeddings(modelID string) ([]Embedding, error)
- func (s *Store) GetBackwardLinks(pageName string) ([]*Link, error)
- func (s *Store) GetBlock(uuid string) (*Block, error)
- func (s *Store) GetBlocksByPage(pageName string) ([]*Block, error)
- func (s *Store) GetEmbedding(blockUUID, modelID string) (*Embedding, error)
- func (s *Store) GetForwardLinks(pageName string) ([]*Link, error)
- func (s *Store) GetMeta(key string) (string, error)
- func (s *Store) GetPage(name string) (*Page, error)
- func (s *Store) GetSource(id string) (*SourceRecord, error)
- func (s *Store) InsertBlock(b *Block) error
- func (s *Store) InsertEmbedding(blockUUID, modelID string, vector []float32, chunkText string) error
- func (s *Store) InsertLink(l *Link) error
- func (s *Store) InsertPage(p *Page) error
- func (s *Store) InsertSource(src *SourceRecord) error
- func (s *Store) ListPages() ([]*Page, error)
- func (s *Store) ListPagesBySource(sourceID string) ([]*Page, error)
- func (s *Store) ListPagesExcludingSource(sourceID string) ([]*Page, error)
- func (s *Store) ListSources() ([]*SourceRecord, error)
- func (s *Store) SearchSimilar(modelID string, queryVec []float32, limit int, threshold float64) ([]SimilarityResult, error)
- func (s *Store) SearchSimilarFiltered(modelID string, queryVec []float32, filters SearchFilters, limit int, ...) ([]SimilarityResult, error)
- func (s *Store) SetMeta(key, value string) error
- func (s *Store) UpdateLastFetched(id string, fetchedAt int64) error
- func (s *Store) UpdatePage(p *Page) error
- func (s *Store) UpdateSourceStatus(id, status, errorMessage string) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsDiskSpaceError ¶
IsDiskSpaceError returns true if the given error indicates disk space exhaustion (e.g., SQLite "database or disk is full", OS "no space left"). Returns false if err is nil. When disk space is insufficient, Dewey should continue operating from the in-memory index without crashing.
Types ¶
type Block ¶
type Block struct {
UUID string
PageName string
ParentUUID sql.NullString
Content string
HeadingLevel int
Position int
}
Block represents a heading-delimited section within a page.
type Embedding ¶
type Embedding struct {
BlockUUID string
ModelID string
Vector []float32
ChunkText string
GeneratedAt int64
}
Embedding represents a vector embedding for a content block. The vector is stored as a float32 BLOB in SQLite for storage efficiency.
type Page ¶
type Page struct {
Name string
OriginalName string
SourceID string
SourceDocID string
Properties string // JSON
ContentHash string
IsJournal bool
CreatedAt int64
UpdatedAt int64
}
Page represents a document in the knowledge graph.
type SearchFilters ¶
SearchFilters constrains similarity search results by metadata.
type SimilarityResult ¶
type SimilarityResult struct {
BlockUUID string `json:"document_id"`
PageName string `json:"page"`
Content string `json:"content"`
Similarity float64 `json:"similarity"`
Source string `json:"source"`
SourceID string `json:"source_id"`
OriginURL string `json:"origin_url,omitempty"`
IndexedAt int64 `json:"indexed_at"`
}
SimilarityResult represents a search result with cosine similarity score. Includes provenance metadata for Observable Quality (Constitution III).
type SourceRecord ¶
type SourceRecord struct {
ID string
Type string
Name string
Config string // JSON
RefreshInterval string
LastFetchedAt int64
Status string
ErrorMessage string
}
SourceRecord represents a content source in the store.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store wraps a SQLite database connection for knowledge graph persistence. It manages pages, blocks, links, embeddings, and index metadata. File-level locking prevents concurrent write corruption (T059).
func New ¶
New opens (or creates) a SQLite database at the given path and applies schema migrations. Pass an empty string or ":memory:" for an in-memory database (useful for testing).
Returns an error if the database cannot be opened, pragma configuration fails, the file lock cannot be acquired (another Dewey process is using the database), or schema migration fails.
The returned Store must be closed with Store.Close when no longer needed to release the database connection and file lock.
The database is configured with:
- WAL journal mode for concurrent read access
- Foreign key enforcement
- Busy timeout of 5 seconds
func (*Store) Close ¶
Close closes the underlying database connection and releases the file lock. Returns an error if the database connection cannot be closed cleanly.
func (*Store) CountBlocks ¶
CountBlocks returns the total number of blocks in the store. Returns 0 and an error if the count query fails.
func (*Store) CountEmbeddings ¶
CountEmbeddings returns the total number of embeddings in the store across all models. Returns 0 and an error if the count query fails.
func (*Store) CountPages ¶
CountPages returns the total number of pages in the store. Returns 0 and an error if the count query fails.
func (*Store) CountPagesBySource ¶
CountPagesBySource returns the number of pages associated with the given source ID. Returns 0 (not an error) if no pages belong to that source. Returns an error if the count query fails.
func (*Store) DB ¶
DB returns the underlying *sql.DB for advanced queries. Prefer using Store methods for standard operations. The returned connection is managed by the Store and must not be closed independently.
func (*Store) DeleteBlocksByPage ¶
DeleteBlocksByPage removes all blocks belonging to the named page. Returns an error if the delete query fails. Does not return an error if no blocks exist for the page (idempotent delete).
func (*Store) DeleteEmbeddingsByBlock ¶
DeleteEmbeddingsByBlock removes all embeddings for a given block UUID across all models. Returns an error if the delete query fails. Does not return an error if no embeddings exist for the block (idempotent delete).
func (*Store) DeleteEmbeddingsByModel ¶
DeleteEmbeddingsByModel removes all embeddings generated by the given model ID. Used when the embedding model changes and all vectors need regeneration. Returns an error if the delete query fails. Does not return an error if no embeddings exist for the model (idempotent delete).
func (*Store) DeleteLinksByPage ¶
DeleteLinksByPage removes all outgoing links from the named page. Returns an error if the delete query fails. Does not return an error if no links exist for the page (idempotent delete).
func (*Store) DeletePage ¶
DeletePage removes a page and its associated blocks and links (via CASCADE). Returns an error if the delete query fails or if no page exists with the given name (page not found).
func (*Store) DeletePagesBySource ¶ added in v1.0.0
DeletePagesBySource removes all pages (and their associated blocks, links, and embeddings via CASCADE) belonging to the given source ID. Returns the number of rows deleted. Used for orphan auto-purge when a source is removed from sources.yaml (FR-013). The operation is wrapped in a transaction for atomicity.
func (*Store) GetAllEmbeddings ¶
GetAllEmbeddings returns all embeddings for a given model ID, loading all vectors into memory. Used by brute-force cosine similarity search. Returns an empty slice if no embeddings exist for the model. Returns an error if the query or row scanning fails.
func (*Store) GetBackwardLinks ¶
GetBackwardLinks returns all incoming links to the named page (pages that link to this page). Returns an empty slice if no pages link to the given page. Returns an error if the query or row scanning fails.
func (*Store) GetBlock ¶
GetBlock retrieves a block by its UUID. Returns the block and nil error on success, or (nil, nil) if no block exists with the given UUID. Returns a non-nil error if the query fails.
func (*Store) GetBlocksByPage ¶
GetBlocksByPage returns all blocks belonging to the named page, ordered by position. Returns an empty slice if the page has no blocks or does not exist. Returns an error if the query or row scanning fails.
func (*Store) GetEmbedding ¶
GetEmbedding retrieves an embedding by block UUID and model ID. Returns the embedding and nil error on success, or (nil, nil) if no embedding exists for the given block and model combination. Returns a non-nil error if the query fails.
func (*Store) GetForwardLinks ¶
GetForwardLinks returns all outgoing links from the named page (pages that this page links to). Returns an empty slice if the page has no outgoing links. Returns an error if the query or row scanning fails.
func (*Store) GetMeta ¶
GetMeta retrieves a metadata value by key. Returns an empty string and nil error if the key does not exist. Returns a non-nil error if the query fails.
func (*Store) GetPage ¶
GetPage retrieves a page by its normalized name. Returns the page and nil error on success, or (nil, nil) if no page exists with the given name. Returns a non-nil error if the query fails.
func (*Store) GetSource ¶
func (s *Store) GetSource(id string) (*SourceRecord, error)
GetSource retrieves a content source record by its ID. Returns the record and nil error on success, or (nil, nil) if no source exists with the given ID. Returns a non-nil error if the query fails.
func (*Store) InsertBlock ¶
InsertBlock inserts a new block into the store. The block's PageName must reference an existing page (foreign key constraint).
Returns an error if the insert fails (e.g., duplicate UUID or missing parent page).
func (*Store) InsertEmbedding ¶
func (s *Store) InsertEmbedding(blockUUID, modelID string, vector []float32, chunkText string) error
InsertEmbedding stores a vector embedding for a block, replacing any existing embedding with the same (blockUUID, modelID) pair. Uses parameterized queries to prevent SQL injection (FR-028). The vector is serialized as a float32 BLOB (4 bytes per float, little-endian). Sets the generated_at timestamp to the current time.
Returns an error if the insert or replace operation fails.
func (*Store) InsertLink ¶
InsertLink inserts a directed link between two pages. Uses INSERT OR IGNORE to silently skip duplicate links (same from_page, to_page, block_uuid triple). Returns an error if the insert query fails for reasons other than a duplicate.
func (*Store) InsertPage ¶
InsertPage inserts a new page into the store. It sets CreatedAt and UpdatedAt to the current time if they are zero. Uses parameterized queries to prevent SQL injection (FR-028).
Returns an error if the insert fails (e.g., duplicate page name violating the unique constraint).
func (*Store) InsertSource ¶
func (s *Store) InsertSource(src *SourceRecord) error
InsertSource inserts a new content source record into the store. Uses parameterized queries to prevent SQL injection (FR-028).
Returns an error if the insert fails (e.g., duplicate source ID).
func (*Store) ListPages ¶
ListPages returns all pages in the store, ordered alphabetically by name. Returns an empty slice (not nil) if no pages exist. Returns an error if the query or row scanning fails.
func (*Store) ListPagesBySource ¶ added in v1.0.0
ListPagesBySource returns all pages belonging to the given source ID, ordered alphabetically by name. Used by `dewey status` for per-source page count reporting (FR-010). Returns an empty slice if no pages belong to the source.
func (*Store) ListPagesExcludingSource ¶ added in v1.0.0
ListPagesExcludingSource returns all pages whose source_id does NOT match the given sourceID, ordered alphabetically by name. Used by LoadExternalPages() to load all non-local pages from the store into the vault's in-memory index. Returns an empty slice if no matching pages exist.
func (*Store) ListSources ¶
func (s *Store) ListSources() ([]*SourceRecord, error)
ListSources returns all content source records in the store, ordered by ID. Returns an empty slice if no sources exist. Returns an error if the query or row scanning fails.
func (*Store) SearchSimilar ¶
func (s *Store) SearchSimilar(modelID string, queryVec []float32, limit int, threshold float64) ([]SimilarityResult, error)
SearchSimilar performs brute-force cosine similarity search over all embeddings for a given model. Loads all vectors into memory, computes cosine similarity against the query vector, and returns the top-k results sorted by descending similarity score, filtered to those above the threshold.
Returns an empty slice if no embeddings match. Each returned SimilarityResult includes provenance metadata (page name, source type, indexed timestamp) for Observable Quality (Constitution III).
Returns an error if the embeddings cannot be loaded from the store.
Design decision: Brute-force was chosen over ANN indexes because Dewey indexes <10k vectors, making brute-force ~5ms (Decision 2 in research.md). The same interface can be backed by HNSW later.
func (*Store) SearchSimilarFiltered ¶
func (s *Store) SearchSimilarFiltered(modelID string, queryVec []float32, filters SearchFilters, limit int, threshold float64) ([]SimilarityResult, error)
SearchSimilarFiltered performs cosine similarity search with metadata filters. Filters are applied at the SQL level (joining embeddings → blocks → pages) before vector comparison to reduce the number of cosine computations.
Returns the top-k results sorted by descending similarity score above the threshold, constrained to embeddings matching the given filters. Returns an empty slice if no embeddings match. Returns an error if the filtered query or similarity ranking fails.
func (*Store) SetMeta ¶
SetMeta sets a metadata key-value pair, inserting a new entry or updating the value if the key already exists (upsert). Returns an error if the upsert query fails.
func (*Store) UpdateLastFetched ¶
UpdateLastFetched updates a source's last_fetched_at timestamp to the given Unix millisecond value. Returns an error if the update query fails or if no source exists with the given ID (source not found).
func (*Store) UpdatePage ¶
UpdatePage updates an existing page's mutable fields and sets UpdatedAt to the current time. The content_hash comparison enables incremental indexing — only re-index when content changes.
Returns an error if the update query fails or if no page exists with the given name (page not found).
func (*Store) UpdateSourceStatus ¶
UpdateSourceStatus updates a source's status and error message fields. Returns an error if the update query fails or if no source exists with the given ID (source not found).
Source Files