Documentation
¶
Overview ¶
Package store owns every SQL statement in Filiation.
See embed.go for the package contract and the single-writer rules. Files here are organised by table group (works, edges, chunks, jobs) rather than by operation, so the SQL touching one part of the schema stays in one place.
Package store owns every SQL statement in Filiation.
This package is the swap seam (ADR-001): replacing SQLite with Postgres for server mode means rewriting this package and nothing else. Two rules keep that true:
- store never imports internal/sources, and sources never imports store. Only internal/graph knows about both.
- No front door (cmd/, internal/mcpsrv, internal/web) may import store. They call internal/library, which calls this.
Writes are single-threaded. SQLite allows one writer, and Go makes that easy to violate by accident. M0 opens two handles: a read pool, and a write handle pinned to SetMaxOpenConns(1), with WAL and busy_timeout set.
Index ¶
- Constants
- Variables
- type DB
- func (db *DB) AuthorsOf(ctx context.Context, workIDs []string) (map[string][]model.Author, error)
- func (db *DB) CitedBy(ctx context.Context, id string) ([]model.Edge, error)
- func (db *DB) Close() error
- func (db *DB) EachEdge(ctx context.Context, includeStubs bool, fn func(from, to string) error) error
- func (db *DB) EachWork(ctx context.Context, includeStubs bool, fn func(w *model.Work) error) error
- func (db *DB) FindWork(ctx context.Context, by Lookup, value string) (*model.Work, error)
- func (db *DB) GetWork(ctx context.Context, id string) (*model.Work, error)
- func (db *DB) Migrate(ctx context.Context) error
- func (db *DB) NextFrontier(ctx context.Context, limit, maxDepth, maxRefsPerWork int, ...) ([]model.FrontierItem, error)
- func (db *DB) Path() string
- func (db *DB) References(ctx context.Context, id string) ([]model.Edge, error)
- func (db *DB) ShortestPath(ctx context.Context, from, to string, maxHops int, dir Direction) (Path, error)
- func (db *DB) Stats(ctx context.Context) (Stats, error)
- func (db *DB) StubsBeyond(ctx context.Context, maxDepth int) (int, error)
- func (db *DB) Summary(ctx context.Context) (Summary, error)
- func (db *DB) TitledWorks(ctx context.Context) ([]model.Work, error)
- func (db *DB) Tx(ctx context.Context, fn func(*Tx) error) error
- func (db *DB) UpsertWork(ctx context.Context, w *model.Work) error
- func (db *DB) Version(ctx context.Context) (int, error)
- func (db *DB) WorksCitedBy(ctx context.Context, id string) ([]model.Work, error)
- func (db *DB) WorksCiting(ctx context.Context, id string) ([]model.Work, error)
- type Direction
- type Lookup
- type Path
- type Recorded
- type Stats
- type Summary
- type Tx
- func (tx *Tx) GetWork(ctx context.Context, id string) (*model.Work, error)
- func (tx *Tx) MarkSeed(ctx context.Context, id string) error
- func (tx *Tx) MarkUnresolved(ctx context.Context, id string) error
- func (tx *Tx) MergeInto(ctx context.Context, oldID, newID string) error
- func (tx *Tx) RecordEdgesAndStubs(ctx context.Context, h *model.HydratedWork) (Recorded, error)
- func (tx *Tx) UpsertEdge(ctx context.Context, e model.Edge) error
- func (tx *Tx) UpsertStub(ctx context.Context, id string, depth *int, source model.Source) (bool, error)
- func (tx *Tx) UpsertWork(ctx context.Context, w *model.Work) error
- func (tx *Tx) WorkIDByDOI(ctx context.Context, doi string) (string, error)
Constants ¶
const ( // BusyTimeout is how long a connection waits for a lock before giving up. // // It must be set explicitly. The driver applies a one-minute default only // when the DSN carries no _pragma at all — set any other pragma and the // default is silently replaced by zero, which turns every moment of // contention into an immediate "database is locked" instead of a short // wait. Measured against v0.35.3: no pragmas gives 60000, foreign_keys // alone gives 0. BusyTimeout = 10 * time.Second )
const SchemaVersion = 1
SchemaVersion is the schema this build understands. It must match the schema_version row seeded by schema.sql.
Raise it in the same commit that changes the DDL, and add the upgrade step to Migrate. A schema change without a version bump is invisible to every guard in this file.
Variables ¶
var Schema string
Schema is the full DDL, compiled into the binary. Applied on first open and on version upgrade; see the meta table's schema_version row.
Functions ¶
This section is empty.
Types ¶
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB is the open library: one read pool and one write handle.
The split is the single-writer rule made structural (hard rule 5). SQLite permits one writer, and Go makes that trivial to violate by accident — any goroutine holding a *sql.DB can start a write. Here it cannot: writes go through a handle pinned to one connection, so concurrent writers queue on the pool instead of racing for the file lock. WAL keeps readers from blocking on any of it.
func Open ¶
Open opens the library at path, creating its directory if needed.
It does not apply the schema. Opening and migrating are separate so that a caller can inspect an existing library without changing it.
func (*DB) AuthorsOf ¶ added in v0.2.0
AuthorsOf returns the authors of each of the given works, in byline order, in one query however many works are asked about. A work with no authorship rows is absent from the map: either it has none, or it was fetched before authors were stored (v0.1), and only the caller knows which question it is asking.
func (*DB) CitedBy ¶
CitedBy lists the works in this library that cite the given one.
It is the reverse lookup, and it is as common as the forward one — "who built on this?" is the question a researcher actually asks. The idx_cites_to index exists for exactly this query.
func (*DB) EachEdge ¶
func (db *DB) EachEdge(ctx context.Context, includeStubs bool, fn func(from, to string) error) error
EachEdge calls fn for every citation, ordered by citing then cited work. With includeStubs false only edges between two fetched works are visited, so an export never refers to a node it did not write. The citing side is always fetched — only a fetched work has a known reference list — so the test is on the cited side alone.
func (*DB) EachWork ¶
EachWork calls fn for every work in the library, in ID order, one row at a time — a large library is never held in memory whole. With includeStubs false only fetched works are visited: a stub has an ID and nothing else, and in a library grown by expansion stubs outnumber fetched works ten to one.
An error from fn stops the walk and is returned.
func (*DB) FindWork ¶
FindWork returns the work in this library whose column matches value, or errs.ErrNotFound. It looks only at the library — never at OpenAlex — which is what read-only commands like neighbours and path want: a paper not added yet has no neighbours to show.
func (*DB) GetWork ¶
GetWork reads one work by its OpenAlex ID, from the read pool.
Authors are not loaded: Work.Authors stays nil, which is how a caller tells "this query did not ask for authors" from "this work has none". It returns a stub as readily as a hydrated work — stubs are the common case, not the exception, and a read path that treated one as missing would hide most of the graph.
func (*DB) Migrate ¶
Migrate brings the library up to SchemaVersion, or refuses to touch it.
It is idempotent: every statement in schema.sql is IF NOT EXISTS or OR IGNORE, so running it against a current library changes nothing.
The whole thing runs in one transaction on the write handle, so a library is either fully migrated or untouched — a half-applied schema is not a state anything downstream should have to handle. The version check happens inside that transaction and before any DDL, so a refusal writes nothing at all.
func (*DB) NextFrontier ¶
func (db *DB) NextFrontier(ctx context.Context, limit, maxDepth, maxRefsPerWork int, exclude map[string]bool) ([]model.FrontierItem, error)
NextFrontier returns up to limit stubs worth hydrating next, best first.
Best is in-graph in-degree: how many works already in this library cite the stub (§4). A paper five of your papers cite is a better use of budget than one a single paper cites, and the signal is free — it is computed from edges already stored, with no API call. Ties go to the shallower work, then to the ID, so the order is fully deterministic; resume depends on that.
A stub is a candidate only if it is:
- not hydrated and not unresolved (OpenAlex has no record — stop spending budget on it, §7);
- at a known depth no greater than maxDepth. A stub with no depth was reached by no route this library can measure, and hydrating it would grow the graph from nowhere;
- within the first maxRefsPerWork references of at least one work citing it. A review article citing eight hundred papers has all eight hundred edges recorded — they arrived free — but may put only its first maxRefsPerWork forward as candidates, or it would spend the budget alone (§4). References keep the order OpenAlex sent them in, which is insertion order in cites. A stub any other work ranks within the cap stays a candidate.
maxRefsPerWork <= 0 means no cap. exclude lists IDs to leave out even though they qualify — the batch that just failed, so one bad batch cannot be retried in a loop within a run (§7).
func (*DB) Path ¶
Path is where this library lives on disk. The CLI prints it on first run, because the default location differs on every OS (ADR-006).
func (*DB) References ¶
References lists the works this one cites, nearest thing first in ID order so the result is stable between runs. It returns the edges, not the works: the targets are mostly stubs, and a caller that wants them can ask.
func (*DB) ShortestPath ¶
func (db *DB) ShortestPath(ctx context.Context, from, to string, maxHops int, dir Direction) (Path, error)
ShortestPath finds the fewest citation steps from one work to another, within maxHops, or reports errs.ErrNotFound if there is none.
It is a breadth-first search: one set-based query per step fetches every edge leaving the current frontier, and a visited set means each work is reached once. That is what keeps it safe on a graph with cycles (§7: the graph is not a DAG) — and fast.
It is deliberately not a single recursive CTE, which CLAUDE.md's stack table named for traversal. A recursive CTE has no shared visited set: guarding cycles with a path string, it enumerates every path rather than every work. Measured on a real 10,049-edge library (24 Sept 2026): 1.5 s to depth 4, and 20.8 million rows in 125 s to depth 5 — to reach the same 6,555 works. Each query here is plain SQL; the ID list travels as one JSON array parameter, which Postgres handles with jsonb_array_elements_text or ANY.
Ties between equally short paths are broken by ID order, so the same library always gives the same answer.
func (*DB) StubsBeyond ¶
StubsBeyond counts stubs deeper than maxDepth that are otherwise candidates. When the frontier is empty, it tells "the graph ends here" apart from "the depth limit stopped us" — model.StopFrontierEmpty against model.StopMaxDepth, which the CLI answers differently.
func (*DB) TitledWorks ¶
TitledWorks returns every hydrated work that has a title, with only the columns needed to tell one record from another: ID, title, year, type, DOI.
It exists for duplicate detection, which is policy and so lives in internal/graph (D14). This package only fetches the rows.
func (*DB) Tx ¶
Tx runs fn inside one write transaction, committing if it returns nil and rolling back otherwise.
The transaction begins as BEGIN IMMEDIATE (see writeDSN), so it waits for the write lock up front, where busy_timeout can wait on it, rather than discovering it is unavailable partway through.
An error from fn is returned unwrapped. Callers branch on the sentinels in internal/errs, and adding a "store: transaction" prefix here would bury the message that names what actually failed without telling anyone anything new.
func (*DB) UpsertWork ¶
UpsertWork runs UpsertWork in a transaction of its own. It is the one-work convenience; a batch should open one transaction and reuse it.
func (*DB) Version ¶
Version reports the schema version recorded in the library, or 0 if it has never been migrated.
func (*DB) WorksCitedBy ¶
WorksCitedBy returns the works id cites: its references, fetched ones first, newest first within that. WorksCiting is the reverse — the works in this library that cite id.
func (*DB) WorksCiting ¶
WorksCiting returns the works in this library that cite id. It can only see citations from works whose references have been fetched: a stub's own reference list is unknown, so "cited by 3" means "by 3 papers in your library", never "by 3 papers in the world".
type Direction ¶
type Direction int
Direction is which way a path may follow citations.
const ( // Backward follows references: from a paper to what it cites, and on to // what that cites. It is lineage — where an idea came from — and the // direction the project is named for. Backward Direction = iota // Forward follows citations the other way: from a paper to what cites it. Forward // Either ignores direction, finding any chain of citations connecting two // papers. Either )
type Path ¶
type Path struct {
IDs []string
// Cites[i] reports that IDs[i] cites IDs[i+1]; false means IDs[i+1] cites
// IDs[i]. It has one entry fewer than IDs.
Cites []bool
}
Path is a chain of works connected by citations.
type Recorded ¶
type Recorded struct {
// Refs is how many references were considered, after self-citations and
// repeated IDs were dropped.
Refs int
// Stubs is how many works this call introduced to the library — references
// pointing at papers nothing had cited yet. The rest were already there,
// which is deduplication happening rather than deduplication failing.
Stubs int
// Edges is how many citation edges were new. Re-running against the same
// work records none, which is what makes add and expand idempotent.
Edges int
}
Recorded is what one call to RecordEdgesAndStubs wrote. It feeds the counters on model.ExpansionResult, which is a report and not an error type.
func (Recorded) DeadEnd ¶
DeadEnd reports whether this work cannot extend the graph: its references were fetched and there were none.
This is not a defect. Reference coverage is a property of the field — 6% dead ends in medicine against 84% in arts and humanities (D12) — and it has to be surfaced per node from day one, exactly as OA status is, or a graph that stops after one hop reads as a broken tool rather than as missing data.
type Stats ¶
type Stats struct {
Works int // every row in work, stubs included
Stubs int // of those, the ones never hydrated
Edges int // rows in cites
}
Stats counts what the library holds. The CLI prints it after add and expand, and the stub count is the honest half of that report: it is how much of the graph is still only an identifier.
type Summary ¶
type Summary struct {
Works int // every row in work
Hydrated int // metadata fetched
Stubs int // known only by ID, still to fetch
Unresolved int // of the stubs, those OpenAlex has no record of
Seeds int // works the user added
Edges int // citations
// DeadEnds counts hydrated works whose references were fetched and turned
// out to be none: the graph cannot grow through them. It is the per-library
// view of D12's reference coverage, computed from the edges already stored.
DeadEnds int
}
Summary is the library at a glance: what `fil stats` prints.
type Tx ¶
type Tx struct {
// contains filtered or unexported fields
}
Tx is an open write transaction, and the only way anything writes to a library.
It exists so the single-writer rule (hard rule 5) is structural rather than remembered. The write handle is unexported and pinned to one connection, so a caller cannot start a write without coming through DB.Tx, and two goroutines that try queue on the pool instead of racing for the file lock.
The unit of work is deliberately the caller's, not the statement's. Hydrating a work, writing its edges and creating the stubs they point at is one transaction: the citation edges reference work rows that the same call creates, so a partial commit would mean either an edge with no target or a work claiming fetched_refs it never wrote. §4 sizes that unit at one batch — Ctrl-C loses at most one batch of progress and never a half-written graph.
func (*Tx) GetWork ¶
GetWork reads one work inside this transaction, so it sees writes the transaction has made but not yet committed.
func (*Tx) MarkSeed ¶
MarkSeed makes id a seed at depth 0. It is for a seed that turned out to be a duplicate of a record already in the library: the flags belong on the record that survives.
func (*Tx) MarkUnresolved ¶
MarkUnresolved records that OpenAlex has no record of a work some other paper cited, so expansion stops spending budget on it.
The citation edge stays. It remains true that the paper was cited, whatever the index knows, and deleting the edge to tidy up would silently falsify the graph (§7). The row stays a stub: unresolved is not hydration.
func (*Tx) MergeInto ¶
MergeInto folds the work oldID into newID: OpenAlex merged two records for one paper and now answers for the old ID with the new one.
Without this the same paper sits in the graph twice — once as the stub some citing work pointed at, once as the hydrated survivor — which is exactly the quiet rot hard rule 6 exists to prevent. The fold:
- creates newID as a stub if it is not already present, carrying oldID's depth and provenance, and otherwise keeps the shallower depth and a sticky is_seed;
- moves every edge, in both directions, from oldID to newID. An edge that would already exist, or would become a self-citation, is dropped;
- deletes oldID, which cascades to anything still attached to it.
Call it before hydrating newID in the same transaction. A merged-away record can hold a DOI the survivor is about to be written with, and the UNIQUE index on work.doi would refuse the survivor while the old row still held it.
Merging an ID into itself, or one not in the library, does nothing.
func (*Tx) RecordEdgesAndStubs ¶
RecordEdgesAndStubs writes a hydrated work's reference list: one stub per referenced work, one edge per reference, and fetched_refs on the citing work.
This is the call the whole ingestion path is built around, and it must be one transaction. The edges reference work rows this same call creates, so a partial commit leaves either an edge with no target or a work claiming fetched_refs it never wrote. Both are states nothing downstream should have to handle, and neither is visible to the user until the graph looks wrong.
The citing work must already exist — call UpsertWork first. The fetched_refs update doubles as that check and returns errs.ErrNotFound if it matches nothing, which is a better failure than the foreign key error the first edge would otherwise raise.
Every reference is recorded, including a review article's eight hundred. They arrive free inside the response already paid for, and an edge is ground truth (hard rule 2) that nothing here is entitled to discard. §4's cap on how many of them may compete for the budget belongs to the frontier query in M1, where it can be applied per citing work without throwing away what was measured.
func (*Tx) UpsertEdge ¶
UpsertEdge writes one edge with whatever has been learned about it, and enriches an edge already present rather than replacing it.
An edge is recorded long before anything is known about why it was made: the row is written during expansion from OpenAlex's resolved reference list, and the context sentence arrives in M3 once the citing paper's PDF has been read, the intent in M6. So a second write must fill in what it knows and leave everything else alone — a call carrying only a context sentence must not erase an intent something else took the trouble to classify.
Both works must already exist. Edges point at real nodes because the stub exists the moment the edge does (§4), and the foreign keys are enforced on every connection (spike 5) to keep that true.
func (*Tx) UpsertStub ¶
func (tx *Tx) UpsertStub(ctx context.Context, id string, depth *int, source model.Source) (bool, error)
UpsertStub creates a work row holding nothing but an identifier, and reports whether it had to create it.
This is what makes deduplication free (§4): the target of a citation gets a row the moment the edge is written, so "connect to the existing node if the paper is already there" is a primary key conflict rather than matching logic that can be got wrong.
It never touches a row that already exists, with one exception: depth is lowered if this route to the work is shorter than the one already recorded. Expansion is best-first, so a work can be reached the long way round first, and leaving the longer distance in place would misreport how far from the seed the user's graph actually reaches.
func (*Tx) UpsertWork ¶
UpsertWork writes a hydrated work, creating the row if this library has never seen it and filling in the metadata if it was already there as a stub.
Hydration is an update, not a replacement, because a work is almost always already present when it is hydrated — a stub was created the moment some other paper was seen to cite it (§4). Three columns are therefore preserved rather than overwritten, and losing any of them would be a real loss:
- depth keeps the smaller of the two. It is hops from the nearest seed, and a work reached again by a longer route has not moved further away.
- is_seed is sticky. A work the user named stays named even when expansion reaches it again from somewhere else.
- source records how the work first entered the library and is never rewritten, so "this arrived through expansion" survives a later re-fetch.
Everything else is taken from w, which is treated as the authoritative record. The PDF columns are left alone entirely: they belong to M3, and nothing in the metadata path knows anything about them.
func (*Tx) WorkIDByDOI ¶
WorkIDByDOI returns the ID of the work in this library that holds doi, or errs.ErrNotFound. The DOI must already be normalised; identity.NormaliseDOI is what every write path uses.
OpenAlex does not always merge its own duplicates: the same arXiv preprint has been seen under two work IDs with one DOI (W2949614626 and W4294576234, 24 Sept 2026). The UNIQUE index on work.doi refuses the second, and this is how a caller finds the first to fold it into.
Source Files