store

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 30, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package store owns every SQL statement in Filiation.

See embed.go for the package contract and the single-writer rules. Files here are organised by table group (works, edges, chunks, jobs) rather than by operation, so the SQL touching one part of the schema stays in one place.

Package store owns every SQL statement in Filiation.

This package is the swap seam (ADR-001): replacing SQLite with Postgres for server mode means rewriting this package and nothing else. Two rules keep that true:

  • store never imports internal/sources, and sources never imports store. Only internal/graph knows about both.
  • No front door (cmd/, internal/mcpsrv, internal/web) may import store. They call internal/library, which calls this.

Writes are single-threaded. SQLite allows one writer, and Go makes that easy to violate by accident. M0 opens two handles: a read pool, and a write handle pinned to SetMaxOpenConns(1), with WAL and busy_timeout set.

Index

Constants

View Source 
const (
	OutcomeOK        = "ok"
	OutcomeDuplicate = "duplicate"
)

OutcomeOK and OutcomeDuplicate are the attempt outcomes store itself reads. A duplicate is a download whose bytes the library already holds for another work — a publisher's "access denied" PDF served for many papers is the usual cause — and counts as a failure of that link.

View Source 
const (
	OriginDownload = "download"
	OriginImport   = "import"
	OriginFound    = "found"
)

File origins.

View Source 
const (
	ImportImported      = "imported"
	ImportDuplicate     = "duplicate"       // the library already holds these bytes
	ImportUnconfirmed   = "unconfirmed"     // an identifier resolved, but its title is not in the PDF
	ImportCandidates    = "candidates"      // only a title search's candidates; the user chooses
	ImportUnidentified  = "unidentified"    // nothing to go on: a scan, or no identifier and no title
	ImportNotInOpenAlex = "not-in-openalex" // a title search found nothing; may become a local document
	ImportUnreadable    = "unreadable"      // PDFium could not open it
)

Import outcomes. Only OutcomeImported files a PDF; the rest say why not.

View Source 
const (
	IdentityMatch    = "match"    // its first pages carry the paper's title and author
	IdentityMismatch = "mismatch" // they do not: another paper, or another edition
)

File identity, as checked against the paper it is filed under.

View Source 
const (
	MarkStart = "\x02"
	MarkEnd   = "\x03"
)

MarkStart and MarkEnd surround the matched words in a Hit's Snippet. They are control characters, which cleaned text never contains, so a front door can render them as it likes without mistaking a bracket for a match.

View Source 
const (
	// BusyTimeout is how long a connection waits for a lock before giving up.
	//
	// It must be set explicitly. The driver applies a one-minute default only
	// when the DSN carries no _pragma at all — set any other pragma and the
	// default is silently replaced by zero, which turns every moment of
	// contention into an immediate "database is locked" instead of a short
	// wait. Measured against v0.35.3: no pragmas gives 60000, foreign_keys
	// alone gives 0.
	BusyTimeout = 10 * time.Second
)
View Source 
const LinkSourceOpenAlex = "openalex"

LinkSourceOpenAlex marks links that came from OpenAlex's locations.

View Source 
const SchemaVersion = 9

SchemaVersion is the schema this build understands. It must match the schema_version row seeded by schema.sql.

Raise it in the same commit that changes the DDL, and add the upgrade step to upgrades. A schema change without a version bump is invisible to every guard in this file. 

1  v0.1, v0.2: work keyed on openalex_id
2  M3: work keyed on fil_id (D17), work_alias; pdf_sha256 and pdf_license gone
3  M3: work_folder, work_file and folder_collection — the papers/ folder (D18)
4  M3: work_pdf and work_location — open-access PDF links (ADR-013)
5  M3: download_attempt — every download, and why it failed (ADR-013)
6  M3: work_text, and chunk offsets — the text of each PDF, or why there is none
7  M3: import_attempt — what importing each of the user's PDFs gave (ADR-014)
8  M3: mention — every citation found in a PDF, with its sentence (ADR-015)
9  M3: work_file.identity — whether a downloaded PDF is the paper it is filed under

Variables

View Source 
var Schema string

Schema is the full DDL, compiled into the binary. Applied on first open and on version upgrade; see the meta table's schema_version row.

Functions

func MatchQuery added in v0.3.0 

func MatchQuery(typed string) (string, error)

MatchQuery turns what a person typed into an FTS5 query that cannot be a syntax error: every word, and every "quoted phrase", becomes a quoted FTS5 string, and all must match. A word ending in * matches as a prefix. FTS5's own operators — AND, OR, NOT, NEAR, column filters, parentheses — are not offered: typed by someone who does not know them, they fail or surprise.

Types

type Attempt added in v0.3.0 

type Attempt struct {
	URL     string
	Host    string
	Outcome string // pdfs.Outcome, or "duplicate"
	Status  int
	Bytes   int64
	Millis  int64
	At      time.Time
}

Attempt is one row of download_attempt.

type Chunk added in v0.3.0 

type Chunk struct {
	Page       int
	Ordinal    int
	Section    string // "references", or "" for the body
	Start, End int    // byte offsets in the page's cleaned text
	Text       string
}

Chunk is one passage of a paper's text.

type CitedRef added in v0.3.0 

type CitedRef struct {
	FilID   string
	Title   string // "" when not known: a stub never detailed
	Year    int    // 0 when not known
	DOI     string
	Surname string // the first author's, "" when not known
	Authors int    // how many authors, 0 when not known
}

CitedRef is one work a paper cites, with what citation context needs to recognise it in the paper's text.

type CollectionSize added in v0.3.0 

type CollectionSize struct {
	Name       string
	Works      int
	FromFolder bool
}

CollectionSize is one line of a collection listing.

type ContextCounts added in v0.3.0 

type ContextCounts struct {
	Citations   int // edges from papers with text
	WithContext int
	Mentions    int
}

ContextCounts is how far citation context reaches: of the citations made by papers whose PDF has text, how many carry a sentence.

type Counts added in v0.3.0 

type Counts struct {
	Works, Cites, Files, Notes int
}

Counts are the row counts Check compares.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB is the open library: one read pool and one write handle.

The split is the single-writer rule made structural (hard rule 5). SQLite permits one writer, and Go makes that trivial to violate by accident — any goroutine holding a *sql.DB can start a write. Here it cannot: writes go through a handle pinned to one connection, so concurrent writers queue on the pool instead of racing for the file lock. WAL keeps readers from blocking on any of it.

func Open 

func Open(ctx context.Context, path string) (*DB, error)

Open opens the library at path, creating its directory if needed.

It does not apply the schema. Opening and migrating are separate so that a caller can inspect an existing library without changing it.

func (*DB) AttemptsOf added in v0.3.0 

func (db *DB) AttemptsOf(ctx context.Context, ref string) ([]Attempt, error)

AttemptsOf returns a work's download attempts, newest first.

func (*DB) AuthorsOf added in v0.2.0 

func (db *DB) AuthorsOf(ctx context.Context, workIDs []string) (map[string][]model.Author, error)

AuthorsOf returns the authors of each of the given works, in byline order, in one query however many works are asked about. A work may be named by its fil ID or its OpenAlex ID, and the map is keyed by the ID the caller gave.

A work with no authorship rows is absent from the map: either it has none, or it was fetched before authors were stored (v0.1), and only the caller knows which question it is asking.

func (*DB) Check added in v0.3.0 

func (db *DB) Check(ctx context.Context) (Counts, error)

Check runs SQLite's integrity check and reports the row counts that a copy must reproduce.

func (*DB) CitedBy 

func (db *DB) CitedBy(ctx context.Context, id string) ([]model.Edge, error)

CitedBy lists the works in this library that cite the given one.

It is the reverse lookup, and it is as common as the forward one — "who built on this?" is the question a researcher actually asks. The idx_cites_to index exists for exactly this query.

func (*DB) Close 

func (db *DB) Close() error

Close closes both handles, reporting every failure rather than the first.

func (*DB) Collections added in v0.3.0 

func (db *DB) Collections(ctx context.Context) ([]CollectionSize, error)

Collections returns every collection's name and member count, in name order.

func (*DB) ContextCounts added in v0.3.0 

func (db *DB) ContextCounts(ctx context.Context) (ContextCounts, error)

ContextCounts counts citation context across the library.

func (*DB) CopyTo added in v0.3.0 

func (db *DB) CopyTo(ctx context.Context, path string) error

CopyTo writes a consistent copy of the whole library to path, which must not exist. VACUUM INTO reads one snapshot, so it includes what is still in the write-ahead log and is safe while readers are open.

func (*DB) EachEdge 

func (db *DB) EachEdge(ctx context.Context, includeStubs bool, fn func(from, to string) error) error

EachEdge calls fn for every citation, as a pair of fil IDs, ordered by citing then cited work. With includeStubs false only edges between two fetched works are visited, so an export never refers to a node it did not write. The citing side is always fetched — only a fetched work has a known reference list — so the test is on the cited side alone.

func (*DB) EachWork 

func (db *DB) EachWork(ctx context.Context, includeStubs bool, fn func(w *model.Work) error) error

EachWork calls fn for every work in the library, in OpenAlex ID order (then fil ID, for works OpenAlex does not know), one row at a time — a large library is never held in memory whole. With includeStubs false only fetched works are visited: a stub has an ID and nothing else, and in a library grown by expansion stubs outnumber fetched works ten to one.

An error from fn stops the walk and is returned.

func (*DB) Files added in v0.3.0 

func (db *DB) Files(ctx context.Context) ([]File, error)

Files returns every file the library knows, in path order.

func (*DB) FilesOf added in v0.3.0 

func (db *DB) FilesOf(ctx context.Context, filID string) ([]File, error)

FilesOf returns the files of one work, found ones first.

func (*DB) FilesWithHash added in v0.3.0 

func (db *DB) FilesWithHash(ctx context.Context, sha256 string) ([]File, error)

FilesWithHash returns the files holding exactly these bytes, which is how a download or an import learns it already has a paper.

func (*DB) FindWork 

func (db *DB) FindWork(ctx context.Context, by Lookup, value string) (*model.Work, error)

FindWork returns the work in this library whose column matches value, or errs.ErrNotFound. It looks only at the library — never at OpenAlex — which is what read-only commands like neighbours and path want: a paper not added yet has no neighbours to show.

A fil ID or OpenAlex ID also finds a work by an alias it left behind when it was folded into another (D17).

func (*DB) Folders added in v0.3.0 

func (db *DB) Folders(ctx context.Context) ([]Folder, error)

Folders returns every work folder the library knows, in path order.

func (*DB) FoldersOf added in v0.3.0 

func (db *DB) FoldersOf(ctx context.Context, filID string) ([]Folder, error)

FoldersOf returns the folders of one work, found ones first.

func (*DB) GetWork 

func (db *DB) GetWork(ctx context.Context, ref string) (*model.Work, error)

GetWork reads one work, from the read pool. ref is any ID the work has or had: its fil ID, its OpenAlex ID, or an alias left by a fold (D17).

Authors are not loaded: Work.Authors stays nil, which is how a caller tells "this query did not ask for authors" from "this work has none". It returns a stub as readily as a hydrated work — stubs are the common case, not the exception, and a read path that treated one as missing would hide most of the graph.

func (*DB) ImportOf added in v0.3.0 

func (db *DB) ImportOf(ctx context.Context, sha256 string) (ImportAttempt, bool, error)

ImportOf returns what importing the file with this hash last gave; false if it was never tried.

func (*DB) LinksToTry added in v0.3.0 

func (db *DB) LinksToTry(ctx context.Context, ref string, retryAfter time.Duration) ([]model.Location, error)

LinksToTry returns a work's PDF links that have not failed since retryAfter ago, best first. A retryAfter of zero returns every link — what a person asking for one paper by name gets.

func (*DB) List added in v0.3.0 

func (db *DB) List(ctx context.Context, q ListQuery) ([]ListRow, error)

List lists the library's papers: seeds first, then the most cited within the library, then the order they arrived.

func (*DB) MentionsOf added in v0.3.0 

func (db *DB) MentionsOf(ctx context.Context, ref string) ([]Mention, error)

MentionsOf returns the mentions in a paper, in reading order.

func (*DB) Migrate 

func (db *DB) Migrate(ctx context.Context) error

Migrate brings the library up to SchemaVersion, or refuses to touch it.

A new library gets schema.sql. An older one is carried forward through upgrades, one step at a time. A current one is left alone, so running this on every open costs one query.

The whole thing runs in one transaction on the write handle, so a library is either fully migrated or untouched — a half-applied schema is not a state anything downstream should have to handle. The version check happens inside that transaction and before any DDL, so a refusal writes nothing at all.

func (*DB) NextFrontier 

func (db *DB) NextFrontier(ctx context.Context, limit, maxDepth, maxRefsPerWork int, exclude map[string]bool) ([]model.FrontierItem, error)

NextFrontier returns up to limit stubs worth hydrating next, best first.

Best is in-graph in-degree: how many works already in this library cite the stub (§4). A paper five of your papers cite is a better use of budget than one a single paper cites, and the signal is free — it is computed from edges already stored, with no API call. Ties go to the shallower work, then to the ID, so the order is fully deterministic; resume depends on that.

A stub is a candidate only if it is:

  • not hydrated and not unresolved (OpenAlex has no record — stop spending budget on it, §7);
  • at a known depth no greater than maxDepth. A stub with no depth was reached by no route this library can measure, and hydrating it would grow the graph from nowhere;
  • within the first maxRefsPerWork references of at least one work citing it. A review article citing eight hundred papers has all eight hundred edges recorded — they arrived free — but may put only its first maxRefsPerWork forward as candidates, or it would spend the budget alone (§4). References keep the order OpenAlex sent them in, which is insertion order in cites. A stub any other work ranks within the cap stays a candidate.

maxRefsPerWork <= 0 means no cap. exclude lists IDs to leave out even though they qualify — the batch that just failed, so one bad batch cannot be retried in a loop within a run (§7).

func (*DB) PDFCounts added in v0.3.0 

func (db *DB) PDFCounts(ctx context.Context) (map[model.PDFAvailability]int, error)

PDFCounts counts works by what is known about their PDF. Works never looked up count under PDFUnchecked.

func (*DB) PDFLinksOf added in v0.3.0 

func (db *DB) PDFLinksOf(ctx context.Context, ref string) (model.PDFLinks, time.Time, error)

PDFLinksOf returns what is known about a work's open-access PDF, and when it was looked up. A work never looked up has PDFUnchecked and a zero time, unless expansion already found OpenAlex has no record of it.

func (*DB) PDFToRead added in v0.3.0 

func (db *DB) PDFToRead(ctx context.Context, ref string) (ToRead, bool, error)

PDFToRead returns the PDF a paper's text comes from; false if it has none on disk.

func (*DB) PDFsOnDisk added in v0.3.0 

func (db *DB) PDFsOnDisk(ctx context.Context) (int, error)

PDFsOnDisk counts the works with at least one PDF present on disk.

func (*DB) PDFsToRead added in v0.3.0 

func (db *DB) PDFsToRead(ctx context.Context, extractor string, again bool) ([]ToRead, error)

PDFsToRead returns the papers whose PDF has not been read: never read, changed since (another hash), or read by an older extractor. With again, every paper with a PDF on disk. Seeds come first.

func (*DB) Path 

func (db *DB) Path() string

Path is where this library lives on disk. The CLI prints it on first run, because the default location differs on every OS (ADR-006).

func (*DB) References 

func (db *DB) References(ctx context.Context, id string) ([]model.Edge, error)

References lists the works this one cites, in fil ID order so the result is stable between runs. It returns the edges, not the works: the targets are mostly stubs, and a caller that wants them can ask. The work may be named by any of its IDs; the edges carry fil IDs.

func (*DB) RefsOf added in v0.3.0 

func (db *DB) RefsOf(ctx context.Context, ref string) ([]CitedRef, error)

RefsOf returns the works ref cites, as OpenAlex gave them.

func (*DB) ResolveRefs added in v0.3.0 

func (db *DB) ResolveRefs(ctx context.Context, refs []string) (map[string]string, error)

ResolveRefs maps each ID to the fil ID of the work it names — its own fil ID, OpenAlex ID or an alias — leaving out IDs that name nothing. A scan uses it to read the fil IDs in folder names, some of which a fold has turned into aliases.

func (*DB) Search added in v0.3.0 

func (db *DB) Search(ctx context.Context, q SearchQuery) ([]Hit, error)

Search finds passages by keyword, best first (FTS5's bm25), with words matched by stem — "citing" finds "cited".

func (*DB) ShortestPath 

func (db *DB) ShortestPath(ctx context.Context, from, to string, maxHops int, dir Direction) (Path, error)

ShortestPath finds the fewest citation steps from one work to another, within maxHops, or reports errs.ErrNotFound if there is none.

It is a breadth-first search: one set-based query per step fetches every edge leaving the current frontier, and a visited set means each work is reached once. That is what keeps it safe on a graph with cycles (§7: the graph is not a DAG) — and fast.

It is deliberately not a single recursive CTE, which CLAUDE.md's stack table named for traversal. A recursive CTE has no shared visited set: guarding cycles with a path string, it enumerates every path rather than every work. Measured on a real 10,049-edge library (24 Sept 2026): 1.5 s to depth 4, and 20.8 million rows in 125 s to depth 5 — to reach the same 6,555 works. Each query here is plain SQL; the ID list travels as one JSON array parameter, which Postgres handles with jsonb_array_elements_text or ANY.

Ties between equally short paths are broken by OpenAlex ID order, which is the same in every library, so the same graph always gives the same answer.

from and to may be any IDs the two works have; the path is of fil IDs.

func (*DB) Stats 

func (db *DB) Stats(ctx context.Context) (Stats, error)

Stats reads the library's counts in one query.

func (*DB) StubsBeyond 

func (db *DB) StubsBeyond(ctx context.Context, maxDepth int) (int, error)

StubsBeyond counts stubs deeper than maxDepth that are otherwise candidates. When the frontier is empty, it tells "the graph ends here" apart from "the depth limit stopped us" — model.StopFrontierEmpty against model.StopMaxDepth, which the CLI answers differently.

func (*DB) StubsCitedBy added in v0.3.0 

func (db *DB) StubsCitedBy(ctx context.Context, filIDs []string) ([]string, error)

StubsCitedBy returns the OpenAlex IDs of the works the given papers cite that are stubs with no title yet: what citation context needs details of.

func (*DB) Summary 

func (db *DB) Summary(ctx context.Context) (Summary, error)

Summary counts the library in one query.

func (*DB) TextCounts added in v0.3.0 

func (db *DB) TextCounts(ctx context.Context) (map[string]int, error)

TextCounts counts the papers by what reading their PDF gave, keyed by status.

func (*DB) TextOf added in v0.3.0 

func (db *DB) TextOf(ctx context.Context, ref string) (Text, bool, error)

TextOf returns what reading a paper's PDF gave; false if it was never read.

func (*DB) TitledWorks 

func (db *DB) TitledWorks(ctx context.Context) ([]model.Work, error)

TitledWorks returns every hydrated work that has a title, with only the columns needed to tell one record from another: IDs, title, year, type, DOI.

It exists for duplicate detection, which is policy and so lives in internal/graph (D14). This package only fetches the rows.

func (*DB) ToLocate added in v0.3.0 

func (db *DB) ToLocate(ctx context.Context, limit int, maxAge time.Duration) ([]string, error)

ToLocate returns the OpenAlex IDs of up to limit works whose PDF links are not known, or were looked up longer ago than maxAge, in the order fetching would want them: the user's own papers first, then the works the library cites most (ADR-013's ranking, the same signal expansion uses). Works OpenAlex has no record of are left out; there is nothing to ask about them.

func (*DB) Tx 

func (db *DB) Tx(ctx context.Context, fn func(*Tx) error) error

Tx runs fn inside one write transaction, committing if it returns nil and rolling back otherwise.

The transaction begins as BEGIN IMMEDIATE (see writeDSN), so it waits for the write lock up front, where busy_timeout can wait on it, rather than discovering it is unavailable partway through.

An error from fn is returned unwrapped. Callers branch on the sentinels in internal/errs, and adding a "store: transaction" prefix here would bury the message that names what actually failed without telling anyone anything new.

func (*DB) UpsertWork 

func (db *DB) UpsertWork(ctx context.Context, w *model.Work) error

UpsertWork runs UpsertWork in a transaction of its own. It is the one-work convenience; a batch should open one transaction and reuse it.

func (*DB) Version 

func (db *DB) Version(ctx context.Context) (int, error)

Version reports the schema version recorded in the library, or 0 if it has never been migrated.

func (*DB) WithoutText added in v0.3.0 

func (db *DB) WithoutText(ctx context.Context) ([]Text, error)

WithoutText lists the papers whose PDF gave no searchable text, with the status and the reason, grouped by status.

func (*DB) WorksCitedBy 

func (db *DB) WorksCitedBy(ctx context.Context, id string) ([]model.Work, error)

WorksCitedBy returns the works id cites: its references, fetched ones first, newest first within that. WorksCiting is the reverse — the works in this library that cite id.

func (*DB) WorksCiting 

func (db *DB) WorksCiting(ctx context.Context, id string) ([]model.Work, error)

WorksCiting returns the works in this library that cite id. It can only see citations from works whose references have been fetched: a stub's own reference list is unknown, so "cited by 3" means "by 3 papers in your library", never "by 3 papers in the world".

func (*DB) WorksToFetch added in v0.3.0 

func (db *DB) WorksToFetch(ctx context.Context, limit int, retryAfter time.Duration) ([]ToFetch, error)

WorksToFetch returns up to limit works to download, in the order fetching wants them: the user's own papers first, then the works the library cites most (ADR-013). A work qualifies when it has a PDF link, no PDF recorded, and at least one link that has not failed since retryAfter ago. Each comes with those links, best first.

type Direction 

type Direction int

Direction is which way a path may follow citations. 

const (
	// Backward follows references: from a paper to what it cites, and on to
	// what that cites. It is lineage — where an idea came from — and the
	// direction the project is named for.
	Backward Direction = iota
	// Forward follows citations the other way: from a paper to what cites it.
	Forward
	// Either ignores direction, finding any chain of citations connecting two
	// papers.
	Either
)

type File added in v0.3.0 

type File struct {
	Path    string // relative to the library folder
	FilID   string
	SHA256  string
	Size    int64
	ModTime int64  // Unix seconds when it was hashed
	Version string // published | accepted | preprint, or empty
	Origin  string // download | import | found
	Missing bool
	// Identity is IdentityMatch or IdentityMismatch once the file has been
	// checked against its paper, "" before. Read only: written by
	// SetIdentity, never by PutFile.
	Identity string
}

File is a row of work_file: a file that belongs to a work.

type Folder added in v0.3.0 

type Folder struct {
	Path      string // relative to the library folder
	FilID     string
	GivenName string // the name fil gave it; empty when a person named it
	Missing   bool
}

Folder is a row of work_folder: a folder in papers/ named for one work.

type FolderCollection added in v0.3.0 

type FolderCollection struct {
	Name  string
	Works []string // fil IDs
}

FolderCollection is a collection as a folder defines it.

type Hit added in v0.3.0 

type Hit struct {
	FilID   string
	Page    int
	Section string
	// Snippet is the matching stretch of the passage, with each matched word
	// between MarkStart and MarkEnd.
	Snippet string
	Path    string  // the PDF it came from, relative to the library folder
	Rank    float64 // bm25: lower is better
}

Hit is one passage that matched.

type ImportAttempt added in v0.3.0 

type ImportAttempt struct {
	SHA256   string
	Path     string
	Outcome  string
	Reason   string
	FilID    string // the work it was filed under, when it was
	Searched bool   // a title search was spent on it
	At       time.Time
}

ImportAttempt is one row of import_attempt.

type ListQuery added in v0.3.0 

type ListQuery struct {
	Seeds       bool   // only the papers the user added or imported
	WithPDF     bool   // only papers with a PDF on disk
	WithoutText bool   // only papers whose PDF gave no searchable text
	Collection  string // only this collection's papers
	Title       string // only titles containing all these words
	Limit       int    // zero means no limit
}

ListQuery filters and orders a listing of the library.

type ListRow added in v0.3.0 

type ListRow struct {
	FilID      string
	OpenAlexID string
	Title      string
	Year       int
	Author     string // first author's name, "" when not known
	Seed       bool
	Hydrated   bool
	CitedBy    int    // in this library
	HasPDF     bool   // a PDF on disk
	TextStatus string // "", ok, no-text, gibberish, unreadable, timed-out
	Contexts   int    // references cited in a sentence fil found
	Mismatch   bool   // its downloaded PDF names another paper or edition
}

ListRow is one paper in a listing, with its state in one row: what the list shows beside each title.

type Lookup 

type Lookup int

Lookup names the column a work is found by in the library. 

const (
	ByOpenAlexID Lookup = iota
	ByDOI
	ByArXivID
	ByPMID
	ByFilID
)

type Mention added in v0.3.0 

type Mention struct {
	ToWork     string // fil ID of the cited work
	Page       int
	Start, End int // byte offsets of Sentence in its page's cleaned text
	Sentence   string
	Section    string
	Marker     string
	Method     string // numeric | author-year
	Confidence float64
}

Mention is one place a paper cites another.

type Path 

type Path struct {
	IDs []string // fil IDs

	// Cites[i] reports that IDs[i] cites IDs[i+1]; false means IDs[i+1] cites
	// IDs[i]. It has one entry fewer than IDs.
	Cites []bool
}

Path is a chain of works connected by citations.

type Recorded 

type Recorded struct {
	// Refs is how many references were considered, after self-citations and
	// repeated IDs were dropped.
	Refs int

	// Stubs is how many works this call introduced to the library — references
	// pointing at papers nothing had cited yet. The rest were already there,
	// which is deduplication happening rather than deduplication failing.
	Stubs int

	// Edges is how many citation edges were new. Re-running against the same
	// work records none, which is what makes add and expand idempotent.
	Edges int
}

Recorded is what one call to RecordEdgesAndStubs wrote. It feeds the counters on model.ExpansionResult, which is a report and not an error type.

func (Recorded) DeadEnd 

func (r Recorded) DeadEnd() bool

DeadEnd reports whether this work cannot extend the graph: its references were fetched and there were none.

This is not a defect. Reference coverage is a property of the field — 6% dead ends in medicine against 84% in arts and humanities (D12) — and it has to be surfaced per node from day one, exactly as OA status is, or a graph that stops after one hop reads as a broken tool rather than as missing data.

type SearchQuery added in v0.3.0 

type SearchQuery struct {
	Words      string // what the person typed; see MatchQuery
	Limit      int    // passages to return
	PerPaper   int    // most passages from one paper; zero means no limit
	FilID      string // only this paper, when set
	References bool   // include reference lists, left out by default
	// Any matches passages with any of the words rather than all of them —
	// the fallback when no passage has them all. bm25 still ranks a passage
	// with more of the words higher.
	Any bool
}

SearchQuery is one keyword search.

type Stats 

type Stats struct {
	Works int // every row in work, stubs included
	Stubs int // of those, the ones never hydrated
	Edges int // rows in cites
}

Stats counts what the library holds. The CLI prints it after add and expand, and the stub count is the honest half of that report: it is how much of the graph is still only an identifier.

type Summary 

type Summary struct {
	Works      int // every row in work
	Hydrated   int // metadata fetched
	Stubs      int // known only by ID, still to fetch
	Unresolved int // of the stubs, those OpenAlex has no record of
	Seeds      int // works the user added
	Edges      int // citations

	// DeadEnds counts hydrated works whose references were fetched and turned
	// out to be none: the graph cannot grow through them. It is the per-library
	// view of D12's reference coverage, computed from the edges already stored.
	DeadEnds int
}

Summary is the library at a glance: what `fil stats` prints.

type Text added in v0.3.0 

type Text struct {
	FilID     string
	Path      string // the PDF read, relative to the library folder
	SHA256    string
	Status    string // text.Status: ok | no-text | gibberish | unreadable | timed-out
	Reason    string
	Pages     int
	Chars     int
	Extractor string
	Millis    int64
	ReadAt    time.Time
}

Text is one row of work_text: what reading a paper's PDF gave.

type ToFetch added in v0.3.0 

type ToFetch struct {
	FilID string
	Links []model.Location
}

ToFetch is a work fetch should download, with the links to try.

type ToRead added in v0.3.0 

type ToRead struct {
	FilID  string
	Path   string
	SHA256 string
}

ToRead is a PDF to extract text from.

type Tx 

type Tx struct {
	// contains filtered or unexported fields
}

Tx is an open write transaction, and the only way anything writes to a library.

It exists so the single-writer rule (hard rule 5) is structural rather than remembered. The write handle is unexported and pinned to one connection, so a caller cannot start a write without coming through DB.Tx, and two goroutines that try queue on the pool instead of racing for the file lock.

The unit of work is deliberately the caller's, not the statement's. Hydrating a work, writing its edges and creating the stubs they point at is one transaction: the citation edges reference work rows that the same call creates, so a partial commit would mean either an edge with no target or a work claiming fetched_refs it never wrote. §4 sizes that unit at one batch — Ctrl-C loses at most one batch of progress and never a half-written graph.

func (*Tx) AddLocalWork added in v0.3.0 

func (tx *Tx) AddLocalWork(ctx context.Context, title string, year *int) (string, error)

AddLocalWork records a document OpenAlex does not know: a fil ID, the title the user gave it, no OpenAlex ID and no references (ADR-014). It is a seed the user uploaded, and it is never on the expansion frontier.

func (*Tx) DeleteFolder added in v0.3.0 

func (tx *Tx) DeleteFolder(ctx context.Context, path string) error

DeleteFolder forgets a folder. Its files' rows are left for the caller.

func (*Tx) GetWork 

func (tx *Tx) GetWork(ctx context.Context, ref string) (*model.Work, error)

GetWork reads one work inside this transaction, so it sees writes the transaction has made but not yet committed.

func (*Tx) MarkSeed 

func (tx *Tx) MarkSeed(ctx context.Context, id string) error

MarkSeed makes id a seed at depth 0. It is for a seed that turned out to be a duplicate of a record already in the library: the flags belong on the record that survives.

func (*Tx) MarkUnresolved 

func (tx *Tx) MarkUnresolved(ctx context.Context, id string) error

MarkUnresolved records that OpenAlex has no record of a work some other paper cited, so expansion stops spending budget on it.

The citation edge stays. It remains true that the paper was cited, whatever the index knows, and deleting the edge to tidy up would silently falsify the graph (§7). The row stays a stub: unresolved is not hydration.

func (*Tx) MergeInto 

func (tx *Tx) MergeInto(ctx context.Context, oldID, newID string) error

MergeInto folds the work oldID into newID: OpenAlex merged two records for one paper and now answers for the old ID with the new one. Both are OpenAlex IDs.

Without this the same paper sits in the graph twice — once as the stub some citing work pointed at, once as the hydrated survivor — which is exactly the quiet rot hard rule 6 exists to prevent. The fold:

  • creates newID as a stub if it is not already present, carrying oldID's depth and provenance, and otherwise keeps the shallower depth and a sticky is_seed;
  • moves every edge, in both directions, from the old record to the survivor. An edge that would already exist, or would become a self-citation, is dropped;
  • moves what the user attached to the old record — notes, collections, text, folders and files — so a fold never deletes a person's work;
  • records the old record's fil ID and OpenAlex ID as aliases of the survivor (D17), and repoints any aliases the old record already had, so everything that named the old record still finds the paper;
  • deletes the old record.

Call it before hydrating newID in the same transaction. A merged-away record can hold a DOI the survivor is about to be written with, and the UNIQUE index on work.doi would refuse the survivor while the old row still held it.

Merging an ID into itself, or one not in the library, does nothing.

func (*Tx) MoveFile added in v0.3.0 

func (tx *Tx) MoveFile(ctx context.Context, from string, to File) error

MoveFile records that the file at from is now at to.Path.

func (*Tx) MoveFolder added in v0.3.0 

func (tx *Tx) MoveFolder(ctx context.Context, from string, to Folder) error

MoveFolder records that the folder at from is now at to, and moves the recorded paths of every file inside it with it.

func (*Tx) PutDetails added in v0.3.0 

func (tx *Tx) PutDetails(ctx context.Context, h *model.HydratedWork) error

PutDetails gives a stub its title, year and authors without hydrating it: its own references are not recorded, so the graph does not grow, and it stays on the expansion frontier as a work still to fetch. A work already hydrated is left alone.

func (*Tx) PutFile added in v0.3.0 

func (tx *Tx) PutFile(ctx context.Context, f File) error

PutFile records a file, replacing what was recorded at its path. The hash is stored in lower case so lookups compare exactly.

func (*Tx) PutFolder added in v0.3.0 

func (tx *Tx) PutFolder(ctx context.Context, f Folder) error

PutFolder records a work folder, replacing what was recorded at its path.

func (*Tx) PutMentions added in v0.3.0 

func (tx *Tx) PutMentions(ctx context.Context, from string, ms []Mention) error

PutMentions replaces what a paper's PDF says about whom it cites, and sets each edge's display context to its first mention. A mention for a work the paper does not cite in OpenAlex's list is refused: context attaches to edges, never makes them (ADR-015). 

func (tx *Tx) PutPDFLinks(ctx context.Context, ref string, links model.PDFLinks) error

PutPDFLinks records what is known about a work's open-access PDF, replacing what was recorded before. ref is any ID the work has.

func (*Tx) PutText added in v0.3.0 

func (tx *Tx) PutText(ctx context.Context, t Text, chunks []Chunk) error

PutText records what reading a paper's PDF gave, replacing its passages: a paper's text is always one reading of one file, never two mixed.

func (*Tx) RecordAttempt added in v0.3.0 

func (tx *Tx) RecordAttempt(ctx context.Context, ref string, a Attempt) error

RecordAttempt logs one download of the work ref names.

func (*Tx) RecordEdgesAndStubs 

func (tx *Tx) RecordEdgesAndStubs(ctx context.Context, h *model.HydratedWork) (Recorded, error)

RecordEdgesAndStubs writes a hydrated work's reference list: one stub per referenced work, one edge per reference, and fetched_refs on the citing work.

This is the call the whole ingestion path is built around, and it must be one transaction. The edges reference work rows this same call creates, so a partial commit leaves either an edge with no target or a work claiming fetched_refs it never wrote. Both are states nothing downstream should have to handle, and neither is visible to the user until the graph looks wrong.

The citing work must already exist — call UpsertWork first. The fetched_refs update doubles as that check and returns errs.ErrNotFound if it matches nothing, which is a better failure than the foreign key error the first edge would otherwise raise.

Every reference is recorded, including a review article's eight hundred. They arrive free inside the response already paid for, and an edge is ground truth (hard rule 2) that nothing here is entitled to discard. §4's cap on how many of them may compete for the budget belongs to the frontier query in M1, where it can be applied per citing work without throwing away what was measured.

func (*Tx) RecordImport added in v0.3.0 

func (tx *Tx) RecordImport(ctx context.Context, a ImportAttempt) error

RecordImport records what importing a file gave, replacing what an earlier try of the same bytes gave.

func (*Tx) SetIdentity added in v0.3.0 

func (tx *Tx) SetIdentity(ctx context.Context, path, identity string) error

SetIdentity records whether a file's first pages name its paper.

func (*Tx) SyncFolderCollections added in v0.3.0 

func (tx *Tx) SyncFolderCollections(ctx context.Context, want map[string]FolderCollection) error

SyncFolderCollections makes the collections that mirror folders match want, a map from folder path to the collection name and its members' fil IDs. A folder collection not in want is deleted; the user's own collections are never touched, except that one whose name a folder now claims is adopted.

func (*Tx) UpsertEdge 

func (tx *Tx) UpsertEdge(ctx context.Context, e model.Edge) error

UpsertEdge writes one edge with whatever has been learned about it, and enriches an edge already present rather than replacing it.

An edge is recorded long before anything is known about why it was made: the row is written during expansion from OpenAlex's resolved reference list, and the context sentence arrives in M3 once the citing paper's PDF has been read, the intent in M6. So a second write must fill in what it knows and leave everything else alone — a call carrying only a context sentence must not erase an intent something else took the trouble to classify.

Both works must already exist, and may be named by any of their IDs; the edge is stored between their fil IDs. Edges point at real nodes because the stub exists the moment the edge does (§4), and the foreign keys are enforced on every connection (spike 5) to keep that true.

func (*Tx) UpsertStub 

func (tx *Tx) UpsertStub(ctx context.Context, id string, depth *int, source model.Source) (bool, error)

UpsertStub creates a work row holding nothing but an identifier, and reports whether it had to create it.

This is what makes deduplication free (§4): the target of a citation gets a row the moment the edge is written, so "connect to the existing node if the paper is already there" is a primary key conflict rather than matching logic that can be got wrong.

It never touches a row that already exists, with one exception: depth is lowered if this route to the work is shorter than the one already recorded. Expansion is best-first, so a work can be reached the long way round first, and leaving the longer distance in place would misreport how far from the seed the user's graph actually reaches.

func (*Tx) UpsertWork 

func (tx *Tx) UpsertWork(ctx context.Context, w *model.Work) error

UpsertWork writes a hydrated work, creating the row if this library has never seen it and filling in the metadata if it was already there as a stub.

Hydration is an update, not a replacement, because a work is almost always already present when it is hydrated — a stub was created the moment some other paper was seen to cite it (§4). Three columns are therefore preserved rather than overwritten, and losing any of them would be a real loss:

  • depth keeps the smaller of the two. It is hops from the nearest seed, and a work reached again by a longer route has not moved further away.
  • is_seed is sticky. A work the user named stays named even when expansion reaches it again from somewhere else.
  • source records how the work first entered the library and is never rewritten, so "this arrived through expansion" survives a later re-fetch.

Everything else is taken from w, which is treated as the authoritative record.

The work is found by its OpenAlex ID, which is what OpenAlex sent. A work new to the library is given a fresh fil ID (D17); one already present keeps the fil ID it has, forever. Either way w.FilID is set on return.

func (*Tx) WorkIDByDOI 

func (tx *Tx) WorkIDByDOI(ctx context.Context, doi string) (string, error)

WorkIDByDOI returns the OpenAlex ID of the work in this library that holds doi, or errs.ErrNotFound. The DOI must already be normalised; identity.NormaliseDOI is what every write path uses.

OpenAlex does not always merge its own duplicates: the same arXiv preprint has been seen under two work IDs with one DOI (W2949614626 and W4294576234, 24 Sept 2026). The UNIQUE index on work.doi refuses the second, and this is how a caller finds the first to fold it into.

A holder with no OpenAlex ID — a local document the user imported, carrying the DOI printed on it — is reported as an error rather than as not found: the OpenAlex work arriving with that DOI is the same paper, and reconciling the two belongs to import (ARCHITECTURE_PHASE3.md ADR-014), not to a fold.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.