Documentation
¶
Overview ¶
Package graph grows and traverses the citation graph.
Two jobs live here:
- Expansion. Follow referenced_works outward from a seed under a node budget and a priority order. At ~40 references per paper, depth 3 is ~64,000 nodes, so breadth-first without a limit is never acceptable.
- Identity resolution. Deduplicate on openalex_id, and reconcile the DOI, arXiv ID and PMID that arrive attached to the same work.
This is the only package that knows about both internal/store and internal/sources. Keeping that pairing here is what lets store stay swappable.
Traversal (neighbours, shortest path) is SQL — a breadth-first search, one set-based query per step, not a recursive CTE (see store.ShortestPath) — so it lives in store. Whole-graph algorithms (PageRank, communities) load edges into memory through gonum/graph and live here.
Index ¶
- type DuplicateGroup
- type ExpandOptions
- type Graph
- func (g *Graph) AddSeed(ctx context.Context, id identity.ID) (Seeded, error)
- func (g *Graph) AddSeedWork(ctx context.Context, h model.HydratedWork) (Seeded, error)
- func (g *Graph) DetailIDs(ctx context.Context, ids []string) (int, error)
- func (g *Graph) Expand(ctx context.Context, o ExpandOptions) (model.ExpansionResult, error)
- func (g *Graph) HydrateIDs(ctx context.Context, ids []string) (model.ExpansionResult, error)
- func (g *Graph) Locate(ctx context.Context, o LocateOptions) (Located, error)
- func (g *Graph) LocateIDs(ctx context.Context, ids []string, progress func(Located)) (Located, error)
- func (g *Graph) ProbableDuplicates(ctx context.Context) ([]DuplicateGroup, error)
- func (g *Graph) Resolve(ctx context.Context, id identity.ID) (model.HydratedWork, error)
- func (g *Graph) SearchTitle(ctx context.Context, title string, limit int) ([]openalex.Candidate, error)
- type LocateOptions
- type Located
- type Seeded
- type Source
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DuplicateGroup ¶
type DuplicateGroup struct {
Title string // as the first work in the group spells it
Works []model.Work // two or more, ordered by year, then ID
}
DuplicateGroup is a set of fetched works with the same title: probably one paper that OpenAlex holds as more than one record, under different DOIs or none.
type ExpandOptions ¶
type ExpandOptions struct {
// MaxNodes is the budget: the most works this run may try to hydrate.
// It is the primary control (ADR-002). Depth is only a guard.
MaxNodes int
// MaxDepth is the deepest a hydrated work may sit, in hops from the
// nearest seed. Its references are still recorded — edges are free — but
// the stubs they create are not fetched in this run.
MaxDepth int
// MaxRefsPerWork caps how many of one work's references may compete for
// the budget (§4). Zero means no cap.
MaxRefsPerWork int
// BatchSize is how many works to fetch per request. Zero means
// openalex.MaxBatch, the most one request may carry.
BatchSize int
// Progress, if set, is called after each committed batch with the running
// totals. The CLI draws its progress line from it.
Progress func(model.ExpansionResult)
}
ExpandOptions bound one expansion run.
type Graph ¶
type Graph struct {
// contains filtered or unexported fields
}
Graph grows the citation graph: it takes works from a Source and writes them to the store. It is the one place data crosses from sources to store, which is what keeps the two swappable independently.
func (*Graph) AddSeed ¶
AddSeed resolves a deterministic identifier and adds the work it names as a seed. A title is refused; see SearchTitle and AddSeedWork.
Nothing is written unless OpenAlex has the work: an unresolvable seed wraps errs.ErrUnresolved and leaves the library untouched, because there is no node to hang the failure on.
func (*Graph) AddSeedWork ¶
AddSeedWork adds an already-fetched work as a seed: the work itself, a stub for every work it cites, and the edges between them, in one transaction (§4). It is how a title-search candidate the user chose gets added without fetching it a second time.
The seed is depth 0 and marked as a seed. If it was already in the library — reached as a stub from some other seed, or added before — hydration keeps the shorter depth and the original source (see store.UpsertWork), so adding it again is safe and idempotent.
func (*Graph) DetailIDs ¶ added in v0.3.0
DetailIDs fetches the title, year and authors of the named stubs without hydrating them: their own references are not recorded, so the graph does not grow, and they stay on the frontier as works still to fetch. Citation context needs them to recognise a paper's references in its text. One request per openalex.MaxBatch IDs; it returns how many got details.
func (*Graph) Expand ¶
func (g *Graph) Expand(ctx context.Context, o ExpandOptions) (model.ExpansionResult, error)
Expand grows the graph from the frontier under a budget, best first.
Each round asks the store for the stubs with the highest in-graph in-degree, hydrates them in one request, and commits the batch — works, their references as new stubs and edges, merged records folded, unknown IDs marked unresolved — in one transaction. Ctrl-C therefore loses at most the batch in flight, and never leaves a half-written graph (§4).
There is no queue to save. The frontier is a query over persisted state, so a run that stops for any reason resumes by running again, and running again on an exhausted frontier makes no requests and writes nothing.
The run ends normally for one of four reasons, reported in StoppedBecause with a nil error: the budget is spent, the frontier is empty, the only stubs left are past MaxDepth, or ctx was cancelled. An error means something actually failed; the result still reports what was committed before it.
func (*Graph) HydrateIDs ¶ added in v0.3.0
HydrateIDs fetches the named works — stubs, typically — whatever their place on the frontier: fetch calls it for the papers it is about to download, so each gets a title, a folder named for it, and its own references. One request per openalex.MaxBatch IDs; each batch is committed as Expand commits one, so the works keep their depth and source.
func (*Graph) Locate ¶ added in v0.3.0
Locate looks up the open-access PDF links of works whose links are not known or are older than MaxAge — seeds first, then the most cited — and records an answer for every one (ADR-013). Works fetched from now on get their links while being hydrated; this is for the rest, and for stubs.
Each batch of 100 is committed on its own, so cancelling keeps what was done. A work OpenAlex has no record of is recorded as such and marked unresolved, which also stops expansion asking about it again.
func (*Graph) LocateIDs ¶ added in v0.3.0
func (g *Graph) LocateIDs(ctx context.Context, ids []string, progress func(Located)) (Located, error)
LocateIDs looks up and records the PDF links of the works with these OpenAlex IDs, 100 to a request, committing each batch on its own.
func (*Graph) ProbableDuplicates ¶
func (g *Graph) ProbableDuplicates(ctx context.Context) ([]DuplicateGroup, error)
ProbableDuplicates lists groups of fetched works whose titles match once case, punctuation and spacing are ignored.
It reports and never merges (D14). A matching title is evidence, not proof: a book review routinely carries the title of the book it reviews, and a preprint and its journal version are one work to a reader but two records that may differ. Duplicates that are certain — the same OpenAlex ID, a merged record, the same DOI — are already folded on the way in (D2, graph.hydrate); what is left here is for a person to judge.
type LocateOptions ¶ added in v0.3.0
type LocateOptions struct {
// Max is the most works to look up. One request covers 100 works at one
// credit, so 1,000 works cost 10 credits.
Max int
// MaxAge is how old an answer may be before it is asked again: open
// copies appear as repositories catch up, so a "no open copy" is not
// forever.
MaxAge time.Duration
// Progress, if set, is called after each committed batch with the running
// totals.
Progress func(Located)
}
LocateOptions bound one lookup of PDF links.
type Located ¶ added in v0.3.0
type Located struct {
Checked int // works looked up
By map[model.PDFAvailability]int // how many of them have each answer
Requests int // batch requests, one credit each
}
Located is what a lookup found.
type Seeded ¶
type Seeded struct {
// Work is the seed as stored, read back after the commit — so its depth,
// flags and provenance are the library's, not the response's.
Work model.Work
// WasSeed reports the work was already a seed before this call.
WasSeed bool
// Recorded counts what was written for the seed's reference list. Adding a
// seed that is already in the library records no new stubs or edges, which
// is how the CLI can say "already there" without a special case.
Recorded store.Recorded
}
Seeded reports what adding a seed did.
type Source ¶
type Source interface {
Resolve(ctx context.Context, id identity.ID) (model.HydratedWork, error)
SearchByTitle(ctx context.Context, title string, limit int) ([]openalex.Candidate, error)
GetWorksBatch(ctx context.Context, ids []string) (openalex.Batch, error)
GetLocations(ctx context.Context, ids []string) (openalex.Located, error)
}
Source is what the graph needs from an index of works. It is declared here, where it is consumed, and says only what this package uses — not a guess at the shape a second source will one day have. *openalex.Client satisfies it; tests satisfy it with a map.
Source Files