11-reference-matching

Overview ¶

Command reference-matching is spike 11 (ARCHITECTURE_PHASE3.md §6): how well can a reference list be split out of a PDF's text and matched to real works?

Measured on PDFs OpenAlex already knows, so OpenAlex's referenced_works is the answer key and the measurement is free. It decides whether parsed references are built, after M3, for local documents OpenAlex does not know. Hard rule 2 stands either way: for a work OpenAlex knows, its list is used, never a parse.

For each text spike 10 extracted:

1. Find the reference section: the last line that is only "References", "Bibliography", "Literature cited" or the like.
2. Split it into entries — numbered ([1], 1.) when the numbers run in sequence, otherwise at lines that start like an author list.
3. Match each of OpenAlex's references to the entry holding most of its title's words (at least 70% of at least three), each entry used once. A match is confirmed when the entry also holds the year or the first author's surname.

Recall is matched references over OpenAlex references with a title. An entry left unmatched is not necessarily an error: OpenAlex's list misses references too (spike 4).

Usage:

go run ./spikes/11-reference-matching -text <spike 10 out>/text -cache <spike 9 dir>/cache > spike11.md