9-pdf-availability

Overview ¶

Command pdf-availability is spike 9 (ARCHITECTURE_PHASE3.md §6): of the works in the M1 validation libraries, how many have a PDF we can actually download — per field, per OA status — and how many more does Unpaywall add? How long does one download take?

It runs in phases, each resumable from what the last run left in -dir:

1. Rebuild the 20 validation libraries (spike 8's seeds, add + one default expansion each). The HTTP cache makes a re-run nearly free.
2. Read every fetched work's OA locations from OpenAlex, 100 per request (1 credit each), and classify it: no open copy; open but no PDF link; a PDF link.
3. Download a sample — -per-field works with a PDF link from each field — the way ADR-013 says fil will: no cookies, no credentials, a plain User-Agent, one request per second per host, %PDF- or it is not a PDF, and an HTML answer is counted, never read for links. Successful PDFs are kept in -dir/pdfs for spikes 10 and 11.
4. If FIL_SPIKE_UNPAYWALL_EMAIL is set, ask Unpaywall about a sample of works OpenAlex gave no PDF link for, and about every download that failed, and try the PDF links it offers. The address is read from the environment only: it is never written to -dir, the cache or the report.

Usage:

go run ./spikes/9-pdf-availability -dir <scratch> > spike9.md