pdfs

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 30, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package pdfs downloads open-access PDFs from the links OpenAlex gives (ADR-013). It is the one place fil fetches a PDF, so the rules live here:

  • no cookies, no credentials, a plain User-Agent that names fil;
  • one request per second to each host, on every redirect hop — a link through doi.org reaches a publisher, and that publisher's server is the one to spare (spike 9, found in review);
  • the body must start like a PDF. Anything else — a landing page, a login page, a bot wall — is refused as not a PDF and never read for links;
  • the body is streamed, never held in memory or cached: papers/ is where PDFs live.

It does not use internal/httpx, which reads whole bodies and caches them — right for JSON from an API, wrong for files.

Index

Constants

View Source 
const (
	// PerHost is the request rate to any one host.
	PerHost = rate.Limit(1)

	// DefaultTimeout bounds one download, end to end. The 90th-percentile
	// download in spike 9 took 4 s; a minute leaves room for a large file on a
	// slow link without letting a stalled server hold a slot for ever.
	DefaultTimeout = 2 * time.Minute
)

Variables

This section is empty.

Functions

func Host 

func Host(u *url.URL) string

Host is a URL's host as fil records it: lower case, without www.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client downloads PDFs. It is safe for concurrent use; the per-host limits are shared by every download it makes.

func New 

func New(opts Options) (*Client, error)

New builds a Client.

func (*Client) Get 

func (c *Client) Get(ctx context.Context, rawURL string) (*Download, error)

Get starts downloading rawURL. It returns once the response has begun and its first bytes look like a PDF; anything else comes back as a *Failure. The caller streams Body to disk and enforces the size limit.

type Download 

type Download struct {
	Body     io.ReadCloser
	FinalURL string // after redirects
	Started  time.Time
}

Download is a PDF being received. Read Body to the end, then Close it.

type Failure 

type Failure struct {
	Outcome Outcome
	Status  int // the HTTP status, when there was one
	Err     error
}

Failure is a download that did not produce a PDF.

func Classify 

func Classify(ctx context.Context, err error) *Failure

Classify turns an error from reading or requesting into a Failure.

func (*Failure) Error 

func (f *Failure) Error() string

func (*Failure) Unwrap 

func (f *Failure) Unwrap() error

type Options 

type Options struct {
	// UserAgent names fil and where to reach its maintainers. Required.
	UserAgent string
	// Timeout bounds one download; zero means DefaultTimeout.
	Timeout time.Duration
	// Transport replaces the network. Tests point it at a local server.
	Transport http.RoundTripper
	// contains filtered or unexported fields
}

Options configure a Client.

type Outcome 

type Outcome string

Outcome is how one download ended, in words a person can act on. 

const (
	OK           Outcome = "ok"
	Refused      Outcome = "refused"   // 401, 403: the host will not serve fil
	NotFound     Outcome = "not-found" // 404, 410
	HTTPError    Outcome = "http-error"
	NotPDF       Outcome = "not-a-pdf" // an HTML page or anything else
	TooLarge     Outcome = "too-large"
	Timeout      Outcome = "timeout"
	Network      Outcome = "network"
	RedirectLoop Outcome = "redirect-loop"
	Cancelled    Outcome = "cancelled"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.