Help us shape the future of Go! Take the 2026 Go Developer Survey

pdfs

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 30, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package pdfs downloads open-access PDFs from the links OpenAlex gives (ADR-013). It is the one place fil fetches a PDF, so the rules live here:

  • no cookies, no credentials, a plain User-Agent that names fil;
  • one request per second to each host, on every redirect hop — a link through doi.org reaches a publisher, and that publisher's server is the one to spare (spike 9, found in review);
  • the body must start like a PDF. Anything else — a landing page, a login page, a bot wall — is refused as not a PDF and never read for links;
  • the body is streamed, never held in memory or cached: papers/ is where PDFs live.

It does not use internal/httpx, which reads whole bodies and caches them — right for JSON from an API, wrong for files.

Index

Constants

View Source 
const (
	// PerHost is the request rate to any one host.
	PerHost = rate.Limit(1)

	// DefaultTimeout bounds one download, end to end. The 90th-percentile
	// download in spike 9 took 4 s; a minute leaves room for a large file on a
	// slow link without letting a stalled server hold a slot for ever.
	DefaultTimeout = 2 * time.Minute
)

Variables

This section is empty.

Functions

func Host 

func Host(u *url.URL) string

Host is a URL's host as fil records it: lower case, without www.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client downloads PDFs. It is safe for concurrent use; the per-host limits are shared by every download it makes.

func New 

func New(opts Options) (*Client, error)

New builds a Client.

func (*Client) Get 

func (c *Client) Get(ctx context.Context, rawURL string) (*Download, error)

Get starts downloading rawURL. It returns once the response has begun and its first bytes look like a PDF; anything else comes back as a *Failure. The caller streams Body to disk and enforces the size limit.

type Download 

type Download struct {
	Body     io.ReadCloser
	FinalURL string // after redirects
	Started  time.Time
}

Download is a PDF being received. Read Body to the end, then Close it.

type Failure 

type Failure struct {
	Outcome Outcome
	Status  int // the HTTP status, when there was one
	Err     error
}

Failure is a download that did not produce a PDF.

func Classify 

func Classify(ctx context.Context, err error) *Failure

Classify turns an error from reading or requesting into a Failure.

func (*Failure) Error 

func (f *Failure) Error() string

func (*Failure) Unwrap 

func (f *Failure) Unwrap() error

type Options 

type Options struct {
	// UserAgent names fil and where to reach its maintainers. Required.
	UserAgent string
	// Timeout bounds one download; zero means DefaultTimeout.
	Timeout time.Duration
	// Transport replaces the network. Tests point it at a local server.
	Transport http.RoundTripper
	// contains filtered or unexported fields
}

Options configure a Client.

type Outcome 

type Outcome string

Outcome is how one download ended, in words a person can act on. 

const (
	OK           Outcome = "ok"
	Refused      Outcome = "refused"   // 401, 403: the host will not serve fil
	NotFound     Outcome = "not-found" // 404, 410
	HTTPError    Outcome = "http-error"
	NotPDF       Outcome = "not-a-pdf" // an HTML page or anything else
	TooLarge     Outcome = "too-large"
	Timeout      Outcome = "timeout"
	Network      Outcome = "network"
	RedirectLoop Outcome = "redirect-loop"
	Cancelled    Outcome = "cancelled"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.