feeds

package
v0.2.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 11, 2025 License: MIT Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source
const DefaultCacheTTL = 24 * time.Hour // TODO: move to a cache pkg

Variables

This section is empty.

Functions

This section is empty.

Types

type CrossrefHarvester added in v0.2.12

type CrossrefHarvester struct {
	Client              Doer
	ApiEndpoint         string
	ApiFilter           string
	ApiEmail            string
	Rows                int
	UserAgent           string
	MaxRetries          int
	AcceptableMissRatio float64 // recommended: 0.1
}

CrossrefHarvester fetches data from crossref API and by default will write it to files on disk.

func (*CrossrefHarvester) WriteDaySlice added in v0.2.12

func (c *CrossrefHarvester) WriteDaySlice(t time.Time, dir string, prefix string) error

WriteDaySlice is a helper function to atomically write crossref data for a single day to file on disk under dir. Idempotent, once the data has been captured. TODO: add compression.

func (*CrossrefHarvester) WriteSlice added in v0.2.12

func (c *CrossrefHarvester) WriteSlice(w io.Writer, from, until time.Time) error

WriteSlice writes a slice of data from the API into a writer.

type Doer added in v0.2.12

type Doer interface {
	Do(*http.Request) (*http.Response, error)
}

Doer abstracts https://pkg.go.dev/net/http#Client.Do.

type PubMedFetcher

type PubMedFetcher struct {
	BaseURL  string
	CacheTTL time.Duration
	CacheDir string
}

PubMedFetcher handles fetching and parsing PubMed update files list

func NewPubMedFetcher

func NewPubMedFetcher(baseURL string) (*PubMedFetcher, error)

NewPubMedFetcher creates a new fetcher with default settings

func (*PubMedFetcher) FetchFiles

func (pf *PubMedFetcher) FetchFiles() ([]PubMedFile, error)

FetchFiles retrieves and parses the PubMed update files.

type PubMedFile

type PubMedFile struct {
	Filename     string
	URL          string
	LastModified time.Time
	Size         string
}

PubMedFile represents metadata for a PubMed update file, cf. https://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/.

func FilterPubmedFiles

func FilterPubmedFiles(files []PubMedFile, f func(PubMedFile) bool) (result []PubMedFile)

FilterPubmedFiles returns a list of file filtered by a given filter function.

type WorksResponse added in v0.2.12

type WorksResponse struct {
	Message struct {
		Facets struct {
		} `json:"facets"`
		Items        []json.RawMessage `json:"items"`
		ItemsPerPage int64             `json:"items-per-page"`
		NextCursor   string            `json:"next-cursor"` // iterate
		Query        struct {
			SearchTerms interface{} `json:"search-terms"`
			StartIndex  int64       `json:"start-index"`
		} `json:"query"`
		TotalResults int64 `json:"total-results"` // want to estimate total results (and verify download)
	} `json:"message"`
	MessageType    string `json:"message-type"`
	MessageVersion string `json:"message-version"`
	Status         string `json:"status"`
}

WorksResponse, stripped of the actual messages, as we only need the status and mayby total results.

func (*WorksResponse) IsLast added in v0.2.12

func (wr *WorksResponse) IsLast() bool

IsLast returns true, if there are no more records to fetch.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL