goodread

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 13, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package goodread is a library for reading public Goodreads data: books, authors, series, lists, quotes, users, shelves, genres, reviews, and search. Every page is normalized into a rich struct (JSON-LD first, HTML fallback), with explicit empty/zero/[] for genuinely absent fields. It carries no CLI knowledge and depends only on goquery and the standard library.

Index

Constants

View Source 
const (
	BaseURL      = "https://www.goodreads.com"
	RobotsTxtURL = "https://www.goodreads.com/robots.txt"

	DefaultDelay    = 2 * time.Second
	DefaultWorkers  = 2
	DefaultTimeout  = 30 * time.Second
	DefaultRetries  = 3
	DefaultCacheTTL = 24 * time.Hour
)

Site constants.

Variables

View Source 
var ErrBlocked = errors.New("blocked: page requires sign-in")

ErrBlocked signals that Goodreads served a sign-in wall (or otherwise refused the page). Callers map it to the "blocked" exit code and print a hint.

View Source 
var ErrNotFound = fmt.Errorf("not found")

ErrNotFound is returned when a page 404s.

View Source 
var ErrRateLimited = errors.New("rate limited (HTTP 429)")

ErrRateLimited signals a sustained HTTP 429 after retries.

Functions

func AuthorURL 

func AuthorURL(id string) string

func BookURL 

func BookURL(id string) string

func Classify 

func Classify(s string) (entity, id string)

Classify inspects any Goodreads URL or bare id and returns (entity, id). A bare numeric id is treated as a book. Returns ("","") when unrecognizable.

func DefaultDataDir 

func DefaultDataDir() string

DefaultDataDir resolves $XDG_DATA_HOME/goodread (or ~/.local/share/goodread).

func DocFromBytes 

func DocFromBytes(body []byte) (*goquery.Document, error)

DocFromBytes parses raw HTML bytes into a goquery document.

func ExtractID 

func ExtractID(rawurl, prefix string) string

ExtractID is the exported form of extractIDFromPath.

func GenreURL 

func GenreURL(slug string) string

GenreURL keeps the slug as-is (genres have no numeric id).

func InferEntityType 

func InferEntityType(u string) string

InferEntityType maps a URL to an entity type, defaulting to book.

func ListURL 

func ListURL(id string) string

ListURL keeps the "<num>.<slug>" id intact since Listopia uses it verbatim.

func LoadCookies 

func LoadCookies(path string) ([]*http.Cookie, error)

LoadCookies reads a Netscape/Mozilla cookies.txt file (the format exported by most browser extensions and by curl) into http.Cookie values.

func ParseList 

func ParseList(doc *goquery.Document, listID, pageURL string) (*List, []ListBook, error)

ParseList parses a Listopia list page.

func ParseSearchHTMLNextPage 

func ParseSearchHTMLNextPage(doc *goquery.Document) string

ParseSearchHTMLNextPage returns the next search page URL, or "".

func ParseSeries 

func ParseSeries(doc *goquery.Document, seriesID, pageURL string) (*Series, []SeriesBook, error)

ParseSeries parses a Goodreads series page.

func ParseShelf 

func ParseShelf(doc *goquery.Document, userID, shelfName, pageURL string) (*Shelf, []ShelfBook, error)

ParseShelf parses a user's shelf page into ShelfBook rows.

func ParseShelfNextPage 

func ParseShelfNextPage(doc *goquery.Document) string

ParseShelfNextPage returns the next shelf page URL, or "".

func ParseShelfRSS 

func ParseShelfRSS(body []byte, userID, shelfName, pageURL string) (*Shelf, []ShelfBook, error)

ParseShelfRSS parses a shelf RSS feed into a shelf header and its rows.

func QuoteURL 

func QuoteURL(id string) string

func SearchAutocompleteURL 

func SearchAutocompleteURL(q string) string

SearchAutocompleteURL builds the JSON autocomplete endpoint URL.

func SearchHTMLURL 

func SearchHTMLURL(q string, page int) string

SearchHTMLURL builds the full HTML search URL for a page.

func SeriesURL 

func SeriesURL(id string) string

func ShelfRSSURL 

func ShelfRSSURL(userID, shelf string) string

ShelfRSSURL builds the RSS-feed form of a user's shelf.

func ShelfURL 

func ShelfURL(userID, shelf string, page int) string

ShelfURL builds a user's shelf page URL.

func UserURL 

func UserURL(id string) string

Types

type Author 

type Author struct {
	AuthorID       string    `json:"author_id"`
	Name           string    `json:"name"`
	Bio            string    `json:"bio"`
	PhotoURL       string    `json:"photo_url"`
	Website        string    `json:"website"`
	BornDate       string    `json:"born_date"`
	DiedDate       string    `json:"died_date"`
	Hometown       string    `json:"hometown"`
	Influences     []string  `json:"influences"`
	Genres         []string  `json:"genres"`
	AvgRating      float64   `json:"avg_rating"`
	RatingsCount   int64     `json:"ratings_count"`
	BooksCount     int       `json:"books_count"`
	FollowersCount int       `json:"followers_count"`
	NotableBookIDs []string  `json:"notable_book_ids"`
	URL            string    `json:"url"`
	FetchedAt      time.Time `json:"fetched_at"`
}

Author is a Goodreads author page (/author/show/<id>).

func ParseAuthor 

func ParseAuthor(doc *goquery.Document, authorID, pageURL string) (*Author, error)

ParseAuthor parses a Goodreads author page.

type Book 

type Book struct {
	BookID             string    `json:"book_id"`
	Title              string    `json:"title"`
	TitleWithoutSeries string    `json:"title_without_series"`
	Description        string    `json:"description"`
	AuthorID           string    `json:"author_id"`
	AuthorName         string    `json:"author_name"`
	ISBN               string    `json:"isbn"`
	ISBN13             string    `json:"isbn13"`
	ASIN               string    `json:"asin"`
	AvgRating          float64   `json:"avg_rating"`
	RatingsCount       int64     `json:"ratings_count"`
	ReviewsCount       int64     `json:"reviews_count"`
	PublishedYear      int       `json:"published_year"`
	Publisher          string    `json:"publisher"`
	Language           string    `json:"language"`
	Pages              int       `json:"pages"`
	Format             string    `json:"format"`
	SeriesID           string    `json:"series_id"`
	SeriesName         string    `json:"series_name"`
	SeriesPosition     string    `json:"series_position"`
	Genres             []string  `json:"genres"`
	CoverURL           string    `json:"cover_url"`
	URL                string    `json:"url"`
	SimilarBookIDs     []string  `json:"similar_book_ids"`
	FetchedAt          time.Time `json:"fetched_at"`
}

Book is a Goodreads book page (/book/show/<id>).

func ParseAutocompleteBooks 

func ParseAutocompleteBooks(body []byte) []Book

ParseAutocompleteBooks parses the rich JSON autocomplete response into Book records. The autocomplete endpoint is open (not WAF-challenged), so it is the primary way to obtain structured book data without a lent session.

func ParseBook 

func ParseBook(doc *goquery.Document, bookID, pageURL string) (*Book, error)

ParseBook parses a Goodreads book page into a Book (JSON-LD first).

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a content-addressed, gzipped on-disk page cache. A captured page is stored under cache/<ab>/<sha256>.html.gz, keyed by its URL.

func NewCache 

func NewCache(cfg Config) *Cache

NewCache returns a cache rooted at cfg.CacheDir() (created on first write).

func (*Cache) Clear 

func (c *Cache) Clear() error

Clear removes the entire cache directory.

func (*Cache) Get 

func (c *Cache) Get(url string) ([]byte, bool)

Get returns cached bytes for a URL when present and within the TTL.

func (*Cache) Path 

func (c *Cache) Path(url string) string

Path returns the on-disk cache path for a URL (whether or not it exists).

func (*Cache) Put 

func (c *Cache) Put(url string, body []byte) error

Put writes bytes for a URL into the cache.

func (*Cache) Stats 

func (c *Cache) Stats() (files int, bytes int64, err error)

Stats reports the file count and total bytes held in the cache.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client performs polite, retrying HTTP GETs against goodreads.com.

func NewClient 

func NewClient(cfg Config) *Client

NewClient builds an anonymous client.

func NewClientWithCookies 

func NewClientWithCookies(cfg Config, cookies []*http.Cookie) (*Client, error)

NewClientWithCookies builds a client pre-loaded with a lent session.

func (*Client) CachingFetch 

func (c *Client) CachingFetch(ctx context.Context, cache *Cache, cfg Config, url string) ([]byte, error)

CachingFetch returns page bytes, reading the cache unless NoCache/Refresh is set, and populating it on a miss.

func (*Client) CachingFetchHTML 

func (c *Client) CachingFetchHTML(ctx context.Context, cache *Cache, cfg Config, url string) (*goquery.Document, error)

CachingFetchHTML is CachingFetch plus goquery parsing.

func (*Client) Fetch 

func (c *Client) Fetch(ctx context.Context, rawurl string) ([]byte, int, error)

Fetch returns the raw body and status for a URL, retrying transient failures. A 404 returns (nil, 404, nil). A sign-in redirect returns ErrBlocked.

func (*Client) FetchHTML 

func (c *Client) FetchHTML(ctx context.Context, rawurl string) (*goquery.Document, int, error)

FetchHTML fetches a URL and parses it into a goquery document.

func (*Client) GetAuthor 

func (c *Client) GetAuthor(ctx context.Context, id string) (*Author, error)

GetAuthor fetches and parses an author page by id.

func (*Client) GetBook 

func (c *Client) GetBook(ctx context.Context, id string) (*Book, error)

GetBook fetches and parses a book page by id (or id-with-slug).

func (*Client) GetGenre 

func (c *Client) GetGenre(ctx context.Context, slug string) (*Genre, error)

GetGenre fetches and parses a genre page by slug.

func (*Client) GetList 

func (c *Client) GetList(ctx context.Context, id string) (*List, []ListBook, error)

GetList fetches and parses a Listopia list by id (keeps "<num>.<slug>").

func (*Client) GetQuotes 

func (c *Client) GetQuotes(ctx context.Context, pageURL string) ([]Quote, error)

GetQuotes fetches and parses a quotes page (an author's or a book's quotes URL).

func (*Client) GetReviews 

func (c *Client) GetReviews(ctx context.Context, bookID string) ([]Review, error)

GetReviews fetches a book page and returns the reviews embedded on it.

func (*Client) GetSeries 

func (c *Client) GetSeries(ctx context.Context, id string) (*Series, []SeriesBook, error)

GetSeries fetches and parses a series page by id.

func (*Client) GetShelf 

func (c *Client) GetShelf(ctx context.Context, userID, shelfName string, maxPages int) (*Shelf, []ShelfBook, error)

GetShelf walks a user's shelf, following pagination up to maxPages (maxPages <= 0 means all pages). It returns the shelf header and every row.

func (*Client) GetShelfRSS 

func (c *Client) GetShelfRSS(ctx context.Context, userID, shelfName string) (*Shelf, []ShelfBook, error)

GetShelfRSS fetches a user's shelf via the open RSS feed (no WAF challenge).

func (*Client) GetUser 

func (c *Client) GetUser(ctx context.Context, id string) (*User, error)

GetUser fetches and parses a public user profile by id.

func (*Client) RobotsSitemaps 

func (c *Client) RobotsSitemaps(ctx context.Context) ([]SitemapCategory, error)

RobotsSitemaps fetches robots.txt and returns the advertised siteindex URLs.

func (*Client) Search 

func (c *Client) Search(ctx context.Context, query string, limit int) ([]SearchResult, error)

Search runs autocomplete first (fast, JSON) and falls back to HTML search.

func (*Client) SearchBooks 

func (c *Client) SearchBooks(ctx context.Context, query string, limit int) ([]Book, error)

SearchBooks returns rich Book records from the open autocomplete endpoint.

func (*Client) SearchHTML 

func (c *Client) SearchHTML(ctx context.Context, query string, limit int) ([]SearchResult, error)

SearchHTML walks the full HTML search results, following pagination until limit results are collected (limit <= 0 means a single page).

func (*Client) SitemapIndex 

func (c *Client) SitemapIndex(ctx context.Context, indexURL string) ([]string, error)

SitemapIndex fetches a <sitemapindex> document and returns its child URLs.

func (*Client) SitemapURLs 

func (c *Client) SitemapURLs(ctx context.Context, sitemapURL string) ([]string, error)

SitemapURLs fetches one <urlset> shard (gz-aware) and returns its URLs.

type Config 

type Config struct {
	DataDir    string        // root for cache/store; defaults to XDG data dir
	Workers    int           // concurrency for multi-page/bulk work
	Delay      time.Duration // minimum spacing between requests
	Timeout    time.Duration // per-request timeout
	Retries    int           // retry attempts on 429/5xx
	CacheTTL   time.Duration // on-disk page-cache freshness window
	NoCache    bool          // bypass the on-disk page cache entirely
	Refresh    bool          // force re-fetch and overwrite the cache
	CookiePath string        // optional Netscape cookie jar for lent sessions
}

Config holds runtime configuration for the library.

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns sensible, polite defaults rooted at the XDG data dir.

func (Config) CacheDir 

func (c Config) CacheDir() string

CacheDir is where gzipped page captures live.

func (Config) StorePath 

func (c Config) StorePath() string

StorePath is the default path of the optional SQLite store.

type Genre 

type Genre struct {
	Slug        string    `json:"slug"`
	Name        string    `json:"name"`
	Description string    `json:"description"`
	BooksCount  int       `json:"books_count"`
	BookIDs     []string  `json:"book_ids"`
	URL         string    `json:"url"`
	FetchedAt   time.Time `json:"fetched_at"`
}

Genre is a Goodreads genre/taxonomy node (/genres/<slug>).

func ParseGenre 

func ParseGenre(doc *goquery.Document, slug, pageURL string) (*Genre, error)

ParseGenre parses a Goodreads genre page.

type List 

type List struct {
	ListID        string    `json:"list_id"`
	Name          string    `json:"name"`
	Description   string    `json:"description"`
	BooksCount    int       `json:"books_count"`
	VotersCount   int       `json:"voters_count"`
	Tags          []string  `json:"tags"`
	CreatedByUser string    `json:"created_by_user"`
	URL           string    `json:"url"`
	FetchedAt     time.Time `json:"fetched_at"`
}

List is a Goodreads Listopia list (/list/show/<id>).

type ListBook 

type ListBook struct {
	ListID string `json:"list_id"`
	BookID string `json:"book_id"`
	Rank   int    `json:"rank"`
	Votes  int    `json:"votes"`
}

ListBook links a list to a book with rank and votes.

type QueueItem 

type QueueItem struct {
	ID         int64  `json:"id"`
	URL        string `json:"url"`
	EntityType string `json:"entity_type"`
	Priority   int    `json:"priority"`
	HTMLPath   string `json:"html_path"` // set when status='fetched'
}

QueueItem is one row of the crawl queue.

type Quote 

type Quote struct {
	QuoteID    string    `json:"quote_id"`
	Text       string    `json:"text"`
	AuthorID   string    `json:"author_id"`
	AuthorName string    `json:"author_name"`
	BookID     string    `json:"book_id"`
	BookTitle  string    `json:"book_title"`
	LikesCount int       `json:"likes_count"`
	Tags       []string  `json:"tags"`
	URL        string    `json:"url"`
	FetchedAt  time.Time `json:"fetched_at"`
}

Quote is a Goodreads quote.

func ParseQuotes 

func ParseQuotes(doc *goquery.Document, pageURL string) ([]Quote, error)

ParseQuotes parses one or more quotes from a quotes page.

type Review 

type Review struct {
	ReviewID   string    `json:"review_id"`
	BookID     string    `json:"book_id"`
	UserID     string    `json:"user_id"`
	UserName   string    `json:"user_name"`
	Rating     int       `json:"rating"`
	Text       string    `json:"text"`
	DateAdded  time.Time `json:"date_added"`
	LikesCount int       `json:"likes_count"`
	IsSpoiler  bool      `json:"is_spoiler"`
	URL        string    `json:"url"`
	FetchedAt  time.Time `json:"fetched_at"`
}

Review is a Goodreads book review.

func ParseReviews 

func ParseReviews(doc *goquery.Document, bookID string) []Review

ParseReviews extracts embedded reviews from a book page.

type SearchResult 

type SearchResult struct {
	Position   int    `json:"position"`
	URL        string `json:"url"`
	EntityType string `json:"entity_type"` // book or author
	Title      string `json:"title"`
}

SearchResult is one result from autocomplete or full search.

func ParseSearchAutocomplete 

func ParseSearchAutocomplete(body []byte) []SearchResult

ParseSearchAutocomplete parses the JSON autocomplete response.

func ParseSearchHTML 

func ParseSearchHTML(doc *goquery.Document) []SearchResult

ParseSearchHTML parses book/author result rows from a full-search page.

type Series 

type Series struct {
	SeriesID         string    `json:"series_id"`
	Name             string    `json:"name"`
	Description      string    `json:"description"`
	TotalBooks       int       `json:"total_books"`
	PrimaryWorkCount int       `json:"primary_work_count"`
	BookIDs          []string  `json:"book_ids"`
	URL              string    `json:"url"`
	FetchedAt        time.Time `json:"fetched_at"`
}

Series is a Goodreads book series (/series/<id>).

type SeriesBook 

type SeriesBook struct {
	SeriesID string `json:"series_id"`
	BookID   string `json:"book_id"`
	Position int    `json:"position"`
}

SeriesBook links a series to a book at a position.

type Shelf 

type Shelf struct {
	ShelfID    string    `json:"shelf_id"` // "<user_id>/<shelf_name>"
	UserID     string    `json:"user_id"`
	Name       string    `json:"name"`
	BooksCount int       `json:"books_count"`
	URL        string    `json:"url"`
	FetchedAt  time.Time `json:"fetched_at"`
}

Shelf is a user's named reading shelf.

type ShelfBook 

type ShelfBook struct {
	ShelfID    string    `json:"shelf_id"`
	UserID     string    `json:"user_id"`
	BookID     string    `json:"book_id"`
	Title      string    `json:"title"`
	AuthorName string    `json:"author_name,omitempty"`
	ISBN       string    `json:"isbn,omitempty"`
	NumPages   int       `json:"num_pages,omitempty"`
	AvgRating  float64   `json:"avg_rating,omitempty"`
	Rating     int       `json:"rating"`
	Shelves    []string  `json:"shelves,omitempty"`
	ReviewID   string    `json:"review_id,omitempty"`
	ReviewText string    `json:"review_text,omitempty"`
	CoverURL   string    `json:"cover_url,omitempty"`
	DateAdded  time.Time `json:"date_added"`
	DateRead   time.Time `json:"date_read"`
}

ShelfBook links a shelf to a book with reading metadata. The RSS feed populates the richer fields (author/isbn/review/...); the HTML fallback fills only the basics.

type SitemapCategory 

type SitemapCategory struct {
	Category string `json:"category"`
	URL      string `json:"url"`
}

SitemapCategory pairs a siteindex category (author, list, quote, genre, ...) with its siteindex URL.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store is a pure-Go SQLite store for parsed records and a crawl queue.

func OpenStore 

func OpenStore(path string) (*Store, error)

OpenStore opens (creating if needed) the SQLite store at path and migrates it.

func (*Store) Close 

func (s *Store) Close() error

Close closes the underlying database.

func (*Store) Count 

func (s *Store) Count(entityType string) (int, error)

Count returns the number of stored records of a type ("" for all types).

func (*Store) CountsByType 

func (s *Store) CountsByType() (map[string]int, error)

CountsByType returns a map of entity type to record count.

func (*Store) Each 

func (s *Store) Each(entityType string, fn func(id string, data []byte) error) error

Each streams stored records of a type to fn as raw JSON.

func (*Store) Enqueue 

func (s *Store) Enqueue(ctx context.Context, url, entityType string, priority int) error

Enqueue adds a URL to the crawl queue (ignored if already present).

func (*Store) Get 

func (s *Store) Get(entityType, id string) ([]byte, error)

Get reads a record's raw JSON by (entityType, id).

func (*Store) Info 

func (s *Store) Info() (string, error)

Info returns a human summary of the store's contents.

func (*Store) MarkFailed 

func (s *Store) MarkFailed(ctx context.Context, id int64) error

MarkFailed records a queue item as failed.

func (*Store) MarkFetched 

func (s *Store) MarkFetched(ctx context.Context, id int64, htmlPath string) error

MarkFetched records a queue item as fetched with its cache path.

func (*Store) NextPending 

func (s *Store) NextPending(ctx context.Context, n int) ([]QueueItem, error)

NextPending claims and returns up to n pending queue items (status -> 'active').

func (*Store) Put 

func (s *Store) Put(entityType, id, url string, record any) error

Put upserts a record keyed by (entityType, id).

func (*Store) QueueStats 

func (s *Store) QueueStats() (map[string]int, error)

QueueStats returns counts of queue items grouped by status.

func (*Store) ResetActive 

func (s *Store) ResetActive() error

ResetActive returns any 'active' items to 'pending' (recover from a crash).

func (*Store) Vacuum 

func (s *Store) Vacuum() error

Vacuum compacts the database file.

type User 

type User struct {
	UserID          string    `json:"user_id"`
	Name            string    `json:"name"`
	Username        string    `json:"username"`
	Location        string    `json:"location"`
	JoinedDate      time.Time `json:"joined_date"`
	FriendsCount    int       `json:"friends_count"`
	BooksReadCount  int       `json:"books_read_count"`
	RatingsCount    int       `json:"ratings_count"`
	ReviewsCount    int       `json:"reviews_count"`
	AvgRating       float64   `json:"avg_rating"`
	Bio             string    `json:"bio"`
	Website         string    `json:"website"`
	AvatarURL       string    `json:"avatar_url"`
	FavoriteBookIDs []string  `json:"favorite_book_ids"`
	URL             string    `json:"url"`
	FetchedAt       time.Time `json:"fetched_at"`
}

User is a public Goodreads reader profile (/user/show/<id>).

func ParseUser 

func ParseUser(doc *goquery.Document, userID, pageURL string) (*User, error)

ParseUser parses a Goodreads user profile.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.