Documentation
¶
Overview ¶
Package store is the persistence layer for the Douban mirror: a frontier of URLs to crawl, the normalized records harvested from them, crawl cursors, and the raw artifact files captured verbatim.
It uses a pure-Go SQLite (modernc.org/sqlite) so the binary keeps building with CGO_ENABLED=0. The database lives at <dir>/douban.db; raw artifacts go under <dir>/raw/ and JSONL exports under <dir>/export/.
Index ¶
- Constants
- type Frontier
- type Record
- type Store
- func (s *Store) Close() error
- func (s *Store) Complete(url, status string, httpStatus int, errMsg, rawPath, hash string) error
- func (s *Store) DBPath() string
- func (s *Store) Dir() string
- func (s *Store) Enqueue(f Frontier) (bool, error)
- func (s *Store) ExportDir() string
- func (s *Store) ExportJSONL(ctx context.Context, entityType string, w io.Writer) (int, error)
- func (s *Store) FrontierCounts() (map[string]map[string]int, error)
- func (s *Store) GetMeta(key string) (value string, ok bool, err error)
- func (s *Store) NextPending(limit int) ([]Frontier, error)
- func (s *Store) Pending(types []string, source string, limit int) ([]Frontier, error)
- func (s *Store) PutRecord(r Record) error
- func (s *Store) QueueRows(status, entityType string, limit int) ([]Frontier, error)
- func (s *Store) RawDir() string
- func (s *Store) ReadRaw(relPath string) ([]byte, error)
- func (s *Store) RecordCounts() (map[string]int, error)
- func (s *Store) ResetFailed(entityType string) (int64, error)
- func (s *Store) SetMeta(key, value string) error
- func (s *Store) WriteRaw(source, entityType, id, ext string, data []byte) (relPath, hash string, err error)
Constants ¶
const ( StatusPending = "pending" StatusDone = "done" StatusFailed = "failed" StatusBlocked = "blocked" StatusSkipped = "skipped" )
Frontier status values.
const ( SourceFrodo = "frodo" SourceHTML = "html" SourceMobile = "mobile" )
Source values record which surface produced an artifact or record.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Frontier ¶
type Frontier struct {
URL string
EntityType string
EntityID string
Source string
Status string
Attempts int
Priority int
HTTPStatus int
Error string
RawPath string
ContentHash string
DiscoveredAt time.Time
FetchedAt time.Time
}
Frontier is one row of the crawl queue: a URL plus what is known about it.
type Record ¶
type Record struct {
EntityType string
EntityID string
Source string
JSON json.RawMessage
FetchedAt time.Time
}
Record is a normalized structured entity, stored as JSON and exported to JSONL.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store owns the mirror directory and its SQLite database.
func (*Store) Complete ¶
Complete marks a URL with a terminal (or retryable) status, recording the HTTP status, any error, the raw artifact path and hash, and bumping the attempt count. The fetched_at timestamp is set for done/blocked/skipped.
func (*Store) Enqueue ¶
Enqueue inserts a frontier URL if it is not already present. It returns true when a new row was added. Re-seeding an existing URL is a no-op (its status and history are preserved).
func (*Store) ExportJSONL ¶
ExportJSONL streams records of entityType (empty = all) to w, one JSON object per line. It returns the number of records written.
func (*Store) FrontierCounts ¶
FrontierCounts returns frontier row counts grouped by (entity_type, status).
func (*Store) NextPending ¶
NextPending returns up to limit pending frontier rows, highest priority first. Callers coordinate dequeueing (a single feeder goroutine) so the same row is not handed to two workers; rows stay pending while in flight so an interrupted run resumes cleanly.
func (*Store) Pending ¶
Pending returns up to limit pending rows, optionally filtered to the given entity types and/or source, highest priority first. An empty types slice and empty source mean "any". Like NextPending, callers coordinate dequeueing.
func (*Store) QueueRows ¶
QueueRows returns frontier rows filtered by status and/or type, for inspection. Empty status or entityType means "any".
func (*Store) ReadRaw ¶
ReadRaw returns the decompressed bytes of a raw artifact given its relative path.
func (*Store) RecordCounts ¶
RecordCounts returns record counts grouped by entity_type.
func (*Store) ResetFailed ¶
ResetFailed moves failed rows back to pending so a later crawl retries them. An empty entityType resets every type. It returns the number requeued.
func (*Store) WriteRaw ¶
func (s *Store) WriteRaw(source, entityType, id, ext string, data []byte) (relPath, hash string, err error)
WriteRaw gzips data to raw/<source>/<type>/<shard>/<id>.<ext>.gz and returns the path relative to the mirror dir plus the sha256 of the uncompressed bytes. The shard is the last two digits of the id, capping directory fan-out.
Source Files