Documentation
¶
Overview ¶
Package store is the persistence layer for the Baidu mirror: a crawl queue, a visited set, the harvested records (Baike articles, search results, suggest queries), the per-run job log, and the raw artifact files captured verbatim.
It uses a pure-Go SQLite (modernc.org/sqlite) so the binary keeps building with CGO_ENABLED=0. The database lives at <dir>/baidu.db; raw artifacts go under <dir>/raw/ and JSONL/Markdown exports under <dir>/export/.
The schema is the go-mizu mirror schema translated from DuckDB to SQLite: CREATE SEQUENCE / nextval become INTEGER PRIMARY KEY AUTOINCREMENT, NOW() becomes CURRENT_TIMESTAMP, and the final columns are declared up front rather than added by ALTER TABLE migrations.
Index ¶
- Constants
- type Job
- type QueueItem
- type QueueStat
- type Store
- func (s *Store) ArticleCount() (int, error)
- func (s *Store) Close() error
- func (s *Store) CreateJob(kind, detail string) (int64, error)
- func (s *Store) DBPath() string
- func (s *Store) Dir() string
- func (s *Store) Done(it QueueItem) error
- func (s *Store) EachArticle(ctx context.Context, fn func(lemmaID int, title string, js []byte) error) error
- func (s *Store) Enqueue(it QueueItem) (bool, error)
- func (s *Store) EnqueueBatch(items []QueueItem) (int, error)
- func (s *Store) ExportDir() string
- func (s *Store) ExportJSONL(ctx context.Context, kind string, w io.Writer) (int, error)
- func (s *Store) Fail(it QueueItem, cause string) error
- func (s *Store) GetMeta(key string) (value string, ok bool, err error)
- func (s *Store) IsVisited(entityType, ref string) (bool, error)
- func (s *Store) ListJobs(limit int) ([]Job, error)
- func (s *Store) ListQueue(status string, limit int) ([]QueueItem, error)
- func (s *Store) MarkFailed(it QueueItem, cause string) error
- func (s *Store) PendingCount() (int, error)
- func (s *Store) Pop(n int) ([]QueueItem, error)
- func (s *Store) QueueStats() ([]QueueStat, error)
- func (s *Store) RawDir() string
- func (s *Store) ReadRaw(relPath string) ([]byte, error)
- func (s *Store) ResetFailed() (int64, error)
- func (s *Store) SearchResultCount() (int, error)
- func (s *Store) SetMeta(key, value string) error
- func (s *Store) SuggestCount() (int, error)
- func (s *Store) UpdateJob(id int64, status string, processed, errCount int) error
- func (s *Store) UpsertArticle(lemmaID int, title string, js json.RawMessage, rawPath string) error
- func (s *Store) UpsertSearchResult(id, query string, page, position int, js json.RawMessage) error
- func (s *Store) UpsertSuggest(query string, rank int, word string) error
- func (s *Store) VisitedCount() (int, error)
- func (s *Store) WriteRaw(entityType, id, ext string, data []byte) (relPath, hash string, err error)
Constants ¶
const ( StatusPending = "pending" StatusDone = "done" StatusFailed = "failed" )
Queue status values.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Job ¶
type Job struct {
ID int64
Kind string // seed, crawl, export, reset-failed
Status string // running, done, failed
Detail string
Processed int
Errors int
StartedAt time.Time
EndedAt time.Time
}
Job is one row of the per-run job log.
type QueueItem ¶
type QueueItem struct {
ID int64
EntityType string // lemma, serp
Ref string // lemma id, title, or "query|page" for serp
URL string
Priority int
Status string
Attempts int
Error string
}
QueueItem is one row of the crawl queue.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store owns the mirror directory and its SQLite database.
func (*Store) ArticleCount ¶
ArticleCount returns the number of stored Baike articles.
func (*Store) Done ¶
Done marks a queue row done and records the (entity_type, ref) in visited, in one transaction.
func (*Store) EachArticle ¶
func (s *Store) EachArticle(ctx context.Context, fn func(lemmaID int, title string, js []byte) error) error
EachArticle calls fn for every stored Baike article JSON, ordered by lemma id. It is the read side used by the Markdown exporter.
func (*Store) Enqueue ¶
Enqueue inserts a queue item if (entity_type, ref) is not already present. It returns true when a new row was added. Re-seeding an existing ref is a no-op so status and history are preserved.
func (*Store) EnqueueBatch ¶
EnqueueBatch enqueues many items in one transaction. It returns the number of new rows added.
func (*Store) ExportJSONL ¶
ExportJSONL streams the JSON of one entity kind to w, one object per line. kind is one of "article", "search", "suggest". It returns the number of rows written.
func (*Store) Fail ¶
Fail bumps a row's attempt count and records the error. After 3 attempts the row is moved to failed; otherwise it stays pending for a later retry.
func (*Store) ListQueue ¶
ListQueue returns queue rows filtered by status (empty = any), oldest first.
func (*Store) MarkFailed ¶
MarkFailed moves a row straight to failed regardless of attempt count. It is used for terminal outcomes a retry cannot fix, such as a walled SERP.
func (*Store) PendingCount ¶
PendingCount returns how many queue rows are still pending.
func (*Store) Pop ¶
Pop claims up to n pending queue items, marking them done is the caller's job via Done/Fail. It returns the highest-priority pending rows. Rows stay pending while in flight so an interrupted run resumes cleanly; callers coordinate dequeueing through a single feeder.
func (*Store) QueueStats ¶
QueueStats returns queue counts grouped by status.
func (*Store) ReadRaw ¶
ReadRaw returns the decompressed bytes of a raw artifact given its relative path.
func (*Store) ResetFailed ¶
ResetFailed moves failed rows back to pending so a later crawl retries them. It returns the number requeued.
func (*Store) SearchResultCount ¶
func (*Store) SuggestCount ¶
func (*Store) UpsertArticle ¶
UpsertArticle stores a Baike article record.
func (*Store) UpsertSearchResult ¶
UpsertSearchResult stores a single SERP entry.
func (*Store) UpsertSuggest ¶
UpsertSuggest stores one suggest entry.
Source Files