discovery

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 1, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package discovery turns explicit repositories, GitHub repository searches, and GH Archive events into bounded discovery signals.

Search partitioning and archive readers are cancellation-aware and expose checkpoint state to callers. This package identifies candidates; the application and corpus layers own network authorization and persistence.

Index

Constants

View Source
const (
	// DefaultArchiveBaseURL is the canonical GH Archive hourly endpoint.
	DefaultArchiveBaseURL = "https://data.gharchive.org"
	// DefaultArchiveTimeout bounds a single hourly download.
	DefaultArchiveTimeout = 30 * time.Second
	// DefaultArchiveMaxBytes is the maximum compressed response body for one
	// hourly file (256 MiB). GH Archive hour files are typically well under
	// this size.
	DefaultArchiveMaxBytes = 256 << 20
)
View Source
const DefaultSearchLimit = 1000

DefaultSearchLimit is the GitHub Search hard result ceiling that forces query windowing.

Variables

View Source
var (
	// ErrHourNotAvailable is returned when the requested hour file does not
	// exist (HTTP 404).
	ErrHourNotAvailable = errors.New("archive hour not available")
	// ErrResponseTooLarge is returned when a response exceeds the configured
	// byte limit.
	ErrResponseTooLarge = errors.New("archive response exceeds size limit")
	// ErrBadStatus is returned for unexpected HTTP status codes.
	ErrBadStatus = errors.New("unexpected archive status")
)
View Source
var ErrAlreadyImported = errors.New("hour already imported")

ErrAlreadyImported is returned by ArchiveReader when a GH Archive hour has already been recorded as imported.

View Source
var ErrDecompressedTooLarge = errors.New("decompressed archive exceeds size limit")

ErrDecompressedTooLarge is returned when an hour's decompressed payload exceeds ArchiveReader.MaxTotalBytes.

View Source
var ErrLimitExceeded = errors.New("size limit exceeded")

ErrLimitExceeded is the generic size-limit error returned by LimitedReader.

Functions

func ArchiveHourRange

func ArchiveHourRange(since time.Duration, now time.Time) (start, end time.Time)

ArchiveHourRange returns the inclusive hourly bounds for a --since crawl. The latest complete hour is the hour before the current hour, because the current hour's file may not yet be published.

func HourKey

func HourKey(hour time.Time) string

HourKey returns a stable, UTC hour identifier for checkpoint storage.

func IsKnownEventType

func IsKnownEventType(t string) bool

IsKnownEventType reports whether t is a recognized GH Archive event type.

func ParseRepoRef

func ParseRepoRef(s string) (domain.RepoRef, error)

ParseRepoRef converts an explicit owner/repo reference into a validated domain.RepoRef. Supported forms are:

A trailing ".git" and query/fragment components are stripped.

func ParseRepoRefs

func ParseRepoRefs(inputs []string) ([]domain.RepoRef, error)

ParseRepoRefs parses a slice of explicit references. It stops at the first invalid entry and returns the successfully parsed prefix.

Types

type ArchiveClient

type ArchiveClient struct {
	// contains filtered or unexported fields
}

ArchiveClient is a context-aware, bounded HTTP fetcher for GH Archive.

func NewArchiveClientWithOptions

func NewArchiveClientWithOptions(baseURL string, client *http.Client, timeout time.Duration, maxBytes int64) *ArchiveClient

NewArchiveClientWithOptions returns a fetcher using the supplied parameters. It is intended for tests and advanced configuration; callers should avoid exposing arbitrary base URLs to untrusted input.

func (*ArchiveClient) Fetch

func (c *ArchiveClient) Fetch(ctx context.Context, hour time.Time) (io.ReadCloser, error)

Fetch builds the canonical https://data.gharchive.org/YYYY-MM-DD-H.json.gz URL, applies a per-request timeout, checks status and response size, and returns a ReadCloser that enforces maxBytes while streaming.

type ArchiveFetcher

type ArchiveFetcher interface {
	Fetch(ctx context.Context, hour time.Time) (io.ReadCloser, error)
}

ArchiveFetcher downloads a single hourly GH Archive gzip file.

func NewArchiveClient

func NewArchiveClient() ArchiveFetcher

NewArchiveClient returns an ArchiveFetcher with sensible production defaults.

type ArchiveReader

type ArchiveReader struct {
	Include       map[string]bool
	Store         CheckpointStore
	MaxEventBytes int
	// MaxTotalBytes bounds the total decompressed bytes for an hour. Zero
	// disables the limit.
	MaxTotalBytes int
}

ArchiveReader streams an hourly GH Archive gzip file line by line, retains only configured event types, and emits normalized repository/thread signals.

func NewArchiveReader

func NewArchiveReader(include []string, store CheckpointStore) *ArchiveReader

NewArchiveReader creates a reader that retains the given event types. An empty include list retains all events.

func (*ArchiveReader) Read

func (r *ArchiveReader) Read(ctx context.Context, hour time.Time, in io.Reader, emit func(Signal) error) error

Read decompresses the hourly gzip stream, parses JSON lines, and emits a Signal for each retained event. It checks the checkpoint store for hour idempotency and marks the hour imported on successful completion. Malformed lines are skipped. If ctx is cancelled, Read returns the cancellation error.

type CheckpointStore

type CheckpointStore interface {
	// GetTime returns the timestamp checkpoint for key. If no checkpoint exists,
	// the bool is false.
	GetTime(ctx context.Context, key string) (time.Time, bool, error)

	// SetTime stores a timestamp checkpoint for key.
	SetTime(ctx context.Context, key string, t time.Time) error

	// IsImported reports whether the given GH Archive hour has already been
	// imported.
	IsImported(ctx context.Context, hour string) (bool, error)

	// MarkImported records the given GH Archive hour as imported.
	MarkImported(ctx context.Context, hour string) error
}

CheckpointStore is a small durable interface used by discovery sources. Implementations may be in-memory, file-backed, or persisted elsewhere, but must not perform network access or GitHub mutations.

type EventType

type EventType string

EventType identifies a GH Archive event type.

const (
	PushEvent                     EventType = "PushEvent"
	IssuesEvent                   EventType = "IssuesEvent"
	PullRequestEvent              EventType = "PullRequestEvent"
	IssueCommentEvent             EventType = "IssueCommentEvent"
	PullRequestReviewEvent        EventType = "PullRequestReviewEvent"
	PullRequestReviewCommentEvent EventType = "PullRequestReviewCommentEvent"
	ReleaseEvent                  EventType = "ReleaseEvent"
	WatchEvent                    EventType = "WatchEvent"
	ForkEvent                     EventType = "ForkEvent"
	DiscussionEvent               EventType = "DiscussionEvent"
	DiscussionCommentEvent        EventType = "DiscussionCommentEvent"
)

Recognized GH Archive event types.

type LimitedReader

type LimitedReader struct {
	R         io.Reader
	N         int64
	Err       error
	CloseFunc func() error
	// contains filtered or unexported fields
}

LimitedReader caps the number of bytes read from an underlying reader and returns a configured error if the limit is exceeded. It closes the underlying read-closer when closed.

func NewLimitedReader

func NewLimitedReader(r io.Reader, n int64, closeFunc func() error, err error) *LimitedReader

NewLimitedReader returns a reader that returns err after reading more than n bytes. If err is nil, ErrLimitExceeded is used. The close function is called on Close; if nil, Close is a no-op unless R implements io.ReadCloser.

func (*LimitedReader) Close

func (l *LimitedReader) Close() error

Close releases the underlying reader.

func (*LimitedReader) Read

func (l *LimitedReader) Read(p []byte) (int, error)

Read returns up to the remaining allowed bytes. When the limit is reached, the next call returns Err. It returns io.EOF if the underlying stream ends exactly at the limit.

type MemoryCheckpointStore

type MemoryCheckpointStore struct {
	// contains filtered or unexported fields
}

MemoryCheckpointStore is an in-memory CheckpointStore for tests and short-lived local use.

func NewMemoryCheckpointStore

func NewMemoryCheckpointStore() *MemoryCheckpointStore

NewMemoryCheckpointStore returns a new in-memory checkpoint store.

func (*MemoryCheckpointStore) GetTime

func (m *MemoryCheckpointStore) GetTime(ctx context.Context, key string) (time.Time, bool, error)

GetTime returns the timestamp checkpoint for key.

func (*MemoryCheckpointStore) IsImported

func (m *MemoryCheckpointStore) IsImported(ctx context.Context, hour string) (bool, error)

IsImported reports whether the given hour has already been imported.

func (*MemoryCheckpointStore) MarkImported

func (m *MemoryCheckpointStore) MarkImported(ctx context.Context, hour string) error

MarkImported records the given hour as imported.

func (*MemoryCheckpointStore) SetTime

func (m *MemoryCheckpointStore) SetTime(ctx context.Context, key string, t time.Time) error

SetTime stores a timestamp checkpoint for key.

type Qualifier

type Qualifier string

Qualifier selects which GitHub Search timestamp field to window.

const (
	// Created windows historical backfills.
	Created Qualifier = "created"
	// Updated windows incremental refresh.
	Updated Qualifier = "updated"
	// Pushed windows incremental repository-search refreshes. GitHub repository
	// search supports pushed, not updated.
	Pushed Qualifier = "pushed"
)

type SearchItem

type SearchItem struct {
	Repo   domain.RepoRef
	Kind   domain.ThreadKind
	Number int
	Title  string
}

SearchItem is a product-owned search result item used when partitioning.

type SearchPartitioner

type SearchPartitioner struct {
	Searcher Searcher
	// Limit is the result count that triggers splitting. Zero means 1000.
	Limit int
}

SearchPartitioner splits a base query into time windows that each fit under a result-count limit.

func (*SearchPartitioner) Partition

func (p *SearchPartitioner) Partition(ctx context.Context, baseQuery string, start, end time.Time, qual Qualifier) ([]Window, error)

Partition recursively splits baseQuery over [start, end] (inclusive) using the chosen qualifier until each window's total result count is at or below the configured limit or the window cannot be split further. Unsplittable windows with more than the limit are returned with Unsplittable set.

func (*SearchPartitioner) Refresh

func (p *SearchPartitioner) Refresh(ctx context.Context, key, baseQuery string, start, end time.Time, overlap time.Duration, store CheckpointStore) ([]Window, error)

Refresh partitions an incremental updated query from the most recent checkpoint, extended backwards by overlap. The new checkpoint is written to store as end on success.

type SearchResponse

type SearchResponse struct {
	Total      int
	Items      []SearchItem
	Incomplete bool
}

SearchResponse is the product-owned result of a search count/list call. Incomplete means GitHub returned incomplete_results.

type Searcher

type Searcher interface {
	Search(ctx context.Context, query string) (SearchResponse, error)
}

Searcher is the capability required to partition a search query.

type Signal

type Signal struct {
	Source       string
	Hour         time.Time
	ObservedAt   time.Time
	EventType    EventType
	Action       string
	Repo         domain.RepoRef
	RepoID       int64
	Actor        string
	ThreadKind   domain.ThreadKind
	ThreadNumber int
	ThreadTitle  string
	ThreadAuthor string
	ThreadState  domain.ThreadState
	Merged       bool
	Ref          string
	SHA          string
	Size         int
	TagName      string
}

Signal is a normalized, product-owned discovery signal emitted from GH Archive events. Not all fields are populated for every event kind.

type Window

type Window struct {
	Query        string
	Qualifier    Qualifier
	Start        time.Time
	End          time.Time
	Total        int
	Incomplete   bool
	Unsplittable bool
}

Window is a bounded GitHub Search query with its total result count and whether it could not be split further.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL