Documentation
¶
Overview ¶
Package discovery turns explicit repositories, GitHub repository searches, and GH Archive events into bounded discovery signals.
Search partitioning and archive readers are cancellation-aware and expose checkpoint state to callers. This package identifies candidates; the application and corpus layers own network authorization and persistence.
Index ¶
- Constants
- Variables
- func ArchiveHourRange(since time.Duration, now time.Time) (start, end time.Time)
- func HourKey(hour time.Time) string
- func IsKnownEventType(t string) bool
- func ParseRepoRef(s string) (domain.RepoRef, error)
- func ParseRepoRefs(inputs []string) ([]domain.RepoRef, error)
- type ArchiveClient
- type ArchiveFetcher
- type ArchiveReader
- type CheckpointStore
- type EventType
- type LimitedReader
- type MemoryCheckpointStore
- func (m *MemoryCheckpointStore) GetTime(ctx context.Context, key string) (time.Time, bool, error)
- func (m *MemoryCheckpointStore) IsImported(ctx context.Context, hour string) (bool, error)
- func (m *MemoryCheckpointStore) MarkImported(ctx context.Context, hour string) error
- func (m *MemoryCheckpointStore) SetTime(ctx context.Context, key string, t time.Time) error
- type Qualifier
- type SearchItem
- type SearchPartitioner
- type SearchResponse
- type Searcher
- type Signal
- type Window
Constants ¶
const ( // DefaultArchiveBaseURL is the canonical GH Archive hourly endpoint. DefaultArchiveBaseURL = "https://data.gharchive.org" // DefaultArchiveTimeout bounds a single hourly download. DefaultArchiveTimeout = 30 * time.Second // DefaultArchiveMaxBytes is the maximum compressed response body for one // hourly file (256 MiB). GH Archive hour files are typically well under // this size. DefaultArchiveMaxBytes = 256 << 20 )
const DefaultSearchLimit = 1000
DefaultSearchLimit is the GitHub Search hard result ceiling that forces query windowing.
Variables ¶
var ( // ErrHourNotAvailable is returned when the requested hour file does not // exist (HTTP 404). ErrHourNotAvailable = errors.New("archive hour not available") // ErrResponseTooLarge is returned when a response exceeds the configured // byte limit. ErrResponseTooLarge = errors.New("archive response exceeds size limit") // ErrBadStatus is returned for unexpected HTTP status codes. ErrBadStatus = errors.New("unexpected archive status") )
var ErrAlreadyImported = errors.New("hour already imported")
ErrAlreadyImported is returned by ArchiveReader when a GH Archive hour has already been recorded as imported.
var ErrDecompressedTooLarge = errors.New("decompressed archive exceeds size limit")
ErrDecompressedTooLarge is returned when an hour's decompressed payload exceeds ArchiveReader.MaxTotalBytes.
var ErrLimitExceeded = errors.New("size limit exceeded")
ErrLimitExceeded is the generic size-limit error returned by LimitedReader.
Functions ¶
func ArchiveHourRange ¶
ArchiveHourRange returns the inclusive hourly bounds for a --since crawl. The latest complete hour is the hour before the current hour, because the current hour's file may not yet be published.
func IsKnownEventType ¶
IsKnownEventType reports whether t is a recognized GH Archive event type.
func ParseRepoRef ¶
ParseRepoRef converts an explicit owner/repo reference into a validated domain.RepoRef. Supported forms are:
- owner/repo
- github.com/owner/repo
- https://github.com/owner/repo
- http://github.com/owner/repo
- git@github.com:owner/repo
- ssh://git@github.com/owner/repo
A trailing ".git" and query/fragment components are stripped.
Types ¶
type ArchiveClient ¶
type ArchiveClient struct {
// contains filtered or unexported fields
}
ArchiveClient is a context-aware, bounded HTTP fetcher for GH Archive.
func NewArchiveClientWithOptions ¶
func NewArchiveClientWithOptions(baseURL string, client *http.Client, timeout time.Duration, maxBytes int64) *ArchiveClient
NewArchiveClientWithOptions returns a fetcher using the supplied parameters. It is intended for tests and advanced configuration; callers should avoid exposing arbitrary base URLs to untrusted input.
func (*ArchiveClient) Fetch ¶
func (c *ArchiveClient) Fetch(ctx context.Context, hour time.Time) (io.ReadCloser, error)
Fetch builds the canonical https://data.gharchive.org/YYYY-MM-DD-H.json.gz URL, applies a per-request timeout, checks status and response size, and returns a ReadCloser that enforces maxBytes while streaming.
type ArchiveFetcher ¶
ArchiveFetcher downloads a single hourly GH Archive gzip file.
func NewArchiveClient ¶
func NewArchiveClient() ArchiveFetcher
NewArchiveClient returns an ArchiveFetcher with sensible production defaults.
type ArchiveReader ¶
type ArchiveReader struct {
Include map[string]bool
Store CheckpointStore
MaxEventBytes int
// MaxTotalBytes bounds the total decompressed bytes for an hour. Zero
// disables the limit.
MaxTotalBytes int
}
ArchiveReader streams an hourly GH Archive gzip file line by line, retains only configured event types, and emits normalized repository/thread signals.
func NewArchiveReader ¶
func NewArchiveReader(include []string, store CheckpointStore) *ArchiveReader
NewArchiveReader creates a reader that retains the given event types. An empty include list retains all events.
func (*ArchiveReader) Read ¶
func (r *ArchiveReader) Read(ctx context.Context, hour time.Time, in io.Reader, emit func(Signal) error) error
Read decompresses the hourly gzip stream, parses JSON lines, and emits a Signal for each retained event. It checks the checkpoint store for hour idempotency and marks the hour imported on successful completion. Malformed lines are skipped. If ctx is cancelled, Read returns the cancellation error.
type CheckpointStore ¶
type CheckpointStore interface {
// GetTime returns the timestamp checkpoint for key. If no checkpoint exists,
// the bool is false.
GetTime(ctx context.Context, key string) (time.Time, bool, error)
// SetTime stores a timestamp checkpoint for key.
SetTime(ctx context.Context, key string, t time.Time) error
// IsImported reports whether the given GH Archive hour has already been
// imported.
IsImported(ctx context.Context, hour string) (bool, error)
// MarkImported records the given GH Archive hour as imported.
MarkImported(ctx context.Context, hour string) error
}
CheckpointStore is a small durable interface used by discovery sources. Implementations may be in-memory, file-backed, or persisted elsewhere, but must not perform network access or GitHub mutations.
type EventType ¶
type EventType string
EventType identifies a GH Archive event type.
const ( PushEvent EventType = "PushEvent" IssuesEvent EventType = "IssuesEvent" PullRequestEvent EventType = "PullRequestEvent" IssueCommentEvent EventType = "IssueCommentEvent" PullRequestReviewEvent EventType = "PullRequestReviewEvent" PullRequestReviewCommentEvent EventType = "PullRequestReviewCommentEvent" ReleaseEvent EventType = "ReleaseEvent" WatchEvent EventType = "WatchEvent" ForkEvent EventType = "ForkEvent" DiscussionEvent EventType = "DiscussionEvent" DiscussionCommentEvent EventType = "DiscussionCommentEvent" )
Recognized GH Archive event types.
type LimitedReader ¶
type LimitedReader struct {
R io.Reader
N int64
Err error
CloseFunc func() error
// contains filtered or unexported fields
}
LimitedReader caps the number of bytes read from an underlying reader and returns a configured error if the limit is exceeded. It closes the underlying read-closer when closed.
func NewLimitedReader ¶
NewLimitedReader returns a reader that returns err after reading more than n bytes. If err is nil, ErrLimitExceeded is used. The close function is called on Close; if nil, Close is a no-op unless R implements io.ReadCloser.
func (*LimitedReader) Close ¶
func (l *LimitedReader) Close() error
Close releases the underlying reader.
type MemoryCheckpointStore ¶
type MemoryCheckpointStore struct {
// contains filtered or unexported fields
}
MemoryCheckpointStore is an in-memory CheckpointStore for tests and short-lived local use.
func NewMemoryCheckpointStore ¶
func NewMemoryCheckpointStore() *MemoryCheckpointStore
NewMemoryCheckpointStore returns a new in-memory checkpoint store.
func (*MemoryCheckpointStore) IsImported ¶
IsImported reports whether the given hour has already been imported.
func (*MemoryCheckpointStore) MarkImported ¶
func (m *MemoryCheckpointStore) MarkImported(ctx context.Context, hour string) error
MarkImported records the given hour as imported.
type Qualifier ¶
type Qualifier string
Qualifier selects which GitHub Search timestamp field to window.
type SearchItem ¶
SearchItem is a product-owned search result item used when partitioning.
type SearchPartitioner ¶
type SearchPartitioner struct {
Searcher Searcher
// Limit is the result count that triggers splitting. Zero means 1000.
Limit int
}
SearchPartitioner splits a base query into time windows that each fit under a result-count limit.
func (*SearchPartitioner) Partition ¶
func (p *SearchPartitioner) Partition(ctx context.Context, baseQuery string, start, end time.Time, qual Qualifier) ([]Window, error)
Partition recursively splits baseQuery over [start, end] (inclusive) using the chosen qualifier until each window's total result count is at or below the configured limit or the window cannot be split further. Unsplittable windows with more than the limit are returned with Unsplittable set.
func (*SearchPartitioner) Refresh ¶
func (p *SearchPartitioner) Refresh(ctx context.Context, key, baseQuery string, start, end time.Time, overlap time.Duration, store CheckpointStore) ([]Window, error)
Refresh partitions an incremental updated query from the most recent checkpoint, extended backwards by overlap. The new checkpoint is written to store as end on success.
type SearchResponse ¶
type SearchResponse struct {
Total int
Items []SearchItem
Incomplete bool
}
SearchResponse is the product-owned result of a search count/list call. Incomplete means GitHub returned incomplete_results.
type Searcher ¶
type Searcher interface {
Search(ctx context.Context, query string) (SearchResponse, error)
}
Searcher is the capability required to partition a search query.
type Signal ¶
type Signal struct {
Source string
Hour time.Time
ObservedAt time.Time
EventType EventType
Action string
Repo domain.RepoRef
RepoID int64
Actor string
ThreadKind domain.ThreadKind
ThreadNumber int
ThreadTitle string
ThreadAuthor string
ThreadState domain.ThreadState
Merged bool
Ref string
SHA string
Size int
TagName string
}
Signal is a normalized, product-owned discovery signal emitted from GH Archive events. Not all fields are populated for every event kind.