Documentation
¶
Overview ¶
Package connector defines the Connector and Sink interfaces and the Provenance struct returned by each extraction.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsFullSHA ¶ added in v0.4.14
IsFullSHA reports whether s is a valid 40-character hex git commit SHA. Both lowercase and uppercase hex digits are accepted.
func NextLink ¶ added in v0.4.13
NextLink returns the URL of the rel="next" entry in a Link header, or the empty string if absent. A results="false" parameter (Sentry pagination sentinel) also returns empty — the caller's for-loop exits in both cases.
func NormalizeFullSHA ¶ added in v0.4.14
NormalizeFullSHA returns the lowercase form of s and true when s is a valid 40-character hex git commit SHA; otherwise it returns ("", false). Use this instead of IsFullSHA when the result is stored or compared — all SHA columns in the schema are lowercase, so callers must not store uppercase variants.
Types ¶
type EndpointStatus ¶
type Prefetcher ¶
Prefetcher is the optional interface a Connector may implement to allow its slow per-repo work to start during the run.go clone phase, in parallel with the actual clone. Prefetch must be safe to call concurrently against distinct slugs and must stash its result somewhere Extract can find it later (typically a per-slug cache held on the connector itself). Extract remains the canonical entry point for row emission; Prefetch is purely a wall-clock hint. See ADR 022 for the connector-contract baseline and the #71 ADR for this extension.
type Provenance ¶
type Provenance struct {
Connector string `json:"connector"`
Repo string `json:"repo"`
WindowCovered Window `json:"window_covered"`
RowsReturned map[string]int `json:"rows_returned"`
PaginationComplete bool `json:"pagination_complete"`
RateLimitTruncated bool `json:"rate_limit_truncated"`
Errors map[string]string `json:"errors"`
Endpoints map[string]EndpointStatus `json:"endpoints,omitempty"`
// Flags carries boolean per-extraction signals the manifest aggregates
// across repos. Currently used for "mailmap_applied". A flag absent from
// the map reads as false; the aggregator ANDs across all provenances of
// the same connector to derive the manifest-wide value.
Flags map[string]bool `json:"flags,omitempty"`
// GraphQLPointsUsed is the total number of GraphQL rate-limit points
// consumed across all requests in this extraction. Zero when no GraphQL
// calls were made (e.g. REST-only connectors).
GraphQLPointsUsed int `json:"graphql_points_used,omitempty"`
// GraphQLPointsRemaining is the remaining GitHub GraphQL rate-limit budget
// as of the last observed response. Zero when no GraphQL calls were made.
GraphQLPointsRemaining int `json:"graphql_points_remaining,omitempty"`
// StreamCancelRetries counts the number of HTTP/2 stream CANCEL frames
// received from GitHub during this extraction and retried. Non-zero means
// at least one GraphQL walk hit GitHub's server-side CPU/time budget and
// had to back off. A successful run with retries > 0 is distinguishable
// from a clean run; a truncated run also has PaginationComplete = false.
StreamCancelRetries int `json:"stream_cancel_retries,omitempty"`
// ConfigDepth records operator-declared extraction-depth overrides that
// narrow what data was captured. Absent keys mean the connector ran at
// full depth. The analyser reads this to interpret reduced row counts as
// "out of scope" rather than "no signal". Currently used by the github
// connector for "pr_window" and "pr_history_sample".
ConfigDepth map[string]string `json:"config_depth,omitempty"`
// Sampling holds per-bucket statistics when sparse-historical PR sampling
// is active (pr_inflection + pr_history_sample configured). Nil when the
// connector ran at full PR fidelity. The analyser uses the per-bucket
// target/actual/total counts to compute confidence intervals on metrics
// derived from pre-bracket sparse data.
Sampling *SamplingProvenance `json:"sampling,omitempty"`
}
func NewProvenance ¶
func NewProvenance(name, repo string, w Window) Provenance
func (*Provenance) Merge ¶
func (p *Provenance) Merge(other Provenance)
Merge folds other into p in place. Used when a single Extract pass runs multiple goroutines that write disjoint provenance fragments (see `github.Connector.Extract`'s clone-bound vs API-bound split, #71).
Policy:
- RowsReturned counters are summed.
- Errors are first-wins per context: p's existing entry sticks, other's fills only previously-empty contexts. Callers organise goroutines so contexts don't collide; on collision the deterministic policy is to keep p's.
- PaginationComplete is ANDed.
- RateLimitTruncated is ORed.
- Endpoints and Flags: other's entry fills if p has none for the key; existing entries on p are preserved.
Window, Connector, Repo are not merged — those are set at NewProvenance time and other's values must match. The caller is expected to pass a fragment built from the same (connector, repo, window).
type SampleBucket ¶ added in v0.4.9
type SampleBucket struct {
Month string `json:"month"` // "2022-01" or "2022-01-W1" for weekly sub-buckets
Target int `json:"target"` // requested N per bucket
Actual int `json:"actual"` // PRs emitted
Total int `json:"total"` // totalCount from GraphQL search
Truncated bool `json:"truncated,omitempty"` // true when Total > 1000 (search cap)
}
SampleBucket records the extraction result for one month (or week) bucket in the pre-bracket sparse slice.
type SamplingProvenance ¶ added in v0.4.9
type SamplingProvenance struct {
InflectionDate string `json:"inflection_date"` // "2023-06-01"
BracketWindow string `json:"bracket_window"` // "12m"
BracketStart string `json:"bracket_start"` // "2022-06-01"
BracketEnd string `json:"bracket_end"` // "2026-06-15"
Strategy string `json:"strategy"` // "search_default_relevance" | "random"
Buckets []SampleBucket `json:"buckets"`
}
SamplingProvenance records the sparse-historical PR sampling configuration and per-bucket extraction results. Present only when pr_inflection is set.
type Sink ¶
type Sink interface {
InsertRepo(model.Repo) error
InsertTeamRepo(team, repo string) error
InsertRepoLanguage(model.RepoLanguage) error
InsertBranch(model.Branch) error
InsertBranchProtection(model.BranchProtection) error
InsertCodeowner(model.Codeowner) error
InsertCommit(model.Commit) error
InsertCommitFile(model.CommitFile) error
InsertPR(model.PR) error
InsertPRCommit(model.PRCommit) error
InsertReview(model.Review) error
InsertPRComment(model.PRComment) error
InsertPRReviewRequest(model.PRReviewRequest) error
InsertPRLabel(model.PRLabel) error
InsertBuild(model.Build) error
InsertBuildJob(model.BuildJob) error
InsertDeploy(model.Deploy) error
InsertRelease(model.Release) error
InsertIncident(model.Incident) error
InsertDefect(model.Defect) error
InsertFileMetric(model.FileMetric) error
InsertHarnessArtifact(model.HarnessArtifact) error
InsertFileComplexityHistory(model.FileComplexityHistory) error
InsertRepoFile(model.RepoFile) error
}
Sink is the typed insertion surface every connector writes against. Methods are typed per canonical table so the compiler enforces that connectors cannot invent tables outside the schema.