Documentation
¶
Index ¶
Constants ¶
const DEFAULT_BRANCH string = "main"
Variables ¶
This section is empty.
Functions ¶
func NewGitHubAPIIterator ¶
NewGitHubAPIIterator() returns a new `GitHubAPIIterator` instance configured by 'uri' in the form of:
githubapi://{ORGANIZATION}/{REPO}/{BRANCH}?{PARAMETERS}
Where {ORGANIZATION} is the name of the owner (organization) of the repository being iterated over and {REPO} is the name of that repository. {BRANCH} is an optional path to indicate a specific branch to use (default is "main"). {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules. * `?concurrent=` A boolean value indicating whether the repository contents should be iterated over concurrently.
func NewOrganizationIterator ¶
NewOrganizationIterator returns a new `OrganizationIterator` configured by 'uri' which takes the form of:
org://{PATH}?{PARAMETERS}
Where {PATH} is an optional path where individual Git repositories should be downloaded for processing; {PARAMETERS} is optional and may be any of the valid parameters used in URIs to create a new `whosonfirst/go-whosonfirst-iterate-git.GitIterator`. If {PATH} is not defined then Git repositories are download in to, and processed from, memory. If {PATH} is defined any Git repositories downloaded will be remove after processing (unless the `?preserve=1` query parameter is present).
Types ¶
type GitHubAPIIterator ¶
type GitHubAPIIterator struct {
iterate.Iterator
// contains filtered or unexported fields
}
GitHubIterator implements the `Iterator` interface for crawling records in a GitHub repository using the GitHub API.
func (*GitHubAPIIterator) Close ¶
func (it *GitHubAPIIterator) Close() error
Close performs any implementation specific tasks before terminating the iterator.
func (*GitHubAPIIterator) IsIterating ¶
func (it *GitHubAPIIterator) IsIterating() bool
IsIterating() returns a boolean value indicating whether 'it' is still processing documents.
func (*GitHubAPIIterator) Iterate ¶
func (it *GitHubAPIIterator) Iterate(ctx context.Context, uris ...string) iter.Seq2[*iterate.Record, error]
Iterate will return an `iter.Seq2[*Record, error]` for each record encountered in 'uris'.
func (*GitHubAPIIterator) Seen ¶
func (it *GitHubAPIIterator) Seen() int64
Seen() returns the total number of records processed so far.
type OrganizationIterator ¶
type OrganizationIterator struct {
iterate.Iterator
// contains filtered or unexported fields
}
type OrganizationIterator implements the `iterator.Iterator` interface for iterating multiple repositories in a GitHub organization.
func (*OrganizationIterator) Close ¶
func (it *OrganizationIterator) Close() error
Close performs any implementation specific tasks before terminating the iterator.
func (*OrganizationIterator) IsIterating ¶
func (it *OrganizationIterator) IsIterating() bool
IsIterating() returns a boolean value indicating whether 'it' is still processing documents.
func (*OrganizationIterator) Iterate ¶
func (it *OrganizationIterator) Iterate(ctx context.Context, uris ...string) iter.Seq2[*iterate.Record, error]
Iterate will return an `iter.Seq2[*Record, error]` for each record encountered in 'uris'. The list of files to process is determined by 'uris' which takes the form of:
{GITHUB_ORGANIZATION}://?prefix={STRING}&exclude={STRING}&access_token={STRING}
Where {PREFIX} is zero or more "prefix" parameters to filter the list of repositories by for inclusion; {EXCLUDE} is zero or more "exclude" query parameters to filter the list of repositories by for exclusion; {ACCESS_TOKEN} is an optional GitHub API access token to include with the underlying calls to the GitHub API.
func (*OrganizationIterator) Seen ¶
func (it *OrganizationIterator) Seen() int64
Seen() returns the total number of records processed so far.
Source Files