Documentation
¶
Overview ¶
Package artifact contains a data pipeline that will read workflow event records from BigQuery and ingest any available logs into cloud storage. A mapping from the original GitHub event to the cloud storage location is persisted in BigQuery along with an indicator for the status of the copy. The pipeline acts as a GitHub App for authentication purposes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ExecuteJob ¶
ExecuteJob runs the ingestion pipeline job to read GitHub action workflow logs from GitHub and store them into GCS.
Types ¶
type ArtifactRecord ¶
type ArtifactRecord struct {
DeliveryID string `bigquery:"delivery_id" json:"delivery_id"`
ProcessedAt time.Time `bigquery:"processed_at" json:"processed_at"`
Status string `bigquery:"status" json:"status"`
WorkflowURI string `bigquery:"workflow_uri" json:"workflow_uri"`
LogsURI string `bigquery:"logs_uri" json:"logs_uri"`
GitHubActor string `bigquery:"github_actor" json:"github_actor"`
OrganizationName string `bigquery:"organization_name" json:"organization_name"`
RepositoryName string `bigquery:"repository_name" json:"repository_name"`
RepositorySlug string `bigquery:"repository_slug" json:"repository_slug"`
JobName string `bigquery:"job_name" json:"job_name"`
}
ArtifactRecord is the output data structure that maps to the leech pipeline's output table schema.
type Config ¶
type Config struct {
GitHubAppID string `env:"GITHUB_APP_ID,required"` // The GitHub App ID
GitHubInstallID string `env:"GITHUB_INSTALL_ID,required"` // The provisioned GitHub App Installation reference
GitHubPrivateKeySecret string `env:"GITHUB_PRIVATE_KEY_SECRET,required"` // The secret name & version containing the GitHub App private key
BatchSize int `env:"BATCH_SIZE,default=100"` // The number of items to process in this pipeline run
ProjectID string `env:"PROJECT_ID,required"` // The project id where the tables live
DatasetID string `env:"DATASET_ID,required"` // The dataset id where the tables live
EventsTableID string `env:"EVENTS_TABLE_ID,required"` // The table_name of the events table
ArtifactsTableID string `env:"ARTIFACTS_TABLE_ID,required"` // The table_name of the artifact_status table
BucketName string `env:"BUCKET_NAME,required"` // The name of the GCS bucket to store artifact logs
}
Config defines the set of environment variables required for running the artifact job.
func (*Config) ToFlags ¶
ToFlags binds the config to the cli.FlagSet and returns it.
type EventRecord ¶
type EventRecord struct {
DeliveryID string `bigquery:"delivery_id" json:"delivery_id"`
RepositorySlug string `bigquery:"repo_slug" json:"repo_slug"`
RepositoryName string `bigquery:"repo_name" json:"repo_name"`
OrganizationName string `bigquery:"org_name" json:"org_name"`
LogsURL string `bigquery:"logs_url" json:"logs_url"`
GitHubActor string `bigquery:"github_actor" json:"github_actor"`
WorkflowURL string `bigquery:"workflow_url" json:"workflow_url"`
}
EventRecord maps the columns from the driving BigQuery query to a usable structure.
type ObjectStore ¶
type ObjectStore struct {
// contains filtered or unexported fields
}
ObjectStore is an implementation of the ObjectWriter interface that writes to Cloud Storage.
func NewObjectStore ¶
func NewObjectStore(ctx context.Context) (*ObjectStore, error)
NewObjectStore creates a ObjectWriter implementation that uses cloud storage to store its objects.
Source Files