cron/

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

Links

Open Source Insights

README ¶

Batch scanning pipeline

cron/ is the batch scanning pipeline behind the weekly public Scorecard scan of 1M+ repositories. It was imported from ossf/scorecard with its full commit history — 466 commits dating to 2020 — because the operational history is the runbook: quota workarounds, shard sizing, PubSub ack deadlines, and BigQuery schema migrations exist only as commit messages. initial-graft.md records the terms of that import, including what could not come across and where to trace it.

Read it before working in this tree. Two rules apply here and nowhere else in the repository:

Keep the tree equivalent to what ossf/scorecard builds until the production cutover to AWS completes. That equivalence is the cutover's acceptance test and its rollback path, so refactors, restructuring into this repository's internal/ + cmd/ layout, and cleanups are deferred by design — not overlooked.
cron/internal/format serializes a data model that lives upstream, so a schema edit here without the corresponding engine change breaks the published contract silently (schema_gen_test.go is what catches it).

Pipeline components

Component	Path	Image
PubSub batch controller	`cron/internal/controller/`	`scorecard-batch-controller`
Batch scan worker	`cron/internal/worker/`	`scorecard-batch-worker`
CII best-practices worker	`cron/internal/cii/`	`scorecard-cii-worker`
BigQuery transfer	`cron/internal/bq/`	`scorecard-bq-transfer`
Release webhook	`cron/internal/webhook/`	`scorecard-webhook-releasetest`
GitHub token-pool server	`cron/internal/githubserver/`	`scorecard-github-server`

Deployment manifests are in k8s/; image build configs in cron/cloudbuild/. cron/internal/format/ owns the published BigQuery and JSON schema contract, verified against the Scorecard engine's data model by schema_gen_test.go.

Scan inventories

The repositories scanned each week are listed in cron/internal/data/:

File	Scope
`projects.csv`	GitHub
`gitlab-projects.csv`	GitLab
`gitlab-projects-releasetest.csv`	GitLab release testing

To add a repository, edit the relevant file and run make add-projects, which normalizes the result. CI enforces both that the inventories are valid and that add-projects is a no-op against them, so a hand-edit the tooling would not produce fails the build. See CONTRIBUTING.md for the full contribution path.

[!NOTE] These inventories previously lived in ossf/scorecard. If you followed a link there, this is the right place now.

Prerequisites

To build the images or regenerate the protobufs, ko and protoc must be on PATH. They are expected rather than vendored; the Makefile explains why and fails with actionable errors when they are missing.

Running the pipeline itself additionally requires its GCP dependencies (PubSub, GCS, BigQuery) and the cluster described in k8s/README.md.

Pipeline targets

Run from the repository root:

make help                  # every target, grouped
make build-controller      # one pipeline binary (also build-worker, …)
make dockerbuild           # build all six images
make ko-images             # the same six via ko
make validate-projects     # validate the scan inventories
make build-proto           # regenerate protobufs (requires protoc on PATH)

Also in this tree

k8s/README.md — the cluster the pipeline runs on.
data/README.md — the scan inventory data files.
internal/emulator/README.md — local PubSub emulation for development.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
config Package config defines the configuration values for the cron job.	Package config defines the configuration values for the cron job.
data Package data contains util fns for reading and writing data handled by the cron job.	Package data contains util fns for reading and writing data handled by the cron job.
internal
bq command Package main implements the BQ transfer job.	Package main implements the BQ transfer job.
cdn Package cdn implements clients for CDN operations.	Package cdn implements clients for CDN operations.
cii command Package main implements the PubSub controller.	Package main implements the PubSub controller.
controller command Package main implements the PubSub controller.	Package main implements the PubSub controller.
data/add command Package main adds new project repositories to the projects file.	Package main adds new project repositories to the projects file.
data/validate command Package main validates an input file with a list of projects.	Package main validates an input file with a list of projects.
format
githubserver command Package main implements the GitHub token server.	Package main implements the GitHub token server.
provenance Package provenance reports which Scorecard engine produced a result.	Package provenance reports which Scorecard engine produced a result.
pubsub Package pubsub handles interactions with PubSub framework.	Package pubsub handles interactions with PubSub framework.
shuffle command Package main implements the PubSub controller.	Package main implements the PubSub controller.
webhook command Package main implements a Webhook which runs a bash script when called.	Package main implements a Webhook which runs a bash script when called.
worker command Package main implements cron worker job.	Package main implements cron worker job.
monitoring Package monitoring defines exporters to be used by opencensus package in the cron job.	Package monitoring defines exporters to be used by opencensus package in the cron job.
worker Package worker implements the generic cron worker logic.	Package worker implements the generic cron worker logic.