README
¶
SubmitQueue
SubmitQueue is a high-performance speculative merge queue that keeps your trunk consistently green at scale. Rather than validating changes one at a time, SubmitQueue speculatively rebases and validates multiple changes in parallel against predicted future states of HEAD. When validations pass, changes land automatically. When they fail, SubmitQueue isolates the offending change and retries the rest — all without human intervention.
Designed for large monorepos and fast-moving teams where concurrent changes can introduce subtle conflicts and destabilize builds.
Quick Start
Requires Docker and Docker Compose. See Development Setup for full prerequisites.
# Build everything
make build
# Run unit tests
make test
# Start full stack locally (Gateway + Orchestrator + MySQL via Docker Compose)
make local-submitqueue-start
# Test with grpcurl
grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping
# Stop services
make local-stop
See example/README.md for more examples including running individual services and clients.
Documentation
| Document | Description |
|---|---|
| Development Setup | Prerequisites, build, environment, IDE setup |
| Contributing | How to contribute, workflow, guidelines |
| Testing Guide | Unit, integration, and E2E testing patterns |
| Architecture Guide | Project layout, patterns, conventions |
| Examples | Running services, clients, API reference |
| RFCs | Design documents and proposals |
Project Status
SubmitQueue is under active development. We welcome contributions and feedback.
Contributing
See CONTRIBUTING.md for how to get started.
License
Licensed under the Apache License 2.0.
Directories
¶
| Path | Synopsis |
|---|---|
|
core
|
|
|
consumer/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
errs/generic
Package generic provides an errs.Classifier for errors that are not tied to any particular backend.
|
Package generic provides an errs.Classifier for errors that are not tied to any particular backend. |
|
errs/mysql
Package mysql provides an errs.Classifier for errors originating from the go-sql-driver/mysql driver and the standard database/sql + net packages commonly seen when talking to a MySQL backend.
|
Package mysql provides an errs.Classifier for errors originating from the go-sql-driver/mysql driver and the standard database/sql + net packages commonly seen when talking to a MySQL backend. |
|
Package entity holds domain entities shared across SubmitQueue, Stovepipe, and other repo-local services.
|
Package entity holds domain entities shared across SubmitQueue, Stovepipe, and other repo-local services. |
|
change
Package change holds the shared code-change identity used across SubmitQueue, Stovepipe, and other repo-local domains.
|
Package change holds the shared code-change identity used across SubmitQueue, Stovepipe, and other repo-local domains. |
|
change/changeutil
Package changeutil holds small helpers shared across the entity/change provider subpackages (github, git, ...).
|
Package changeutil holds small helpers shared across the entity/change provider subpackages (github, git, ...). |
|
change/git
Package git parses change IDs that use the git:// URI scheme.
|
Package git parses change IDs that use the git:// URI scheme. |
|
mergestrategy
Package mergestrategy holds the shared source-control integration strategy used across SubmitQueue, runway, and other repo-local domains.
|
Package mergestrategy holds the shared source-control integration strategy used across SubmitQueue, runway, and other repo-local domains. |
|
example
|
|
|
stovepipe/gateway/client
command
|
|
|
stovepipe/gateway/server
command
|
|
|
stovepipe/orchestrator/client
command
|
|
|
stovepipe/orchestrator/server
command
|
|
|
submitqueue/gateway/client
command
|
|
|
submitqueue/gateway/server
command
|
|
|
submitqueue/orchestrator/client
command
|
|
|
submitqueue/orchestrator/server
command
|
|
|
Package extension holds extension interfaces and implementations shared across SubmitQueue, Stovepipe, and other repo-local services.
|
Package extension holds extension interfaces and implementations shared across SubmitQueue, Stovepipe, and other repo-local services. |
|
counter/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
messagequeue/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
messagequeue/mysql
Package mysql is a generated GoMock package.
|
Package mysql is a generated GoMock package. |
|
messagequeue/mysql/ctl
command
|
|
|
stovepipe
|
|
|
core
Package core groups infrastructure shared across Stovepipe's own services (gateway and orchestrator) — the Stovepipe-scoped analogue of the repo-level core/.
|
Package core groups infrastructure shared across Stovepipe's own services (gateway and orchestrator) — the Stovepipe-scoped analogue of the repo-level core/. |
|
core/topickey
Package topickey defines Stovepipe pipeline stage identifiers.
|
Package topickey defines Stovepipe pipeline stage identifiers. |
|
entity
Package entity holds Stovepipe-specific domain types (distinct from shared repo entity/).
|
Package entity holds Stovepipe-specific domain types (distinct from shared repo entity/). |
|
extension
Package extension holds Stovepipe-specific extension implementations.
|
Package extension holds Stovepipe-specific extension implementations. |
|
submitqueue
|
|
|
core
Package core groups infrastructure shared across SubmitQueue's own services (gateway and orchestrator) — the SubmitQueue-scoped analogue of the repo-level core/.
|
Package core groups infrastructure shared across SubmitQueue's own services (gateway and orchestrator) — the SubmitQueue-scoped analogue of the repo-level core/. |
|
core/changeset
Package changeset resolves batch identity into the changes a batch contains.
|
Package changeset resolves batch identity into the changes a batch contains. |
|
core/changeset/fake
Package fake provides an in-memory changeset.Resolver for tests and examples.
|
Package fake provides an in-memory changeset.Resolver for tests and examples. |
|
core/changeset/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
core/fakemarker
Package fakemarker holds the shared "sq-fake=<token>" change-URI marker convention used by the extension fakes to inject failures from a request payload.
|
Package fakemarker holds the shared "sq-fake=<token>" change-URI marker convention used by the extension fakes to inject failures from a request payload. |
|
core/topickey
Package topickey defines SubmitQueue pipeline stage identifiers.
|
Package topickey defines SubmitQueue pipeline stage identifiers. |
|
extension/buildrunner/buildkite
Package buildkite implements buildrunner.BuildRunner backed by the Buildkite CI platform.
|
Package buildkite implements buildrunner.BuildRunner backed by the Buildkite CI platform. |
|
extension/buildrunner/fake
Package fake provides a buildrunner.BuildRunner whose outcome is driven by the triggered changes.
|
Package fake provides a buildrunner.BuildRunner whose outcome is driven by the triggered changes. |
|
extension/buildrunner/githubactions
Package githubactions implements buildrunner.BuildRunner backed by GitHub Actions workflow_dispatch.
|
Package githubactions implements buildrunner.BuildRunner backed by GitHub Actions workflow_dispatch. |
|
extension/buildrunner/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/changeprovider/fake
Package fake provides a changeprovider.ChangeProvider whose outcome is driven by the input change.
|
Package fake provides a changeprovider.ChangeProvider whose outcome is driven by the input change. |
|
extension/changeprovider/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/conflict/all
Package all provides a conflict.Analyzer that pessimistically reports a conflict against every in-flight batch.
|
Package all provides a conflict.Analyzer that pessimistically reports a conflict against every in-flight batch. |
|
extension/conflict/fake
Package fake provides a conflict.Analyzer that decorates an existing analyzer: it delegates to the wrapped implementation for the happy path, but injects an error when a caller-supplied predicate matches.
|
Package fake provides a conflict.Analyzer that decorates an existing analyzer: it delegates to the wrapped implementation for the happy path, but injects an error when a caller-supplied predicate matches. |
|
extension/conflict/fileoverlap
Package fileoverlap provides a conflict.Analyzer that reports a conflict between two batches when they change one or more of the same files.
|
Package fileoverlap provides a conflict.Analyzer that reports a conflict between two batches when they change one or more of the same files. |
|
extension/conflict/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/conflict/none
Package none provides a conflict.Analyzer that never reports a conflict.
|
Package none provides a conflict.Analyzer that never reports a conflict. |
|
extension/mergechecker/fake
Package fake provides a mergechecker.MergeChecker whose outcome is driven by the input change.
|
Package fake provides a mergechecker.MergeChecker whose outcome is driven by the input change. |
|
extension/mergechecker/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/pusher/fake
Package fake provides a pusher.Pusher whose outcome is driven by the input changes.
|
Package fake provides a pusher.Pusher whose outcome is driven by the input changes. |
|
extension/pusher/git
Package git is a simple Pusher implementation backed by a local git checkout.
|
Package git is a simple Pusher implementation backed by a local git checkout. |
|
extension/pusher/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/queueconfig/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/queueconfig/yaml
Package yaml provides a YAML-file-backed implementation of queueconfig.Store.
|
Package yaml provides a YAML-file-backed implementation of queueconfig.Store. |
|
extension/scorer/fake
Package fake provides a scorer.Scorer that decorates an existing scorer: it delegates to the wrapped implementation for the happy path, but injects an error when a change URI carries a failure marker of the form "sq-fake=<token>":
|
Package fake provides a scorer.Scorer that decorates an existing scorer: it delegates to the wrapped implementation for the happy path, but injects an error when a change URI carries a failure marker of the form "sq-fake=<token>": |
|
extension/scorer/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/storage/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
orchestrator/controller/buildsignal
Package buildsignal implements the build poll loop.
|
Package buildsignal implements the build poll loop. |
|
orchestrator/controller/cancel
Package cancel implements the orchestrator-side cancel controller.
|
Package cancel implements the orchestrator-side cancel controller. |
|
orchestrator/controller/dlq
Package dlq contains controllers that consume messages from per-topic dead-letter queues and reconcile the affected request and batch entities into a terminal failed state.
|
Package dlq contains controllers that consume messages from per-topic dead-letter queues and reconcile the affected request and batch entities into a terminal failed state. |
|
test
|
|
|
tool
|
|
|
linter/licenseheader
command
|
Click to show internal directories.
Click to hide internal directories.