README
¶
SubmitQueue
SubmitQueue is a high-performance speculative merge queue that keeps your trunk consistently green at scale. Rather than validating changes one at a time, SubmitQueue speculatively rebases and validates multiple changes in parallel against predicted future states of HEAD. When validations pass, changes land automatically. When they fail, SubmitQueue isolates the offending change and retries the rest — all without human intervention.
Designed for large monorepos and fast-moving teams where concurrent changes can introduce subtle conflicts and destabilize builds.
Repository layout
Cross-domain Go code (errors, metrics, consumer framework, HTTP helpers, shared entities, shared extension contracts) lives under platform/. Each product domain has its own tree (submitqueue/, stovepipe/, …) and grows into gateway/, orchestrator/, entity/, extension/, and domain-local core/ — though a domain may start smaller (Stovepipe is currently a single Ping-only service with just controller/). See CLAUDE.md for conventions and import paths.
Quick Start
Requires Docker and Docker Compose. See Development Setup for full prerequisites.
# Build everything
make build
# Run unit tests
make test
# Start full stack locally (Gateway + Orchestrator + MySQL via Docker Compose)
make local-submitqueue-start
# Test with grpcurl
grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping
# Stop services
make local-stop
See service/README.md for more examples including running individual services and clients.
Documentation
| Document | Description |
|---|---|
| Development Setup | Prerequisites, build, environment, IDE setup |
| Contributing | How to contribute, workflow, guidelines |
| Testing Guide | Unit, integration, and E2E testing patterns |
| Architecture Guide | Project layout, patterns, conventions |
| Examples | Running services, clients, API reference |
| RFCs | Design documents and proposals |
Project Status
SubmitQueue is under active development. We welcome contributions and feedback.
Contributing
See CONTRIBUTING.md for how to get started.
License
Licensed under the Apache License 2.0.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
runway/messagequeue
Package messagequeue holds Runway's external message-queue contract: the wire payloads for the merge queues Runway owns, defined by the proto files in proto/ and generated into protopb/.
|
Package messagequeue holds Runway's external message-queue contract: the wire payloads for the merge queues Runway owns, defined by the proto files in proto/ and generated into protopb/. |
|
platform
|
|
|
base
Package base is the root of shared domain entities used across SubmitQueue, Stovepipe, and other repo-local services.
|
Package base is the root of shared domain entities used across SubmitQueue, Stovepipe, and other repo-local services. |
|
base/change
Package change holds the shared code-change identity used across SubmitQueue, Stovepipe, and other repo-local domains.
|
Package change holds the shared code-change identity used across SubmitQueue, Stovepipe, and other repo-local domains. |
|
base/change/changeutil
Package changeutil holds small helpers shared across the entity/change provider subpackages (github, git, ...).
|
Package changeutil holds small helpers shared across the entity/change provider subpackages (github, git, ...). |
|
base/change/git
Package git parses change IDs that use the git:// URI scheme.
|
Package git parses change IDs that use the git:// URI scheme. |
|
base/mergestrategy
Package mergestrategy holds the shared source-control integration strategy used across SubmitQueue, runway, and other repo-local domains.
|
Package mergestrategy holds the shared source-control integration strategy used across SubmitQueue, runway, and other repo-local domains. |
|
base/page
Package page defines a generic, cursor-paginated result envelope shared across domains.
|
Package page defines a generic, cursor-paginated result envelope shared across domains. |
|
consumer/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
errs/generic
Package generic provides an errs.Classifier for errors that are not tied to any particular backend.
|
Package generic provides an errs.Classifier for errors that are not tied to any particular backend. |
|
errs/mysql
Package mysql provides an errs.Classifier for errors originating from the go-sql-driver/mysql driver and the standard database/sql + net packages commonly seen when talking to a MySQL backend.
|
Package mysql provides an errs.Classifier for errors originating from the go-sql-driver/mysql driver and the standard database/sql + net packages commonly seen when talking to a MySQL backend. |
|
extension
Package extension holds extension interfaces and implementations shared across SubmitQueue, Stovepipe, and other repo-local services.
|
Package extension holds extension interfaces and implementations shared across SubmitQueue, Stovepipe, and other repo-local services. |
|
extension/consumergate
Package consumergate defines the consumer-gate extension: runtime stop/start of individual queue controllers without stopping the service that hosts them.
|
Package consumergate defines the consumer-gate extension: runtime stop/start of individual queue controllers without stopping the service that hosts them. |
|
extension/consumergate/file
Package file implements the consumergate contract with plain files in a shared directory.
|
Package file implements the consumergate contract with plain files in a shared directory. |
|
extension/consumergate/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/consumergate/noop
Package noop provides a no-op consumergate.Gate that admits every delivery immediately.
|
Package noop provides a no-op consumergate.Gate that admits every delivery immediately. |
|
extension/counter/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/messagequeue/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/messagequeue/mysql
Package mysql is a generated GoMock package.
|
Package mysql is a generated GoMock package. |
|
extension/messagequeue/mysql/ctl
command
|
|
|
http
Package http provides shared HTTP client helpers (e.g.
|
Package http provides shared HTTP client helpers (e.g. |
|
lifecycle
Package lifecycle provides the Component interface and Group type for managing ordered start/stop lifecycles.
|
Package lifecycle provides the Component interface and Group type for managing ordered start/stop lifecycles. |
|
pipeline
Package pipeline provides a typed engine for assembling queue-driven service pipelines from declarative data.
|
Package pipeline provides a typed engine for assembling queue-driven service pipelines from declarative data. |
|
runway
|
|
|
controller/dlq
Package dlq reconciles dead-lettered merge requests back to the client.
|
Package dlq reconciles dead-lettered merge requests back to the client. |
|
controller/merge
Package merge consumes committing merge requests from Runway's merge queue.
|
Package merge consumes committing merge requests from Runway's merge queue. |
|
controller/mergeconflictcheck
Package mergeconflictcheck consumes dry-run merge-conflict check requests from Runway's merge-conflict-check queue.
|
Package mergeconflictcheck consumes dry-run merge-conflict check requests from Runway's merge-conflict-check queue. |
|
extension
Package extension holds Runway-specific extension implementations.
|
Package extension holds Runway-specific extension implementations. |
|
extension/merger
Package merger defines the pluggable interface for version-control merge operations that Runway performs on behalf of its callers.
|
Package merger defines the pluggable interface for version-control merge operations that Runway performs on behalf of its callers. |
|
extension/merger/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/merger/noop
Package noop provides a no-op Merger implementation for local development and testing.
|
Package noop provides a no-op Merger implementation for local development and testing. |
|
service
|
|
|
runway/client
command
|
|
|
runway/server
command
|
|
|
stovepipe/client
command
|
|
|
stovepipe/server
command
|
|
|
stovepipe/server/mapper
Package mapper translates stovepipe wire (proto) types to and from the domain entities the controllers operate on.
|
Package mapper translates stovepipe wire (proto) types to and from the domain entities the controllers operate on. |
|
submitqueue/gateway/client
command
|
|
|
submitqueue/gateway/server
command
|
|
|
submitqueue/gateway/server/mapper
Package mapper translates gateway wire (proto) types to and from the domain entities the controllers operate on.
|
Package mapper translates gateway wire (proto) types to and from the domain entities the controllers operate on. |
|
submitqueue/orchestrator/client
command
|
|
|
submitqueue/orchestrator/server
command
|
|
|
stovepipe
|
|
|
controller/build
Package build holds the build-stage queue controller.
|
Package build holds the build-stage queue controller. |
|
controller/buildsignal
Package buildsignal holds the buildsignal-stage queue controller.
|
Package buildsignal holds the buildsignal-stage queue controller. |
|
controller/dlq
Package dlq contains controllers that consume messages from a pipeline stage's dead-letter topic and reconcile the affected request into a terminal state.
|
Package dlq contains controllers that consume messages from a pipeline stage's dead-letter topic and reconcile the affected request into a terminal state. |
|
controller/process
Package process holds the process-stage queue controller.
|
Package process holds the process-stage queue controller. |
|
core/loader
Package loader holds the "load one entity by id from a store" pattern shared by Stovepipe's queue controllers (build, process, buildsignal).
|
Package loader holds the "load one entity by id from a store" pattern shared by Stovepipe's queue controllers (build, process, buildsignal). |
|
core/messagequeue
Package messagequeue holds Stovepipe's internal message-queue contract: the wire payloads for the pipeline queues Stovepipe owns, defined by the proto files in proto/ and generated into protopb/.
|
Package messagequeue holds Stovepipe's internal message-queue contract: the wire payloads for the pipeline queues Stovepipe owns, defined by the proto files in proto/ and generated into protopb/. |
|
extension/buildrunner
Package buildrunner defines the contract through which Stovepipe triggers and polls builds against an external build system.
|
Package buildrunner defines the contract through which Stovepipe triggers and polls builds against an external build system. |
|
extension/buildrunner/fake
Package fake provides a buildrunner.BuildRunner whose outcome is driven by the triggered head URI.
|
Package fake provides a buildrunner.BuildRunner whose outcome is driven by the triggered head URI. |
|
extension/buildrunner/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/queueconfig/default
Package defaultconfig provides a queueconfig.Store that returns global wiring defaults for any non-empty queue name.
|
Package defaultconfig provides a queueconfig.Store that returns global wiring defaults for any non-empty queue name. |
|
extension/queueconfig/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/sourcecontrol
Package sourcecontrol defines the contract through which Stovepipe talks to a version control system.
|
Package sourcecontrol defines the contract through which Stovepipe talks to a version control system. |
|
extension/sourcecontrol/fake
Package fake provides an in-memory sourcecontrol.SourceControl seeded with a single queue's linear ref history, ordered newest-first.
|
Package fake provides an in-memory sourcecontrol.SourceControl seeded with a single queue's linear ref history, ordered newest-first. |
|
extension/sourcecontrol/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/storage/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
submitqueue
|
|
|
core
Package core groups infrastructure shared across SubmitQueue's own services (gateway and orchestrator) — the SubmitQueue-scoped analogue of the repo-level core/.
|
Package core groups infrastructure shared across SubmitQueue's own services (gateway and orchestrator) — the SubmitQueue-scoped analogue of the repo-level core/. |
|
core/changeset
Package changeset resolves batch identity into the changes a batch contains.
|
Package changeset resolves batch identity into the changes a batch contains. |
|
core/changeset/fake
Package fake provides an in-memory changeset.Resolver for tests and examples.
|
Package fake provides an in-memory changeset.Resolver for tests and examples. |
|
core/changeset/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
core/fakemarker
Package fakemarker holds the shared "sq-fake=<token>" change-URI marker convention used by the extension fakes to inject failures from a request payload.
|
Package fakemarker holds the shared "sq-fake=<token>" change-URI marker convention used by the extension fakes to inject failures from a request payload. |
|
core/topickey
Package topickey defines SubmitQueue pipeline stage identifiers.
|
Package topickey defines SubmitQueue pipeline stage identifiers. |
|
extension/buildrunner/buildkite
Package buildkite implements buildrunner.BuildRunner backed by the Buildkite CI platform.
|
Package buildkite implements buildrunner.BuildRunner backed by the Buildkite CI platform. |
|
extension/buildrunner/fake
Package fake provides a buildrunner.BuildRunner whose outcome is driven by the triggered changes.
|
Package fake provides a buildrunner.BuildRunner whose outcome is driven by the triggered changes. |
|
extension/buildrunner/githubactions
Package githubactions implements buildrunner.BuildRunner backed by GitHub Actions workflow_dispatch.
|
Package githubactions implements buildrunner.BuildRunner backed by GitHub Actions workflow_dispatch. |
|
extension/buildrunner/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/changeprovider/fake
Package fake provides a changeprovider.ChangeProvider whose outcome is driven by the input change.
|
Package fake provides a changeprovider.ChangeProvider whose outcome is driven by the input change. |
|
extension/changeprovider/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/changeprovider/routing
Package routing provides a ChangeProvider that dispatches URIs to downstream change providers based on the URI's change type.
|
Package routing provides a ChangeProvider that dispatches URIs to downstream change providers based on the URI's change type. |
|
extension/conflict/all
Package all provides a conflict.Analyzer that pessimistically reports a conflict against every in-flight batch.
|
Package all provides a conflict.Analyzer that pessimistically reports a conflict against every in-flight batch. |
|
extension/conflict/fake
Package fake provides a conflict.Analyzer that decorates an existing analyzer: it delegates to the wrapped implementation for the happy path, but injects an error when a caller-supplied predicate matches.
|
Package fake provides a conflict.Analyzer that decorates an existing analyzer: it delegates to the wrapped implementation for the happy path, but injects an error when a caller-supplied predicate matches. |
|
extension/conflict/fileoverlap
Package fileoverlap provides a conflict.Analyzer that reports a conflict between two batches when they change one or more of the same files.
|
Package fileoverlap provides a conflict.Analyzer that reports a conflict between two batches when they change one or more of the same files. |
|
extension/conflict/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/conflict/none
Package none provides a conflict.Analyzer that never reports a conflict.
|
Package none provides a conflict.Analyzer that never reports a conflict. |
|
extension/mergechecker/fake
Package fake provides a mergechecker.MergeChecker whose outcome is driven by the input change.
|
Package fake provides a mergechecker.MergeChecker whose outcome is driven by the input change. |
|
extension/mergechecker/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/queueconfig/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/queueconfig/yaml
Package yaml provides a YAML-file-backed implementation of queueconfig.Store.
|
Package yaml provides a YAML-file-backed implementation of queueconfig.Store. |
|
extension/scorer/fake
Package fake provides a scorer.Scorer that decorates an existing scorer: it delegates to the wrapped implementation for the happy path, but injects an error when a change URI carries a failure marker of the form "sq-fake=<token>":
|
Package fake provides a scorer.Scorer that decorates an existing scorer: it delegates to the wrapped implementation for the happy path, but injects an error when a change URI carries a failure marker of the form "sq-fake=<token>": |
|
extension/scorer/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/speculation/speculator/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/storage/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
extension/validator/fake
Package fake provides a validator.Validator that always passes and a validator.Factory that returns it.
|
Package fake provides a validator.Validator that always passes and a validator.Factory that returns it. |
|
extension/validator/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
orchestrator
Package orchestrator declares the SubmitQueue orchestrator's pipeline topology, extension seams, and controller set.
|
Package orchestrator declares the SubmitQueue orchestrator's pipeline topology, extension seams, and controller set. |
|
orchestrator/controller/buildsignal
Package buildsignal implements the build poll loop.
|
Package buildsignal implements the build poll loop. |
|
orchestrator/controller/cancel
Package cancel implements the orchestrator-side cancel controller.
|
Package cancel implements the orchestrator-side cancel controller. |
|
orchestrator/controller/dlq
Package dlq contains controllers that consume messages from per-topic dead-letter queues and reconcile the affected request and batch entities into a terminal failed state.
|
Package dlq contains controllers that consume messages from per-topic dead-letter queues and reconcile the affected request and batch entities into a terminal failed state. |
|
orchestrator/controller/merge
Package merge implements the trigger stage for the asynchronous merge.
|
Package merge implements the trigger stage for the asynchronous merge. |
|
orchestrator/controller/mergeconflictsignal
Package mergeconflictsignal consumes merge-conflict check results from runway's signal queue, correlates them to the request by the echoed id, and either advances the request to the batch stage (mergeable) or fails it (conflicted).
|
Package mergeconflictsignal consumes merge-conflict check results from runway's signal queue, correlates them to the request by the echoed id, and either advances the request to the batch stage (mergeable) or fails it (conflicted). |
|
orchestrator/controller/mergesignal
Package mergesignal consumes merge results from runway's merge-signal queue, correlates them to the batch by the echoed id, and transitions the batch to a terminal state — Succeeded when runway merged the batch, Failed when it could not — then fans the batch out to conclude (so member requests pick up the outcome) and speculate (so dependents can re-plan).
|
Package mergesignal consumes merge results from runway's merge-signal queue, correlates them to the batch by the echoed id, and transitions the batch to a terminal state — Succeeded when runway merged the batch, Failed when it could not — then fans the batch out to conclude (so member requests pick up the outcome) and speculate (so dependents can re-plan). |
|
test
|
|
|
tool
|
|
|
linter/licenseheader
command
|
Click to show internal directories.
Click to hide internal directories.