cloudrig

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 13, 2026 License: MIT Imports: 37 Imported by: 0

README

CloudRig

A local Google Cloud environment for realistic, deterministic integration testing.

CloudRig gives you a connected GCP environment on your machine. Your real application code and GCP clients interact with realistic local services, while tests can deterministically control time, failures, and state.

                         CloudRig
                            │
       ┌────────────────────┼────────────────────┐
       │                    │                    │
       ▼                    ▼                    ▼
   GCP Services         Real Workloads       Test Controls
       │                    │                    │
       │                    │              ┌─────┼─────┐
       │                    │              ▼     ▼     ▼
 Storage / PubSub       Functions/GKE    Time  Fault  Fork
 Firestore / Tasks      Cloud Run
 Scheduler / Secrets
       │
       └────────── Connected Event-Driven ──────────┘

Quick Start

Install CloudRig from a checkout, start it, and point gcloud at your local GCP (Go 1.25+):

git clone https://github.com/monirz/cloudrig && cd cloudrig
go install ./cmd/cloudrig    # puts cloudrig on your PATH

cloudrig start &
. ./cloudrig-env.sh          # points gcloud at CloudRig, with no credentials

Now use it exactly like the real thing:

gcloud storage buckets create gs://demo
gcloud storage cp README.md gs://demo/
gcloud storage ls gs://demo
# gs://demo/README.md

Point the client libraries at the same port with the standard emulator variables:

export PUBSUB_EMULATOR_HOST=localhost:4599
export FIRESTORE_EMULATOR_HOST=localhost:4599

Prefer a local binary instead? make build produces ./cloudrig in the repo.

See It in Action

One workflow, end to end: an object landing in a bucket triggers a function that publishes to Pub/Sub, a worker enqueues a Cloud Task, and fast-forwarding the clock fires its retries. Run the whole thing with examples/pipeline/run.sh, or step through it:

# Start CloudRig with a virtual clock. It points functions at its own services
# automatically; SINK_URL is this app's own setting.
SINK_URL=http://localhost:8080/ cloudrig start --clock virtual &
. ./cloudrig-env.sh

# Provision a bucket, a Pub/Sub topic, and a task queue.
gcloud storage buckets create gs://intake
curl -X PUT localhost:4599/v1/projects/cloudrig-local/topics/orders -d '{}'
gcloud tasks queues create pipeline --location=us-central1 --max-attempts=3

# Deploy the pipeline: ingest (storage -> Pub/Sub), worker (Pub/Sub -> Cloud Task).
cloudrig fn deploy ingest --source ./examples/pipeline/ingest --trigger-bucket intake
cloudrig fn deploy worker --source ./examples/pipeline/worker --trigger-topic orders

# Drop an object in. It flows through the whole chain and enqueues a task.
echo '{"id":42}' > order.json
gcloud storage cp order.json gs://intake/

# Fast-forward time. The task fires and retries, deterministically.
cloudrig clock advance 1h
#   sink: task attempt      <- the retries, with no real waiting
#   sink: task attempt

Every stage is real: the functions run as processes, Pub/Sub and Cloud Tasks are live, and the clock is injected so the retries need no real waiting.

Why CloudRig?

  • Connected and event-driven. A storage write triggers a function; a scheduler job publishes to Pub/Sub which triggers another. Whole workflows run locally, in one process.
  • Deterministic. Time is injected, so a test reproduces exactly, with no flaky time.Sleeps.
  • Built for testing. Travel through time, inject failures, and fork state.
  • Real clients. gcloud, Terraform, and the Google client libraries work unchanged, over gRPC and REST.

How To

Supported Services

CloudRig emulates the following services:

Service Status Notes
Cloud Functions ✅ Real local execution (Go & Node), HTTP & event triggers
Cloud Storage ✅ Buckets, objects, uploads, versioning, signed URLs
Pub/Sub ✅ Publish, streaming pull, ack/nack, redelivery
Firestore ✅ Documents, queries, transactions, over gRPC
Cloud Tasks ✅ Deferred HTTP work, dispatched on the clock
Cloud Scheduler ✅ Cron jobs firing HTTP or Pub/Sub, on the clock
Secret Manager ✅ Secrets, versions, the latest alias
Cloud Logging ✅ Write and read structured entries with a filter
Service Usage ✅ gcloud services enable/disable/list
Cloud Run ✅ Runs a real container (needs Docker)
GKE ✅ Real local Kubernetes (needs k3d or kind)

Full compatibility and limitations: service guides · what's not supported.

Testing

Tutorials

Guides

Architecture

How CloudRig serves every service on one port, with an injected clock: ARCHITECTURE.md.

CLI Reference

Every subcommand and flag: docs/cli.md.

Development

git clone https://github.com/monirz/cloudrig && cd cloudrig
make build              # -> ./cloudrig
make check              # build, vet, lint, gofmt, tests

See CONTRIBUTING.md for conventions.

Roadmap

Planned services, distribution, and testing features: ROADMAP.md.

Contributing

Contributions, bug reports and ideas are welcome. See CONTRIBUTING.md.

License

MIT. See LICENSE.

Documentation

Overview

Package cloudrig is a local emulator for Google Cloud APIs.

Start runs it as a server; MustStart runs it in-process inside a Go test: 

func TestUpload(t *testing.T) {
	t.Parallel()
	emu := cloudrig.MustStart(t)
	// ... point a GCP client at emu.BaseURL()
}

MustStart is the reason the project exists: no container, no daemon, and one isolated instance per test.

Index

Constants

View Source 
const DefaultAddr = ":4599"

DefaultAddr binds every interface so the binary is reachable from a container. MustStart ignores it and takes a random free loopback port.

Variables

This section is empty.

Functions

This section is empty.

Types

type Emulator 

type Emulator struct {
	// contains filtered or unexported fields
}

Emulator is a running instance.

func MustStart 

func MustStart(t testing.TB, opts ...Options) *Emulator

MustStart runs the emulator in-process for one test, on a random free port with a FakeClock and t.Cleanup registered. Every call is fully isolated.

func Start 

func Start(ctx context.Context, o Options) (*Emulator, error)

Start runs the emulator on a real listener; the caller owns shutdown. ctx bounds startup only and does not stop the server.

func (*Emulator) BaseURL 

func (e *Emulator) BaseURL() string

BaseURL is the endpoint as a URL: what STORAGE_EMULATOR_HOST and the SDK endpoint overrides want.

func (*Emulator) Clock 

func (e *Emulator) Clock() clock.Clock

Clock returns the emulator's clock.

func (*Emulator) CloudRun 

func (e *Emulator) CloudRun() *cloudrun.Registry

CloudRun is the registry of deployed Cloud Run services.

func (*Emulator) Endpoint 

func (e *Emulator) Endpoint() string

Endpoint is the host:port a client should dial, e.g. "127.0.0.1:53412".

func (*Emulator) FakeClock 

func (e *Emulator) FakeClock(t testing.TB) *clock.FakeClock

FakeClock returns the clock as a *clock.FakeClock so a test can Advance it, failing t if the emulator is running a real one.

func (*Emulator) Faults 

func (e *Emulator) Faults() *faults.Set

Faults is the live fault-injection rule set.

A rule armed here fails matching requests before they reach any service, so a test can prove its own retry and error handling rather than hoping the path is exercised.

func (*Emulator) Fork 

func (e *Emulator) Fork(t testing.TB) *Emulator

Fork returns a second emulator carrying a copy of this one's state.

Metadata is copied; object payloads are shared, because they are content-addressed and immutable — the same bytes under the same name. So a fork of a hundred gigabytes of objects costs the size of the metadata.

What is copied is state, not processes: deployed functions, armed faults and the clock do not travel. The fork gets its own port, its own event bus and its own fault set, and starts with no functions deployed.

func (*Emulator) FunctionURL 

func (e *Emulator) FunctionURL(name string) string

FunctionURL is where the named function is served, or "" if it is not deployed. It returns the short form, valid for the default project and location; the prefixed form is also served.

func (*Emulator) Functions 

func (e *Emulator) Functions() *functions.Registry

Functions is the registry backing this emulator, so a test can deploy into a running instance rather than only at Start.

func (*Emulator) Reset 

func (e *Emulator) Reset(ctx context.Context, project string) error

Reset clears emulator state. An empty project clears everything.

func (*Emulator) Shutdown 

func (e *Emulator) Shutdown(ctx context.Context) error

Shutdown stops the emulator; MustStart registers it with t.Cleanup.

func (*Emulator) SyncEvents 

func (e *Emulator) SyncEvents()

SyncEvents waits for every event published so far to reach its subscribers.

Delivery is asynchronous so a write never waits on a trigger. A test that uploads an object and then asserts a function ran needs this, or it would have to poll.

func (*Emulator) SyncScheduler 

func (e *Emulator) SyncScheduler()

SyncScheduler waits for every fired Cloud Scheduler HTTP job to finish, the same way SyncTasks does for tasks.

func (*Emulator) SyncTasks 

func (e *Emulator) SyncTasks()

SyncTasks waits for every immediately-dispatched Cloud Tasks task to finish, including arming any retry. A test advances the clock for scheduled tasks; this covers the due-now ones that run off a goroutine.

type Options 

type Options struct {
	// Addr is the listen address. Empty means DefaultAddr for Start, a random
	// free loopback port for MustStart.
	Addr string

	// Clock is the only source of time. Empty means real for Start, fake for
	// MustStart: a test that cannot control time will eventually sleep.
	Clock clock.Clock

	// Version is reported by /_emu/health. Empty means "dev".
	Version string

	// Runner is "auto", "subprocess" or "none". "auto" resolves to
	// "subprocess" when Functions is non-empty, else "none".
	Runner string

	// Functions are built and launched at Start, and stopped at Shutdown.
	Functions []functions.Function

	// FunctionLog receives function output. Nil discards it; use fn logs.
	FunctionLog io.Writer

	// EventLog receives the emulator's own messages, such as a watched
	// function redeploying. Nil discards them.
	EventLog io.Writer

	// DataDir persists Cloud Storage across restarts: metadata snapshots and
	// object content live under it. Empty keeps everything in memory and a
	// temp directory, which is what a test wants — state surviving a test is a
	// bug, not a feature.
	DataDir string
}

Options configures an Emulator. The zero value is valid.

Source Files

View all Source files

Directories

Path Synopsis
cmd
cloudrig command
Command cloudrig runs the emulator as a server: flags, environment and signals only.
 Command cloudrig runs the emulator as a server: flags, environment and signals only.
core
clock
Package clock is the only place permitted to read wall-clock time.
 Package clock is the only place permitted to read wall-clock time.
events
Package events is the in-process bus services use to reach each other.
 Package events is the in-process bus services use to reach each other.
faults
Package faults injects failures into the emulator's responses.
 Package faults injects failures into the emulator's responses.
gerr
Package gerr is cloudrig's canonical error type: a code, an explicit HTTP status, and the reason string clients branch on.
 Package gerr is cloudrig's canonical error type: a code, an explicit HTTP status, and the reason string clients branch on.
logring
Package logring keeps the last lines a child process wrote.
 Package logring keeps the last lines a child process wrote.
resource
Package resource encodes GCP resources as store keys.
 Package resource encodes GCP resources as store keys.
tmp
Package tmp puts every temporary directory the emulator makes under one process-owned root.
 Package tmp puts every temporary directory the emulator makes under one process-owned root.
examples
hello
Package hello is a sample HTTP Cloud Function.
 Package hello is a sample HTTP Cloud Function.
pubsub command
Command pubsub-demo exercises the emulator's Pub/Sub the way an application would: nothing here names cloudrig, and the only configuration is PUBSUB_EMULATOR_HOST.
 Command pubsub-demo exercises the emulator's Pub/Sub the way an application would: nothing here names cloudrig, and the only configuration is PUBSUB_EMULATOR_HOST.
Package functions builds and runs Go Cloud Functions as child processes.
 Package functions builds and runs Go Cloud Functions as child processes.
services
cloudfunctions
Package cloudfunctions serves the Cloud Functions v1 REST API, so real gcloud and the GCP SDKs can drive the emulator.
 Package cloudfunctions serves the Cloud Functions v1 REST API, so real gcloud and the GCP SDKs can drive the emulator.
cloudlogging
Package cloudlogging is the Cloud Logging emulation.
 Package cloudlogging is the Cloud Logging emulation.
cloudrun
Package cloudrun runs Cloud Run services locally.
 Package cloudrun runs Cloud Run services locally.
cloudscheduler
Package cloudscheduler is the Cloud Scheduler emulation.
 Package cloudscheduler is the Cloud Scheduler emulation.
cloudtasks
Package cloudtasks is the Cloud Tasks emulation.
 Package cloudtasks is the Cloud Tasks emulation.
firestore
Package firestore is the Cloud Firestore emulation.
 Package firestore is the Cloud Firestore emulation.
gke
Package gke emulates the GKE cluster admin API, backed by a real local Kubernetes cluster rather than a stub.
 Package gke emulates the GKE cluster admin API, backed by a real local Kubernetes cluster rather than a stub.
pubsub
Package pubsub is the Cloud Pub/Sub emulation.
 Package pubsub is the Cloud Pub/Sub emulation.
secretmanager
Package secretmanager is the Secret Manager emulation.
 Package secretmanager is the Secret Manager emulation.
serviceusage
Package serviceusage tracks which APIs a project has enabled.
 Package serviceusage tracks which APIs a project has enabled.
storage
Package storage is the Cloud Storage semantics layer: buckets, objects, generations and preconditions.
 Package storage is the Cloud Storage semantics layer: buckets, objects, generations and preconditions.
Package store is cloudrig's metadata layer: a versioned compare-and-swap key/value map.
 Package store is cloudrig's metadata layer: a versioned compare-and-swap key/value map.
blob
Package blob stores object payloads as content-addressed files.
 Package blob stores object payloads as content-addressed files.
Package transport is cloudrig's front door: one handler on one port, serving REST over HTTP/1.1 and gRPC over cleartext HTTP/2 at once.
 Package transport is cloudrig's front door: one handler on one port, serving REST over HTTP/1.1 and gRPC over cleartext HTTP/2 at once.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.