README
¶
Sprue
The Forge upload service in Go (formerly the Storacha upload service).
Running locally
The repo ships a docker-compose.yaml that brings up sprue alongside
PostgreSQL and MinIO for self-hosted development:
docker compose up -d postgres minio
SPRUE_STORAGE_POSTGRES_DSN="postgres://sprue:sprue@localhost:5432/sprue?sslmode=disable" \
./sprue serve
Postgres is the default store backend, so no extra flag is required.
Store backends
Sprue supports two store backends, selected by
storage.type (or SPRUE_STORAGE_TYPE; defaults to postgres):
memory— in-process only; all data is lost on restart. Dev/test only.postgres— PostgreSQL for metadata + S3-compatible storage (MinIO, Ceph, AWS S3) for storing payloads of invocations, receipts, and delegations. Schema is managed by goose migrations embedded ininternal/migrations/sql/and applied on startup.
Logging
Sprue writes all logs to stdout/stderr through zap.
HTTP request logs are routed through the same zap logger (via Echo's
RequestLoggerWithConfig middleware), so application and request logs share one
output and format.
The output format depends on the mode. By default sprue uses zap's production
configuration, which emits a single JSON stream. When the log level is debug or
the deployment environment is development/test, sprue uses zap's development
configuration, which emits a human-readable console format (not JSON).
The JSON output makes it straightforward to collect logs with a sidecar such as
Grafana Alloy or Promtail: point the collector at the container's stdout/stderr
and use a json pipeline stage to extract fields like level, ts, and msg.
Request logs carry method, uri, status, latency, request_id,
content_length, response_size, headers, and related fields, and use the
request completed / client error / server error messages.
Tracing
sprue serve exports OpenTelemetry traces over OTLP/HTTP when
OTEL_EXPORTER_OTLP_ENDPOINT names a collector; with no endpoint, tracing is
off. Each UCAN request is a trace named for the commands it invokes (such as
/space/blob/add), with a span per invocation and the Postgres, S3, piri and
indexer calls it makes as child spans. A caller that sends a traceparent
header, such as ingot, gets sprue's spans in its own trace. The other standard
OTEL_* environment variables apply: OTEL_EXPORTER_OTLP_HEADERS
authenticates to the collector, and OTEL_TRACES_SAMPLER_ARG sets the
fraction of requests traced (0.01 traces 1%; the default traces every
request).
Notes
- Rate limits storage was not implemented. It has never been used in JS implementation, only supports blocking completely and can probably be applied at firewall.
- Plans, provisions, subscriptions, usage are not stores, they are services.
- Using
cid.Cidin new code overipld.Linkto ease transition to UCAN 1.0 when it comes. retrievalAuthis now an array of CIDs - an explicit delegation chain./upload/addnow takes an optionalindexCID, allowing us to track/remove indexes.
Container images
A push to main publishes to GHCR from the Container workflow. The prod
target becomes ghcr.io/fil-forge/sprue:main, a stripped binary on a slim
Debian base. The dev target becomes ghcr.io/fil-forge/sprue:main-dev and
adds delve plus a handful of debugging tools. Both cover linux/amd64 and
linux/arm64, and both also carry a sha-<short-sha> tag, the dev image with a
-dev suffix.
Deploying to dev
The same run asks infra-central to deploy the prod image. It dispatches a
bump-deployed-image event carrying the manifest digest it just pushed, and
infra-central's Bump deployed image workflow opens a pull request
pinning that digest in terraform/envs/dev/apps/terraform.tfvars, with
auto-merge enabled. infra-central's Check and deploy workflow runs
tofu apply on dev/apps on every push to its main, so merging that pull
request is what deploys.
The dispatch runs as the fil-forge-bot GitHub App and needs the
FORGE_BOT_APP_ID variable and the FORGE_BOT_PRIVATE_KEY secret. Prod pins
are promoted by hand.
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
fx/store/postgres
Package postgres wires the Postgres-backed store implementations into the application via uber-go/fx.
|
Package postgres wires the Postgres-backed store implementations into the application via uber-go/fx. |
|
migrations
Package migrations embeds the sprue Postgres migrations and exposes a runner that applies them via goose.
|
Package migrations embeds the sprue Postgres migrations and exposes a runner that applies them via goose. |
|
tracing
Package tracing is sprue's OpenTelemetry instrumentation: the per-request server span, the per-invocation handler spans and the instrumented HTTP transport every outbound call goes through.
|
Package tracing is sprue's OpenTelemetry instrumentation: the per-request server span, the per-invocation handler spans and the instrumented HTTP transport every outbound call goes through. |
|
pkg
|
|
|
commands/admin/provider/gen
command
|
|
|
internal/revision
Package revision provides the vcs revision, embedded by the compiler, as a global variable.
|
Package revision provides the vcs revision, embedded by the compiler, as a global variable. |
|
store/agent/postgres
Package postgres provides a PostgreSQL-backed implementation of agent.Store.
|
Package postgres provides a PostgreSQL-backed implementation of agent.Store. |
|
store/blob_registry/memory
Package memory holds the in-memory blob registry, for local development and tests.
|
Package memory holds the in-memory blob registry, for local development and tests. |
|
store/blob_registry/postgres
Package postgres provides a PostgreSQL-backed implementation of blob_registry.Store.
|
Package postgres provides a PostgreSQL-backed implementation of blob_registry.Store. |
|
store/consumer/postgres
Package postgres provides a PostgreSQL-backed implementation of consumer.Store.
|
Package postgres provides a PostgreSQL-backed implementation of consumer.Store. |
|
store/customer/postgres
Package postgres provides a PostgreSQL-backed implementation of customer.Store.
|
Package postgres provides a PostgreSQL-backed implementation of customer.Store. |
|
store/delegation/postgres
Package postgres provides a PostgreSQL-backed implementation of delegation.Store.
|
Package postgres provides a PostgreSQL-backed implementation of delegation.Store. |
|
store/metrics/postgres
Package postgres provides PostgreSQL-backed implementations of metrics.Store and metrics.SpaceStore.
|
Package postgres provides PostgreSQL-backed implementations of metrics.Store and metrics.SpaceStore. |
|
store/replica/postgres
Package postgres provides a PostgreSQL-backed implementation of replica.Store.
|
Package postgres provides a PostgreSQL-backed implementation of replica.Store. |
|
store/revocation/postgres
Package postgres provides a PostgreSQL-backed implementation of revocation.Store.
|
Package postgres provides a PostgreSQL-backed implementation of revocation.Store. |
|
store/routing_policy
Package routingpolicy stores routing policies (a policy DID with its set of storage node candidates) and the policy each space references.
|
Package routingpolicy stores routing policies (a policy DID with its set of storage node candidates) and the policy each space references. |
|
store/routing_policy/postgres
Package postgres provides a PostgreSQL-backed implementation of routingpolicy.Store.
|
Package postgres provides a PostgreSQL-backed implementation of routingpolicy.Store. |
|
store/space_diff/postgres
Package postgres provides a PostgreSQL-backed implementation of space_diff.Store.
|
Package postgres provides a PostgreSQL-backed implementation of space_diff.Store. |
|
store/storage_provider/postgres
Package postgres provides a PostgreSQL-backed implementation of storage_provider.Store.
|
Package postgres provides a PostgreSQL-backed implementation of storage_provider.Store. |
|
store/subscription/postgres
Package postgres provides a PostgreSQL-backed implementation of subscription.Store.
|
Package postgres provides a PostgreSQL-backed implementation of subscription.Store. |
|
store/upload/postgres
Package postgres provides a PostgreSQL-backed implementation of upload.Store.
|
Package postgres provides a PostgreSQL-backed implementation of upload.Store. |
|
store/upload_diff
Package uploaddiff records per-space object-count changes as an append-only signed-delta log, so a caller can reconstruct the object count at any past time and bucket it into windows.
|
Package uploaddiff records per-space object-count changes as an append-only signed-delta log, so a caller can reconstruct the object count at any past time and bucket it into windows. |
|
store/upload_diff/postgres
Package postgres provides a PostgreSQL-backed implementation of upload_diff.Store.
|
Package postgres provides a PostgreSQL-backed implementation of upload_diff.Store. |
|
usage
Package usage reconstructs per-space usage time series from the space diff log and the space's running byte counters.
|
Package usage reconstructs per-space usage time series from the space diff log and the space's running byte counters. |