internal/

directory
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0

README

internal

Overview

The Go packages behind the openstorage server (module github.com/coder/openstorage). This directory has no Go files of its own. Requests enter through api (REST) and githttp (git smart HTTP, Git LFS, and the git hooks). Both rely on auth for JWT verification, on meta for org, repo, and upstream metadata, on ref for the ref authority, pack registry, and quota ledger, and on blob for pack and LFS bytes. gitread and gitlimit run bounded git subprocesses. jobs and sync run background work: sweeps, purges, repacks, LFS reclamation, webhook delivery through events, and upstream mirroring. simtest is the deterministic simulation harness over these packages.

Subdirectories & Boundaries

  • api: ./api/README.md - REST /api/v1 routes and the /api/repos/{repo_name} aliases: auth, rate limits, git-read budgets, repo and ref CRUD, reads, commit, merge, fork, sync, credential, and webhook handlers
  • auth: ./auth/README.md - JWT claims, scopes, per-org key verifier, token minting, and ref-policy parsing and matching
  • blob: ./blob/README.md - BlobStore contract with S3 and filesystem backends, pack and LFS key layout, usage listing, and restore covers
  • envconfig: ./envconfig/README.md - OPENSTORAGE_* environment lookup. It never reads the old CODESTORAGE_* names: [envconfig/envconfig.go:CheckLegacy] rejects a process that sets one without its OPENSTORAGE_* replacement
  • events: ./events/README.md - signed webhook delivery with an SSRF dial guard, and a ref-store decorator that emits push events
  • githttp: ./githttp/README.md - git smart-HTTP and LFS server, cache-repo hydration, push holds, and the pre-receive, update, and reference-transaction hooks
  • gitlimit: ./gitlimit/README.md - admission slots, tracked git process groups, and the Linux heap cap
  • gitread: ./gitread/README.md - budgeted read-side git plumbing: trees, blobs, logs, blame, diff, grep, archives, tags, notes
  • jobs: ./jobs/README.md - River and in-process job queues, orphan sweeper, repo purge, repack, LFS reclamation, and absence reconciler
  • lfsscan: ./lfsscan/README.md - LFS pointer parser and bounded reachable-OID walk
  • meta: ./meta/README.md - metadata store (Postgres, SQLite, S3WAL) for orgs, keys, repos, webhooks, credentials, upstream claim lanes, and the local schedule
  • metrics: ./metrics/README.md - Prometheus metrics on a private registry, HTTP instrumentation, and the /metrics listener
  • ref: ./ref/README.md - ref authority (Postgres, SQLite, S3WAL): CAS refs, pack registry, quota ledger, LFS registry, purge, scan state; child s3wal holds the conditional-object substrate
  • s3oplog: ./s3oplog/README.md - opt-in per-process S3 round-trip log
  • secrets: ./secrets/README.md - AES-256-GCM envelope encryption with key rotation
  • simtest: ./simtest/README.md - deterministic simulation testing of a multi-node cluster with fault injection and invariant oracles
  • sync: ./sync/README.md - upstream pull sync and push-forward worker, egress proxy, queues, and local scheduler
  • testutil: ./testutil/README.md - test helpers: pinned container images and container log capture
  • version: ./version/README.md - one-line build version string

Directories

Path Synopsis
Named-repo route aliases /api/repos/{repo_name}/...
Named-repo route aliases /api/repos/{repo_name}/...
Package auth declares the JWT authentication contract for the HTTP API (docs/reference/authentication.md): customer-signed tokens verified against per-org public keys.
Package auth declares the JWT authentication contract for the HTTP API (docs/reference/authentication.md): customer-signed tokens verified against per-org public keys.
Package blob provides immutable pack-object storage: the BlobStore contract plus the S3/MinIO implementation (gocloud.dev s3blob).
Package blob provides immutable pack-object storage: the BlobStore contract plus the S3/MinIO implementation (gocloud.dev s3blob).
Package envconfig resolves OPENSTORAGE_* settings.
Package envconfig resolves OPENSTORAGE_* settings.
Package events implements the docs/reference/webhooks.md outbound-webhook surface: HMAC delivery signing, the per-attempt delivery worker (re-signing on every retry), the storage contract for registered webhooks, and the ref.RefStore decorator that turns won non-ephemeral CAS ref flips into `push` events (docs/explanation/domain-model.md invariant 7).
Package events implements the docs/reference/webhooks.md outbound-webhook surface: HMAC delivery signing, the per-attempt delivery worker (re-signing on every retry), the storage contract for registered webhooks, and the ref.RefStore decorator that turns won non-ephemeral CAS ref flips into `push` events (docs/explanation/domain-model.md invariant 7).
commitgraph.go refreshes the split commit-graph of a cache repo after a full hydrate (issue #307, ADR-47).
commitgraph.go refreshes the split commit-graph of a cache repo after a full hydrate (issue #307, ADR-47).
Package gitlimit is the shared chokepoint for git subprocesses that untrusted input can make expensive (#313, #322): the process-wide admission pool, the per-child heap cap, and their error values.
Package gitlimit is the shared chokepoint for git subprocesses that untrusted input can make expensive (#313, #322): the process-wide admission pool, the per-child heap cap, and their error values.
Package gitread is the read-side git plumbing layer for the /api/v1 git:read endpoints (docs/reference/http-api.md): ref resolution, ls-tree, log, diff, blame, grep, and tar.gz archive streaming over a hydrated bare cache repo.
Package gitread is the read-side git plumbing layer for the /api/v1 git:read endpoints (docs/reference/http-api.md): ref resolution, ls-tree, log, diff, blame, grep, and tar.gz archive streaming over a hydrated bare cache repo.
Package jobs is the phase-5 background-work seam (plan Decision 1): one Queue interface with two implementations sharing the same worker code — River (durable Postgres jobs, at-least-once with backoff retries and unique-by-repo sync jobs) for postgres deployments, and an in-process goroutine queue for local/sqlite mode, whose queued work is lost on process restart (the documented ADR trade-off; local mode keeps working, no silent breakage).
Package jobs is the phase-5 background-work seam (plan Decision 1): one Queue interface with two implementations sharing the same worker code — River (durable Postgres jobs, at-least-once with backoff retries and unique-by-repo sync jobs) for postgres deployments, and an in-process goroutine queue for local/sqlite mode, whose queued work is lost on process restart (the documented ADR trade-off; local mode keeps working, no silent breakage).
Package lfsscan is the leaf reachability layer of unreferenced-LFS-object reclamation (issue #73): a lenient Git LFS pointer-file decoder and a bounded `git rev-list`/`cat-file` walk collecting the OIDs reachable pointer files reference.
Package lfsscan is the leaf reachability layer of unreferenced-LFS-object reclamation (issue #73): a lenient Git LFS pointer-file decoder and a bounded `git rev-list`/`cat-file` walk collecting the OIDs reachable pointer files reference.
Phase-5 eventing/sync persistence contract (docs/reference/webhooks.md webhooks, docs/reference/http-api.md git-credentials + upstream sync): webhook registrations, git credentials, and repo upstream links live in the meta database next to the repo rows they reference.
Phase-5 eventing/sync persistence contract (docs/reference/webhooks.md webhooks, docs/reference/http-api.md git-credentials + upstream sync): webhook registrations, git credentials, and repo upstream links live in the meta database next to the repo rows they reference.
Package metrics owns the opt-in Prometheus exposition (ADR-48, issue #325).
Package metrics owns the opt-in Prometheus exposition (ADR-48, issue #325).
ref
absence_sqlite.go mirrors absence.go's fleet-wide registry paging for the SQLite backend (issue #101) — see absence.go for the rationale.
absence_sqlite.go mirrors absence.go's fleet-wide registry paging for the SQLite backend (issue #101) — see absence.go for the rationale.
s3wal
Package s3wal is the S3-manifest-WAL RefStore experiment (issue #133, docs/plans/s3wal-plan.md): one JSON manifest object per base repo group, advanced by conditional PUT (If-Match / If-None-Match: *).
Package s3wal is the S3-manifest-WAL RefStore experiment (issue #133, docs/plans/s3wal-plan.md): one JSON manifest object per base repo group, advanced by conditional PUT (If-Match / If-None-Match: *).
s3wal/s3waltest
Package s3waltest carries the shared s3wal test fixtures: the pinned SeaweedFS/MinIO testcontainers, raw path-style S3 client construction, bucket bootstrap/cleanup helpers, and the Phase 1a {seq, nonce} payload codec.
Package s3waltest carries the shared s3wal test fixtures: the pinned SeaweedFS/MinIO testcontainers, raw path-style S3 client construction, bucket bootstrap/cleanup helpers, and the Phase 1a {seq, nonce} payload codec.
Package s3oplog is the per-request S3 round-trip log seam (issue #182): when OPENSTORAGE_S3_OP_LOG names a file, every S3 HTTP attempt made through the openstorage SDK clients — the s3wal conditional client (ref and meta authority) and the blob client — appends one whitespace- separated line:
Package s3oplog is the per-request S3 round-trip log seam (issue #182): when OPENSTORAGE_S3_OP_LOG names a file, every S3 HTTP attempt made through the openstorage SDK clients — the s3wal conditional client (ref and meta authority) and the blob client — appends one whitespace- separated line:
Package secrets implements optional application-level envelope encryption for secret columns in the metadata store (issue #54): webhook HMAC signing keys and upstream-sync credential passwords.
Package secrets implements optional application-level envelope encryption for secret columns in the metadata store (issue #54): webhook HMAC signing keys and upstream-sync credential passwords.
Package simtest is the deterministic simulation testing harness (issue #3, ported from waltier's SimStore + DST design): fault-injection decorators for blob.BlobStore and ref.RefStore plus a seeded, serialized multi-node simulation with invariant oracles.
Package simtest is the deterministic simulation testing harness (issue #3, ported from waltier's SimStore + DST design): fault-injection decorators for blob.BlobStore and ref.RefStore plus a seeded, serialized multi-node simulation with invariant oracles.
Package sync implements upstream mirror synchronization (docs/explanation/domain-model.md invariant 11, docs/reference/http-api.md "Upstream sync", docs/reference/webhooks.md repo.sync.* events): a worker mirror-fetches from an upstream git provider and loopback-pushes into this server's own git smart-HTTP endpoint, reusing pack storage, ref policy, CAS, and push-event emission (phase-5 Decision 4).
Package sync implements upstream mirror synchronization (docs/explanation/domain-model.md invariant 11, docs/reference/http-api.md "Upstream sync", docs/reference/webhooks.md repo.sync.* events): a worker mirror-fetches from an upstream git provider and loopback-pushes into this server's own git smart-HTTP endpoint, reusing pack storage, ref policy, CAS, and push-event emission (phase-5 Decision 4).
testutil
seaweedcapture
Package seaweedcapture retains opt-in, private logs for SeaweedFS test fixtures.
Package seaweedcapture retains opt-in, private logs for SeaweedFS test fixtures.
testimages
Package testimages pins the container images that tests and fixtures pull.
Package testimages pins the container images that tests and fixtures pull.
Package version reports the build version of the openstorage binary.
Package version reports the build version of the openstorage binary.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL