gonimbus

Gonimbus

High-scale cloud object storage inspection and crawl engine

Gonimbus is a Go-first library + CLI + server for large-scale inspection and crawl of cloud object storage. It produces machine-friendly outputs (JSONL baseline) and favors prefix-first listing with doublestar matching to stay fast and predictable.

Scale: Tested with 32M+ object buckets. Path-scoped index builds reduce listing costs by 99%+ on date-partitioned data, and recurring --since top-ups narrow re-listing to recent partitions instead of re-enumerating the whole scope every run.

Modes

• CLI: Run validated crawl/inspect jobs from manifests; stream JSONL to stdout/files or index sinks
• Server: Long-running runner with streaming results; intended to live near the data and accept remote job submissions
• Library: Embeddable components (matcher, crawler, outputs, provider backends, and the Experimental reflow engine) for Go apps. See docs/library-consumers.md for the supported import surface and embedding contract

Quick Start

Prerequisites

• Go 1.26.10 or newer (install), the minimum supported toolchain to ensure secure operation; the module selects toolchain go1.26.10 by default
• golangci-lint (install)

Install

Homebrew (macOS Apple Silicon darwin/arm64, Linux amd64 & arm64):

brew install 3leaps/tap/gonimbus
gonimbus version

Use the full 3leaps/tap/gonimbus formula name (or run brew tap 3leaps/tap first); Intel macOS (darwin/amd64) is intentionally unpublished.

Scoop (Windows amd64 & arm64):

scoop bucket add 3leaps https://github.com/3leaps/scoop-bucket
scoop install 3leaps/gonimbus
gonimbus version

Exact-version / verified binary install (sfetch or GitHub Releases):

sfetch --repo 3leaps/gonimbus --tag v0.5.0 --dest-dir ~/.local/bin

Prebuilt standalone executables for darwin-arm64, linux-amd64, linux-arm64, windows-amd64, and windows-arm64 are attached to each immutable GitHub release, alongside SHA256SUMS and SHA512SUMS manifests, detached minisign (.minisig) and OpenPGP (.asc) signatures, the tagged public verification keys (gonimbus-minisign.pub, gonimbus-release-signing-key.asc), and Decernor fingerprint anchors (expected-fingerprints.txt, expected-fingerprints.ndjson). See docs/security/README.md to verify the signed tag, release assets, and published registries.

Pinned & digest-verified automation runners: 3leaps/tap/gonimbus and 3leaps/scoop-bucket track the latest stable release (with the terminal 0.4.x build preserved as 3leaps/tap/gonimbus@0.4 and 3leaps/gonimbus-0.4). Automation runners that gate execution on an exact release SHA256SUMS digest should install the tagged raw release binary into a runner-private directory (via sfetch or verified GitHub Release assets) or hold workstation upgrades explicitly (brew pin gonimbus / scoop hold gonimbus). On Windows Scoop, ~\scoop\shims\gonimbus.exe on PATH is Scoop's generic launcher shim; hash the resolved target of scoop which gonimbus (~\scoop\apps\gonimbus\current\gonimbus-windows-<arch>.exe) when verifying against SHA256SUMS, and use gonimbus version --json [--extended] for machine-readable preflight checks. See Version-gated downstream inventory.

From source (go install, requires Go 1.26.10+):

go install github.com/3leaps/gonimbus/cmd/gonimbus@latest
gonimbus version

go install-built binaries report the module version via runtime/debug.ReadBuildInfo + an embedded VERSION file (no -ldflags injection required), but are compiled locally and will not match official release SHA256SUMS digests.

Run

# Quick inspection of an S3 prefix
gonimbus inspect s3://my-bucket/path/to/data/

# ...or a Google Cloud Storage prefix (auth via Application Default Credentials)
gonimbus inspect gs://my-bucket/path/to/data/

# Run a crawl job from manifest
gonimbus crawl --job crawl-manifest.yaml

# Check environment and auth
gonimbus doctor

# Check with specific AWS profile
gonimbus doctor --provider s3 --profile my-sso-profile

# Start server mode
gonimbus serve

Build from Source

git clone https://github.com/3leaps/gonimbus.git
cd gonimbus
make bootstrap
make build
./bin/gonimbus version

Core Capabilities

Providers

• S3/S3-compatible: First-class support with access key/secret
• AWS profiles: Assume-role chains, SSO, cached tokens
• Embedded S3 auth controls: Anonymous public reads and injected AWS SDK credential providers for Go library consumers
• GCS (Google Cloud Storage): First-class gs:// source and transfer reflow destination — read (List/Head/Get/range) and reflow-destination (Put + conditional IfAbsent). GCS reports the same IfAbsent honored/probe-status fields as S3 and maps 429 and 403+RESOURCE_EXHAUSTED to provider.ErrThrottled (and 5xx, including 503, to provider.ErrProviderUnavailable), so adaptive --parallel and the capability-aware collision fallback apply unchanged. Authentication uses Application Default Credentials or service-account keys. Not supported: ETag-based If-Match conditional writes (GCS uses generation preconditions), so --on-collision overwrite-if-source-newer is unavailable on GCS destinations and fails closed; IfAbsent-based skip-if-duplicate is fully supported

Authentication

Uses SDK default auth chains - no reinventing the wheel:

• AWS: env vars, shared config/credentials, profiles, SSO, web identity/IRSA
• GCS: Application Default Credentials (gcloud auth application-default login, workload identity, or GOOGLE_APPLICATION_CREDENTIALS service-account key). Credentials resolve from the environment only — gonimbus never accepts a credential filepath from a URI or manifest (credential-source discipline)
• Enterprise SSO: --profile flag with aws sso login workflow
• Raw keys supported as explicit fallback (Wasabi, DigitalOcean Spaces)
• Library consumers can opt into unsigned public reads with s3.Config{Anonymous: true} or inject caller-managed AWS SDK credentials with s3.Config.CredentialsProvider

See docs/auth/aws-profiles.md for enterprise authentication patterns.

Matching

• Doublestar semantics over normalized keys
• Derives strongest possible list prefix per pattern (critical for scale)
• Include/exclude pattern support
• Path-scoped index builds for date-partitioned data (see docs/user-guide/index.md)

Reflow and Index Operations

transfer reflow can write explicit destination user metadata, preserve source content type, set or propagate destination storage class, and derive per-object destination metadata from named source metadata fields or source system fields.

Metadata writes are opt-in and disclosure-sensitive: destination metadata is durable, visible to destination readers, and not redacted at destination. Use --metadata-policy clear plus explicit --metadata-set, --metadata-set-from-source-key, and --metadata-set-from-source-derived allow-lists when source metadata may contain sensitive values. See docs/user-guide/transfer.md and docs/releases/v0.2.1.md for the full operator posture, including the destination-system-metadata non-goal.

Local directory sources are supported with transfer reflow file://...; hidden files and dot-directories are skipped by default unless --hidden=include is set. Indexes retain LIST-derived storage class and can be enriched with HEAD-derived archive/restore/content-type metadata via index enrich-with-head. Local directory trees can also be backed up into object storage by piping crawl --emit reflow-input into transfer reflow --stdin, which preserves nested paths without rewrite templates. Long-running job-backed index build, index enrich-with-head, and transfer reflow runs are now failed-resumable: an interrupted run can be safely continued with --resume-run <run_id>. (Stdin-streamed reflow is not --resume-run-resumable.) stream put can upload raw stdin or framed stream get batches, reflow can use overwrite-if-source-newer for freshness-based collision handling, and inspect-pair can verify terminal reflow write claims against destination HEAD results. Large index export artifacts and reflow writes use multipart upload for S3-compatible destinations when they cross the default threshold; see Local Index and Reflow for local-resource and lifecycle guidance. Recurring index jobs no longer have to re-enumerate their whole listing scope to catch the day's arrivals: index build --since <timestamp>|auto narrows the listing plan on date-partitioned layouts and reports whether that reduction actually applied, so a top-up never silently pays full enumeration cost. Downstream consumers can then use index query --since-run <run_id> to process only the current objects added or changed after a known run; see Steady-State Index Operations. Adaptive transfer reflow --parallel behavior and throughput tuning are documented in docs/user-guide/reflow.md and docs/user-guide/concurrency-and-throughput.md. Terminal reflow records can carry what the destination acknowledged for each write, and content probe ends with a summary record and exits 60 for a completed run with object data errors; see Terminal write receipts and Probe termination and process result. Durable is the default index format (index build publishes durable-index/v0 parquet-inventory/v0 ZSTD segments + manifest unless you pass --format sqlite or --format both, while retaining full dual-read compatibility with 0.4.x durable-v2 indexes). Durable builds compile canonical unicode-prefix/v0 plans (including date_partitions and discovered segments scopes), refuse unexplained successor prefix coverage reductions (scope_loss) unless all four --acknowledge-scope-loss-{parent,plan,by,at} flags are supplied, ingest all enumerated keys in confirmed-complete scopes (dot_segment_policy: "include", dot_segment_rows), and record per-run gonimbus_version / gonimbus_commit / gonimbus_vcs_modified provenance and portable relative marker paths. A finished durable run can be taken into current hub custody and queried as an exact acquired bundle without listing the live object store. Format-aware local consumers include query, list, stats, doctor, enrich-with-head, and whole-set gc. SQLite remains a first-class compatibility path; it is still required for latest-selected query --since-run, stats --prefixes, and full --resume-run checkpoint recovery. Durable forward deltas require exact --index-set, current --run-id, and --since-run pins. Streaming publication uses operator-tunable capacity budgets (16 GiB workspace / 16 MiB record defaults; flag > env > config > default). See Durable Index Format, Reuse a finished inventory, Version-gated downstream inventory, v0.5.0 release notes, v0.4.7 release notes, v0.4.6 release notes, v0.4.5 release notes, v0.4.4 release notes, v0.4.3 release notes, v0.4.2 release notes, v0.4.1 release notes, and v0.4.0 release notes.

Outputs

Stream-friendly JSONL records:

{"type":"gonimbus.object.v1","ts":"2025-01-15T10:30:00.000Z","job_id":"abc123","provider":"s3","data":{...}}

Two output modes for content access:

• Content inspection (JSONL-only): content head reads first N bytes with base64 encoding. See docs/releases/v0.1.6.md.
• Content streaming (mixed framing): stream get delivers full content with JSONL headers + raw bytes. See docs/releases/v0.1.5.md.

Optional local index sink backed by an embedded SQLite database for fast repeated queries on large buckets — see Local Index.

Examples

See docs/user-guide/examples/README.md for copy/paste recipes (advanced filtering, s3-compatible endpoints, and more as this project grows).

CLI Commands

# Explore workflow (no index required)
gonimbus tree <uri>            # Prefix summary (directory-like view)
gonimbus inspect <uri>         # Quick inspection with filters
gonimbus crawl --job <path>    # Full crawl to JSONL
gonimbus atlas build --from-index <id> --output <dir> # Build content-addressed atlas artifacts

# Index workflow (for large buckets)
gonimbus index init            # Initialize local index database
gonimbus index build --job <path>  # Build index (default format: durable)
gonimbus index build --job <path> --format sqlite  # SQLite compatibility path
gonimbus index build --job <path> --format both    # Dual-format + LIST parity report
gonimbus index build --since auto --job <path>  # Incremental top-up (narrows date-partitioned re-listing)
gonimbus index build --background --job <path>  # Background build with job tracking
gonimbus index query <uri>     # Query indexed objects (format-aware: durable or SQLite)
gonimbus index query <uri> --since-run <run_id>  # Latest-selected delta (SQLite)
gonimbus index query --index-set <full-id> --run-id <current> --since-run <baseline>  # Exact durable delta
gonimbus index enrich-with-head <index-set-id>  # HEAD enrich (format-aware: durable or SQLite)
gonimbus index compare durable-delta  # Temporal delta between two durable snapshots
gonimbus index list            # List local indexes (format-aware)
gonimbus index stats           # Show index statistics (format-aware; some flags SQLite-only)
gonimbus index doctor          # Validate index integrity (format-aware)
gonimbus index gc              # Reclaim old local sets from an audited format-aware plan
gonimbus index export          # Export an index run to a hub (auto prefers durable)
gonimbus index hydrate         # Download an index run from a hub (format-aware)
gonimbus index acquire         # Materialize one exact durable run as a verified query bundle
gonimbus index hub             # Manage index hubs
gonimbus index hub bridge-durable  # Copy one already-complete durable run into current custody

# Job management (for long-running builds)
gonimbus index jobs list       # List running and recent jobs
gonimbus index jobs status <id>  # Check job state and progress
gonimbus index jobs logs <id>  # Stream job logs
gonimbus index jobs stop <id>  # Safe cancellation
gonimbus index jobs gc         # Clean up old job records

# Content inspection (JSONL-only, for routing decisions)
gonimbus content head <uri>    # Read first N bytes (base64 in JSONL)

# Content streaming (for pipeline integration)
gonimbus stream head <uri>     # Object metadata (JSONL)
gonimbus stream get <uri>      # Stream full content (JSONL + raw bytes)
gonimbus stream put <uri>      # Upload raw/framed stdin, multipart for large objects

# Operations
gonimbus transfer --job <path> # Copy/move objects between buckets
gonimbus transfer reflow <source> --dest <uri> # Copy objects to a new key layout
gonimbus inspect-pair --from-reflow <path> --expected-dest-prefix <uri> # Verify reflow writes
gonimbus preflight --job <path> # Verify permissions before transfer
gonimbus doctor                # Environment/auth checks
gonimbus envinfo               # Environment summary for support/debugging
gonimbus health                # Self-health check
gonimbus serve                 # Run server mode
gonimbus version               # Version info (--json [--extended] for machine output)

# Safety latch: hard-disable provider-side mutations
# gonimbus --readonly <command>

Configuration

Gonimbus uses three-layer configuration via gofulmen:

1. Template Defaults: config/gonimbus/v1.0.0/gonimbus-defaults.yaml
2. User Overrides: ~/.config/3leaps/gonimbus.yaml
3. Runtime: Environment variables (GONIMBUS_*) and CLI flags

Environment Variables

GONIMBUS_PORT=8080              # Server port
GONIMBUS_HOST=localhost         # Server host
GONIMBUS_LOG_LEVEL=info         # Log level (trace/debug/info/warn/error)
GONIMBUS_METRICS_PORT=9090      # Metrics port
GONIMBUS_READONLY=1             # Disable provider-side mutations
GONIMBUS_DATA_DIR=/mnt/gonimbus-data  # One-process operational data root override
# GONIMBUS_DATA_ROOT is accepted as an alias for GONIMBUS_DATA_DIR.

Copy .env.example to .env for local development.

Server Endpoints

When running in server mode:

• GET /health/* - Liveness/readiness probes
• GET /version - Full version info with SSOT versions
• GET /metrics - Prometheus metrics

Non-Goals

• Mounts, sync engines, FUSE/desktop UX
• "List everything by default" for broad patterns (scale requires explicit sharding)
• Pinning/offline queues

Development

make help                 # Show all targets
make bootstrap            # Install dependencies
make build                # Build binary
make test                 # Run unit tests
make test-cloud           # Run cloud integration tests (requires moto)
make lint                 # Run linting
make surface-scan         # Run public-surface hygiene and content checks
make release-tooling-test # Run release provenance and surface-hygiene test suites
make check-all            # Format, lint, test, API stability, provenance, surface-scan
make pr-final             # Non-mutating gate run by release preflight

Cloud Integration Tests

Cloud integration tests run against a local S3-compatible endpoint (moto):

make moto-start    # Start moto server (Docker)
make test-cloud    # Run cloud integration tests
make moto-stop     # Stop moto server

See docs/development/ for detailed development guides including testing strategy.

Contributing

Contributions are welcome — see AGENTS.md for the role model, commit-attribution format, and quality gates. Gonimbus is open source and conforms to the 3 Leaps OSS policies: keep the repository free of proprietary or client-specific material (client names, proprietary product or brand names, account or bucket identifiers, internal paths), and keep any sensitive local data outside the repository tree per ADR-0005.

Architecture

See docs/architecture.md for component design:

• Provider Layer
• Match Layer (Cloud-Doublestar)
• Crawl Engine
• Durable Storage & State Strata (docs/architecture/durable-storage.md and canonical-state conformance)
• Job Manifest schemas
• Output formats

Fulmen Ecosystem

Gonimbus is part of the Fulmen ecosystem:

Level 4: Production Apps (Gonimbus)
Level 3: DX Tools (goneat, fulward)
Level 2: Templates (forge-workhorse-*)
Level 1: Libraries (gofulmen, pyfulmen)
Level 0: Crucible (SSOT - schemas, standards)

Dependencies

• gofulmen - Config path API, three-layer config, schema validation, Crucible shim
• AWS SDK v2 - Default configuration loading

License

Licensed under the Apache License 2.0. See LICENSE for details.

Trademarks: "Fulmen" and "3 Leaps" are trademarks of 3 Leaps, LLC.