gonimbus

Gonimbus

High-scale cloud object storage inspection and crawl engine

Gonimbus is a Go-first library + CLI + server for large-scale inspection and crawl of cloud object storage. It produces machine-friendly outputs (JSONL baseline) and favors prefix-first listing with doublestar matching to stay fast and predictable.

Scale: Tested with 32M+ object buckets. Path-scoped index builds reduce listing costs by 99%+ on date-partitioned data, and recurring --since top-ups narrow re-listing to recent partitions instead of re-enumerating the whole scope every run.

Modes

• CLI: Run validated crawl/inspect jobs from manifests; stream JSONL to stdout/files or index sinks
• Server: Long-running runner with streaming results; intended to live near the data and accept remote job submissions
• Library: Embeddable components (matcher, crawler, outputs, provider backends, and the Experimental reflow engine) for Go apps. See docs/library-consumers.md for the supported import surface and embedding contract

Quick Start

Prerequisites

• Go 1.26.10 or newer (install), the minimum supported toolchain to ensure secure operation; the module selects toolchain go1.26.10 by default
• golangci-lint (install)

Install

go install github.com/3leaps/gonimbus/cmd/gonimbus@latest
gonimbus version

go install-built binaries report the correct version via runtime/debug.ReadBuildInfo + an embedded VERSION file (no -ldflags injection required).

Prebuilt binaries for darwin-arm64, linux-amd64, linux-arm64, windows-amd64 and windows-arm64 are attached to each GitHub release, with SHA256SUMS and SHA512SUMS manifests signed with minisign and GPG. See docs/security/README.md to verify the signed tag and release assets.

Run

# Quick inspection of an S3 prefix
gonimbus inspect s3://my-bucket/path/to/data/

# ...or a Google Cloud Storage prefix (auth via Application Default Credentials)
gonimbus inspect gs://my-bucket/path/to/data/

# Run a crawl job from manifest
gonimbus crawl --job crawl-manifest.yaml

# Check environment and auth
gonimbus doctor

# Check with specific AWS profile
gonimbus doctor --provider s3 --profile my-sso-profile

# Start server mode
gonimbus serve

Build from Source

git clone https://github.com/3leaps/gonimbus.git
cd gonimbus
make bootstrap
make build
./bin/gonimbus version

Core Capabilities

Providers

• S3/S3-compatible: First-class support with access key/secret
• AWS profiles: Assume-role chains, SSO, cached tokens
• Embedded S3 auth controls: Anonymous public reads and injected AWS SDK credential providers for Go library consumers
• GCS (Google Cloud Storage): First-class gs:// source and transfer reflow destination — read (List/Head/Get/range) and reflow-destination (Put + conditional IfAbsent). GCS reports the same IfAbsent honored/probe-status fields as S3 and maps 429 and 403+RESOURCE_EXHAUSTED to provider.ErrThrottled (and 5xx, including 503, to provider.ErrProviderUnavailable), so adaptive --parallel and the capability-aware collision fallback apply unchanged. Authentication uses Application Default Credentials or service-account keys. Not supported: ETag-based If-Match conditional writes (GCS uses generation preconditions), so --on-collision overwrite-if-source-newer is unavailable on GCS destinations and fails closed; IfAbsent-based skip-if-duplicate is fully supported

Authentication

Uses SDK default auth chains - no reinventing the wheel:

• AWS: env vars, shared config/credentials, profiles, SSO, web identity/IRSA
• GCS: Application Default Credentials (gcloud auth application-default login, workload identity, or GOOGLE_APPLICATION_CREDENTIALS service-account key). Credentials resolve from the environment only — gonimbus never accepts a credential filepath from a URI or manifest (credential-source discipline)
• Enterprise SSO: --profile flag with aws sso login workflow
• Raw keys supported as explicit fallback (Wasabi, DigitalOcean Spaces)
• Library consumers can opt into unsigned public reads with s3.Config{Anonymous: true} or inject caller-managed AWS SDK credentials with s3.Config.CredentialsProvider

See docs/auth/aws-profiles.md for enterprise authentication patterns.

Matching

• Doublestar semantics over normalized keys
• Derives strongest possible list prefix per pattern (critical for scale)
• Include/exclude pattern support
• Path-scoped index builds for date-partitioned data (see docs/user-guide/index.md)

Reflow and Index Operations

transfer reflow can write explicit destination user metadata, preserve source content type, set or propagate destination storage class, and derive per-object destination metadata from named source metadata fields or source system fields.

Metadata writes are opt-in and disclosure-sensitive: destination metadata is durable, visible to destination readers, and not redacted at destination. Use --metadata-policy clear plus explicit --metadata-set, --metadata-set-from-source-key, and --metadata-set-from-source-derived allow-lists when source metadata may contain sensitive values. See docs/user-guide/transfer.md and docs/releases/v0.2.1.md for the full operator posture, including the destination-system-metadata non-goal.

Local directory sources are supported with transfer reflow file://...; hidden files and dot-directories are skipped by default unless --hidden=include is set. Indexes retain LIST-derived storage class and can be enriched with HEAD-derived archive/restore/content-type metadata via index enrich-with-head. Local directory trees can also be backed up into object storage by piping crawl --emit reflow-input into transfer reflow --stdin, which preserves nested paths without rewrite templates. Long-running job-backed index build, index enrich-with-head, and transfer reflow runs are now failed-resumable: an interrupted run can be safely continued with --resume-run <run_id>. (Stdin-streamed reflow is not --resume-run-resumable.) stream put can upload raw stdin or framed stream get batches, reflow can use overwrite-if-source-newer for freshness-based collision handling, and inspect-pair can verify terminal reflow write claims against destination HEAD results. Large index export artifacts and reflow writes use multipart upload for S3-compatible destinations when they cross the default threshold; see Local Index and Reflow for local-resource and lifecycle guidance. Recurring index jobs no longer have to re-enumerate their whole listing scope to catch the day's arrivals: index build --since <timestamp>|auto narrows the listing plan on date-partitioned layouts and reports whether that reduction actually applied, so a top-up never silently pays full enumeration cost. Downstream consumers can then use index query --since-run <run_id> to process only the current objects added or changed after a known run; see Steady-State Index Operations. Adaptive transfer reflow --parallel behavior and throughput tuning are documented in docs/user-guide/reflow.md and docs/user-guide/concurrency-and-throughput.md. Terminal reflow records can carry what the destination acknowledged for each write, and content probe ends with a summary record and exits 60 for a completed run with object data errors; see Terminal write receipts and Probe termination and process result. Durable is the default index format (index build publishes durable-v2 segments + manifest unless you pass --format sqlite or --format both). A finished durable run can be taken into current hub custody and queried as an exact acquired bundle without listing the live object store. Format-aware local consumers include query, list, stats, doctor, enrich-with-head, and whole-set gc. SQLite remains a first-class compatibility path; it is still required for latest-selected query --since-run, stats --prefixes, and full --resume-run checkpoint recovery. Durable forward deltas require exact --index-set, current --run-id, and --since-run pins. Streaming publication uses operator-tunable capacity budgets (16 GiB workspace / 16 MiB record defaults; flag > env > config > default). See Durable Index Format, Reuse a finished inventory, Version-gated downstream inventory, v0.4.6 release notes, v0.4.5 release notes, v0.4.4 release notes, v0.4.3 release notes, v0.4.2 release notes, v0.4.1 release notes, and v0.4.0 release notes.

Outputs

Stream-friendly JSONL records:

{"type":"gonimbus.object.v1","ts":"2025-01-15T10:30:00.000Z","job_id":"abc123","provider":"s3","data":{...}}

Two output modes for content access:

• Content inspection (JSONL-only): content head reads first N bytes with base64 encoding. See docs/releases/v0.1.6.md.
• Content streaming (mixed framing): stream get delivers full content with JSONL headers + raw bytes. See docs/releases/v0.1.5.md.

Optional local index sink backed by an embedded SQLite database for fast repeated queries on large buckets — see Local Index.

Examples

See docs/user-guide/examples/README.md for copy/paste recipes (advanced filtering, s3-compatible endpoints, and more as this project grows).

For automated workflow testing and validation, see fulseed - a companion tool for building reproducible test scenarios.

CLI Commands

# Explore workflow (no index required)
gonimbus tree <uri>            # Prefix summary (directory-like view)
gonimbus inspect <uri>         # Quick inspection with filters
gonimbus crawl --job <path>    # Full crawl to JSONL
gonimbus atlas build --from-index <id> --output <dir> # Build content-addressed atlas artifacts

# Index workflow (for large buckets)
gonimbus index init            # Initialize local index database
gonimbus index build --job <path>  # Build index (default format: durable)
gonimbus index build --job <path> --format sqlite  # SQLite compatibility path
gonimbus index build --job <path> --format both    # Dual-format + LIST parity report
gonimbus index build --since auto --job <path>  # Incremental top-up (narrows date-partitioned re-listing)
gonimbus index build --background --job <path>  # Background build with job tracking
gonimbus index query <uri>     # Query indexed objects (format-aware: durable or SQLite)
gonimbus index query <uri> --since-run <run_id>  # Latest-selected delta (SQLite)
gonimbus index query --index-set <full-id> --run-id <current> --since-run <baseline>  # Exact durable delta
gonimbus index enrich-with-head <index-set-id>  # HEAD enrich (format-aware: durable or SQLite)
gonimbus index compare durable-delta  # Temporal delta between two durable snapshots
gonimbus index list            # List local indexes (format-aware)
gonimbus index stats           # Show index statistics (format-aware; some flags SQLite-only)
gonimbus index doctor          # Validate index integrity (format-aware)
gonimbus index gc              # Reclaim old local sets from an audited format-aware plan
gonimbus index export          # Export an index run to a hub (auto prefers durable)
gonimbus index hydrate         # Download an index run from a hub (format-aware)
gonimbus index acquire         # Materialize one exact durable run as a verified query bundle
gonimbus index hub             # Manage index hubs
gonimbus index hub bridge-durable  # Copy one already-complete durable run into current custody

# Job management (for long-running builds)
gonimbus index jobs list       # List running and recent jobs
gonimbus index jobs status <id>  # Check job state and progress
gonimbus index jobs logs <id>  # Stream job logs
gonimbus index jobs stop <id>  # Safe cancellation
gonimbus index jobs gc         # Clean up old job records

# Content inspection (JSONL-only, for routing decisions)
gonimbus content head <uri>    # Read first N bytes (base64 in JSONL)

# Content streaming (for pipeline integration)
gonimbus stream head <uri>     # Object metadata (JSONL)
gonimbus stream get <uri>      # Stream full content (JSONL + raw bytes)
gonimbus stream put <uri>      # Upload raw/framed stdin, multipart for large objects

# Operations
gonimbus transfer --job <path> # Copy/move objects between buckets
gonimbus transfer reflow <source> --dest <uri> # Copy objects to a new key layout
gonimbus inspect-pair --from-reflow <path> --expected-dest-prefix <uri> # Verify reflow writes
gonimbus preflight --job <path> # Verify permissions before transfer
gonimbus doctor                # Environment/auth checks
gonimbus envinfo               # Environment summary for support/debugging
gonimbus health                # Self-health check
gonimbus serve                 # Run server mode
gonimbus version               # Version info

# Safety latch: hard-disable provider-side mutations
# gonimbus --readonly <command>

Configuration

Gonimbus uses three-layer configuration via gofulmen:

1. Template Defaults: config/gonimbus/v1.0.0/gonimbus-defaults.yaml
2. User Overrides: ~/.config/3leaps/gonimbus.yaml
3. Runtime: Environment variables (GONIMBUS_*) and CLI flags

Environment Variables

GONIMBUS_PORT=8080              # Server port
GONIMBUS_HOST=localhost         # Server host
GONIMBUS_LOG_LEVEL=info         # Log level (trace/debug/info/warn/error)
GONIMBUS_METRICS_PORT=9090      # Metrics port
GONIMBUS_READONLY=1             # Disable provider-side mutations
GONIMBUS_DATA_DIR=/mnt/gonimbus-data  # One-process operational data root override
# GONIMBUS_DATA_ROOT is accepted as an alias for GONIMBUS_DATA_DIR.

Copy .env.example to .env for local development.

Server Endpoints

When running in server mode:

• GET /health/* - Liveness/readiness probes
• GET /version - Full version info with SSOT versions
• GET /metrics - Prometheus metrics

Non-Goals

• Mounts, sync engines, FUSE/desktop UX
• "List everything by default" for broad patterns (scale requires explicit sharding)
• Pinning/offline queues

Development

make help          # Show all targets
make bootstrap     # Install dependencies
make build         # Build binary
make test          # Run unit tests
make test-cloud    # Run cloud integration tests (requires moto)
make lint          # Run linting
make check-all     # Format, lint, test, API stability, release provenance checks
make pr-final      # Non-mutating gate run by release preflight

Cloud Integration Tests

Cloud integration tests run against a local S3-compatible endpoint (moto):

make moto-start    # Start moto server (Docker)
make test-cloud    # Run cloud integration tests
make moto-stop     # Stop moto server

See docs/development/ for detailed development guides including testing strategy.

Contributing

Contributions are welcome — see AGENTS.md for the role model, commit-attribution format, and quality gates. Gonimbus is open source and conforms to the 3 Leaps OSS policies: keep the repository free of proprietary or client-specific material (client names, proprietary product or brand names, account or bucket identifiers, internal paths), and keep any sensitive local data outside the repository tree per ADR-0005.

Architecture

See docs/architecture.md for component design:

• Provider Layer
• Match Layer (Cloud-Doublestar)
• Crawl Engine
• Job Manifest schemas
• Output formats

Fulmen Ecosystem

Gonimbus is part of the Fulmen ecosystem:

Level 4: Production Apps (Gonimbus)
Level 3: DX Tools (goneat, fulward)
Level 2: Templates (forge-workhorse-*)
Level 1: Libraries (gofulmen, pyfulmen)
Level 0: Crucible (SSOT - schemas, standards)

Dependencies

• gofulmen - Config path API, three-layer config, schema validation, Crucible shim
• AWS SDK v2 - Default configuration loading

License

Licensed under the Apache License 2.0. See LICENSE for details.

Trademarks: "Fulmen" and "3 Leaps" are trademarks of 3 Leaps, LLC.