trove

Trove: object storage for Go over S3, GCS, Azure Blob and local disk

Trove is an open-source Go library for object and file storage. You write against one API (Put, Get, List, Copy, buckets) and pick the backend at startup: local disk, memory, Amazon S3 and S3-compatible stores like MinIO, Google Cloud Storage, Azure Blob Storage or SFTP. On top of that you get a middleware pipeline for encryption, compression, virus scanning and deduplication, routing of keys to different backends, content-addressable storage, and an io/fs view of a bucket. It runs on its own or as a Forge extension.

Documentation · Quick start · Drivers · Middleware · Routing · CAS and VFS · Forge · FAQ

Quick start

You need Go 1.25 or later. The local disk driver lives in the root module, so one go get is enough to try it:

mkdir hello-trove
cd hello-trove
go mod init example.com/hello-trove
go get github.com/xraph/trove@latest

Save this as main.go:

package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/xraph/trove"
	"github.com/xraph/trove/driver"
	"github.com/xraph/trove/drivers/localdriver"
)

func main() {
	ctx := context.Background()

	local := localdriver.New()
	if err := local.Open(ctx, "local://./storage"); err != nil {
		log.Fatal(err)
	}

	t, err := trove.Open(local, trove.WithDefaultBucket("uploads"))
	if err != nil {
		log.Fatal(err)
	}
	defer t.Close(ctx)

	if err := t.CreateBucket(ctx, "uploads"); err != nil && !errors.Is(err, trove.ErrBucketExists) {
		log.Fatal(err)
	}

	info, err := t.Put(ctx, "uploads", "notes/hello.txt", strings.NewReader("Hello, Trove!"),
		driver.WithContentType("text/plain"))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("stored", info.Key, info.Size, "bytes")

	obj, err := t.Get(ctx, "uploads", "notes/hello.txt")
	if err != nil {
		log.Fatal(err)
	}
	defer obj.Close()

	body, err := io.ReadAll(obj)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(body))

	it, err := t.List(ctx, "uploads", driver.WithPrefix("notes/"))
	if err != nil {
		log.Fatal(err)
	}
	objects, err := it.All(ctx)
	if err != nil {
		log.Fatal(err)
	}
	for _, o := range objects {
		fmt.Println("listed", o.Key)
	}
}

Run it:

go run .
# stored notes/hello.txt 13 bytes
# Hello, Trove!
# listed notes/hello.txt

Look in ./storage/uploads/notes/ and you'll find hello.txt next to a hello.txt.meta.json sidecar, which is where the local driver keeps the content type and metadata. To move the same code to S3, import github.com/xraph/trove/drivers/s3driver, call s3driver.New() and open it with an s3:// DSN. Nothing else changes.

Why Trove?

If you've written an upload service, you've probably written the same wrapper twice: once over the AWS SDK for production and once over the filesystem for tests and local dev. Trove is that wrapper, with the parts you'd bolt on later (encryption at rest, compression, scanning, routing old files to cheap storage) already built as middleware you can scope to a bucket or a key pattern.

The architecture guide shows how the facade, the router, the middleware resolver and the drivers fit together.

Storage drivers

localdriver and memdriver ship inside the root module. The cloud and SFTP drivers are separate Go modules, so you only pull the AWS, Google or Azure SDK into your build when you import that driver, and a service that only ever writes to local disk keeps a short go.sum.

Not every backend can do everything. Capabilities are separate interfaces in the driver package, and you check for one with a type assertion:

if p, ok := t.Driver().(driver.PresignDriver); ok {
	url, err := p.PresignGet(ctx, "uploads", "report.pdf", 15*time.Minute)
	// ...
}

Errors are classified the same way across drivers. errors.Is(err, trove.ErrObjectNotFound) works whether the object was missing on disk or in S3, and trove.Permanent(err) tells you whether a retry has any chance of working.

Middleware

Middleware wraps the write path, the read path, or both. You can register it for everything, or give it a scope so it only touches the buckets and keys you choose:

t, err := trove.Open(drv,
	trove.WithMiddleware(encrypt.New(
		encrypt.WithKeyProvider(encrypt.NewStaticKeyProvider(key)),
	)),
	trove.WithScopedMiddleware(
		&middleware.ScopeBucket{Buckets: []string{"logs"}},
		compress.New(compress.WithMinSize(4096)),
	),
)

Scopes are ScopeBucket, ScopeKeyPattern, ScopeContentType and ScopeFunc, and ScopeAnd, ScopeOr and ScopeNot combine them. You can also add and remove middleware at runtime with UseMiddleware and RemoveMiddleware. One thing to plan for: the middleware buffers the whole object in memory while it works, so if you accept uploads of a few gigabytes through an encrypted or compressed path, size your memory and your upload limits for that. The middleware guide has the details.

Route keys to different backends

One Trove can hold several named backends. Routing rules decide which one a key lands on, and anything no rule claims goes to the driver you passed to Open:

t, err := trove.Open(primary,
	trove.WithBackend("archive", archive),
	trove.WithRoute("*.log", "archive"),
	trove.WithRouteFunc(func(bucket, key string) string {
		if strings.HasPrefix(key, "tmp/") {
			return "archive"
		}
		return ""
	}),
)

Patterns use filepath.Match, so * stops at a / and *.log matches app.log but not 2026/app.log. t.Backend("archive") gives you a Trove bound to that backend if you need to reach it directly. See multi-backend routing.

Content-addressable storage

Turn on CAS with trove.WithCAS(cas.AlgSHA256) (or cas.AlgBlake3, cas.AlgXXHash) and t.CAS() stores content under its hash. Writing the same bytes twice keeps one copy:

hash, info, err := t.CAS().Store(ctx, strings.NewReader("same bytes, stored once"))

Retrieve, Exists and Stat read it back, Pin and Unpin protect an entry, and GC removes what nothing references. More in the CAS docs.

Virtual filesystem

An object store has no folders, only keys. t.VFS("uploads") gives you a directory-shaped view anyway, built from the / in your keys, and vfs.NewIOFS adapts it to io/fs.FS so the standard library can serve it:

fsys := t.VFS("uploads")
entries, err := fsys.ReadDir(ctx, "notes")

http.Handle("/files/", http.StripPrefix("/files/", http.FileServer(http.FS(vfs.NewIOFS(ctx, fsys)))))

The stream package handles chunked transfers with backpressure, resumable streams and progress hooks. Start with the VFS docs and the streaming engine.

Use it with Forge

The github.com/xraph/trove/extension module plugs Trove into a Forge app. It opens the storage driver from a DSN during Register, creates the default bucket, provides *trove.Trove through Forge's DI container, reports health and closes the driver on shutdown. It needs Go 1.26, the same as Forge.

go get github.com/xraph/trove/extension@latest
go get github.com/xraph/trove/drivers/s3driver@latest

import (
	"github.com/xraph/forge"
	"github.com/xraph/trove"
	_ "github.com/xraph/trove/drivers/s3driver"
	troveext "github.com/xraph/trove/extension"
)

app := forge.NewApp(forge.AppConfig{
	Name:        "files",
	HTTPAddress: ":8080",
	Extensions: []forge.Extension{
		troveext.New(troveext.WithConfig(troveext.Config{
			StorageDriver: "s3://us-east-1/my-bucket",
			DefaultBucket: "uploads",
		})),
	},
})

err := app.Router().POST("/notes", func(ctx forge.Context) error {
	t, err := forge.Inject[*trove.Trove](app.Container())
	if err != nil {
		return err
	}

	info, err := t.Put(ctx.Context(), "uploads", "notes/hello.txt", strings.NewReader("hello"))
	if err != nil {
		return err
	}

	return ctx.JSON(http.StatusCreated, info)
})

The extension registers the file, local and mem schemes itself. For S3, GCS, Azure or SFTP you blank-import the driver package so its scheme is registered. With no storage_driver at all, you get an in-memory store, which is handy in tests and a surprise in production.

If the container holds a *grove.DB (from the Grove extension, on Postgres, SQLite or MongoDB), the Trove extension keeps object and bucket metadata there and runs its own migrations during Register. It looks the database up while it registers, so list the Grove extension before Trove. It also registers a contributor for the Forge Dashboard, with a content route at /dashboard/trove/content that serves previews, downloads and uploads through short-lived signed tickets.

The REST handlers in extension/handler (buckets, objects, multipart uploads, CAS) form a plain http.Handler that you build with handler.New and mount yourself. Integrations with Chronicle, Vault, Warden and Dispatch are in extension/hooks, and you wire those yourself too.

Telemetry needs no wiring. Every storage operation records a span under your request's span, and the extension puts forge_trove_operation_duration_seconds, forge_trove_object_bytes_total and the scan and CAS counters on /_/metrics. Outside Forge, pass an observe.Recorder to trove.WithMetrics. The observability page lists every span and metric.

Configuration

The extension reads extensions.trove from your Forge config, then falls back to a top-level trove key. YAML wins for string fields, and a true you set in code turns a flag on even when YAML leaves it off.

extensions:
  trove:
    storage_driver: s3://us-east-1/my-bucket
    default_bucket: uploads
    enable_compression: true

WithTroveOption and WithTroveOptionFor(store, opt) pass any trove.Option, such as your own middleware or routing rules, through to the instances the extension builds. The Forge extension guide and the configuration reference go further.

Documentation

The exported Go API is on pkg.go.dev, and the docs source is in docs/.

FAQ

How do I use S3 and local storage with the same Go code?

Write your code against *trove.Trove and choose the driver when you start up: localdriver with a file:// DSN in development, s3driver with an s3:// DSN in production. The calls and the error values stay the same.

Does Trove work with MinIO, Cloudflare R2 or other S3-compatible storage?

The S3 driver takes an endpoint and path_style parameter in its DSN, as in s3://KEY:SECRET@us-east-1/bucket?endpoint=http://localhost:9000&path_style=true, and the integration tests run against MinIO. Other S3-compatible services work the same way if they accept the S3 API at a custom endpoint.

How do I encrypt files at rest in Go?

Add encrypt.New(encrypt.WithKeyProvider(...)) with trove.WithMiddleware. Objects are sealed with AES-256-GCM before they reach the driver and opened again on Get, and the key provider can fetch the 32-byte key from wherever you keep secrets.

Can I use Trove without Forge?

Yes. The root module has no xraph dependencies at all, so you can drop it into any Go service, and you only need Forge if you import extension.

Contributing

Issues and focused pull requests are welcome on GitHub. Build with GOWORK=off, and test each module from its own directory, since ./... stops at module boundaries:

git clone https://github.com/xraph/trove.git
cd trove
GOWORK=off go build ./...
make test-race
make lint
(cd extension && GOWORK=off go test -race -count=1 ./...)
(cd drivers/s3driver && GOWORK=off go test -race -count=1 ./...)

The extension, gcsdriver and sftpdriver modules need Go 1.26. The driver integration tests sit behind the integration build tag and need a local MinIO, Azurite, fake-gcs-server or SFTP container, which the top of each integration_test.go shows how to start. Benchmarks need -tags bench. Several -all Makefile targets don't work in this repo, and AGENTS.md lists which ones, along with the CI jobs and the release workflow. Commits follow Conventional Commits.

License

MIT. See LICENSE.

Built by Rex Raphael and contributors, as part of the Forge forgery stack.