go-filer

go-filer

A file storage server and Go SDK with a unified HTTP API across local disk, S3-compatible object stores, and in-memory storage.

Features

• Multiple backends: file:// (local disk), s3:// (AWS S3 and S3-compatible), mem:// (in-memory)
• HTTP API server: REST endpoints for listing, uploading, downloading, and deleting objects with streaming multipart uploads
• CLI: List, upload, download, head, and delete objects against a running server
• Go SDK: Typed client library (pkg/httpclient) for embedding filer into Go applications
• Conditional create: If-None-Match: * support to prevent overwriting existing objects
• User metadata: Arbitrary key-value metadata on objects, preserved across backends
• Pagination: Offset/limit pagination with count-only mode
• Concurrent uploads: Parallel multi-file uploads with per-file progress callbacks
• Docker image: Multi-arch (linux/amd64, linux/arm64) image published to ghcr.io/mutablelogic/filer

Quick Start

Server

Run the server with a local data directory mounted at /data:

docker volume create filer-data
docker run -d --name filer \
  -v filer-data:/data -p 8087:8087 \
  ghcr.io/mutablelogic/filer

The server defaults to run file://data/data, which creates a backend named data rooted at /data. Override the listen address or UID/GID as needed:

docker run -d --name filer \
  -v /host/path:/data -p 8087:8087 \
  -e FILER_UID=1234 -e FILER_GID=1234 \
  ghcr.io/mutablelogic/filer

Build from source

git clone https://github.com/mutablelogic/go-filer
cd go-filer

# Full binary (server + client)
make build

# Client-only binary (no server component)
make filer-client

Binaries are placed in build/filer. Pre-built client-only binaries for Linux, macOS, and Windows are also available on the releases page.

Server Configuration

Backends

One or more backend URLs are passed as positional arguments to filer run. Multiple backends can be registered simultaneously.

# Two backends: a local disk backend and an S3 backend
filer run file://local/var/data s3://my-bucket/

AWS / S3 credentials

Credential priority: --aws.profile → --aws.access-key → anonymous.

HTTP

CLI Commands

Released binaries are client-only. The full binary (built with make build) includes the run server command. All client commands point at a running server via --http.addr.

Server

Client

REST API

All endpoints are prefixed with --http.prefix (default /api/filer).

Object metadata is returned in the X-Object-Meta response header as a JSON blob. Multi-file uploads are streamed as multipart requests and return JSON on completion.

Object JSON fields

Go SDK

import httpclient "github.com/mutablelogic/go-filer/pkg/httpclient"

c, err := httpclient.New("http://localhost:8087/api/filer")

// List objects
resp, err := c.ListObjects(ctx, "data", schema.ListObjectsRequest{
    Path:      "/docs",
    Recursive: true,
    Limit:     100,
})

// Upload
obj, err := c.CreateObject(ctx, "data", schema.CreateObjectRequest{
    Path:        "/docs/hello.txt",
    Body:        strings.NewReader("hello world"),
    ContentType: "text/plain",
})

// Download
obj, err := c.ReadObject(ctx, "data", schema.ReadObjectRequest{Path: "/docs/hello.txt"},
    func(chunk []byte) error {
        _, err := os.Stdout.Write(chunk)
        return err
    })

// Delete
obj, err := c.DeleteObject(ctx, "data", schema.DeleteObjectRequest{Path: "/docs/hello.txt"})

Architecture

cmd/filer/          CLI + HTTP server binary
pkg/backend/        Storage backends (file, s3, mem) via gocloud.dev/blob
pkg/httpclient/     Typed Go HTTP client
pkg/httphandler/    HTTP handler registration + request/response encoding
pkg/manager/        Backend registry and lifecycle management
pkg/schema/         Shared request/response types
pkg/version/        Build metadata

License

Apache 2.0 — see LICENSE.