Help us shape the future of Go! Take the 2026 Go Developer Survey

s3api

package
v0.46.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 26, 2026 License: MIT Imports: 44 Imported by: 0

Documentation

Overview

Package s3api serves the S3 wire protocol on top of filex storages.

This file is the half that has to be exactly right before anything else can be built: verifying that a request really was signed by the holder of a secret.

Why this is written by hand

aws-sdk-go-v2 ships a SigV4 signer, and it cannot help here: it SIGNS. There is no verify direction, because a client never needs one. The server does, so the canonical-request construction is reimplemented here — and the tests use the SDK's signer as the golden source, which is the strongest check available: if the two disagree about a single byte, the signatures differ.

The traps, in the order they bite

  • The Host header is SIGNED. A proxy that rewrites it breaks every request with SignatureDoesNotMatch and nothing in the error says so. Go keeps Host off Header, so it is read from r.Host explicitly.
  • Content-Length is likewise absent from Header (Go promotes it to a field) and some clients sign it.
  • S3 does NOT normalize or double-encode the path, unlike every other AWS service. `a//b` and `a/./b` are distinct keys, so the raw escaped path is what gets signed.
  • The payload hash comes from the client (X-Amz-Content-Sha256) and is NOT verified here — verifying it would mean buffering the whole body. It is an input to the signature, so a client cannot lie about it without invalidating its own request, but the bytes themselves are checked where they are read, not here.

Index

Constants

View Source
const (

	// MaxSkew is how far a request's timestamp may be from ours. AWS uses 15
	// minutes; matching it means a client whose clock is tolerated by S3 is
	// tolerated here, and one that is not gets the same error from both.
	MaxSkew = 15 * time.Minute
)
View Source
const Prefix = "/s3"

Prefix is where the endpoint is mounted when it shares a host with the app. A dedicated host (s3.filex.sh) mounts it at the root instead.

Variables

View Source
var (
	// ErrChunkFraming means the body is not valid aws-chunked.
	ErrChunkFraming = errors.New("s3api: malformed aws-chunked body")
	// ErrChunkSignature means a chunk's MAC did not match — the body was
	// altered between the client and here.
	ErrChunkSignature = errors.New("s3api: chunk signature does not match")
)

Errors the caller maps onto S3 codes.

View Source
var (
	// ErrNoCredentials means the request carried no SigV4 material at all.
	ErrNoCredentials = errors.New("s3api: request is not signed")
	// ErrMalformed covers anything unparseable in the signing material.
	ErrMalformed = errors.New("s3api: malformed authorization")
	// ErrSignatureMismatch is the one that matters: the request was signed,
	// and not by the holder of this secret.
	ErrSignatureMismatch = errors.New("s3api: signature does not match")
	// ErrExpired covers both a stale X-Amz-Date and an elapsed presign window.
	ErrExpired = errors.New("s3api: request has expired")
)

Errors a caller maps onto S3 error codes.

View Source
var ErrChecksumMismatch = errors.New("s3api: the object did not match the checksum the client sent")

ErrChecksumMismatch means the bytes did not hash to what the caller said.

Functions

func EndpointURL

func EndpointURL(publicURL, domain string) string

EndpointURL is the address a client program must be given.

⚠⚠ Not the application's own URL. With a dedicated host the endpoint IS that host's root; without one it lives under `/s3`, and a client pointed at the application root reaches the web app and parses an HTML page as XML. Handing out the wrong one is the difference between a paste that works and an error nobody can read, which is why this is computed in one place and returned by the key endpoints rather than assembled in the UI.

func ErrorCode

func ErrorCode(err error) (code string, status int)

ErrorCode maps a verification failure onto the S3 error code a client expects. Clients parse the code, not the HTTP status, so getting this wrong makes a correct refusal look like a broken server.

func HostMatches

func HostMatches(domain, host string) bool

HostMatches reports whether a Host header targets the S3 endpoint — either the endpoint host itself or a `<bucket>.<host>` virtual-hosted address.

⚠ This is what makes the endpoint usable at all. Every S3 client talks to the ROOT of its endpoint URL: `GET /` is ListBuckets, `PUT /bucket/key` is an upload. Mounted only under a path prefix, `GET /` reaches the web app instead and the client is handed an HTML redirect to parse as XML — which is exactly how this was found, with rclone reporting "XML syntax error on line 10" against a page that says "Found".

⚠⚠ Pointing this at the host the web app already serves would hand the whole app to the S3 handler. It must be a DEDICATED host (s3.example.com), which is also what virtual-hosted addressing needs a wildcard record for.

func IsChunked

func IsChunked(payloadHash string) bool

IsChunked reports whether a payload hash marks an aws-chunked body.

func PathStyleRequired

func PathStyleRequired(domain string) bool

PathStyleRequired reports whether clients must be told to use path-style addressing.

Without a dedicated domain there is no `bucket.host` to resolve, and a current SDK — which defaults to virtual-hosted — fails at DNS with an error that names neither filex nor the cause.

func PrincipalFrom

func PrincipalFrom(ctx context.Context) (*protocolauth.Principal, bool)

PrincipalFrom returns the verified caller, if any.

func Verify

func Verify(r *http.Request, sr *Request, lookup SecretLookup, now time.Time) error

Verify recomputes the signature and compares it in constant time.

⚠ It must be called with the request EXACTLY as received — before any middleware rewrites a header, normalizes the path, or strips a query parameter. Everything in the canonical request is part of the MAC.

func WithPrincipal

func WithPrincipal(ctx context.Context, p *protocolauth.Principal) context.Context

WithPrincipal stashes the verified caller for the handlers downstream.

func WriteError

func WriteError(w http.ResponseWriter, r *http.Request, status int, code, message string)

WriteError renders an S3 error. The message is for a human reading a log; the code is the contract.

func WriteXML

func WriteXML(w http.ResponseWriter, r *http.Request, status int, v any)

WriteXML renders a successful XML response.

Types

type Authenticator

type Authenticator struct {
	Res *protocolauth.Resolver
	// Now is injectable for tests; nil means time.Now.
	Now func() time.Time
}

Authenticator turns a signed S3 request into a filex caller.

It exists so the two halves cannot drift apart: looking a key up (protocolauth.AccessKey) and proving the request was signed by its holder (Verify) are useless separately, and a handler that did the first and forgot the second would authenticate anybody who knows an access key id — which is public by design. Nothing in this package hands out a Principal without a verified signature.

func NewAuthenticator

func NewAuthenticator(res *protocolauth.Resolver) *Authenticator

NewAuthenticator builds one.

func (*Authenticator) Authenticate

func (a *Authenticator) Authenticate(r *http.Request) (*protocolauth.Principal, error)

Authenticate verifies the signature and returns the caller.

The returned Principal already carries the tenant scope, the confinement and the ACL resolver, so a handler cannot forget to attach one of them — see internal/protocolauth for why that matters more here than anywhere else.

type Bucket

type Bucket struct {
	Name         string `xml:"Name"`
	CreationDate string `xml:"CreationDate"`
}

Bucket is one storage, seen as an S3 bucket.

type BucketList

type BucketList struct {
	Bucket []Bucket `xml:"Bucket"`
}

BucketList wraps the bucket array. The nesting is not decoration: clients unmarshal <Buckets><Bucket/></Buckets> and a flattened list silently produces zero buckets rather than an error.

type ChecksumError

type ChecksumError struct {
	Header string
	Want   string
	Got    string
}

ChecksumError names which digest disagreed.

func (*ChecksumError) Error

func (e *ChecksumError) Error() string

func (*ChecksumError) Is

func (e *ChecksumError) Is(target error) bool

type CommonPrefix

type CommonPrefix struct {
	Prefix string `xml:"Prefix"`
}

CommonPrefix is a folder, as S3 sees one.

type CompleteMultipartUpload

type CompleteMultipartUpload struct {
	XMLName xml.Name        `xml:"CompleteMultipartUpload"`
	Parts   []CompletedPart `xml:"Part"`
}

CompleteMultipartUpload is the request body listing the parts to assemble.

type CompleteMultipartUploadResult

type CompleteMultipartUploadResult struct {
	XMLName  xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CompleteMultipartUploadResult"`
	Location string   `xml:"Location"`
	Bucket   string   `xml:"Bucket"`
	Key      string   `xml:"Key"`
	ETag     string   `xml:"ETag"`
}

CompleteMultipartUploadResult answers a successful completion.

type CompletedPart

type CompletedPart struct {
	PartNumber int    `xml:"PartNumber"`
	ETag       string `xml:"ETag"`
}

CompletedPart is one part the client believes it uploaded.

type Config

type Config struct {
	// Enabled — FILEX_S3 kill switch. When false the whole subtree 404s, the
	// same shape /dav uses.
	Enabled bool
	Store   db.Store
	// Auth is the shared credential resolver. Required: without it nothing can
	// authenticate, and answering 500 to every request would look like an
	// outage rather than a configuration gap.
	Auth *protocolauth.Resolver
	// ACL resolves per-user grants.
	ACL *acl.Resolver
	// Resolver returns the live driver for a storage id.
	Resolver func(int64) (storage.Driver, error)
	// Staging holds in-flight multipart parts — the SAME area the browser's
	// resumable upload uses, so there is one place on disk with half-finished
	// uploads, one sweeper and one disk guard. Nil disables multipart.
	Staging *staging.Area
	// PublicURL is echoed in a completed upload's Location.
	PublicURL string
	// Quota is the per-user ceiling, checked before bytes land. Nil disables
	// the check, which is what a single-user install wants.
	Quota *quota.Service
	// Index and Thumbs feed the shared post-write bookkeeping. Both optional.
	Index  *search.Index
	Thumbs *thumb.Pipeline
	// Body is the ONE door every read surface goes through: it knows whether
	// the bytes are on the driver or still in the staging area of an upload
	// that has not finished transferring. Reading the driver directly here
	// would make a just-uploaded object 404 over S3 while it is readable
	// everywhere else.
	Body *filebody.Resolver
	// MultiTenant mirrors config.MultiTenant.
	MultiTenant bool
	// Domain is the base host for virtual-hosted-style addressing
	// (bucket.s3.filex.sh). Empty disables that style, leaving path-style —
	// which is what restic and the older SDKs use anyway.
	Domain string
}

Config wires the handler to the shared services.

type CopyObjectResult

type CopyObjectResult struct {
	XMLName      xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CopyObjectResult"`
	ETag         string   `xml:"ETag"`
	LastModified string   `xml:"LastModified"`
}

CopyObjectResult is the response body.

⚠ The result is XML in the BODY of a 200, not a status — including when the copy fails. S3 does that because the connection is already open and streaming for a long copy; a client that only checks the status code will read a failure as success. filex's copies are short enough that this cannot bite in practice, but the shape is what clients parse, so it is the shape we send.

type Credentials

type Credentials struct {
	AccessKeyID string
	SecretKey   string
}

Credentials is what the verifier needs about the key that signed a request.

type DeleteErrorEntry

type DeleteErrorEntry struct {
	Key     string `xml:"Key"`
	Code    string `xml:"Code"`
	Message string `xml:"Message"`
}

DeleteErrorEntry reports one failure.

type DeleteObjectsEntry

type DeleteObjectsEntry struct {
	Key string `xml:"Key"`
}

DeleteObjectsEntry is one key to delete.

type DeleteObjectsRequest

type DeleteObjectsRequest struct {
	XMLName xml.Name `xml:"Delete"`
	// Quiet suppresses the per-key success entries, which is what rclone and
	// mc ask for — a thousand <Deleted> elements per batch is a lot of XML
	// nobody reads.
	Quiet   bool                 `xml:"Quiet"`
	Objects []DeleteObjectsEntry `xml:"Object"`
}

DeleteObjectsRequest is the body of a bulk delete (POST /bucket?delete).

type DeleteResult

type DeleteResult struct {
	XMLName xml.Name           `xml:"http://s3.amazonaws.com/doc/2006-03-01/ DeleteResult"`
	Deleted []DeletedEntry     `xml:"Deleted,omitempty"`
	Errors  []DeleteErrorEntry `xml:"Error,omitempty"`
}

DeleteResult is the response.

type DeletedEntry

type DeletedEntry struct {
	Key string `xml:"Key"`
}

DeletedEntry reports one success.

type Error

type Error struct {
	XMLName   xml.Name `xml:"Error"`
	Code      string   `xml:"Code"`
	Message   string   `xml:"Message"`
	Resource  string   `xml:"Resource,omitempty"`
	RequestID string   `xml:"RequestId,omitempty"`
}

Error is the body every failure returns.

type Handler

type Handler struct {
	// contains filtered or unexported fields
}

Handler serves the S3 protocol.

func NewHandler

func NewHandler(cfg Config) *Handler

NewHandler builds the handler.

func (*Handler) ServeHTTP

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type InitiateMultipartUploadResult

type InitiateMultipartUploadResult struct {
	XMLName  xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ InitiateMultipartUploadResult"`
	Bucket   string   `xml:"Bucket"`
	Key      string   `xml:"Key"`
	UploadID string   `xml:"UploadId"`
}

InitiateMultipartUploadResult answers CreateMultipartUpload.

type ListAllMyBucketsResult

type ListAllMyBucketsResult struct {
	XMLName xml.Name   `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListAllMyBucketsResult"`
	Owner   Owner      `xml:"Owner"`
	Buckets BucketList `xml:"Buckets"`
}

ListAllMyBucketsResult is the ListBuckets response.

type ListBucketResult

type ListBucketResult struct {
	XMLName      xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListBucketResult"`
	Name         string   `xml:"Name"`
	Prefix       string   `xml:"Prefix"`
	Delimiter    string   `xml:"Delimiter,omitempty"`
	MaxKeys      int      `xml:"MaxKeys"`
	IsTruncated  bool     `xml:"IsTruncated"`
	EncodingType string   `xml:"EncodingType,omitempty"`

	// V2 only.
	KeyCount              int    `xml:"KeyCount,omitempty"`
	ContinuationToken     string `xml:"ContinuationToken,omitempty"`
	NextContinuationToken string `xml:"NextContinuationToken,omitempty"`
	StartAfter            string `xml:"StartAfter,omitempty"`

	// V1 only.
	Marker     string `xml:"Marker,omitempty"`
	NextMarker string `xml:"NextMarker,omitempty"`

	Contents       []ObjectEntry  `xml:"Contents"`
	CommonPrefixes []CommonPrefix `xml:"CommonPrefixes"`
}

ListBucketResult serves both list versions. The difference between them is four fields, not four code paths — ⚠ and V1 is NOT optional: clients that still use it (older SDKs, embedded devices, some backup tools) see an empty bucket rather than an error if only V2 exists.

type ListPartsResult

type ListPartsResult struct {
	XMLName     xml.Name    `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListPartsResult"`
	Bucket      string      `xml:"Bucket"`
	Key         string      `xml:"Key"`
	UploadID    string      `xml:"UploadId"`
	MaxParts    int         `xml:"MaxParts"`
	IsTruncated bool        `xml:"IsTruncated"`
	Parts       []PartEntry `xml:"Part"`
}

ListPartsResult answers ListParts.

type LocationConstraint

type LocationConstraint struct {
	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ LocationConstraint"`
	Region  string   `xml:",chardata"`
}

LocationConstraint is the GetBucketLocation response.

type ObjectEntry

type ObjectEntry struct {
	Key          string `xml:"Key"`
	LastModified string `xml:"LastModified"`
	ETag         string `xml:"ETag"`
	Size         int64  `xml:"Size"`
	StorageClass string `xml:"StorageClass"`
}

ObjectEntry is one key in a listing.

type Owner

type Owner struct {
	ID          string `xml:"ID"`
	DisplayName string `xml:"DisplayName"`
}

Owner identifies the caller. S3 requires the element; the values are opaque to every client we care about, so filex uses its own stable identifiers rather than inventing AWS-shaped ones.

type PartEntry

type PartEntry struct {
	PartNumber int    `xml:"PartNumber"`
	Size       int64  `xml:"Size"`
	ETag       string `xml:"ETag"`
}

PartEntry is one staged part, as ListParts reports it.

type Request

type Request struct {
	AccessKeyID   string
	Region        string
	Service       string
	Date          string // yyyymmdd, the credential scope date
	SignedHeaders []string
	Signature     string
	// Presigned requests carry their timestamp and lifetime in the query.
	Presigned bool
	Expires   time.Duration
	Timestamp time.Time
	// PayloadHash is what the client claims the body hashes to, or
	// UNSIGNED-PAYLOAD.
	PayloadHash string
}

Request is the parsed signing material, filled by Parse and consumed by Verify. It is exposed because the caller needs the access key id BEFORE it can look up a secret.

func Parse

func Parse(r *http.Request) (*Request, error)

Parse extracts the signing material from a request without verifying it.

type SecretLookup

type SecretLookup func(accessKeyID string) (string, error)

SecretLookup returns the secret for an access key id, or an error if there is no such key. Returning an error and returning a wrong secret must be indistinguishable to the caller — both end as ErrSignatureMismatch — so an unauthenticated client cannot probe which key ids exist.

type VersioningConfiguration

type VersioningConfiguration struct {
	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ VersioningConfiguration"`
	Status  string   `xml:"Status,omitempty"`
}

VersioningConfiguration is the GetBucketVersioning response. filex keeps version history of its own, but not the S3 object-version model, so the truthful answer is that S3 versioning is off.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL