Documentation
¶
Overview ¶
Package s3api serves the S3 wire protocol on top of filex storages.
This file is the half that has to be exactly right before anything else can be built: verifying that a request really was signed by the holder of a secret.
Why this is written by hand ¶
aws-sdk-go-v2 ships a SigV4 signer, and it cannot help here: it SIGNS. There is no verify direction, because a client never needs one. The server does, so the canonical-request construction is reimplemented here — and the tests use the SDK's signer as the golden source, which is the strongest check available: if the two disagree about a single byte, the signatures differ.
The traps, in the order they bite ¶
- The Host header is SIGNED. A proxy that rewrites it breaks every request with SignatureDoesNotMatch and nothing in the error says so. Go keeps Host off Header, so it is read from r.Host explicitly.
- Content-Length is likewise absent from Header (Go promotes it to a field) and some clients sign it.
- S3 does NOT normalize or double-encode the path, unlike every other AWS service. `a//b` and `a/./b` are distinct keys, so the raw escaped path is what gets signed.
- The payload hash comes from the client (X-Amz-Content-Sha256) and is NOT verified here — verifying it would mean buffering the whole body. It is an input to the signature, so a client cannot lie about it without invalidating its own request, but the bytes themselves are checked where they are read, not here.
Index ¶
- Constants
- Variables
- func EndpointURL(publicURL, domain string) string
- func ErrorCode(err error) (code string, status int)
- func HostMatches(domain, host string) bool
- func IsChunked(payloadHash string) bool
- func PathStyleRequired(domain string) bool
- func PrincipalFrom(ctx context.Context) (*protocolauth.Principal, bool)
- func Verify(r *http.Request, sr *Request, lookup SecretLookup, now time.Time) error
- func WithPrincipal(ctx context.Context, p *protocolauth.Principal) context.Context
- func WriteError(w http.ResponseWriter, r *http.Request, status int, code, message string)
- func WriteXML(w http.ResponseWriter, r *http.Request, status int, v any)
- type Authenticator
- type Bucket
- type BucketList
- type ChecksumError
- type CommonPrefix
- type CompleteMultipartUpload
- type CompleteMultipartUploadResult
- type CompletedPart
- type Config
- type CopyObjectResult
- type Credentials
- type DeleteErrorEntry
- type DeleteObjectsEntry
- type DeleteObjectsRequest
- type DeleteResult
- type DeletedEntry
- type Error
- type Handler
- type InitiateMultipartUploadResult
- type ListAllMyBucketsResult
- type ListBucketResult
- type ListPartsResult
- type LocationConstraint
- type ObjectEntry
- type Owner
- type PartEntry
- type Request
- type SecretLookup
- type VersioningConfiguration
Constants ¶
const ( // MaxSkew is how far a request's timestamp may be from ours. AWS uses 15 // minutes; matching it means a client whose clock is tolerated by S3 is // tolerated here, and one that is not gets the same error from both. MaxSkew = 15 * time.Minute )
const Prefix = "/s3"
Prefix is where the endpoint is mounted when it shares a host with the app. A dedicated host (s3.filex.sh) mounts it at the root instead.
Variables ¶
var ( // ErrChunkFraming means the body is not valid aws-chunked. ErrChunkFraming = errors.New("s3api: malformed aws-chunked body") // ErrChunkSignature means a chunk's MAC did not match — the body was // altered between the client and here. ErrChunkSignature = errors.New("s3api: chunk signature does not match") // ErrDecodedLength means the decoded body did not add up to the // x-amz-decoded-content-length the client declared: shorter, or longer. ErrDecodedLength = errors.New("s3api: the decoded body does not match x-amz-decoded-content-length") )
Errors the caller maps onto S3 codes.
var ( // ErrNoCredentials means the request carried no SigV4 material at all. ErrNoCredentials = errors.New("s3api: request is not signed") // ErrMalformed covers anything unparseable in the signing material. ErrMalformed = errors.New("s3api: malformed authorization") // ErrSignatureMismatch is the one that matters: the request was signed, // and not by the holder of this secret. ErrSignatureMismatch = errors.New("s3api: signature does not match") // ErrExpired covers both a stale X-Amz-Date and an elapsed presign window. ErrExpired = errors.New("s3api: request has expired") )
Errors a caller maps onto S3 error codes.
var ErrChecksumMismatch = errors.New("s3api: the object did not match the checksum the client sent")
ErrChecksumMismatch means the bytes did not hash to what the caller said.
Functions ¶
func EndpointURL ¶
EndpointURL is the address a client program must be given.
⚠⚠ Not the application's own URL. With a dedicated host the endpoint IS that host's root; without one it lives under `/s3`, and a client pointed at the application root reaches the web app and parses an HTML page as XML. Handing out the wrong one is the difference between a paste that works and an error nobody can read, which is why this is computed in one place and returned by the key endpoints rather than assembled in the UI.
func ErrorCode ¶
ErrorCode maps a verification failure onto the S3 error code a client expects. Clients parse the code, not the HTTP status, so getting this wrong makes a correct refusal look like a broken server.
func HostMatches ¶
HostMatches reports whether a Host header targets the S3 endpoint — either the endpoint host itself or a `<bucket>.<host>` virtual-hosted address.
⚠ This is what makes the endpoint usable at all. Every S3 client talks to the ROOT of its endpoint URL: `GET /` is ListBuckets, `PUT /bucket/key` is an upload. Mounted only under a path prefix, `GET /` reaches the web app instead and the client is handed an HTML redirect to parse as XML — which is exactly how this was found, with rclone reporting "XML syntax error on line 10" against a page that says "Found".
⚠⚠ Pointing this at the host the web app already serves would hand the whole app to the S3 handler. It must be a DEDICATED host (s3.example.com), which is also what virtual-hosted addressing needs a wildcard record for.
func PathStyleRequired ¶
PathStyleRequired reports whether clients must be told to use path-style addressing.
Without a dedicated domain there is no `bucket.host` to resolve, and a current SDK — which defaults to virtual-hosted — fails at DNS with an error that names neither filex nor the cause.
func PrincipalFrom ¶
func PrincipalFrom(ctx context.Context) (*protocolauth.Principal, bool)
PrincipalFrom returns the verified caller, if any.
func Verify ¶
Verify recomputes the signature and compares it in constant time.
⚠ It must be called with the request EXACTLY as received — before any middleware rewrites a header, normalizes the path, or strips a query parameter. Everything in the canonical request is part of the MAC.
func WithPrincipal ¶
WithPrincipal stashes the verified caller for the handlers downstream.
func WriteError ¶
WriteError renders an S3 error. The message is for a human reading a log; the code is the contract.
Types ¶
type Authenticator ¶
type Authenticator struct {
Res *protocolauth.Resolver
// Now is injectable for tests; nil means time.Now.
Now func() time.Time
}
Authenticator turns a signed S3 request into a filex caller.
It exists so the two halves cannot drift apart: looking a key up (protocolauth.AccessKey) and proving the request was signed by its holder (Verify) are useless separately, and a handler that did the first and forgot the second would authenticate anybody who knows an access key id — which is public by design. Nothing in this package hands out a Principal without a verified signature.
func NewAuthenticator ¶
func NewAuthenticator(res *protocolauth.Resolver) *Authenticator
NewAuthenticator builds one.
func (*Authenticator) Authenticate ¶
func (a *Authenticator) Authenticate(r *http.Request) (*protocolauth.Principal, error)
Authenticate verifies the signature and returns the caller.
The returned Principal already carries the tenant scope, the confinement and the ACL resolver, so a handler cannot forget to attach one of them — see internal/protocolauth for why that matters more here than anywhere else.
type BucketList ¶
type BucketList struct {
Bucket []Bucket `xml:"Bucket"`
}
BucketList wraps the bucket array. The nesting is not decoration: clients unmarshal <Buckets><Bucket/></Buckets> and a flattened list silently produces zero buckets rather than an error.
type ChecksumError ¶
ChecksumError names which digest disagreed.
func (*ChecksumError) Error ¶
func (e *ChecksumError) Error() string
func (*ChecksumError) Is ¶
func (e *ChecksumError) Is(target error) bool
type CommonPrefix ¶
type CommonPrefix struct {
Prefix string `xml:"Prefix"`
}
CommonPrefix is a folder, as S3 sees one.
type CompleteMultipartUpload ¶
type CompleteMultipartUpload struct {
XMLName xml.Name `xml:"CompleteMultipartUpload"`
Parts []CompletedPart `xml:"Part"`
}
CompleteMultipartUpload is the request body listing the parts to assemble.
type CompleteMultipartUploadResult ¶
type CompleteMultipartUploadResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CompleteMultipartUploadResult"`
Location string `xml:"Location"`
Bucket string `xml:"Bucket"`
Key string `xml:"Key"`
ETag string `xml:"ETag"`
}
CompleteMultipartUploadResult answers a successful completion.
type CompletedPart ¶
CompletedPart is one part the client believes it uploaded.
type Config ¶
type Config struct {
// Enabled — FILEX_S3 kill switch. When false the whole subtree 404s, the
// same shape /dav uses.
Enabled bool
Store db.Store
// Auth is the shared credential resolver. Required: without it nothing can
// authenticate, and answering 500 to every request would look like an
// outage rather than a configuration gap.
Auth *protocolauth.Resolver
// ACL resolves per-user grants.
ACL *acl.Resolver
// E2EPolicy is who may encrypt (internal/e2epolicy): a write that CREATES
// an encrypted folder's key file or a `.fxe` asks it
// (protoperm.EncryptionAllowed). The router's own, handed over like ACL.
// nil: NewHandler builds one over Store (protoperm.EncryptionPolicy), so a
// lost wiring line never switches the rule off.
E2EPolicy *e2epolicy.Service
// Resolver returns the live driver for a storage id.
Resolver func(int64) (storage.Driver, error)
// Staging holds in-flight multipart parts — the SAME area the browser's
// resumable upload uses, so there is one place on disk with half-finished
// uploads, one sweeper and one disk guard. Nil disables multipart.
Staging *staging.Area
// PublicURL is echoed in a completed upload's Location.
PublicURL string
// Quota is the per-user ceiling, checked before bytes land. Nil disables
// the check, which is what a single-user install wants.
Quota *quota.Service
// Index and Thumbs feed the shared post-write bookkeeping. Both optional.
Index *search.Index
Thumbs *thumb.Pipeline
// Body is the ONE door every read surface goes through: it knows whether
// the bytes are on the driver or still in the staging area of an upload
// that has not finished transferring. Reading the driver directly here
// would make a just-uploaded object 404 over S3 while it is readable
// everywhere else.
Body *filebody.Resolver
// MultiTenant mirrors config.MultiTenant.
MultiTenant bool
// Domain is the base host for virtual-hosted-style addressing
// (bucket.s3.filex.sh). Empty disables that style, leaving path-style —
// which is what restic and the older SDKs use anyway.
Domain string
}
Config wires the handler to the shared services.
type CopyObjectResult ¶
type CopyObjectResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CopyObjectResult"`
ETag string `xml:"ETag"`
LastModified string `xml:"LastModified"`
}
CopyObjectResult is the response body.
⚠ The result is XML in the BODY of a 200, not a status — including when the copy fails. S3 does that because the connection is already open and streaming for a long copy; a client that only checks the status code will read a failure as success. filex's copies are short enough that this cannot bite in practice, but the shape is what clients parse, so it is the shape we send.
type Credentials ¶
Credentials is what the verifier needs about the key that signed a request.
type DeleteErrorEntry ¶
type DeleteErrorEntry struct {
Key string `xml:"Key"`
Code string `xml:"Code"`
Message string `xml:"Message"`
}
DeleteErrorEntry reports one failure.
type DeleteObjectsEntry ¶
type DeleteObjectsEntry struct {
Key string `xml:"Key"`
}
DeleteObjectsEntry is one key to delete.
type DeleteObjectsRequest ¶
type DeleteObjectsRequest struct {
XMLName xml.Name `xml:"Delete"`
// Quiet suppresses the per-key success entries, which is what rclone and
// mc ask for — a thousand <Deleted> elements per batch is a lot of XML
// nobody reads.
Quiet bool `xml:"Quiet"`
Objects []DeleteObjectsEntry `xml:"Object"`
}
DeleteObjectsRequest is the body of a bulk delete (POST /bucket?delete).
type DeleteResult ¶
type DeleteResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ DeleteResult"`
Deleted []DeletedEntry `xml:"Deleted,omitempty"`
Errors []DeleteErrorEntry `xml:"Error,omitempty"`
}
DeleteResult is the response.
type DeletedEntry ¶
type DeletedEntry struct {
Key string `xml:"Key"`
}
DeletedEntry reports one success.
type Error ¶
type Error struct {
XMLName xml.Name `xml:"Error"`
Code string `xml:"Code"`
Message string `xml:"Message"`
Resource string `xml:"Resource,omitempty"`
RequestID string `xml:"RequestId,omitempty"`
}
Error is the body every failure returns.
type Handler ¶
type Handler struct {
// contains filtered or unexported fields
}
Handler serves the S3 protocol.
type InitiateMultipartUploadResult ¶
type InitiateMultipartUploadResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ InitiateMultipartUploadResult"`
Bucket string `xml:"Bucket"`
Key string `xml:"Key"`
UploadID string `xml:"UploadId"`
}
InitiateMultipartUploadResult answers CreateMultipartUpload.
type ListAllMyBucketsResult ¶
type ListAllMyBucketsResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListAllMyBucketsResult"`
Owner Owner `xml:"Owner"`
Buckets BucketList `xml:"Buckets"`
}
ListAllMyBucketsResult is the ListBuckets response.
type ListBucketResult ¶
type ListBucketResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListBucketResult"`
Name string `xml:"Name"`
Prefix string `xml:"Prefix"`
Delimiter string `xml:"Delimiter,omitempty"`
MaxKeys int `xml:"MaxKeys"`
IsTruncated bool `xml:"IsTruncated"`
EncodingType string `xml:"EncodingType,omitempty"`
// V2 only.
KeyCount int `xml:"KeyCount,omitempty"`
ContinuationToken string `xml:"ContinuationToken,omitempty"`
NextContinuationToken string `xml:"NextContinuationToken,omitempty"`
StartAfter string `xml:"StartAfter,omitempty"`
// V1 only.
Marker string `xml:"Marker,omitempty"`
NextMarker string `xml:"NextMarker,omitempty"`
Contents []ObjectEntry `xml:"Contents"`
CommonPrefixes []CommonPrefix `xml:"CommonPrefixes"`
}
ListBucketResult serves both list versions. The difference between them is four fields, not four code paths — ⚠ and V1 is NOT optional: clients that still use it (older SDKs, embedded devices, some backup tools) see an empty bucket rather than an error if only V2 exists.
type ListPartsResult ¶
type ListPartsResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListPartsResult"`
Bucket string `xml:"Bucket"`
Key string `xml:"Key"`
UploadID string `xml:"UploadId"`
MaxParts int `xml:"MaxParts"`
IsTruncated bool `xml:"IsTruncated"`
Parts []PartEntry `xml:"Part"`
}
ListPartsResult answers ListParts.
type LocationConstraint ¶
type LocationConstraint struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ LocationConstraint"`
Region string `xml:",chardata"`
}
LocationConstraint is the GetBucketLocation response.
type ObjectEntry ¶
type ObjectEntry struct {
Key string `xml:"Key"`
LastModified string `xml:"LastModified"`
ETag string `xml:"ETag"`
Size int64 `xml:"Size"`
StorageClass string `xml:"StorageClass"`
}
ObjectEntry is one key in a listing.
type Owner ¶
Owner identifies the caller. S3 requires the element; the values are opaque to every client we care about, so filex uses its own stable identifiers rather than inventing AWS-shaped ones.
type PartEntry ¶
type PartEntry struct {
PartNumber int `xml:"PartNumber"`
Size int64 `xml:"Size"`
ETag string `xml:"ETag"`
}
PartEntry is one staged part, as ListParts reports it.
type Request ¶
type Request struct {
AccessKeyID string
Region string
Service string
Date string // yyyymmdd, the credential scope date
SignedHeaders []string
Signature string
// Presigned requests carry their timestamp and lifetime in the query.
Presigned bool
Expires time.Duration
Timestamp time.Time
// PayloadHash is what the client claims the body hashes to, or
// UNSIGNED-PAYLOAD.
PayloadHash string
}
Request is the parsed signing material, filled by Parse and consumed by Verify. It is exposed because the caller needs the access key id BEFORE it can look up a secret.
type SecretLookup ¶
SecretLookup returns the secret for an access key id, or an error if there is no such key. Returning an error and returning a wrong secret must be indistinguishable to the caller — both end as ErrSignatureMismatch — so an unauthenticated client cannot probe which key ids exist.
type VersioningConfiguration ¶
type VersioningConfiguration struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ VersioningConfiguration"`
Status string `xml:"Status,omitempty"`
}
VersioningConfiguration is the GetBucketVersioning response. filex keeps version history of its own, but not the S3 object-version model, so the truthful answer is that S3 versioning is off.