upload

package
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 9, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package upload implements client-side S3 multipart uploads via presigned URLs.

The Adapto API runs on AWS Lambda behind API Gateway, which base64-encodes binary request bodies into the invocation event (a 4/3 expansion) and caps that event at 6 MB. S3 separately requires every multipart part except the last to be at least 5 MiB. A 5 MiB part base64-encodes to about 6.99 MB against a 6.29 MB cap, and the largest raw body that survives the expansion is 4.50 MiB. No chunk size satisfies both limits, so file bytes cannot travel through the API at all and must go from the client straight to S3.

The API is therefore used only for the control plane: init, presign, complete and abort. Bytes go client -> S3.

Index

Constants

View Source
const (
	// PartSize is the size of every part except the last. S3 rejects non-final
	// parts smaller than 5 MiB, so this must not be lowered.
	PartSize int64 = 5 * 1024 * 1024

	// MaxParts is the S3 limit on the number of parts in one multipart upload.
	MaxParts = 10000
)

Variables

This section is empty.

Functions

func ParseETag

func ParseETag(raw string) (string, error)

ParseETag normalises the ETag S3 returns on a part PUT. S3 sends it quoted, and occasionally as a weak validator; the complete endpoint is sent the bare value.

Types

type APIPresigner

type APIPresigner struct {
	BaseURL string
	Config  config.Config
	HTTP    *http.Client
}

APIPresigner asks the Adapto API for presigned S3 PUT URLs.

The generated OpenAPI client does not cover this route yet, so the request is built by hand. It is an ordinary authenticated Management API call and goes through the shared transport, so it carries the standard headers and gets the same one-shot token refresh on 401 as every other command.

func NewAPIPresigner

func NewAPIPresigner(cfg config.Config) *APIPresigner

NewAPIPresigner builds a presigner from the resolved CLI configuration.

func (*APIPresigner) Presign

func (a *APIPresigner) Presign(ctx context.Context, fileID, uploadID string, partNumbers []int) ([]PresignedURL, error)

Presign requests presigned S3 PUT URLs for the given part numbers.

type Lifecycle

type Lifecycle struct {
	// Init starts a multipart upload and returns the S3 upload id.
	Init func(ctx context.Context, fileID string) (string, error)
	// Complete finalises the upload from the collected part ETags.
	Complete func(ctx context.Context, fileID, uploadID string, parts []Part) error
	// Abort discards an incomplete upload so its parts stop accruing S3
	// storage charges.
	Abort func(ctx context.Context, fileID, uploadID string) error
}

Lifecycle is the set of Adapto API control-plane calls the uploader drives. Complete is a func rather than an interface method so callers can capture the server's response for printing.

type Part

type Part struct {
	ETag       string `json:"ETag"`
	PartNumber int    `json:"PartNumber"`
}

Part is one uploaded part, in the shape the complete endpoint expects.

The ETag is stored unquoted. S3 returns it wrapped in double quotes (and occasionally as a weak validator); the server normalises quoted, unquoted and weak forms, and this package consistently sends the bare value.

type PartPlan

type PartPlan struct {
	// Number is the 1-based S3 part number.
	Number int
	// Offset is the byte offset of the chunk within the file.
	Offset int64
	// Size is the length of the chunk in bytes.
	Size int64
}

PartPlan describes one chunk of the source file.

func PlanParts

func PlanParts(size, partSize int64) ([]PartPlan, error)

PlanParts splits a file of the given size into chunks of partSize.

A zero-length file still yields a single empty part, so complete always has at least one part to reference.

type PresignRequest

type PresignRequest struct {
	UploadID    string `json:"upload_id"`
	PartNumbers []int  `json:"part_numbers"`
}

PresignRequest is the body of POST /manage/files/{file_id}/multipart/presign.

type PresignResponse

type PresignResponse struct {
	UploadID string         `json:"upload_id"`
	URLs     []PresignedURL `json:"urls"`
}

PresignResponse is the 200 body of the presign endpoint.

type PresignedURL

type PresignedURL struct {
	PartNumber int    `json:"part_number"`
	URL        string `json:"url"`
	ExpiresAt  string `json:"expires_at"`
}

PresignedURL is one entry of the presign response.

type Presigner

type Presigner interface {
	Presign(ctx context.Context, fileID, uploadID string, partNumbers []int) ([]PresignedURL, error)
}

Presigner requests presigned S3 PUT URLs for a set of part numbers.

type StatusError

type StatusError struct {
	Code int
	Body string
}

StatusError is a non-2xx response from a presigned S3 URL PUT.

func (*StatusError) Error

func (e *StatusError) Error() string

func (*StatusError) Expired

func (e *StatusError) Expired() bool

Expired reports whether the status means the presigned URL itself was rejected, which calls for a fresh URL rather than a retry of the same one.

func (*StatusError) Retryable

func (e *StatusError) Retryable() bool

Retryable reports whether retrying the same URL could plausibly succeed.

type Uploader

type Uploader struct {
	// Presigner obtains presigned PUT URLs. Required.
	Presigner Presigner
	// HTTP is used for the PUTs to S3. Defaults to http.DefaultClient. It must
	// not be the Management API client: presigned URLs carry their own
	// credentials and reject extra auth headers.
	HTTP *http.Client
	// PartSize overrides PartSize. Only tests should set this; S3 rejects
	// non-final parts under 5 MiB.
	PartSize int64
	// PresignBatch is how many URLs to request per presign call.
	PresignBatch int
	// MaxAttempts bounds transient retries per part.
	MaxAttempts int
	// MaxRefreshes bounds presigned-URL refreshes per part.
	MaxRefreshes int
	// RetryDelay is the base backoff between transient retries.
	RetryDelay time.Duration
	// Progress receives human-readable progress lines. Callers should point
	// this at stderr so stdout stays machine-parseable. Nil discards.
	Progress io.Writer
	// Clock reads the current time when judging whether a presigned URL is
	// still fresh. Defaults to time.Now; tests override it to travel forward.
	Clock func() time.Time
}

Uploader streams a file to S3 one presigned part at a time. Memory use is bounded by the transport's buffers, not by the size of the file: each part is an io.SectionReader over the file on disk.

func (*Uploader) Run

func (u *Uploader) Run(ctx context.Context, lc Lifecycle, fileID string, src io.ReaderAt, size int64) (err error)

Run performs the whole upload for an existing file record: init, presign, PUT every part straight to S3, then complete.

If anything fails after init, the multipart upload is aborted so its parts do not keep accruing S3 storage charges. The original failure is returned; a failure to abort is appended to it rather than replacing it.

func (*Uploader) UploadSinglePart

func (u *Uploader) UploadSinglePart(ctx context.Context, fileID, uploadID string, partNumber int, src io.ReaderAt, size int64) (Part, error)

UploadSinglePart uploads the whole of src as one numbered part of an existing multipart upload. It backs the low-level multipart-upload command, where the caller has already split the file and owns the part numbering.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL