multipartupload

package

v0.4.0 Latest Latest Go to latest Published: Oct 22, 2026 License: MIT Imports: 8 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/baum-ai/goredact

Links

Open Source Insights

Documentation ¶

Overview ¶

Package multipartupload demonstrates a bounded-memory redact, Zstandard, and multipart-upload pipeline. Plaintext flows directly from src into the redactor; it is never written to an intermediate file.

Index ¶

Constants
func RedactCompressUpload(ctx context.Context, client Client, engine *goredact.Engine, src io.Reader, ...) (goredact.Stats, error)
type Client
type Options
type Part
type Progress
type Upload

Constants ¶

View Source

const (
	DefaultPartSize = 8 * 1024 * 1024
	// MaxParts matches the common S3 multipart limit and also bounds retained
	// part metadata independently of input size.
	MaxParts = 10_000
)

Variables ¶

This section is empty.

Functions ¶

func RedactCompressUpload ¶

func RedactCompressUpload(ctx context.Context, client Client, engine *goredact.Engine, src io.Reader, opts Options) (goredact.Stats, error)

RedactCompressUpload streams src through engine and a Zstandard writer into multipart upload. At most PartSize plus the engine's fixed buffers are held in memory. Any scan, compression, upload, completion, or cancellation error causes Abort to be attempted with a fresh bounded context.

Types ¶

type Client ¶

type Client interface {
	Begin(context.Context) (Upload, error)
	UploadPart(context.Context, Upload, int, io.Reader, int64) (Part, error)
	Complete(context.Context, Upload, []Part) error
	Abort(context.Context, Upload) error
}

Client is the storage-specific multipart adapter. Implementations should make UploadPart consume body before returning.

type Options ¶

type Options struct {
	PartSize int
	Progress func(Progress)

	// MaxParts overrides the package-level MaxParts limit when non-zero.
	// Tests use this to exercise the part-count boundary without
	// depending on zstd's compression ratio to hit an exact byte count.
	MaxParts int
}

type Part ¶

type Part any

type Progress ¶

type Progress struct{ BytesRead int64 }

Progress contains counts only and is safe to log. It never contains input or matched bytes.

type Upload ¶

type Upload any

Upload and Part are opaque values owned by Client.

Source Files ¶

View all Source files

upload.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL