session

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package session is the upload session shared by the object-store blob backends (s3, gcs, azure; decision 0027). Object stores cannot append, so each backend buffers appended bytes into parts and uploads a part whenever one is full; between HTTP requests the session must survive in the metadata store so a Docker chunked upload or a CLI multipart upload resumes in a later request or another process.

A Session holds what every backend needs: the size, the declared digest, the saved blob.Hasher state and the pending tail shorter than one part, plus a backend-specific document D (S3's multipart upload ID and part list, GCS's part objects, Azure's block count). It is persisted through a Store after every append. InUploads keeps sessions in the uploads table under a per-store backend name such as "s3:hot", so two stores of one kind never share or misattribute sessions.

Role

platform layer: the upload session of the object-store blob backends, and the port of the uploads table it is persisted through. Blob storage belongs to no module, so the sessions sit in the platform beside platform/blob, whose hasher, upload options and sentinels they use. The s3, gcs and azure stores keep their sessions through it, and so does the mirror for the uploads it spreads over its members. Upload and UploadStore are declared here, by the package that persists through them, so that it imports no store (decision 0044, "Narrowing"). The platform's store adapters (platform/store/sql and platform/store/inmem) implement the port over the uploads table. Through the metadata store, the Docker registry and the JFrog CLI multipart service keep their own sessions in the same table, and the stale-upload reaper of garbage collection lists the idle ones with Stale. The package imports platform/blob, the kernel and the standard library.

Inputs

  • From a backend: the upload options for Begin (a declared SHA-256), an upload ID for Resume, the appended bytes as an io.Reader with the backend's part size and its FlushFunc, and the digests to Verify.
  • A Store to persist through: InUploads over an UploadStore, the metadata store's uploads, and a backend name for the store instance (the composition root uses "<backend>:<tier>", such as "s3:blob" or "gcs:cold_blob", "<backend>:<tier>.<member>" for a mirror member and "mirror:<tier>" for the mirror); or NewMemory, which keeps sessions in process memory for tests and embedders without a metadata store.

Outputs

  • Session[D], from Begin or Resume, with Append, Flush, Save, Live, Verify and Forget; FlushFunc, which hands one part to the backend.
  • Store, what a session is persisted through (Put, Get, Delete), with Memory (NewMemory) and InUploads.
  • Upload, one row of the uploads table, and UploadStore, the port the platform's stores implement: Put, Get, Delete, and Stale for the reaper.

Invariants

  • A session is saved after every append, also when the flush or the read fails, so Size counts every byte read and hashed and the client can continue from it. A part whose flush fails stays pending, and the next Append or Commit retries it.
  • A session stored under another backend name is unknown to InUploads, so the stale-upload reaper always knows which store aborts it: each row names its store, and a store resumes only its own.
  • Get of an unknown ID wraps blob.ErrUploadUnknown, and Delete of a missing session succeeds, in Memory and InUploads alike. Resume of an unknown, malformed or damaged session also wraps blob.ErrUploadUnknown: there is no local copy of the uploaded parts to rehash, so the session ends.
  • InUploads keeps a session's Started on every save and sets Updated to the time of the save, which is what Stale compares.
  • The stored form is JSON, the record's fields and each backend's document under its json tags. A session saved by one process is resumed by another, so the encoding stays as it is.

References

  • Decision 0004: docs/research/decisions/0004-storage-sql-metadata-and-blobs.md ("Metadata" and "Blobs": the metadata store, and upload sessions that resume across requests).
  • Decision 0027: docs/research/decisions/0027-cloud-blob-backends.md ("Shared session layer": object stores cannot append, and sessions live in the uploads table under a per-instance backend name).
  • Research 15, section 4 (the uploads table): docs/research/15-data-model.md; research 18: docs/research/18-cloud-blob-storage.md.
  • Decision 0044: docs/research/decisions/0044-modular-monolith-migration.md ("Narrowing").
  • Target layout: docs/research/architecture-options/5-modular-monolith.md, line 149 (platform/blob/session) and line 1049 (the mapping row).
  • Plan: docs/implementation/plan.md#p2-stores, #p2b-cloud-blob-storage-and-mirroring and #p9d-modular-monolith-migration, epic PLT (https://github.com/weaveplatform/artifactweave/issues/938).
  • https://github.com/weaveplatform/artifactweave/issues/8
  • https://github.com/weaveplatform/artifactweave/issues/982

Status

Implemented P2: Upload and UploadStore, with Stale for the stale-upload reaper that garbage collection runs from P3 (pull request 3). P2b: the shared session layer, with Session, Store, Memory and InUploads under per-store backend names (issue 8, pull request 30). Moved here in P9d.

Testing

  • TestUnitAppendFlushesFullPartsAndResumes: full parts go to the flush and the tail stays pending; a resumed session continues where it stopped, Verify gives the digests of everything appended, and after Forget, Live and Append report blob.ErrUploadUnknown.
  • TestUnitVerify: the declared SHA-256 with a matching MD5 passes, and a declared digest of other content is blob.ErrDigestMismatch.
  • TestUnitAppendFailures: a failed flush keeps the bytes pending and counted; a failed read, save or forget, data that cannot be encoded, a cancelled context and an invalid declared digest each fail with their cause.
  • TestUnitResumeRejects: a malformed record, hasher state or document, an invalid ID and an ID never issued are blob.ErrUploadUnknown.
  • TestUnitMemoryCopies: Memory copies on Put and Get, and a deleted session is unknown.
  • TestUnitInUploadsSeparatesStores: decision 0027's check that two stores of one kind share the uploads table without seeing each other's sessions, and that each row names its store.
  • TestUnitInUploadsRoundTripKeepsStart: a save keeps Started and moves Updated, and forgetting twice succeeds.
  • TestUnitInUploadsErrors: the upload store's errors are wrapped and not taken for an unknown session, and a missing row lets Put start one.

The InUploads tests keep their sessions in the platform's in-memory store. The platform's upload contract (the Uploads subtest of platform/store/storetest) runs on the platform's in-memory and SQLite stores and on the metadata store's in-memory, SQLite and PostgreSQL stores, and the s3, gcs, azure and mirror stores run the blob contract with sessions kept through InUploads.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type FlushFunc

type FlushFunc func(ctx context.Context, part []byte) error

FlushFunc hands one part to the backend. On success the session drops the part from Pending; on error the bytes stay pending (and hashed), so the next Append or Commit retries the same part.

type Memory

type Memory struct {
	// contains filtered or unexported fields
}

Memory is a Store in process memory. Sessions do not survive a restart; it serves tests and embedders without a metadata store.

func NewMemory

func NewMemory() *Memory

NewMemory returns an empty Memory.

func (*Memory) Delete

func (m *Memory) Delete(_ context.Context, id string) error

Delete forgets a session.

func (*Memory) Get

func (m *Memory) Get(_ context.Context, id string) ([]byte, error)

Get returns a copy of the session.

func (*Memory) Put

func (m *Memory) Put(_ context.Context, id string, state []byte, _ int64) error

Put stores a copy of state.

type Session

type Session[D any] struct {
	// ID names the session for Resume.
	ID string
	// Size counts every appended byte, uploaded or pending.
	Size int64
	// Expected is the SHA-256 declared at BeginUpload, or "".
	Expected string
	// Pending is the tail not yet handed to the backend.
	Pending []byte
	// Data is the backend's state.
	Data D
	// contains filtered or unexported fields
}

Session is one upload session. D is the backend's own state; it is stored as JSON, so its fields carry json tags and are the storage format.

func Begin

func Begin[D any](ctx context.Context, st Store, opts ...blob.UploadOption) (*Session[D], error)

Begin starts a session with a new upload ID and saves it.

func Resume

func Resume[D any](ctx context.Context, st Store, id string) (*Session[D], error)

Resume loads a saved session. Unknown or malformed IDs, and sessions whose state cannot be restored, return an error wrapping blob.ErrUploadUnknown: there is no local copy of the uploaded parts to rehash, so a damaged session ends.

func (*Session[D]) Append

func (s *Session[D]) Append(
	ctx context.Context,
	r io.Reader,
	partSize int64,
	flush FlushFunc,
) (int64, error)

Append reads r into Pending, calling flush whenever partSize bytes are buffered, and saves the session. Bytes read are hashed and counted even when flush or the read fails, so the session stays consistent and the client may continue from Size.

func (*Session[D]) Flush

func (s *Session[D]) Flush(ctx context.Context, flush FlushFunc) error

Flush hands the pending bytes to flush when there are any and drops them on success.

func (*Session[D]) Forget

func (s *Session[D]) Forget(ctx context.Context) error

Forget removes the saved session.

func (*Session[D]) Live

func (s *Session[D]) Live(ctx context.Context) error

Live reports, with an error wrapping blob.ErrUploadUnknown, a session that was committed or aborted since this handle was resumed.

func (*Session[D]) Save

func (s *Session[D]) Save(ctx context.Context) error

Save persists the session.

func (*Session[D]) Verify

func (s *Session[D]) Verify(want blob.Digests) (blob.Digests, error)

Verify compares every non-empty field of want, with SHA256 defaulting to the declared digest, against the digests of everything appended, and returns those digests. A mismatch wraps blob.ErrDigestMismatch; the caller aborts the session.

type Store

type Store interface {
	Put(ctx context.Context, id string, state []byte, size int64) error
	Get(ctx context.Context, id string) ([]byte, error)
	Delete(ctx context.Context, id string) error
}

Store keeps encoded sessions between requests. Get returns an error wrapping blob.ErrUploadUnknown for an unknown ID; Delete of a missing session succeeds.

func InUploads

func InUploads(us UploadStore, backend string) Store

InUploads keeps sessions in the metadata store's uploads table (decision 0004) under backend, which names the blob store instance, for example "s3:hot" or "gcs:archive". A session stored under another backend name is unknown here, so the stale-upload reaper always knows which store aborts a session.

type Upload

type Upload struct {
	ID      string
	Backend string
	RepoKey string
	Path    string
	State   []byte
	Bytes   int64
	Started time.Time
	Updated time.Time
}

Upload is a blob upload session that outlives one request: a Docker chunked upload, the CLI's multipart upload, or an S3 multipart session. State is the blob backend's own session record, opaque here.

type UploadStore

type UploadStore interface {
	// Put creates or replaces the session with u.ID.
	Put(ctx context.Context, u Upload) error
	// Get returns the session.
	Get(ctx context.Context, id string) (Upload, error)
	// Delete removes the session.
	Delete(ctx context.Context, id string) error
	// Stale lists sessions last updated (or started, when never updated)
	// before t, oldest first, at most limit (kernel.DefaultLimit when
	// zero), for the reaper.
	Stale(ctx context.Context, before time.Time, limit int) ([]Upload, error)
}

UploadStore persists upload sessions.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL