blob

package
v1.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 23, 2026 License: MIT Imports: 36 Imported by: 0

Documentation

Index

Constants

View Source
const (

	// MaxOriginalImageSizeBytes bounds the declared size of an ORIGINAL image upload.
	// It is pinned into the upload policy, so storage rejects anything larger before a
	// single byte lands.
	MaxOriginalImageSizeBytes = 8 * 1024 * 1024 // 8 MiB

)

Variables

View Source
var (
	// ErrImageCorrupt means the bytes could not be read or decoded as an image.
	ErrImageCorrupt = errors.New("image is corrupt or undecodable")

	// ErrImageUnsupportedType means the bytes are a kind the service does not
	// accept — an unsupported format, or an animated image.
	ErrImageUnsupportedType = errors.New("unsupported image type")

	// ErrImageTooLarge means the image's pixel dimensions exceed the limits.
	ErrImageTooLarge = errors.New("image exceeds dimension limits")

	// ErrImagePrivacyMetadata means the image still carries embedded
	// privacy-sensitive metadata that the client was required to strip.
	ErrImagePrivacyMetadata = errors.New("image carries privacy-sensitive metadata")
)

InspectImage failure categories. They are wrapped into the descriptive error it returns so finalization can classify a rejection into the right RejectionReason without re-deriving why the bytes were unacceptable. A failure carrying none of these is an internal processing fault.

View Source
var (
	// ErrBlobNotFound means the blob does not exist OR is not owned by the caller.
	// The two are deliberately indistinguishable: a BlobId is a bearer capability,
	// so confirming that someone else's id exists would leak its existence.
	ErrBlobNotFound = errors.New("blob not found")

	// ErrBlobNotReady means the blob's bytes are still being processed. This is
	// transient — the caller may retry once the blob reaches READY.
	ErrBlobNotReady = errors.New("blob not ready")

	// ErrBlobRejected means the blob failed validation or moderation. This is
	// terminal for that id, since the bytes behind it are immutable: to try again
	// the client must upload a new blob.
	ErrBlobRejected = errors.New("blob rejected")

	// ErrBlobInvalid means the blob is READY but unusable on this surface — it is a
	// server-derived rendition rather than an original, or it is not an image.
	ErrBlobInvalid = errors.New("blob invalid")
)

The granular reasons a blob cannot be attached to a surface. They exist because SetAsProfilePicture attaches a single blob and its caller must tell the client which of these happened — whether to retry (not ready) or upload again (rejected).

View Source
var (
	// ErrNotFound is returned when no blob exists for the given id.
	ErrNotFound = errors.New("blob not found")

	// ErrExists is returned when a blob with the given id already exists.
	ErrExists = errors.New("blob already exists")

	// ErrCannotAdvanceToRejected is returned by Store.Advance when StateRejected is
	// passed as the target. Rejection is terminal and carries metadata, so it is
	// reached only through Store.Reject, never Advance.
	ErrCannotAdvanceToRejected = errors.New("cannot advance to rejected state; use Reject")
)
View Source
var ErrBlobNotShareable = errors.New("blob not shareable")

ErrBlobNotShareable is returned by Integration.ShareIntoChat when a referenced blob cannot be attached to a chat — it does not exist, is not owned by the sharer, is not a READY original, or is not an image. When it is returned none of the blobs are granted.

It is deliberately coarse: a chat share is all-or-nothing over a batch, so there is no single blob whose specific failure could be reported. Surfaces that attach exactly one blob (SetAsProfilePicture) get the granular errors below instead, since they can act on the distinction.

View Source
var ErrInvalidGrant = errors.New("invalid grant")

ErrInvalidGrant is returned by AccessStore methods when a grant — or the key used to look one up — is not well-formed: a missing blob id, an unknown principal type, an empty principal id, or an unknown permission.

View Source
var ErrObjectNotFound = errors.New("object not found")

ErrObjectNotFound is returned by ObjectStorage.GetUploaded when no bytes have been uploaded for the key yet (the upload never happened or is incomplete).

View Source
var SupportedImageMimeTypes = func() map[string]bool {
	set := make(map[string]bool, len(imageFormatToMimeType))
	for _, mimeType := range imageFormatToMimeType {
		set[mimeType] = true
	}
	return set
}()

SupportedImageMimeTypes is the set of MIME types a client may declare for an upload. It is derived from the decodable formats, so the type pinned into the upload policy is always one the server can re-derive and validate from the stored bytes. It is the single source of truth for "is this an image we accept".

Functions

func IDString added in v1.20.0

func IDString(id *blobpb.BlobId) string

func MustGenerateID added in v1.20.0

func MustGenerateID() *blobpb.BlobId

func StorageKey

func StorageKey(id *blobpb.BlobId, mimeType string) (string, error)

StorageKey derives the object key for an image blob's bytes from its id and mime type. Images can have server-derived renditions (display, thumbnail, ...), so an image's bytes live under a per-media-item directory keyed by its id, leaving room to group its renditions under the same prefix:

images/<uuid>/original.jpg

Only images are supported today, so a non-image mime type is rejected outright rather than being silently forced into the image layout. Other kinds (videos, files) may want a different prefix and layout — keyed off the kind, not merely a resolvable extension — so adding one has to make a deliberate decision here. The extension is derived from the (immutable) mime type, and the same key is used in both the upload and origin stores.

Types

type AccessStore

type AccessStore interface {
	// Grant records that the grant's principal may exercise its permission on its
	// blob. It is idempotent: re-granting the same (blob, principal, permission)
	// is a no-op. It returns ErrInvalidGrant if the grant is not well-formed.
	Grant(ctx context.Context, g *Grant) error

	// HasGrant reports whether a grant exists for the exact (blob, principal,
	// permission) triple. A missing grant is (false, nil), not an error. It
	// returns ErrInvalidGrant if the lookup key is not well-formed.
	HasGrant(ctx context.Context, blobID *blobpb.BlobId, p Principal, perm Permission) (bool, error)

	// Revoke removes a grant. It is idempotent: revoking a grant that does not
	// exist is a no-op. It returns ErrInvalidGrant if the key is not well-formed.
	Revoke(ctx context.Context, blobID *blobpb.BlobId, p Principal, perm Permission) error
}

AccessStore persists blob ACL grants. A grant's existence authorizes its principal to exercise its permission on the blob; there is no other state.

The store resolves a grant by its exact (blob, principal, permission) key and performs no membership resolution, so it never depends on the chat — or any other principal — subsystem. Authorizing a concrete user against a non-user principal (e.g. checking chat membership for a PrincipalTypeChat grant) is the caller's responsibility.

type Blob

type Blob struct {
	ID *blobpb.BlobId

	// Rendition is which rendition of its media this blob holds. An ORIGINAL has
	// a nil ParentID; any other rendition type is a server-derived variant with
	// ParentID pointing at the original.
	Rendition RenditionType

	// ParentID is set when this blob is a server-derived rendition of another
	// blob; it points at the ORIGINAL the client uploaded. It is nil for an
	// ORIGINAL. Renditions are never uploaded by clients — the server derives
	// them from the original's bytes.
	ParentID *blobpb.BlobId

	Owner *commonpb.UserId

	// State is the blob's internal lifecycle state — the source of truth for how
	// far processing has progressed. The public blobpb.BlobStatus is derived from
	// it via State.ToBlobStatus.
	State State

	// StorageKey is the object key the bytes live under in the backing store. It
	// is derived from the ID and never leaves the server.
	StorageKey string

	// MimeType is the declared MIME type, pinned at reservation and immutable.
	MimeType string

	// SizeBytes is the declared size, pinned at reservation and immutable.
	SizeBytes uint64

	// Image is the derived IMAGE metadata, set only when this blob is an image
	// and READY. It is the image variant of the blob's kind-specific metadata;
	// as additional content kinds are supported they will be carried by their
	// own sibling fields here (e.g. Video, Audio), one per blobpb.BlobMetadata
	// kind variant. Only images exist today.
	Image *ImageMetadata

	// Renditions is the manifest of derived renditions, populated ONLY on an
	// ORIGINAL and only once its renditions have been generated. Each entry is a
	// compact, immutable copy of a child rendition blob's servable metadata,
	// denormalized onto the parent so a media's whole rendition set resolves in the
	// single read that fetches the original — no per-original index query. The child
	// rendition blobs remain the canonical, independently-addressable records (a
	// rendition id resolves through GetBlobs and inherits the parent's ACL); this is
	// purely a read-path manifest. It is nil on a rendition blob itself and on an
	// original whose ladder produced nothing.
	Renditions []RenditionRef

	// Rejection records why this blob was rejected, set only when State is
	// StateRejected; it is nil for any non-rejected blob.
	Rejection *RejectionMetadata
}

Blob is the server-authoritative record for a stored blob. It is the durable identity behind a BlobId and tracks the blob through its lifecycle.

The MimeType and SizeBytes are declared by the client on reservation and pinned into the signed upload policy, so storage rejects any upload that does not match them. They are immutable for the life of the blob: finalization re-validates the stored bytes against them and REJECTs the blob on any mismatch rather than overwriting them. Only the derived kind-specific metadata is filled in at finalization.

func (*Blob) Clone

func (b *Blob) Clone() *Blob

Clone returns a deep copy of the blob, so stores can hand out values callers cannot mutate in place.

func (*Blob) ContentKind added in v1.22.0

func (b *Blob) ContentKind() ContentKind

ContentKind is the blob's processing family, derived from its pinned MIME type.

type ChatResolver

type ChatResolver struct {
	// contains filtered or unexported fields
}

ChatResolver is the PrincipalResolver for chat-scoped grants: a user is covered by a PrincipalTypeChat principal iff they are a member of the chat the principal identifies. It resolves membership directly against the chat store.

func (*ChatResolver) Covers

func (r *ChatResolver) Covers(ctx context.Context, principal Principal, user *commonpb.UserId) (bool, error)

Covers reports whether user is covered by principal. A PrincipalTypeChat principal is covered iff user is a member of the chat identified by the principal id. Any other principal type is outside this resolver's scope — it is chat-only — so it reports not-covered rather than guessing; supporting another scope is a matter of layering a different resolver, not changing this one.

type CompositeResolver

type CompositeResolver struct {
	// contains filtered or unexported fields
}

CompositeResolver is the top-level PrincipalResolver: it routes each principal to the domain resolver registered for its type. The server holds one of these, so adding an access surface is a matter of registering its resolver here — the read path does not change. A principal whose type has no registered resolver is not covered, mirroring how an unknown access scope is treated.

func (*CompositeResolver) Covers

func (r *CompositeResolver) Covers(ctx context.Context, principal Principal, user *commonpb.UserId) (bool, error)

Covers routes principal to the resolver registered for its type and returns that resolver's decision. A principal whose type has no registered resolver is not covered (false, nil) — an unroutable scope authorizes nothing.

type ContentKind added in v1.22.0

type ContentKind int

ContentKind identifies which processing family a blob's bytes belong to: which validation, moderation, and rendition pipeline they go through, and which finalization queue they wait in. It is derived from the blob's pinned MIME type, never stored on its own. Images are the only kind supported today; video, audio, etc. each become their own kind — with their own queue and worker tuning — as they are added.

The values are persisted (in finalization queue partition keys), so they must be stable forever.

const (
	ContentKindUnknown ContentKind = iota

	// ContentKindImage is a still image.
	ContentKindImage
)

func ContentKindForMimeType added in v1.22.0

func ContentKindForMimeType(mimeType string) ContentKind

ContentKindForMimeType maps a declared MIME type to its processing family. An unsupported type maps to ContentKindUnknown, which nothing may be queued under.

func (ContentKind) String added in v1.22.0

func (k ContentKind) String() string

String names the kind for logs and metric dimensions.

type FinalizationQueueStats added in v1.22.0

type FinalizationQueueStats struct {
	// Depth is how many blobs are queued, due or not — a delayed retry is
	// still backlog.
	Depth uint64

	// OldestEnqueuedAt is when the longest-queued blob was FIRST enqueued; its
	// distance from now is the queue's max age. Re-marks and retry delays never
	// reset it, so it exposes a blob stuck cycling through backoff — which the
	// depth alone hides. It is the zero time when the queue is empty.
	OldestEnqueuedAt time.Time
}

FinalizationQueueStats is a point-in-time gauge of one content kind's finalization queue.

type FinalizationTask added in v1.22.0

type FinalizationTask struct {
	ID *blobpb.BlobId

	// Attempts is the number of failed finalization attempts recorded so far
	// (via DelayFinalization). The worker uses it to pace backoff and to stop
	// retrying an unfinalizable blob.
	Attempts uint32

	// NextAttemptAt is when the task next becomes due.
	NextAttemptAt time.Time
}

FinalizationTask is a queued unit of finalization work: a blob whose uploaded bytes are awaiting processing, with the retry bookkeeping the worker schedules off of.

type Finalizer added in v1.22.0

type Finalizer struct {
	// contains filtered or unexported fields
}

Finalizer drives an uploaded blob through the processing pipeline to a terminal state: confirm the upload landed, validate + derive metadata + moderate, copy into the origin store, generate renditions, and clean up. It is the single owner of that pipeline — the background worker runs it off the finalization queue (see Store.MarkForFinalization), while the RPCs only queue the work.

func NewFinalizer added in v1.22.0

func NewFinalizer(
	log *zap.Logger,
	blobs Store,
	storage ObjectStorage,
	moderator moderation.Client,
) *Finalizer

NewFinalizer returns a Finalizer over the given blob metadata store, object storage, and (optional) moderation client.

func (*Finalizer) Fail added in v1.22.0

func (f *Finalizer) Fail(ctx context.Context, id *blobpb.BlobId) error

Fail terminally rejects a blob whose finalization attempts are exhausted, so the client sees a definitive (internal) rejection instead of an eternal PROCESSING. It is idempotent: a blob that reached a terminal state first keeps that state, and one already reclaimed (TTL) is a no-op.

func (*Finalizer) Finalize added in v1.22.0

func (f *Finalizer) Finalize(ctx context.Context, record *Blob) (blobpb.BlobStatus, error)

Finalize drives a blob through its processing pipeline, resuming from whatever state it is already in: confirm the upload landed, validate + derive metadata + moderate, copy into the origin store, and clean up. Each step checkpoints its completed state, so a replay (a worker retry, or a concurrent worker) skips the steps already done — notably re-moderation and the copy. It is idempotent and safe to run concurrently for the same blob (the store's forward-only transitions resolve races), and returns the blob's resulting public status. A returned error means the pipeline stopped before a terminal state and the attempt should be retried.

type Grant

type Grant struct {
	BlobID     *blobpb.BlobId
	Principal  Principal
	Permission Permission
}

Grant authorizes a Principal to exercise a Permission on a blob. It is the ACL entry: its existence is the authorization and it carries no other state. A blob's Owner holds every permission implicitly and needs no grant. Grants are made against the ORIGINAL blob id; server-derived renditions inherit their original's grants.

func (*Grant) Validate

func (g *Grant) Validate() error

Validate reports whether the grant is well-formed. Stores call it before persisting or looking up a grant so a malformed principal or permission can never be written or silently miss.

type ImageInspection

type ImageInspection struct {
	MimeType string
	Metadata *ImageMetadata

	// Decoded is the decoded image, retained so callers can derive other
	// renderings (e.g. the moderation payload) without decoding the bytes again.
	Decoded image.Image
}

ImageInspection is the result of decoding image bytes: the MIME type authoritatively derived from the bytes plus the intrinsic image metadata.

func InspectImage

func InspectImage(data []byte) (*ImageInspection, error)

InspectImage decodes the bytes as an image and derives their authoritative MIME type, pixel dimensions, and BlurHash. It returns an error if the bytes are not a decodable image of a supported format, if the image's pixel count exceeds maxImagePixels, or if it still carries privacy-sensitive metadata; callers treat any of these as a rejection.

type ImageMetadata

type ImageMetadata struct {
	Width    uint32
	Height   uint32
	Blurhash string
	HasAlpha bool
}

ImageMetadata holds the server-derived, intrinsic descriptors of a still image. Every field is derived once from the stored bytes and is immutable.

This is the IMAGE variant of a blob's kind-specific metadata. It is populated only for blobs whose bytes are an image; other content kinds (video, audio, ...) will each carry their own distinct metadata type, mirroring the blobpb.BlobMetadata.kind oneof. Images are simply the only kind supported today.

type Integration added in v1.20.0

type Integration struct {
	// contains filtered or unexported fields
}

Integration is the surface other domains (messaging and profile today) use to attach blobs to a resource they own: it validates and grants read access when the blob is attached (ShareIntoChat, SetAsProfilePicture), and resolves the blobs' metadata on read (Resolve).

func NewIntegration added in v1.20.0

func NewIntegration(blobs Store, storage ObjectStorage, access AccessStore) *Integration

NewIntegration returns an Integration backed by the given blob metadata store, object storage, and ACL store.

func (*Integration) ResolveRenditions added in v1.22.0

func (i *Integration) ResolveRenditions(ctx context.Context, ids []*blobpb.BlobId) (map[string][]*blobpb.Rendition, error)

ResolveRenditions returns the full, hydrated rendition set for each READY original among ids, keyed by string(BlobId.Value): the ORIGINAL first, then every derived rendition recorded in the original's manifest, each carrying a freshly minted, short-lived download URL alongside its mime type, size, and image descriptors. Unknown or not-yet-READY ids are omitted; an empty input yields a nil map.

It reads only the originals — the whole rendition set is denormalized onto each original's record as a manifest — so resolving a page of media costs a single batched store read rather than a per-original index query. Each rendition's wire metadata is minted through the same buildMetadata path as an original's, from the manifest entry, without a second read of the child rendition records.

It performs NO authorization. Callers must only pass original ids they have already established the reader may see — e.g. ids drawn from a chat message the reader is a member of, which were granted to the chat when the message was sent.

func (*Integration) SetAsProfilePicture added in v1.22.0

func (i *Integration) SetAsProfilePicture(ctx context.Context, ownerID *commonpb.UserId, blobID *blobpb.BlobId) error

SetAsProfilePicture attaches a blob to ownerID's public profile: it verifies that ownerID owns the blob and that it is a READY image original, then grants the profile read access to it. It is idempotent, so re-setting the same picture re-grants harmlessly.

Granting the profile — rather than each viewer — is what makes a profile picture public: every caller is covered by the profile principal (see ProfileResolver), so exactly the blobs granted to it are readable through it. Grants are never revoked, so a picture stays readable through the profile once set.

The ownership check matters for the same reason it does on a chat share: a BlobId is a bearer capability, so without it a user could publish a blob they merely learned the id of — including one that was never moderated for them.

It returns one of ErrBlobNotFound, ErrBlobNotReady, ErrBlobRejected, or ErrBlobInvalid when the blob cannot back a picture. Nothing is granted then.

func (*Integration) ShareIntoChat added in v1.20.0

func (i *Integration) ShareIntoChat(ctx context.Context, sharerID *commonpb.UserId, chatID *commonpb.ChatId, blobIDs []*blobpb.BlobId) error

ShareIntoChat attaches blobs to a chat: it verifies that sharerID owns every blob in blobIDs and that each is a READY image original, then grants the chat read access to each. It is all-or-nothing — if any blob fails validation nothing is granted and ErrBlobNotShareable is returned — and idempotent, so a re-sent message re-grants harmlessly. An empty blobIDs is a no-op.

Only the owner may introduce a blob into a chat: a BlobId is a bearer capability, so without the ownership check a member could attach a blob they merely learned the id of. Only a READY original is servable and grantable (renditions inherit their original's grants), so a pending, rejected, or rendition blob is rejected; and chat media is images only today, so a non-image blob is rejected too.

type ObjectStorage

type ObjectStorage interface {
	// PresignUpload mints a short-lived, presigned target the client uploads the
	// bytes to directly, into the UPLOAD store under the given key. The target
	// pins the declared content type and exact size, so storage rejects any
	// upload that does not match.
	PresignUpload(ctx context.Context, key, mimeType string, sizeBytes uint64) (*blobpb.UploadTarget, error)

	// GetUploaded returns the bytes a client uploaded under the key in the UPLOAD
	// store, or ErrObjectNotFound if no bytes are present yet. Finalization reads
	// them back to validate the content before promoting it.
	GetUploaded(ctx context.Context, key string) ([]byte, error)

	// UploadExists reports whether bytes are present under the key in the UPLOAD
	// store, without fetching them. Completion uses it to cheaply distinguish an
	// upload that landed from one that never happened before queueing the blob
	// for finalization.
	UploadExists(ctx context.Context, key string) (bool, error)

	// CopyToOrigin copies a validated object from the UPLOAD store to the ORIGIN
	// store under the same key, making it servable through the CDN. It overwrites
	// any object already served under the key, so it is safe to call more than
	// once. The upload copy is intentionally left in place — the caller removes it
	// with DeleteUpload only after the blob's terminal state is durably recorded,
	// so an interrupted finalization is always replayable from the upload bytes.
	CopyToOrigin(ctx context.Context, key string) error

	// PutOrigin writes server-derived bytes directly into the ORIGIN store under
	// the given key, pinning the content type so the CDN serves them correctly. It
	// is how derived renditions land: unlike a client upload, these bytes were
	// produced by the server from an original that already passed inspection and
	// moderation, so they are trusted and bypass the upload quarantine entirely
	// rather than being staged and copied. It overwrites any object already under
	// the key, so a replayed finalization that regenerates a rendition is
	// idempotent.
	PutOrigin(ctx context.Context, key, mimeType string, data []byte) error

	// DeleteUpload removes an object from the UPLOAD store. It is best-effort
	// cleanup run after a blob reaches a terminal state, and is idempotent:
	// deleting an absent key is not an error.
	DeleteUpload(ctx context.Context, key string) error

	// SignDownloadURL mints a fresh, short-lived CDN URL for fetching a promoted
	// object's bytes from the ORIGIN store, paired with the instant it expires. It
	// is authorized at mint time and expires on its own, so callers mint a new one
	// rather than persisting it.
	SignDownloadURL(ctx context.Context, key string) (*blobpb.DownloadUrl, error)
}

ObjectStorage is the bytes backend behind blobs. It spans two object stores: an UPLOAD store clients write to directly, and an ORIGIN store — fronted by a CDN — that only validated bytes are promoted into. Quarantining untrusted uploads this way means nothing is ever served until the server has read it back, validated it, and promoted it.

The server never proxies blob bytes: it presigns upload targets, reads uploaded bytes back to validate them, promotes the good ones, and mints signed CDN URLs for serving. It is deliberately provider-agnostic. The production implementation is two S3 buckets (presigned PUT into the upload bucket, GetObject to read it back, a server-side copy into the origin bucket) with a CloudFront CDN in front of the origin bucket; an in-memory implementation backs tests.

type Permission

type Permission int

Permission is what a grant authorizes a principal to do with a blob. READ is the only permission today; WRITE, DELETE, and SHARE can be added without changing the store. Persisted by value, so the numbering is significant.

const (
	PermissionUnknown Permission = iota

	// PermissionRead authorizes resolving the blob and minting a download URL for
	// it — the access GetBlobs grants.
	PermissionRead
)

type Principal

type Principal struct {
	Type PrincipalType
	ID   []byte
}

Principal is the typed subject a grant is made to. ID is the type-specific identifier bytes: a user id for PrincipalTypeUser, a chat id for PrincipalTypeChat, and the profile owner's user id for PrincipalTypeProfile.

func PrincipalForChat

func PrincipalForChat(chatID *commonpb.ChatId) Principal

PrincipalForChat returns the principal for the members of a chat.

func PrincipalForProfile added in v1.22.0

func PrincipalForProfile(userID *commonpb.UserId) Principal

PrincipalForProfile returns the principal for the public profile of a user.

func PrincipalForUser

func PrincipalForUser(userID *commonpb.UserId) Principal

PrincipalForUser returns the principal for a single user.

type PrincipalResolver

type PrincipalResolver interface {
	Covers(ctx context.Context, principal Principal, user *commonpb.UserId) (bool, error)
}

PrincipalResolver reports whether a concrete user is covered by a principal — that is, whether a grant made to that principal authorizes the user. A PrincipalTypeUser principal is covered by identity; a group principal such as PrincipalTypeChat is covered by live membership.

It is the single extension point the read path consults to resolve a grant's principal, so supporting a new access surface is a matter of adding a PrincipalType and teaching the resolver about it — the server does not change. Defining it here, rather than depending on the chat (or any future scope) subsystem, keeps blob decoupled from what backs each principal type; the wiring supplies a resolver that knows how to resolve them (e.g. mapping a PrincipalTypeChat to chat membership).

func NewChatResolver

func NewChatResolver(chats chat.Store) PrincipalResolver

NewChatResolver returns a ChatResolver backed by the given chat store.

func NewCompositeResolver

func NewCompositeResolver(byType map[PrincipalType]PrincipalResolver) PrincipalResolver

NewCompositeResolver returns a CompositeResolver that dispatches a principal to the resolver registered for its PrincipalType. The routing table is copied, so later mutation of the passed map does not affect the resolver.

func NewProfileResolver added in v1.22.0

func NewProfileResolver() PrincipalResolver

NewProfileResolver returns a ProfileResolver.

type PrincipalType

type PrincipalType int

PrincipalType identifies the kind of subject a grant is made to. It is the extension point for "where a blob can be accessed from": each access surface (a chat today; a feed, a profile, a public link later) is a principal type with its own server-side membership check. The constants are persisted by value, so their numbering is significant and must not be reordered.

const (
	PrincipalTypeUnknown PrincipalType = iota

	// PrincipalTypeUser is a single user, identified by their user id. A user
	// principal is resolved by direct identity, with no membership lookup.
	PrincipalTypeUser

	// PrincipalTypeChat is every current member of a chat, identified by the chat
	// id. A chat principal is resolved against live chat membership, so a single
	// grant covers the whole chat and stays correct as membership changes.
	PrincipalTypeChat

	// PrincipalTypeProfile is a user's public profile, identified by that user's
	// id. It covers every caller — a profile is public — so the grant alone gates
	// the read: exactly the blobs granted to the profile are readable through it.
	PrincipalTypeProfile
)

type ProfileResolver added in v1.22.0

type ProfileResolver struct{}

ProfileResolver is the PrincipalResolver for profile-scoped grants: a profile is public, so every caller is covered by a PrincipalTypeProfile principal — there is no membership to resolve, and it therefore needs no store.

Coverage being universal is precisely why the grant carries the whole decision here: only the blobs granted to a profile resolve through it, so a picture stops being readable the moment it is superseded and its grant revoked. Read authorization still requires both halves (see Server.canRead), so this is not a blanket authorization of any blob id.

func (*ProfileResolver) Covers added in v1.22.0

func (r *ProfileResolver) Covers(_ context.Context, principal Principal, _ *commonpb.UserId) (bool, error)

Covers reports whether user is covered by principal. Every user is covered by a PrincipalTypeProfile principal, since a profile is public. Any other principal type is outside this resolver's scope, so it reports not-covered rather than guessing.

type RejectionMetadata

type RejectionMetadata struct {
	Reason RejectionReason

	// FlaggedCategory is the moderation category that tripped, set only when
	// Reason is RejectionReasonModeration; it is NONE (the zero value) otherwise.
	FlaggedCategory moderationpb.FlaggedCategory
}

RejectionMetadata records why a blob was rejected during finalization. It is set only on a StateRejected blob and is immutable thereafter.

func (*RejectionMetadata) ToProto

ToProto renders the rejection metadata for the wire. A nil receiver renders to nil, so a non-rejected blob simply carries no rejection.

type RejectionReason

type RejectionReason int

RejectionReason is the internal mirror of blobpb.RejectionReason: why a blob's uploaded bytes failed finalization. It is set only on a StateRejected blob, is immutable thereafter, and is its own type (persisted as its own value) so the stored representation does not depend on the wire enum's numbering.

const (
	RejectionReasonUnknown RejectionReason = iota
	RejectionReasonModeration
	RejectionReasonUnsupportedType
	RejectionReasonMismatchedType
	RejectionReasonTooLarge
	RejectionReasonCorrupt
	RejectionReasonInternal
	RejectionReasonPrivacyMetadataPresent
)

func (RejectionReason) ToProto

ToProto maps the internal reason onto the public blobpb.RejectionReason.

type RenditionRef added in v1.22.0

type RenditionRef struct {
	// ID is the child rendition blob's id — the same id GetBlobs resolves.
	ID *blobpb.BlobId

	// Rendition is which rendition role these bytes serve (display, thumbnail).
	Rendition RenditionType

	// MimeType is the rendition's own encoded type, which may differ from the
	// original's (e.g. an opaque PNG original yields JPEG renditions).
	MimeType string

	// SizeBytes is the size of the encoded rendition bytes.
	SizeBytes uint64

	// StorageKey is the object key the rendition's bytes live under, used to sign
	// its download URL. It never leaves the server.
	StorageKey string

	// Image is the rendition's derived image metadata — its own dimensions, and the
	// BlurHash copied from the original. Reuses the same type the original carries.
	Image *ImageMetadata
}

RenditionRef is a single entry in an original's rendition manifest: the servable metadata of a derived rendition blob, enough to mint its wire Rendition (role, handle, image descriptors, and a freshly signed download URL) without reading the child blob record. Its fields mirror the same servable subset of a Blob — MimeType, SizeBytes, StorageKey, and the reused ImageMetadata — so a ref converts to the Blob that buildMetadata already understands (see asBlob). It never changes once written, because a rendition's bytes are immutable.

type RenditionType

type RenditionType int

RenditionType identifies which rendition of a piece of media a blob holds. The ORIGINAL is the exact bytes the client uploaded; every other type is a variant the server derives from that original. It is a server-internal concept: clients only ever upload and reference ORIGINALs, and the server derives and serves the rest.

const (
	RenditionUnknown RenditionType = iota

	// RenditionOriginal is the exact bytes the client uploaded.
	RenditionOriginal

	// RenditionDisplay is a server-derived variant sized and optimized for
	// inline display — e.g. rendering the image inline within a chat message or
	// feed — rather than serving the full-resolution original.
	RenditionDisplay

	// RenditionThumbnail is a small, server-derived preview image — e.g. a
	// grid/list thumbnail — smaller than the display rendition.
	RenditionThumbnail
)

func (RenditionType) ToProtoRole added in v1.22.0

func (r RenditionType) ToProtoRole() blobpb.Rendition_Role

ToProtoRole maps an internal RenditionType onto the wire Rendition.Role a hydrated rendition carries. The roles are kind-agnostic — a video, too, has an ORIGINAL, a downscaled DISPLAY, and a THUMBNAIL still.

type Server

type Server struct {
	blobpb.UnimplementedBlobStorageServer
	// contains filtered or unexported fields
}

func NewServer

func NewServer(
	log *zap.Logger,
	authz auth.Authorizer,
	accounts account.Store,
	blobs Store,
	storage ObjectStorage,
	access AccessStore,
	resolver PrincipalResolver,
	requireStaff bool,
) *Server

func (*Server) CompleteExternalUpload

CompleteExternalUpload confirms the client's upload landed and queues the blob for finalization. The processing pipeline itself — validation, moderation, promotion, rendition generation — runs on the background worker, so the RPC returns PROCESSING and the client observes the terminal status by polling GetBlobs (or re-completing). Completion is idempotent: once the blob is terminal it reports that committed status, with the rejection metadata when the blob was rejected.

func (*Server) GetBlobs

func (*Server) GetUploadPolicy

GetUploadPolicy returns the upload constraints in force for the caller. It is advisory and cacheable: a client uses it to validate and resize before reserving an upload, but InitiateExternalUpload remains authoritative. Access is gated identically to initiating an upload, so a caller who could not upload does not receive a policy.

type State

type State int

State is the blob's internal, fine-grained lifecycle state. It records how far processing has progressed so an interrupted finalize can resume from the last completed checkpoint instead of repeating expensive steps — re-reading the bytes, re-deriving metadata, re-moderating, re-copying. It is deliberately finer-grained than the public blobpb.BlobStatus, which it maps onto, so the proto enum stays a derived view and this state is the source of truth.

The success path advances strictly forward — Pending → Uploaded → Inspected → Promoted → GeneratingRenditions → Ready — with Rejected an alternative terminal. The ordering of the constants is significant: a blob is only ever advanced to a higher-ranked state.

const (
	// StatePending is a freshly reserved blob awaiting the client's upload.
	StatePending State = iota

	// StateUploaded means the client's upload is complete and the bytes are
	// present in the upload store. This is the signal a processing worker keys off
	// of to begin deriving metadata, moderating, and promoting the blob.
	StateUploaded

	// StateInspected means the uploaded bytes were validated against the declared
	// type/size, the metadata was derived, and moderation passed. The derived
	// metadata is persisted at this checkpoint, so resuming skips re-moderation.
	StateInspected

	// StatePromoted means the original's bytes were copied into the origin (CDN)
	// store. The blob is NOT client-ready yet — its renditions have not been
	// generated. Resuming skips the copy.
	StatePromoted

	// StateGeneratingRenditions means the original is in the origin store and the
	// server is deriving its renditions (display, thumbnail) from it. The blob is
	// still not client-ready, so clients do not see READY until this completes.
	StateGeneratingRenditions

	// StateReady means processing is complete — the renditions are generated and
	// the upload-store bytes have been cleaned up — so the blob is client-ready.
	// Terminal.
	StateReady

	// StateRejected means the bytes failed validation or moderation. Terminal.
	StateRejected
)

func (State) Terminal

func (s State) Terminal() bool

Terminal reports whether no further processing is possible from this state.

func (State) ToBlobStatus

func (s State) ToBlobStatus() blobpb.BlobStatus

ToBlobStatus maps the internal state onto the public lifecycle status. A blob is reported READY only once it is fully processed — its renditions generated — so a client never references it (e.g. in a message) before the renditions it will use exist.

type Store

type Store interface {
	// CreatePending inserts a freshly reserved blob in the PENDING state. The
	// blob may be an ORIGINAL (nil ParentID) or a server-created rendition of an
	// existing original (ParentID set).
	//
	// ErrExists is returned if a blob with the same id already exists.
	CreatePending(ctx context.Context, blob *Blob) error

	// GetByID returns the blob with the given id, or ErrNotFound. Ownership is
	// not enforced here; callers that act on behalf of the uploader (e.g.
	// finalization) compare Owner themselves.
	GetByID(ctx context.Context, id *blobpb.BlobId) (*Blob, error)

	// GetByIDs returns the blobs among the given ids that exist, in unspecified
	// order. A BlobId is an opaque capability, so resolution is not scoped to an
	// owner; ids that do not exist are simply omitted. A BlobId resolves to at
	// most one record, so duplicate ids collapse to a single result. An original's
	// rendition manifest (Blob.Renditions) is included.
	GetByIDs(ctx context.Context, ids []*blobpb.BlobId) ([]*Blob, error)

	// AttachRenditions records an original's rendition manifest onto its own
	// record, so the whole set resolves in the single read that fetches the
	// original. It overwrites any manifest already present, so a replayed
	// generation that recomputes the same set is idempotent. It is only ever
	// called on an ORIGINAL, and refs may be empty (an original whose ladder
	// produced nothing).
	//
	// ErrNotFound is returned if no blob exists for the given id.
	AttachRenditions(ctx context.Context, id *blobpb.BlobId, refs []RenditionRef) error

	// Advance moves a blob forward along the success path to a later lifecycle
	// state, persisting derived metadata when provided (image is set only on the
	// transition into StateInspected). It advances strictly forward and never out
	// of a terminal state, so a replayed or concurrent finalize is idempotent:
	// advancing to a state the blob is already at or past is a no-op. The declared
	// MimeType and SizeBytes are never changed. Reaching StateReady also removes
	// the blob from the finalization queue, atomically with the transition.
	//
	// StateRejected is not a valid target — rejection is terminal and carries
	// metadata, so it is reached only through Reject. Passing it returns
	// ErrCannotAdvanceToRejected.
	//
	// It reports whether this call actually performed the transition. A false
	// return with a nil error means the blob was already at or past the target
	// (or terminal) — a concurrent or replayed finalize lost the race — so the
	// caller can stop instead of applying further side effects on a stale view.
	//
	// ErrNotFound is returned if no blob exists for the given id.
	Advance(ctx context.Context, id *blobpb.BlobId, to State, image *ImageMetadata) (bool, error)

	// Reject moves a non-terminal blob to the terminal StateRejected, recording
	// why. Like Advance it transitions only out of a non-terminal state and is
	// idempotent: it reports whether it performed the transition, and a false with
	// a nil error means the blob was already terminal — a concurrent or replayed
	// finalize won the race, so the committed rejection (or readiness) stands and
	// must not be overwritten. Rejection also removes the blob from the
	// finalization queue, atomically with the transition.
	//
	// ErrNotFound is returned if no blob exists for the given id.
	Reject(ctx context.Context, id *blobpb.BlobId, rejection *RejectionMetadata) (bool, error)

	// MarkForFinalization queues a blob on its content kind's finalization
	// queue, due at nextAttemptAt. It is idempotent: re-marking an already-queued
	// blob resets its due time (and moves it if the kind changed, though a blob's
	// kind never legitimately changes) but preserves its failed-attempt count,
	// and marking a blob that already reached a terminal state is a no-op (the
	// work is done, so nothing is queued).
	//
	// ErrNotFound is returned if no blob exists for the given id.
	MarkForFinalization(ctx context.Context, id *blobpb.BlobId, kind ContentKind, nextAttemptAt time.Time) error

	// GetDueForFinalization returns up to limit blobs queued under kind whose due
	// time is at or before asOf, soonest first.
	GetDueForFinalization(ctx context.Context, kind ContentKind, asOf time.Time, limit int) ([]*FinalizationTask, error)

	// GetFinalizationQueueStats reports kind's queue depth and the enqueue time
	// of its longest-queued blob, in one walk of the queue. It backs the queue
	// gauges: depth trending up faster than workers drain it means the queue is
	// growing, and a max age climbing while depth stays flat means something is
	// stuck retrying rather than merely busy.
	GetFinalizationQueueStats(ctx context.Context, kind ContentKind) (*FinalizationQueueStats, error)

	// ClaimForFinalization pushes a queued blob's due time out to until, provided
	// it is still queued and due as of asOf. It reports whether the claim was
	// performed: false means the blob left the queue (it reached a terminal
	// state) or is no longer due (another worker claimed or delayed it first).
	//
	// A claim is an efficiency guard, not a lock — finalization is idempotent and
	// safe to run concurrently; claiming only keeps workers from duplicating
	// expensive work (e.g. moderation calls). A crashed claimant needs no
	// recovery: the claim expires when its until passes and the task simply
	// becomes due again.
	ClaimForFinalization(ctx context.Context, id *blobpb.BlobId, asOf, until time.Time) (bool, error)

	// DelayFinalization reschedules a queued blob after a failed attempt: its due
	// time moves to nextAttemptAt and its failed-attempt count increments. It is
	// a no-op on a blob that is no longer queued (a concurrent finalize drove it
	// terminal, dequeuing it).
	DelayFinalization(ctx context.Context, id *blobpb.BlobId, nextAttemptAt time.Time) error
}

Store persists server-authoritative blob metadata. The bytes themselves live in an ObjectStorage; this store only tracks the lifecycle and derived metadata keyed by BlobId.

It also carries the finalization queues: the durable record of which blobs have uploaded bytes awaiting processing, which the background workers drain (Mark/GetDue/Claim/Delay below). There is one queue per ContentKind — each kind's pipeline is drained by its own worker, tuned to that kind's cost — and the queue is bookkeeping ON the blob record, so reaching a terminal state removes a blob from its queue atomically. The store does not interpret the kind; callers queue a blob under the kind they derived from it.

type Worker added in v1.22.0

type Worker struct {
	// contains filtered or unexported fields
}

Worker drains one content kind's finalization queue: it polls the blob store for uploads of that kind awaiting processing and drives each through the finalization pipeline (validation, moderation, promotion, rendition generation).

Each ContentKind gets its own Worker, so a kind's tuning (concurrency, timeout, backoff) matches its pipeline's cost — an image resample and a video transcode should not share a knob — and a backlog in one kind never starves another.

It implements the OCP worker.Runtime interface, so the parent application registers it alongside its other background runtimes and controls the poll interval. It is safe to run on every server instance: workers claim tasks before processing them, and the pipeline itself is idempotent, so overlap costs duplicate work at worst, never a wrong state.

Construct with NewWorker and either run it via Start (the OCP runtime entry point) or drive single ticks with Process (tests).

func NewWorker added in v1.22.0

func NewWorker(log *zap.Logger, blobs Store, finalizer *Finalizer, kind ContentKind, opts ...WorkerOption) *Worker

NewWorker returns a Worker draining kind's finalization queue over the given blob store and finalizer.

func (*Worker) Process added in v1.22.0

func (w *Worker) Process(runtimeCtx context.Context) (int, error)

Process runs one worker tick: it pulls the due tasks and processes them with bounded concurrency, reporting how many this worker actually took on (claimed, or terminally failed for exhaustion). Zero means the due queue is drained — tasks other workers hold claims on are not counted.

func (*Worker) Start added in v1.22.0

func (w *Worker) Start(ctx context.Context, interval time.Duration) error

Start satisfies the OCP worker.Runtime interface: it polls the finalization queue every interval until ctx is cancelled, whose error it returns. The cadence is fixed-rate, not fixed-delay: each sleep is the interval minus the time the tick's processing took, so slow ticks do not stretch the gap between polls (a tick that overran the interval polls again immediately). A full batch also polls again immediately, so a burst drains at processing speed rather than one batch per interval. It also runs the queue gauges (depth, max age) for its kind alongside the processing loop.

type WorkerOption added in v1.22.0

type WorkerOption func(*Worker)

WorkerOption overrides one of the worker's tuning knobs.

func WithWorkerBackoff added in v1.22.0

func WithWorkerBackoff(base, maxDelay time.Duration) WorkerOption

WithWorkerBackoff overrides the retry backoff's base and maximum delay.

func WithWorkerBatchSize added in v1.22.0

func WithWorkerBatchSize(n int) WorkerOption

WithWorkerBatchSize overrides how many due tasks one tick pulls from the queue.

func WithWorkerClaimLease added in v1.22.0

func WithWorkerClaimLease(lease time.Duration) WorkerOption

WithWorkerClaimLease overrides how far a claim pushes a task's due time out.

func WithWorkerFinalizeTimeout added in v1.22.0

func WithWorkerFinalizeTimeout(timeout time.Duration) WorkerOption

WithWorkerFinalizeTimeout overrides the bound on a single finalization attempt.

func WithWorkerMaxAttempts added in v1.22.0

func WithWorkerMaxAttempts(n uint32) WorkerOption

WithWorkerMaxAttempts overrides how many failed attempts a blob gets before it is terminally rejected.

func WithWorkerMaxConcurrency added in v1.22.0

func WithWorkerMaxConcurrency(n int) WorkerOption

WithWorkerMaxConcurrency overrides how many blobs are processed at once.

Directories

Path Synopsis
Package s3 implements blob.ObjectStorage on top of Amazon S3 for storage and a CloudFront CDN for delivery.
Package s3 implements blob.ObjectStorage on top of Amazon S3 for storage and a CloudFront CDN for delivery.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL