Help us shape the future of Go! Take the 2026 Go Developer Survey

files

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 6, 2026 License: AGPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package files owns the record of a stored file: who owns it, what it is called, what it is for, how large it is, and when it stops being readable.

The bytes live in internal/blob under an opaque key. That key lives in this record and nowhere else, so the only way to reach a file is through a record this package agrees to hand out.

Index

Constants

View Source 
const (
	// StorageSchemaVersion identifies the only file record schema.
	StorageSchemaVersion = 5
	// StoragePrefix is the file record v1 namespace.
	StoragePrefix = "files:v1:account:"
)
View Source 
const DefaultPendingGrace = time.Hour

DefaultPendingGrace is how long a pending record may stay pending before a sweep treats it as abandoned.

It is longer than any upload this gateway accepts, because a sweep that ran while an upload was still streaming would delete the bytes out from under it.

View Source 
const DefaultRetention = 30 * 24 * time.Hour

DefaultRetention is how long a stored file stays readable when an upload names no shorter window.

Every file expires. OpenAI keeps an upload until a caller deletes it, and Starport does not, because storage that only grows is an unbounded cost and an unbounded liability. An operator raises or lowers the window, and an upload shortens it, but no upload escapes it.

View Source 
const MaxFilenameLength = 255

MaxFilenameLength bounds the name a caller may attach to an upload. The name is a label a person reads back, not a path anything opens.

View Source 
const MinRetention = time.Hour

MinRetention is the shortest window an upload may ask for. A window under an hour would expire a file while the request that stored it is still running.

Variables

View Source 
var (
	// ErrInvalidFile reports a record that cannot be stored as given.
	ErrInvalidFile = errors.New("files: invalid file")
	// ErrInvalidPurpose reports a purpose this gateway does not serve.
	ErrInvalidPurpose = errors.New("files: unsupported purpose")
)
View Source 
var (
	// ErrOutputIncomplete reports output without complete retained bytes.
	ErrOutputIncomplete = errors.New("files: output is incomplete")
	// ErrOutputExpired refuses publication beyond the original retention deadline.
	ErrOutputExpired = errors.New("files: output retention expired")
	// ErrOutputConflict preserves the retained output identity and digest.
	ErrOutputConflict = errors.New("files: output differs from retained evidence")
)
View Source 
var (
	// ErrRepositoryRequired reports an absent file record storage adapter.
	ErrRepositoryRequired = errors.New("files: record storage is required")
	// ErrFileNotFound reports a file this account cannot see.
	//
	// A file another account owns produces this error rather than a refusal. A
	// refusal would confirm that the identifier exists, and an identifier is
	// the only thing a caller needs to guess.
	ErrFileNotFound = errors.New("files: file not found")
	// ErrFileExists reports an identifier already in use.
	ErrFileExists = errors.New("files: file already exists")
	// ErrCorruptRecord reports durable file data this package cannot read.
	ErrCorruptRecord = errors.New("files: file record is invalid")
)
View Source 
var (
	// ErrServiceRequired reports a service built without a record store or a
	// byte store.
	ErrServiceRequired = errors.New("files: a record store and a byte store are required")

	// ErrRetentionTooLong reports an upload asking to outlive the window this
	// deployment set. An upload shortens the window and never extends it.
	ErrRetentionTooLong = errors.New("files: the requested retention exceeds the deployment window")

	// ErrRetentionTooShort reports an upload asking for a window under
	// MinRetention.
	ErrRetentionTooShort = errors.New("files: the requested retention is shorter than one hour")
)

Functions

func PrepareRecoveryReplay added in v1.3.0 

func PrepareRecoveryReplay(ctx context.Context, source RecoveryReader, assets blob.PublicationReader, at time.Time, changes []RecoveryChange) ([]storage.CompareAndSwapMutation, error)

PrepareRecoveryReplay preserves retained identity, expiry, and cleanup history. The coordinator combines file and byte-account owner mutations under the import barrier. Deletion requires durable retirement. Missing bytes do not prove retirement.

Types

type File 

type File struct {
	ID        string
	Account   string
	Filename  string
	Purpose   Purpose
	Bytes     int64
	State     FileState
	CreatedAt time.Time
	ExpiresAt time.Time
	// contains filtered or unexported fields
}

File is one stored file.

Every field a caller may read is exported. The blob key is not one of them. It stays unexported, so no encoder, no template, and no response body can carry it out of this package by accident.

func ReadRecoveryFile added in v1.3.0 

func ReadRecoveryFile(ctx context.Context, records RecoveryRecordReader, account, id string) (File, error)

ReadRecoveryFile checks the account and canonical identity of a referenced file. It does not make expired or pending files readable to callers.

func VerifyRecoveryRecord added in v1.3.0 

func VerifyRecoveryRecord(ctx context.Context, key string, data []byte, source blob.PublicationReader, capturedAt time.Time) (File, error)

VerifyRecoveryRecord checks canonical identity and retained bytes at the capture time. Pending, deleting, and expired records can legitimately have no live bytes. It preserves those states and never publishes, deletes, or grants access.

func (File) Expired 

func (f File) Expired(now time.Time) bool

Expired reports whether the file stopped being readable at the given time. A zero expiry never expires.

func (File) RecoveryByteAttachment added in v1.3.0 

func (f File) RecoveryByteAttachment() storedbytes.RecoveryAttachment

RecoveryByteAttachment projects only the facts owned by the stored-byte meter.

func (File) RecoveryOutputMatches added in v1.3.0 

func (f File) RecoveryOutputMatches(size int64, digest string, ready bool) bool

RecoveryOutputMatches permits an interrupted digest bind only for unfinished output.

func (File) Validate 

func (f File) Validate() error

Validate reports whether the record can be stored.

func (File) VerifyRecoveryByteAttachment added in v1.3.0 

func (f File) VerifyRecoveryByteAttachment(ctx context.Context, source blob.RecoveryPublicationReader) (storedbytes.RecoveryAttachment, error)

VerifyRecoveryByteAttachment includes positive retirement evidence for a deleting file.

type FileState 

type FileState string

FileState is where one record sits between an upload and a deletion.

The states exist because a file is two writes, a record and its bytes, and a process can stop between them. A state tells a later sweep which of the two writes it has to finish or undo. 

const (
	// FileStatePending marks a record written ahead of its bytes. It is not
	// readable, and a sweep deletes it and the bytes it names.
	FileStatePending FileState = "pending"
	// FileStateReady marks a record whose bytes landed.
	FileStateReady FileState = "ready"
	// FileStateDeleting marks a record on its way out. It reads as not found,
	// and a sweep finishes the delete that left it.
	FileStateDeleting FileState = "deleting"
)

type Meter 

type Meter interface {
	InitializeEmpty(context.Context, string) error
	Total(context.Context, string) (int64, error)
	Reserve(ctx context.Context, holder, id string, size, bound int64) error
	Resize(context.Context, string, string, int64) error
	Attachment(context.Context, string, string) (storage.CompareAndSwapMutation, error)
	Release(ctx context.Context, holder, id string) error
	Abort(context.Context, string, string) error
	RecoverPending(context.Context) error
}

Meter bounds how many bytes one account keeps in storage at a time.

This package names the primitive rather than importing the limit vocabulary. A stored file knows its size and its owner, and nothing about who set the bound or where the number came from. The storage meter in internal/limits satisfies the contract.

type Option 

type Option func(*Service)

Option changes one service setting.

func WithClock 

func WithClock(now func() time.Time) Option

WithClock replaces the source of time. A test uses it to age a record without waiting.

func WithMeter 

func WithMeter(meter Meter) Option

WithMeter counts and bounds account storage. Production supplies a meter even when an account has no configured bound.

func WithPendingGrace 

func WithPendingGrace(grace time.Duration) Option

WithPendingGrace sets how long a pending record may stay pending.

func WithRetention 

func WithRetention(window time.Duration) Option

WithRetention sets the window every upload gets and no upload exceeds.

type Purpose 

type Purpose string

Purpose states what a caller intends to do with a stored file.

The set is deliberately small. OpenAI names several more, and each of the others belongs to a product Starport does not run: an assistant or a fine-tune. A gateway that accepted them would take an upload it can never use and bill storage for it. 

const (
	// PurposeUserData is a file a model reads as part of a request.
	PurposeUserData Purpose = "user_data"
	// PurposeVision is an image a model reads as part of a request.
	PurposeVision Purpose = "vision"
	// PurposeBatch is a JSONL input file a batch reads one line at a time.
	PurposeBatch Purpose = "batch"
	// PurposeBatchOutput is a JSONL result file a batch writes. A caller never
	// uploads one, so the upload route refuses it and only the batch runner
	// stores it.
	PurposeBatchOutput Purpose = "batch_output"
)

func Purposes 

func Purposes() []Purpose

Purposes lists every purpose this gateway accepts.

func UploadPurposes added in v1.2.0 

func UploadPurposes() []Purpose

UploadPurposes lists every purpose a caller may name on an upload. The batch output purpose is absent, because an upload claiming it would put caller bytes where a reader expects batch results.

func (Purpose) Valid 

func (p Purpose) Valid() bool

Valid reports whether this gateway serves the purpose.

type RecordPage added in v1.3.0 

type RecordPage struct {
	Records []File
	Next    string
	Scanned bool
}

RecordPage carries one native scan step. An empty continuation ends the pass. Scanned distinguishes a failed scan from a valid page with no readable records.

type RecoveryChange added in v1.3.0 

type RecoveryChange struct {
	Account        string        `json:"account"`
	ID             string        `json:"id"`
	ExpectedSHA256 string        `json:"expected_sha256"`
	After          *RecoveryFile `json:"after"`
}

RecoveryChange names an exact expected file and a later record or explicit deletion.

type RecoveryFile added in v1.3.0 

type RecoveryFile struct {
	// contains filtered or unexported fields
}

RecoveryFile retains private file identity, metering, output lineage, and expiry.

func CaptureRecoveryFile added in v1.3.0 

func CaptureRecoveryFile(ctx context.Context, source RecoveryReader, account, id string) (RecoveryFile, error)

CaptureRecoveryFile copies and validates the complete private durable record.

func (RecoveryFile) Format added in v1.3.0 

func (RecoveryFile) Format(s fmt.State, _ rune)

Format hides private recovery evidence for every formatting verb.

func (RecoveryFile) MarshalJSON added in v1.3.0 

func (r RecoveryFile) MarshalJSON() ([]byte, error)

MarshalJSON encodes private operator recovery evidence.

func (RecoveryFile) SHA256 added in v1.3.0 

func (r RecoveryFile) SHA256() string

SHA256 binds exact retained bytes, including the private durable fields.

func (*RecoveryFile) UnmarshalJSON added in v1.3.0 

func (r *RecoveryFile) UnmarshalJSON(data []byte) error

UnmarshalJSON rejects unsupported private record fields and schemas.

type RecoveryReader added in v1.3.0 

type RecoveryReader interface {
	ReadCaptured(context.Context, string, int) (storage.TransferRecord, error)
}

RecoveryReader supplies immutable complete owner bytes and original TTLs.

type RecoveryRecordReader added in v1.3.0 

type RecoveryRecordReader interface {
	GetBounded(context.Context, string, int) ([]byte, error)
}

RecoveryRecordReader supplies bounded reads from a captured record image.

type Repository 

type Repository interface {
	Create(context.Context, File) error
	CreateClaimed(context.Context, File, storage.CompareAndSwapMutation) error
	Get(context.Context, string, string) (File, error)
	List(context.Context, string, int) ([]File, error)
	Replace(context.Context, File) error
	Delete(context.Context, string, string) error
	Scan(context.Context, int) ([]File, error)
	Page(context.Context, string, string) (RecordPage, error)
}

Repository is the durable file record contract. It stores records and never bytes, which is why every method here is cheap and none of them streams.

func OpenRepository 

func OpenRepository(store storage.KVStore) (Repository, error)

OpenRepository returns a storage-backed file record repository.

type Service 

type Service struct {
	// contains filtered or unexported fields
}

Service writes a file as two writes and keeps them consistent.

The record comes first, then the bytes, then the commit. The order matters: a process that stops after the first write leaves a pending record that names its bytes, and the sweep can find and delete both. The opposite order would leave bytes that no record names, and the byte store lists no keys, so nothing could ever find them again.

func NewService 

func NewService(records Repository, blobs blob.Store, options ...Option) (*Service, error)

NewService builds a file service over a record store and a byte store.

func (*Service) CommitOutput added in v1.3.0 

func (s *Service) CommitOutput(ctx context.Context, account, id string, size int64, digest string, content io.Reader) (File, error)

CommitOutput pins the expected bytes before writing them. A retry can only write that same content. Failed writes retain metadata for recovery.

func (*Service) Delete 

func (s *Service) Delete(ctx context.Context, account, id string) error

Delete removes a file this account owns.

The record is marked first, then the bytes go, then the record goes. Marking first is what makes the delete resumable: a process that stops part way leaves a record in the deleting state, which reads as not found and which the next sweep finishes. A delete that removed the bytes without marking would leave a ready record over bytes that no longer exist.

func (*Service) ExposeOutput added in v1.3.0 

func (s *Service) ExposeOutput(ctx context.Context, account, id string) error

ExposeOutput verifies a completed aggregate before public file listing.

func (*Service) Get 

func (s *Service) Get(ctx context.Context, account, id string) (File, error)

Get returns one readable file. A pending record reads as not found, because a caller that could see it could also read bytes that never finished landing. So does a record on its way out, and so does an expired one.

Expiry is decided on the read rather than by the sweep. The sweep runs on an interval, and a file that answered for the length of that interval past its stated window would make the window a suggestion.

func (*Service) List 

func (s *Service) List(ctx context.Context, account string, limit int) ([]File, error)

List returns the readable files one account owns.

func (*Service) Open 

func (s *Service) Open(ctx context.Context, account, id string) (File, io.ReadCloser, error)

Open returns the record and a reader over its bytes.

func (*Service) PrepareOutput added in v1.3.0 

func (s *Service) PrepareOutput(ctx context.Context, account, identity, name string, bound int64) (File, error)

PrepareOutput retains one output identity before its producer starts. Repeated preparation preserves the original expiry and byte claim.

func (*Service) RecoverOutput added in v1.3.0 

func (s *Service) RecoverOutput(ctx context.Context, account, id string) (File, error)

RecoverOutput verifies retained bytes before completing interrupted publication. Missing bytes remain incomplete. It never asks a provider to generate them.

func (*Service) Retention 

func (s *Service) Retention() time.Duration

Retention reports the window this deployment gives a stored file.

func (*Service) Sweep 

func (s *Service) Sweep(ctx context.Context) (SweepResult, error)

Sweep reclaims the storage that nothing reads any more.

It handles three cases, and every one of them deletes the bytes before the record. An interrupted sweep therefore leaves a record naming bytes that may already be gone, which the next sweep finishes, rather than an object that no record names and that nothing can ever find again.

One failing record does not stop the pass. A sweep that returned on the first error would let one unreachable object hold every later one hostage, and the caller runs on a ticker that would repeat the same failure forever.

func (*Service) Upload 

func (s *Service) Upload(ctx context.Context, request UploadRequest, r io.Reader) (File, error)

Upload stores the bytes and returns the readable record.

type SweepResult 

type SweepResult struct {
	// Abandoned counts pending records older than the grace window.
	Abandoned int
	// Expired counts ready records that passed their retention window.
	Expired int
	// Resumed counts records an interrupted delete left in the deleting state.
	Resumed int
}

SweepResult counts what one sweep finished. An operator reads it to tell a quiet deployment from a sweep that never runs.

func (SweepResult) Total 

func (r SweepResult) Total() int

Total counts every record this sweep removed.

type UploadRequest 

type UploadRequest struct {
	Account  string
	Filename string
	Purpose  Purpose

	// Size is what the caller says the upload weighs, and the service claims
	// it against the bound before it writes a byte. A claim after the write
	// has already spent the storage it was supposed to protect.
	//
	// The service reconciles the claim against the real size once the write
	// lands, so a caller that understated the upload gains nothing.
	Size int64

	// StoredBytesBound is the total this account may keep. Zero leaves the
	// account unbounded, and the service still counts its bytes so a bound set
	// later reads a true number.
	StoredBytesBound int64

	// Retention shortens the window this file gets. A zero value takes the
	// window the deployment set. A longer one is refused rather than clamped,
	// because a caller that asked for a year and silently got a month would
	// find out when the file stopped reading.
	Retention time.Duration
}

UploadRequest names everything about a file except its bytes.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.