Documentation
¶
Overview ¶
Package files owns the record of a stored file: who owns it, what it is called, what it is for, how large it is, and when it stops being readable.
The bytes live in internal/blob under an opaque key. That key lives in this record and nowhere else, so the only way to reach a file is through a record this package agrees to hand out.
Index ¶
- Constants
- Variables
- type File
- type FileState
- type Meter
- type Option
- type Purpose
- type Repository
- type Service
- func (s *Service) Delete(ctx context.Context, account, id string) error
- func (s *Service) Get(ctx context.Context, account, id string) (File, error)
- func (s *Service) List(ctx context.Context, account string, limit int) ([]File, error)
- func (s *Service) Open(ctx context.Context, account, id string) (File, io.ReadCloser, error)
- func (s *Service) Retention() time.Duration
- func (s *Service) Sweep(ctx context.Context) (SweepResult, error)
- func (s *Service) Upload(ctx context.Context, request UploadRequest, r io.Reader) (File, error)
- type SweepResult
- type UploadRequest
Constants ¶
const ( // StorageSchemaVersion identifies the only file record schema. StorageSchemaVersion = 1 // StoragePrefix is the file record v1 namespace. StoragePrefix = "files:v1:account:" )
const DefaultPendingGrace = time.Hour
DefaultPendingGrace is how long a pending record may stay pending before a sweep treats it as abandoned.
It is longer than any upload this gateway accepts, because a sweep that ran while an upload was still streaming would delete the bytes out from under it.
const DefaultRetention = 30 * 24 * time.Hour
DefaultRetention is how long a stored file stays readable when an upload names no shorter window.
Every file expires. OpenAI keeps an upload until a caller deletes it, and Starport does not, because storage that only grows is an unbounded cost and an unbounded liability. An operator raises or lowers the window, and an upload shortens it, but no upload escapes it.
const MaxFilenameLength = 255
MaxFilenameLength bounds the name a caller may attach to an upload. The name is a label a person reads back, not a path anything opens.
const MinRetention = time.Hour
MinRetention is the shortest window an upload may ask for. A window under an hour would expire a file while the request that stored it is still running.
Variables ¶
var ( // ErrInvalidFile reports a record that cannot be stored as given. ErrInvalidFile = errors.New("files: invalid file") // ErrInvalidPurpose reports a purpose this gateway does not serve. ErrInvalidPurpose = errors.New("files: unsupported purpose") )
var ( // ErrRepositoryRequired reports an absent file record storage adapter. ErrRepositoryRequired = errors.New("files: record storage is required") // ErrFileNotFound reports a file this account cannot see. // // A file another account owns produces this error rather than a refusal. A // refusal would confirm that the identifier exists, and an identifier is // the only thing a caller needs to guess. ErrFileNotFound = errors.New("files: file not found") // ErrFileExists reports an identifier already in use. ErrFileExists = errors.New("files: file already exists") // ErrCorruptRecord reports durable file data this package cannot read. ErrCorruptRecord = errors.New("files: file record is invalid") )
var ( // ErrServiceRequired reports a service built without a record store or a // byte store. ErrServiceRequired = errors.New("files: a record store and a byte store are required") // ErrRetentionTooLong reports an upload asking to outlive the window this // deployment set. An upload shortens the window and never extends it. ErrRetentionTooLong = errors.New("files: the requested retention exceeds the deployment window") // ErrRetentionTooShort reports an upload asking for a window under // MinRetention. ErrRetentionTooShort = errors.New("files: the requested retention is shorter than one hour") )
Functions ¶
This section is empty.
Types ¶
type File ¶
type File struct {
ID string
Account string
Filename string
Purpose Purpose
Bytes int64
State FileState
CreatedAt time.Time
ExpiresAt time.Time
// contains filtered or unexported fields
}
File is one stored file.
Every field a caller may read is exported. The blob key is not one of them. It stays unexported, so no encoder, no template, and no response body can carry it out of this package by accident.
type FileState ¶
type FileState string
FileState is where one record sits between an upload and a deletion.
The states exist because a file is two writes, a record and its bytes, and a process can stop between them. A state tells a later sweep which of the two writes it has to finish or undo.
const ( // FileStatePending marks a record written ahead of its bytes. It is not // readable, and a sweep deletes it and the bytes it names. FileStatePending FileState = "pending" // FileStateReady marks a record whose bytes landed. FileStateReady FileState = "ready" // FileStateDeleting marks a record on its way out. It reads as not found, // and a sweep finishes the delete that left it. FileStateDeleting FileState = "deleting" )
type Meter ¶
type Meter interface {
Reserve(ctx context.Context, holder string, size, bound int64) error
Release(ctx context.Context, holder string, size int64) error
}
Meter bounds how many bytes one account keeps in storage at a time.
This package names the primitive rather than importing the limit vocabulary. A stored file knows its size and its owner, and nothing about who set the bound or where the number came from. The storage meter in internal/limits satisfies the contract.
type Option ¶
type Option func(*Service)
Option changes one service setting.
func WithClock ¶
WithClock replaces the source of time. A test uses it to age a record without waiting.
func WithMeter ¶
WithMeter bounds the bytes each account keeps. Without one the service stores without counting, which is what a deployment that set no bound wants.
func WithPendingGrace ¶
WithPendingGrace sets how long a pending record may stay pending.
func WithRetention ¶
WithRetention sets the window every upload gets and no upload exceeds.
type Purpose ¶
type Purpose string
Purpose states what a caller intends to do with a stored file.
The set is deliberately small. OpenAI names several more, and each of the others belongs to a product Starport does not run: an assistant or a fine-tune. A gateway that accepted them would take an upload it can never use and bill storage for it.
const ( // PurposeUserData is a file a model reads as part of a request. PurposeUserData Purpose = "user_data" // PurposeVision is an image a model reads as part of a request. PurposeVision Purpose = "vision" // PurposeBatch is a JSONL input file a batch reads one line at a time. PurposeBatch Purpose = "batch" // PurposeBatchOutput is a JSONL result file a batch writes. A caller never // uploads one, so the upload route refuses it and only the batch runner // stores it. PurposeBatchOutput Purpose = "batch_output" )
func UploadPurposes ¶ added in v1.2.0
func UploadPurposes() []Purpose
UploadPurposes lists every purpose a caller may name on an upload. The batch output purpose is absent, because an upload claiming it would put caller bytes where a reader expects batch results.
type Repository ¶
type Repository interface {
Create(context.Context, File) error
Get(context.Context, string, string) (File, error)
List(context.Context, string, int) ([]File, error)
Replace(context.Context, File) error
Delete(context.Context, string, string) error
Scan(context.Context, int) ([]File, error)
}
Repository is the durable file record contract. It stores records and never bytes, which is why every method here is cheap and none of them streams.
func OpenRepository ¶
func OpenRepository(store storage.KVStore) (Repository, error)
OpenRepository returns a storage-backed file record repository.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service writes a file as two writes and keeps them consistent.
The record comes first, then the bytes, then the commit. The order matters: a process that stops after the first write leaves a pending record that names its bytes, and the sweep can find and delete both. The opposite order would leave bytes that no record names, and the byte store lists no keys, so nothing could ever find them again.
func NewService ¶
NewService builds a file service over a record store and a byte store.
func (*Service) Delete ¶
Delete removes a file this account owns.
The record is marked first, then the bytes go, then the record goes. Marking first is what makes the delete resumable: a process that stops part way leaves a record in the deleting state, which reads as not found and which the next sweep finishes. A delete that removed the bytes without marking would leave a ready record over bytes that no longer exist.
func (*Service) Get ¶
Get returns one readable file. A pending record reads as not found, because a caller that could see it could also read bytes that never finished landing. So does a record on its way out, and so does an expired one.
Expiry is decided on the read rather than by the sweep. The sweep runs on an interval, and a file that answered for the length of that interval past its stated window would make the window a suggestion.
func (*Service) Sweep ¶
func (s *Service) Sweep(ctx context.Context) (SweepResult, error)
Sweep reclaims the storage that nothing reads any more.
It handles three cases, and every one of them deletes the bytes before the record. An interrupted sweep therefore leaves a record naming bytes that may already be gone, which the next sweep finishes, rather than an object that no record names and that nothing can ever find again.
One failing record does not stop the pass. A sweep that returned on the first error would let one unreachable object hold every later one hostage, and the caller runs on a ticker that would repeat the same failure forever.
type SweepResult ¶
type SweepResult struct {
// Abandoned counts pending records older than the grace window.
Abandoned int
// Expired counts ready records that passed their retention window.
Expired int
// Resumed counts records an interrupted delete left in the deleting state.
Resumed int
}
SweepResult counts what one sweep finished. An operator reads it to tell a quiet deployment from a sweep that never runs.
func (SweepResult) Total ¶
func (r SweepResult) Total() int
Total counts every record this sweep removed.
type UploadRequest ¶
type UploadRequest struct {
Account string
Filename string
Purpose Purpose
// Size is what the caller says the upload weighs, and the service claims
// it against the bound before it writes a byte. A claim after the write
// has already spent the storage it was supposed to protect.
//
// The service reconciles the claim against the real size once the write
// lands, so a caller that understated the upload gains nothing.
Size int64
// StoredBytesBound is the total this account may keep. Zero leaves the
// account unbounded, and the service still counts its bytes so a bound set
// later reads a true number.
StoredBytesBound int64
// Retention shortens the window this file gets. A zero value takes the
// window the deployment set. A longer one is refused rather than clamped,
// because a caller that asked for a year and silently got a month would
// find out when the file stopped reading.
Retention time.Duration
}
UploadRequest names everything about a file except its bytes.
Source Files