Documentation
¶
Overview ¶
Package artifacts is the control plane's half of the object store: the S3 client, the key layout, presigned URLs, and the service that records what has been stored.
Nodes never talk to S3. A node hands a file to NodeService.UploadArtifact and the server is what writes it — "nodes only ever talk to the server" is the invariant the whole networking design rests on, and a presigned PUT straight from a node would break it.
Index ¶
- Constants
- Variables
- func ArtifactKey(taskID, artifactID, name string) string
- func LogKey(taskID, stream string) string
- func SanitizeName(name string) string
- type Config
- type S3
- func (s *S3) Bucket() string
- func (s *S3) EnsureBucket(ctx context.Context) error
- func (s *S3) Get(ctx context.Context, key string) (io.ReadCloser, error)
- func (s *S3) PresignGet(ctx context.Context, key, filename string, ttl time.Duration) (*url.URL, error)
- func (s *S3) PresignPut(ctx context.Context, key, contentType string, maxSize int64, ttl time.Duration) (*url.URL, error)
- func (s *S3) Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) (int64, error)
- func (s *S3) Ready(ctx context.Context) error
- func (s *S3) Remove(ctx context.Context, key string) error
- type Service
- func (s *Service) Enabled() bool
- func (s *Service) EnsureBucket(ctx context.Context) error
- func (s *Service) Get(ctx context.Context, id string) (store.Artifact, error)
- func (s *Service) List(ctx context.Context, taskID string) ([]store.Artifact, error)
- func (s *Service) Open(ctx context.Context, a store.Artifact) (io.ReadCloser, error)
- func (s *Service) PresignGet(ctx context.Context, a store.Artifact) (*url.URL, time.Time, error)
- func (s *Service) Put(ctx context.Context, up Upload) (store.Artifact, error)
- func (s *Service) Ready(ctx context.Context) error
- type Upload
Constants ¶
const DefaultRegion = "us-east-1"
DefaultRegion is what an S3 endpoint that does not care is told.
const LogContentType = "text/plain+zstd"
LogContentType is what a rolled-up log stream is stored as.
const MaxArtifactBytes int64 = 512 << 20
MaxArtifactBytes is the largest single artifact Podium will store: 512 MB.
const PresignTTL = 15 * time.Minute
PresignTTL is how long a presigned URL is good for.
Variables ¶
var ErrDisabled = errors.New("artifacts: no object store is configured (set PODIUM_S3_ENDPOINT)")
ErrDisabled is returned by every call when no object store is configured. It is not a failure of the task that asked: a Podium with no PODIUM_S3_ENDPOINT is still a task runner, it just cannot keep files.
var ErrTooLarge = fmt.Errorf("artifacts: larger than the %d MB limit", MaxArtifactBytes>>20)
ErrTooLarge is returned for an artifact over MaxArtifactBytes.
Functions ¶
func ArtifactKey ¶
ArtifactKey is where one of a task's files lives: tasks/<task>/artifacts/<id>-<name>. The artifact ID makes the key unique even when two files share a name, and the sanitised name makes it readable in a bucket browser.
func LogKey ¶
LogKey is where a rolled-up log stream lives: tasks/<task>/logs/<stream>.log.zst. stream is "stdout", "stderr", or "sidecar-<name>".
func SanitizeName ¶
SanitizeName reduces a producer's name to [A-Za-z0-9._-], at most 128 characters. The original is kept in the database; this is only what reaches the object key.
A path separator becomes an underscore rather than a slash, so an auto-collected "shots/a.png" cannot climb out of its task's prefix, and a leading dot is dropped so a name can never be ".." or an invisible file.
Types ¶
type Config ¶
type Config struct {
// Endpoint is PODIUM_S3_ENDPOINT: host:port, or a full URL whose scheme picks TLS.
Endpoint string
// Bucket is PODIUM_S3_BUCKET. It is created on start if it does not exist.
Bucket string
// AccessKey is PODIUM_S3_ACCESS_KEY.
AccessKey string
// SecretKey is PODIUM_S3_SECRET_KEY. SENSITIVE: never log it.
SecretKey string
// Region is PODIUM_S3_REGION, default us-east-1. The bundled object store ignores it; a
// real S3 does not.
Region string
// UseSSL is derived from the endpoint's scheme, or PODIUM_S3_USE_SSL when the endpoint
// carries none.
UseSSL bool
}
Config is the PODIUM_S3_* environment. An empty Endpoint means artifacts are disabled: tasks still run, they just cannot produce files and their logs are never rolled up.
func ConfigFromEnv ¶
func ConfigFromEnv() Config
ConfigFromEnv reads the canonical PODIUM_S3_* variables.
type S3 ¶
type S3 struct {
// contains filtered or unexported fields
}
S3 is a bucket on an S3-compatible endpoint. It is safe for concurrent use.
func NewS3 ¶
NewS3 builds a client for cfg. It does not touch the network: the endpoint may be down when the server starts and that must not stop it, because a task does not need artifacts to run.
func (*S3) EnsureBucket ¶
EnsureBucket creates the bucket if it is missing. It is called at start and again by every readiness probe that finds the store unready, so an endpoint that comes up late heals without a restart.
func (*S3) PresignGet ¶
func (s *S3) PresignGet(ctx context.Context, key, filename string, ttl time.Duration) (*url.URL, error)
PresignGet mints a short-lived download URL. filename, when set, becomes the Content-Disposition the object store sends back, so a browser saves the artifact under the name the task gave it rather than under its key.
func (*S3) PresignPut ¶
func (s *S3) PresignPut(ctx context.Context, key, contentType string, maxSize int64, ttl time.Duration) (*url.URL, error)
PresignPut mints a short-lived upload URL bound to a content type.
Nothing in Podium uses it yet: the node upload path goes through the server (NodeService.UploadArtifact) precisely so that a node never needs a route to S3. It is the interface that direct-to-S3 upload would be built on, and it is what makes the signing path testable from both ends. maxSize is advisory — a presigned PUT cannot carry a size limit in the URL, only a POST policy can — so the caller must still refuse anything larger when it records the artifact.
func (*S3) Put ¶
func (s *S3) Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) (int64, error)
Put streams size bytes from r to key. A negative size streams until EOF, which costs a multipart upload; the upload paths all know their length.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service records artifacts: it writes the bytes to the object store and the row to Postgres, in that order, so a row always describes an object that exists.
func (*Service) EnsureBucket ¶
EnsureBucket creates the bucket if it is missing.
func (*Service) PresignGet ¶
PresignGet mints a download URL for an artifact and says when it stops working.
func (*Service) Put ¶
Put streams an upload into the object store and records it.
The bytes go first and the row second: a row that names a missing object is a download that fails for no visible reason, while an object with no row is invisible and costs only space. If the row cannot be written the object is removed again.
type Upload ¶
type Upload struct {
TaskID string
Kind string
Name string
ContentType string
// Size is what the producer expects to send, or 0 when it does not know. It is used
// for an early rejection only; what is counted is what arrives.
Size int64
Body io.Reader
}
Upload is one artifact on its way in.
Source Files
Directories