Documentation
¶
Overview ¶
Package spool is ynf's side of the spool layout in ynr ADR-003 and ADR-004: the root's three folders (factory/ for ynf's own writer, runs/<run id>/ one per run, manifests/<run id>.json one per run), the manifest, the quota on a run's folder, ynr serve for the length of a factory job, and what is swept into the run capture (ADR-007, ADR-009, ADR-010, ADR-011).
Nothing here ever fails a step: telemetry is the factory's side effect, and a full or broken spool costs telemetry, never work (ynr ADR-006, rule 12).
Index ¶
- Constants
- Variables
- func Env(host string, inContainer bool) map[string]string
- func StripOTLP(environ []string) []string
- func UpstreamFromEnv(getenv func(string) string) string
- func ValidName(s string) bool
- type Collector
- type Detection
- type Manifest
- type Run
- type Serve
- type Spool
- func (s *Spool) Begin(m Manifest, imageUser bool) (*Run, error)
- func (s *Spool) BeginWith(m Manifest, imageUser, volume bool) (*Run, error)
- func (s *Spool) Capture(runID, dir string)
- func (s *Spool) CleanStale(capture string)
- func (s *Spool) Close(dir string)
- func (s *Spool) FactoryDir() string
- func (s *Spool) RunDir(id string) string
- func (s *Spool) Sweep(dir string) int
- func (s *Spool) WriteManifest(m Manifest) error
- type Volumes
Constants ¶
const ( FactoryDir = "factory" RunsDir = "runs" ManifestsDir = "manifests" )
The folders under the spool root.
const ( RunCaptureLimit = 32 << 20 JobCaptureLimit = 256 << 20 )
The most that is put into a run capture (ADR-010): per run, and per job for what is swept at its end. A spool file past the limit is left where it is and counted in the log.
const ContainerPath = "/run/ynr/spool"
ContainerPath is where a docker run sees its own folder, and nothing else of the spool.
const DefaultArchive = 30 * time.Second
DefaultArchive is how long ynr serve is given, at a job's end, to ship what is left.
const DefaultDrain = 10 * time.Second
DefaultDrain is how long a run's end waits for ynr serve to ship the run's closed files before they are captured and the run's volume is taken away.
const DefaultQuota = 64 << 20
DefaultQuota is the size a run's folder is held to when the configuration names none.
const MaxManifest = 64 << 10
MaxManifest is the largest manifest ynr reads (64 KiB).
const Ynr = "ynr"
Ynr is the ynr ynf starts and asks: whatever `ynr` resolves to on the PATH ynf was given, as a shell would find it. There is no other lookup; without ynr on PATH, ynf carries on.
Variables ¶
var ErrNoVolume = errors.New("no size-limited volume here")
ErrNoVolume means the host does not let ynf make a size-limited volume of its own: the quota watcher bounds the run's folder instead. It is expected, not a fault, in a job container without CAP_SYS_ADMIN, for an unprivileged user on Linux, and on a host with no implementation.
Functions ¶
func Env ¶
Env is the environment that points a run at its folder: where the run sees it, which is its own path except in a container.
func StripOTLP ¶
StripOTLP removes the operator's OTEL_EXPORTER_OTLP_* from an environment, for a run when the collector is on: its records go to its own folder, and the operator's endpoint is served by ynr serve's upstream instead (ynr ADR-004).
func UpstreamFromEnv ¶
UpstreamFromEnv is the endpoint the operator set for OpenTelemetry, which ynr serve ships to when the collector is on (ynr ADR-004): OTEL_EXPORTER_OTLP_ENDPOINT, else a signal's own endpoint with its /v1/<signal> path taken off, else YNR_UPSTREAM, which ynr reads itself. It is empty when there is none.
Types ¶
type Collector ¶
type Collector struct {
Enabled bool
// ID is the collector's identity: the runner pool or the host, never one job (ynr ADR-003).
ID string
// Instance is the job within the pool: data, not identity.
Instance string
// Upstream is the OTLP/HTTP endpoint ynr serve ships to. ynr needs one until it has an object
// store of its own.
Upstream string
// Archive is how long ynr serve has at a job's end to ship what is left; default 30s.
Archive time.Duration
}
Collector is the configuration's collector settings (ADR-009): whether ynf starts ynr serve for a factory job, and how. It is off by default, and nothing else turns it on: not finding ynr.
type Detection ¶
type Detection struct {
Found bool `json:"found"`
Version string `json:"version,omitempty"`
Build string `json:"build,omitempty"` // full or slim
Capabilities string `json:"capabilities,omitempty"`
// Serving is true when a ynr serve holds the spool root ynr reports.
Serving bool `json:"serving,omitempty"`
Detail string `json:"detail,omitempty"` // why it was not detected
}
Detection is what asking ynr found (ADR-012). Detecting it starts nothing: only the configuration starts ynr serve.
type Manifest ¶
type Manifest struct {
Run string `json:"run"`
Lane string `json:"lane"`
Harness string `json:"harness"`
Focus string `json:"focus"`
Item string `json:"item"`
Step string `json:"step"`
// UID is the user the run writes as, when that is not the owner of its folder: a ynh-built
// image's own user, or the inline run user. ynr then also accepts the files that user owns in
// the run's folder. It comes from the image or the executor's own setting, never from the run,
// and is omitted when the run writes as the folder's owner.
UID *uint32 `json:"uid,omitempty"`
}
Manifest is what ynf writes for each run, out of the run's reach (ynr ADR-003): ynr stamps the factory attributes on everything from the run's folder from it, so a run cannot claim another factory. It holds ids and names, never content.
type Run ¶
Run is one run's folder, with its quota being held.
type Serve ¶
type Serve struct {
// contains filtered or unexported fields
}
Serve is a ynr serve that ynf started for a job.
func StartServe ¶
StartServe starts ynr serve on the spool root for the length of a job. It never fails the job: when ynr is missing or does not start, it says so in the log and returns nil, and the job runs on without a collector. bin is the ynr to start (Ynr); environ is the environment it gets.
func (*Serve) Running ¶
Running reports whether ynr serve is still running. It is safe on a nil Serve.
func (*Serve) Stop ¶
Stop ends ynr serve at a job's end. ynr serve ships a closed file as it finds it and deletes the file once it is shipped, and it polls, so what ynf wrote last may still be waiting: Stop first waits for the closed files in factory/ and runs/ to be gone, then sends SIGTERM and gives ynr serve the rest of the archive time to ship what is left (at least 5 seconds, or all of a shorter archive time), then kills it. It reports whether ynr serve ended within the time. It is safe on a nil Serve.
type Spool ¶
type Spool struct {
Root string
// Quota is how large a run's folder may grow; default DefaultQuota.
Quota int64
// Interval is how often a run's folder is measured; default 250ms.
Interval time.Duration
Log *slog.Logger
// Shipping reports whether ynr serve is running on this spool, so it ships the files and a
// run's end captures nothing. Nil means it is not.
Shipping func() bool
// Volumes make a run's folder a size-limited filesystem of its own, so the quota is hard. Nil,
// or a host that does not allow one, keeps the quota watcher as the bound.
Volumes Volumes
// Drain is how long a run's end waits for ynr serve to ship the run's files before its volume
// is taken away; default DefaultDrain.
Drain time.Duration
// Alive reports whether the ynf process that holds a run volume's lease is running, for the
// volumes a killed ynf left (CleanStale). Nil checks the process table.
Alive func(pid int, start string) bool
// contains filtered or unexported fields
}
Spool is the spool root as ynf uses it.
func New ¶
New makes the layout under root: factory/ and manifests/ for ynf alone, runs/ that a run user can enter but not list. It never follows a link: a root's folder that is one is an error.
func (*Spool) Begin ¶
Begin gives a run its folder and its manifest, before the run starts, and holds the folder to its quota: as a volume of its own with a hard limit where the host allows one, else with the quota watcher. It is all or nothing: without a manifest ynr could not say which factory the run belongs to, so no folder is left behind for it. imageUser is for a run whose container keeps the image's own user, who needs the folder open to them.
The caller runs on without a spool when it returns an error: a spool that cannot be written never fails a step.
func (*Spool) BeginWith ¶
BeginWith is Begin for a run whose executor may not be able to use a volume: with volume false the folder is a plain one, bound by the quota watcher, whatever the host allows.
func (*Spool) Capture ¶
Capture records where a run's leftovers go, so the sweep at a job's end puts them beside the run's own capture.
func (*Spool) CleanStale ¶
CleanStale takes away the run volumes a ynf that is no longer running left behind, at the start of a job. A volume is stale only when it has a lease (owner) in manifests/ written on this host by a process that is gone, and its run is not one of this process's own. A run folder without a lease, a lease whose process is running (this ynf or another one sharing the root), and anything that is not ynf's own volume are left exactly as they are. For a stale one it copies the spool files into the run's capture as the run's end would, within the capture limits, unmounts the volume, and removes the folder. capture is the job's capture folder, for a run that has no capture of its own.
It never fails the job: each volume that could not be cleaned is logged and left for the next job, and what was cleaned is logged once.
func (*Spool) Close ¶
Close takes away any run volume still mounted, such as one whose run never reached its end, after keeping what is in it in the run capture: the run's own when it has one, else under dir. It is safe to call more than once.
func (*Spool) FactoryDir ¶
FactoryDir is ynf's own writer folder.
func (*Spool) Sweep ¶
Sweep puts any spool file still in the spool, in runs/ and factory/, into the run capture at a job's end (ADR-010): a run's into its own capture when it has one, the rest under dir. It is called after ynr serve has had its archive time, so what is left is what was not shipped.
func (*Spool) WriteManifest ¶
WriteManifest writes manifests/<run id>.json as ynr reads it: the run id equal to the folder's, a non-empty lane, under 64 KiB, a regular file. It goes to a temporary name in manifests/ and is renamed into place, so ynr never reads half of it, and it refuses to replace anything that is not a regular file, such as a link.
type Volumes ¶
type Volumes interface {
// Name says what kind of volume it is, for the log.
Name() string
// Mount makes dir, an empty folder, a filesystem of at most size bytes and entries files and
// folders, owned by the user ynf runs as. It returns how to take it away again, which must
// work whether or not anything is left in it. It returns an error wrapping ErrNoVolume when
// the host does not allow a volume.
Mount(dir string, size int64, entries int) (unmount func() error, err error)
// Backing is what, besides the mount point, identifies the volume Mount made at dir, for a
// later ynf to find it by: the disk image's path on macOS, empty where the mount point is all
// there is.
Backing(dir string) string
// Attached finds the volume that ynf made at dir (with that backing) and returns how to take
// it away, which removes whatever Backing names too. It returns nil when there is none. It is
// for a volume whose ynf was killed, and it must match only what ynf itself made: never a
// mount or an image that merely is attached to the host.
Attached(dir, backing string) (detach func() error, err error)
}
Volumes make a folder a filesystem of its own with a hard size limit, which ynr serve reads as a run folder on a device of its own (ynr ADR-003). An implementation is the host's: a tmpfs on Linux, a disk image on macOS.
func HostVolumes ¶
func HostVolumes() Volumes
HostVolumes is a tmpfs per run: memory sized by the quota, with no backing file to clean up. It needs CAP_SYS_ADMIN, which an unprivileged user or a job container does not have, and then the quota watcher is the bound.
Source Files