samples

package
v2.25.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 4, 2026 License: GPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package samples records the executions of a continuous step (a step that repeats without a limit) as samples, rolls them up in tiers, and answers the queries of the API. See specs/083-continuous-runs.md.

Layout

The store of one step is a directory (see Dir) with these files: 

raw/YYYYMMDDHH.jsonl     one line for each sample, one file for each hour
minute/YYYYMMDD.jsonl    one aggregate for each minute, one file for each day
hour/YYYYMM.jsonl        one aggregate for each hour, one file for each month
day/YYYY.jsonl           one aggregate for each day, one file for each year
events.jsonl             the events; no expiry

All times are UTC. A file holds only whole buckets of the next tier, so a tier can drop a whole file when the next tier holds the aggregates of it.

Format

The files are append-only text lines (JSON or JSON-like). A text format is easy to repair by hand and to read with a tool. The raw line is the one place where size matters, so it is a compact array: 

[offsetMs,durationUs,exitCode]            a sample
[offsetMs,durationUs,exitCode,"message"]  a failed sample whose message differs
                                          from the message of the previous failed
                                          sample of the same file

offsetMs is the time of the sample, in milliseconds after the start of the file. A raw line has about 17 bytes, so one day of 1 sample for each second is about 1.5 MB, which is less than the 3 MiB of the spec.

An aggregate line is a JSON object with the start of the bucket in seconds (t), the count (n), the count of failed samples (f), the least, the greatest and the sum of the durations in microseconds (lo, hi, s), and the histogram (h) as a flat list of index and count pairs, for the used bins only.

Histogram

The histogram has 16 bins for each octave of the duration in microseconds. Bin 0 holds the durations below 1 us. Bin i (i >= 1) holds the durations d with (i-1)/16 <= log2(d/1us) < i/16. The last bin starts at 2^54 us, above the longest time.Duration, so every duration has a bin of finite width. Two histograms add bin by bin, so the aggregate of a bucket is the sum of the aggregates of its parts.

A percentile is the bin that holds the sample of the nearest rank. The value is the geometric middle of the bin, limited to the least and the greatest of the aggregate. A bin is 2^(1/16) wide (4.4%), so the value is within 2.2% of the true value; a bucket of one sample, or of equal samples, is exact.

Roll-up and repair

The Recorder holds the samples of one minute at most. At the end of a minute it appends the raw lines and the aggregate of the minute. At the end of an hour it builds the aggregate of the hour from the minute file, and at the end of a day the aggregate of the day from the hour file. A roll-up builds the aggregates of a whole file and replaces a record only when it differs, so a second run gives the same files. OpenRecorder runs the same roll-up for all files, and removes the temporary files that a crash left, before it accepts a sample. A file is removed when its whole span is older than the retention of its tier, after its roll-up ran.

Close also writes the aggregates of the open buckets, so that a stopped run shows all its samples in every tier. The next roll-up replaces them.

Reader

The Reader opens only the files of the window. The newest bucket of a tier has no aggregate yet while it is open, so the Reader builds it from the finer tier. Samples that the Recorder still holds in memory (less than one minute) are not visible.

Index

Constants

This section is empty.

Variables

View Source 
var ErrInvalidCursor = errors.New("samples: invalid cursor")

ErrInvalidCursor reports an events cursor that the store did not make.

View Source 
var ErrNotImplemented = errors.New("samples: not implemented")

ErrNotImplemented is returned by the functions of this package that a later change fills in.

Functions

func Backfill 

func Backfill(ctx context.Context, dir string, o BackfillOptions) (err error)

Backfill writes the minute, hour and day tiers and the events of a long stretch of history, without a raw tier and without a Recorder. It is for tools and tests that need years of samples at once: the cost is the cost of the Next function and of one histogram update for each sample. The directory must hold no store yet. A Recorder opened after it continues the store from End.

func Dir 

func Dir(attemptDir, stepName string) string

Dir returns the directory of the store of a step, under the directory of the attempt of a run. The step name is made safe for a file name.

Types

type Availability 

type Availability struct {
	Windows       []WindowStat
	Streak        uint64
	LastFailureAt time.Time // zero when none failed
	TotalCount    uint64
}

Availability is the availability of each window, the current streak, and the total count of executions.

type BackfillOptions 

type BackfillOptions struct {
	// Start and End bound the stretch, [Start, End). Both are multiples of one
	// minute.
	Start, End time.Time
	// Interval is the time between two samples, in whole milliseconds: a
	// divisor of one minute, or a whole count of minutes.
	Interval time.Duration
	// Now is the time that the retention of the tiers counts from. A file that
	// the retention of its tier would remove at Now is not written.
	Now time.Time
	// Next gives the sample at a time, in unix milliseconds: the duration,
	// the exit code and the message.
	Next func(ms int64) (time.Duration, int, string)
}

BackfillOptions describes a stretch of history for Backfill.

type Bucket 

type Bucket struct {
	// Time is the start of the bucket, in UTC.
	Time     time.Time
	Count    uint64
	Failures uint64
	Min      time.Duration
	Max      time.Duration
	P50      time.Duration
	P95      time.Duration
}

Bucket summarizes the samples of one time bucket.

type Event 

type Event struct {
	Time     time.Time
	Kind     EventKind
	ExitCode int
	Message  string
	// FailedCount is the count of failed samples of the outage, for a
	// recovered event, for a repeated failed event, and for a stopped event
	// during an outage, so that the next start continues the count.
	FailedCount uint64
	// Outage is the length of the outage, for a recovered event.
	Outage time.Duration
}

Event is a start, a stop, a failure or a recovery of a continuous step.

type EventKind 

type EventKind string

EventKind is the kind of an event. 

const (
	EventStarted   EventKind = "started"
	EventStopped   EventKind = "stopped"
	EventFailed    EventKind = "failed"
	EventRecovered EventKind = "recovered"
)

The kinds of events.

type EventPage 

type EventPage struct {
	Events     []Event
	NextCursor string
}

EventPage is a page of events.

type EventQuery 

type EventQuery struct {
	From   time.Time
	To     time.Time
	Limit  int
	Before string // opaque cursor from EventPage.NextCursor
}

EventQuery lists events, the newest first.

type Options 

type Options struct {
	// Now gives the time; time.Now when nil. Tests set it.
	Now func() time.Time
}

Options configures a Recorder. A zero value uses the defaults of the spec.

type Reader 

type Reader interface {
	Samples(ctx context.Context, q SampleQuery) (Series, error)
	Events(ctx context.Context, q EventQuery) (EventPage, error)
	Availability(ctx context.Context, now time.Time) (Availability, error)
}

Reader is the read side of the store of one continuous step. It never holds more than one response in memory and its cost does not depend on the count of samples.

func OpenReader 

func OpenReader(dir string) Reader

OpenReader opens the store of a step for reading. A step with no store gives an empty reader.

func OpenReaderWith 

func OpenReaderWith(dir string, opts Options) Reader

OpenReaderWith is OpenReader with options. Only Now applies: it gives the time that the retention of the tiers counts from.

type Recorder 

type Recorder interface {
	// Record adds a sample. It holds at most one minute of samples in memory
	// and writes the raw tier, the events and the roll-ups at the boundaries
	// of the buckets.
	Record(ctx context.Context, s Sample) error
	// Flush writes what is held in memory.
	Flush(ctx context.Context) error
	// Close flushes and stops the recorder, and records a stopped event.
	Close(ctx context.Context) error
}

Recorder is the write side of the store of one continuous step. It is safe for use by one goroutine at a time.

func OpenRecorder 

func OpenRecorder(ctx context.Context, dir string, opts Options) (Recorder, error)

OpenRecorder opens the store of a step for writing, and builds the roll-ups that a stop or a crash left out.

type Sample 

type Sample struct {
	// Time is the start of the execution, in UTC. It has millisecond precision.
	Time time.Time
	// Duration is the length of the execution.
	Duration time.Duration
	// ExitCode is the exit code of the execution; -1 when the step did not start.
	ExitCode int
	// Message is the last line of the standard error of a failed execution,
	// 256 bytes at most. It is empty when the execution succeeded.
	Message string
}

Sample is the record of one execution of a continuous step.

func (Sample) Failed 

func (s Sample) Failed() bool

Failed reports whether the sample is a failed sample (exit code not 0).

type SampleQuery 

type SampleQuery struct {
	From      time.Time
	To        time.Time
	MaxPoints int
}

SampleQuery asks for the samples of a window. The reader picks the finest tier whose retention covers From and whose bucket count in the window is not more than MaxPoints, and returns at most MaxPoints buckets.

type Series 

type Series struct {
	Tier    Tier
	From    time.Time
	To      time.Time
	Buckets []Bucket
}

Series is the answer to a sample query.

type Tier 

type Tier int

Tier is one of the four tiers of the spec. 

const (
	TierRaw Tier = iota
	TierMinute
	TierHour
	TierDay
)

The tiers, from the finest to the coarsest.

func (Tier) Retention 

func (t Tier) Retention() time.Duration

Retention is how long the tier keeps its records; 0 means no limit.

func (Tier) Width 

func (t Tier) Width() time.Duration

Width is the bucket width of the tier.

type Window 

type Window string

Window is one availability window. 

const (
	Window24h  Window = "24h"
	Window7d   Window = "7d"
	Window30d  Window = "30d"
	Window365d Window = "365d"
)

The availability windows.

type WindowStat 

type WindowStat struct {
	Window   Window
	Count    uint64
	Failures uint64
}

WindowStat is the availability of one window.

Source Files

View all Source files

Directories

Path Synopsis
cmd
synth command
Command synth fills the sample store of a continuous step with years of synthetic data, and creates the DAG and the run that the web UI needs to show it.
 Command synth fills the sample store of a continuous step with years of synthetic data, and creates the DAG and the run that the web UI needs to show it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.