stream

package
v0.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 15, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package stream defines the VegaLoad run event stream (vegaload.io/stream/v1): the event format, the per-sink queues, and the file and push sinks.

The format is documented in docs/streaming.md. This package does not import the engine, the report package or any driver: the runner hands it plain values through the Emitter interface, so a run with no stream pays nothing and the core engine knows nothing about streaming (AGENTS.md, module boundaries).

Index

Constants

View Source
const (
	TypeRunStarted  = "run.started"
	TypePoint       = "point"
	TypeWarning     = "warning"
	TypeRunFinished = "run.finished"
	TypeRunFailed   = "run.failed"
)

Event types.

View Source
const DefaultQueueDepth = 4096

DefaultQueueDepth bounds how many points a sink may have waiting.

View Source
const Schema = "vegaload.io/stream/v1"

Schema is the value of the schema field on every event.

Variables

This section is empty.

Functions

func IsLoopbackHost

func IsLoopbackHost(host string) bool

IsLoopbackHost reports whether host is localhost or a loopback IP.

func NewRunID

func NewRunID() string

NewRunID returns a random 12-hex-digit run id.

func SanitizeTarget

func SanitizeTarget(target string) string

SanitizeTarget returns target without anything that can carry a credential: the userinfo, the query and the fragment of a URL, and a userinfo before an "@" in a plain host:port. Anything it cannot read as a URL it returns with every "@..." prefix and "?..." suffix cut off, so a secret is never passed on by accident.

func ValidRunID

func ValidRunID(id string) error

ValidRunID reports whether id can be used as a run id. The id ends up in file names and URLs, so the set is small.

func ValidatePushURL

func ValidatePushURL(raw string, insecure bool) (*url.URL, error)

ValidatePushURL checks a push destination. It must be http or https, carry no credentials, and use https unless the host is loopback or insecure is set.

func Write

func Write(w io.Writer, events []Envelope) error

Write writes events as NDJSON, one per line.

Types

type Aborter

type Aborter interface{ Abort() }

Aborter is implemented by sinks that can be stuck in a retry loop. Abort makes them give up, so Close can return.

type Drainer

type Drainer interface{ Drain() }

Drainer is implemented by sinks that retry. Drain is called, from any goroutine, when the run is over and the recorder is about to wait for the sink for a bounded time. The sink should stop backing off and try for the rest of that time, because what is left includes the terminal event.

type Emitter

type Emitter interface {
	// Started begins the stream. at is the instant the run's clock started:
	// offsetS on every event counts from it.
	Started(at time.Time, rs RunStarted)
	// Point reports one closed one-second bucket.
	Point(p Point)
	// Warning reports something the user should know.
	Warning(msg string)
	// Finished ends the stream with the run's result. result is marshalled
	// as it is (the same value `-out` writes); passed and exitCode are added.
	Finished(result any, passed bool, exitCode int)
	// Failed ends the stream for a run that could not run or was interrupted.
	// partial is the partial result, or nil.
	Failed(reason string, partial any)
}

Emitter is what a runner needs from a stream. The runner holds one as a nil-able interface value: with no stream there is no Emitter and no cost. Every method is safe to call from any goroutine and none of them blocks on a sink or returns an error.

type Envelope

type Envelope struct {
	Schema    string            `json:"schema"`
	RunID     string            `json:"runId"`
	Seq       int64             `json:"seq"`
	TS        time.Time         `json:"ts"`
	OffsetS   float64           `json:"offsetS"`
	Labels    map[string]string `json:"labels,omitempty"`
	Synthetic bool              `json:"synthetic,omitempty"`
	Type      string            `json:"type"`
	Data      json.RawMessage   `json:"data"`
}

Envelope wraps every event. Data stays raw so that readers keep fields and event types they do not know.

func (Envelope) Terminal

func (e Envelope) Terminal() bool

Terminal reports whether the event ends a run.

type FileSink

type FileSink struct {
	// contains filtered or unexported fields
}

FileSink writes events as NDJSON, one line per event, unbuffered, so that `tail -f` works and another process can follow the run.

func NewFileSink

func NewFileSink(path string) (*FileSink, error)

NewFileSink creates (or truncates) the file at path, and the directories above it.

func (*FileSink) Close

func (s *FileSink) Close() error

func (*FileSink) Name

func (s *FileSink) Name() string

func (*FileSink) Write

func (s *FileSink) Write(line []byte) error

type Flusher

type Flusher interface {
	// Deadline reports when the pending batch must be sent; ok is false when
	// nothing is pending.
	Deadline() (t time.Time, ok bool)
	Flush() error
}

Flusher is implemented by sinks that batch. The runner calls Flush when the deadline passes with nothing new to write, and once more when the run is over.

type Point

type Point struct {
	OffsetS     float64 `json:"offsetS"`
	RPS         float64 `json:"rps"`
	Requests    int64   `json:"requests"`
	Failed      int64   `json:"failed"`
	ErrorRate   float64 `json:"errorRate"`
	P50Ms       float64 `json:"p50Ms"`
	P95Ms       float64 `json:"p95Ms"`
	P99Ms       float64 `json:"p99Ms"`
	Total       int64   `json:"total"`
	TotalFailed int64   `json:"totalFailed"`
}

Point is the data of point: one closed one-second bucket of the run.

type PushOptions

type PushOptions struct {
	URL     *url.URL
	Token   string
	Version string
	// Zero values take the documented defaults.
	MaxBatch   int           // 100 events
	MaxDelay   time.Duration // 500 ms from the first event of a batch
	Timeout    time.Duration // 10 s per request
	BackoffMin time.Duration // 500 ms
	BackoffMax time.Duration // 10 s
	Client     *http.Client
}

PushOptions configures a PushSink.

type PushSink

type PushSink struct {
	// contains filtered or unexported fields
}

PushSink POSTs batches of NDJSON to an ingest endpoint (docs/streaming.md, "Push"). One batch is in flight at a time, so a receiver sees batches in order, and a retried batch repeats the same sequence range.

2xx acknowledges a batch. Network errors, 408, 429 and 5xx are retried with exponential backoff until the run is over and the deadline given to Recorder.Close passes. Any other 4xx drops the batch and adds a warning.

func NewPushSink

func NewPushSink(opt PushOptions) *PushSink

NewPushSink creates a sink for opt.URL, which must already have passed ValidatePushURL.

func (*PushSink) Abort

func (s *PushSink) Abort()

Abort makes a retrying Flush give up.

func (*PushSink) Close

func (s *PushSink) Close() error

func (*PushSink) Deadline

func (s *PushSink) Deadline() (time.Time, bool)

func (*PushSink) Delivered

func (s *PushSink) Delivered() (delivered, lost int)

Delivered returns how many events the receiver acknowledged and how many were given up on.

func (*PushSink) Drain

func (s *PushSink) Drain()

Drain implements Drainer.

func (*PushSink) Flush

func (s *PushSink) Flush() error

Flush sends the pending batch, retrying where the rules say so. It returns an error only when the sink was aborted, which ends it.

func (*PushSink) Name

func (s *PushSink) Name() string

func (*PushSink) Warnings

func (s *PushSink) Warnings() []string

Warnings implements Warner: each distinct problem once, with a count when it repeated.

func (*PushSink) Write

func (s *PushSink) Write(line []byte) error

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder turns what the runner reports into events and hands each event to every sink. It implements Emitter. Sequence numbers and timestamps are assigned once, so every sink sees the same events.

A Recorder never blocks the caller on a sink and never returns an error to it: a failing sink only adds a warning.

func NewRecorder

func NewRecorder(opt RecorderOptions, sinks ...Sink) *Recorder

NewRecorder creates a Recorder that writes to sinks. With no sinks it still numbers events, which is useful in tests.

func (*Recorder) Close

func (r *Recorder) Close(timeout time.Duration) []string

Close stops accepting events, waits up to timeout for the sinks to write (and, for batching sinks, send) what is queued, and closes them. A sink that is still busy at the deadline is aborted. It returns the names of the sinks that did not finish.

func (*Recorder) Failed

func (r *Recorder) Failed(reason string, partial any)

Failed implements Emitter.

func (*Recorder) Finished

func (r *Recorder) Finished(result any, passed bool, exitCode int)

Finished implements Emitter.

func (*Recorder) HasStarted

func (r *Recorder) HasStarted() bool

HasStarted reports whether run.started has been emitted.

func (*Recorder) Point

func (r *Recorder) Point(p Point)

Point implements Emitter.

func (*Recorder) RunID

func (r *Recorder) RunID() string

RunID returns the id used on every event.

func (*Recorder) Started

func (r *Recorder) Started(at time.Time, rs RunStarted)

Started implements Emitter. Only the first call counts.

func (*Recorder) Warning

func (r *Recorder) Warning(msg string)

Warning implements Emitter.

func (*Recorder) Warnings

func (r *Recorder) Warnings() []string

Warnings returns what has gone wrong with the sinks so far: a sink that stopped accepting events, and events dropped under pressure.

type RecorderOptions

type RecorderOptions struct {
	RunID  string            // default: NewRunID()
	Labels map[string]string // attached to every event
	// QueueDepth overrides DefaultQueueDepth (tests).
	QueueDepth int
}

RecorderOptions configures a Recorder.

type Recording

type Recording struct {
	Path   string
	Events []Envelope
	// Raw holds each event's original line, so that serving a recording
	// forwards fields this version does not know.
	Raw [][]byte
}

Recording is one run's events as read from an NDJSON file.

func Read

func Read(r io.Reader) (*Recording, error)

Read parses an NDJSON event stream and checks its structure: every line is a JSON object with the v1 schema, one run id, a type, data, and a seq that increases by exactly one starting at 1. A recording with no events, or with events after a terminal event, is rejected. Unknown fields are kept.

func ReadFile

func ReadFile(path string) (*Recording, error)

ReadFile reads and checks a recording from a file.

func (*Recording) RunID

func (r *Recording) RunID() string

RunID returns the run id shared by every event in the recording.

func (*Recording) Terminal

func (r *Recording) Terminal() (Envelope, bool)

Terminal returns the last event when it ends a run.

type RunStarted

type RunStarted struct {
	Name          string          `json:"name"`
	Executor      string          `json:"executor"`
	VUs           int             `json:"vus,omitempty"`
	Rate          float64         `json:"rate,omitempty"`
	Duration      string          `json:"duration,omitempty"`
	Stages        []Stage         `json:"stages,omitempty"`
	Target        *Target         `json:"target,omitempty"`
	Thresholds    []ThresholdSpec `json:"thresholds,omitempty"`
	AbortOnBreach bool            `json:"abortOnBreach,omitempty"`
	Version       string          `json:"version,omitempty"`
}

RunStarted is the data of run.started. It carries the shape of the run and nothing a user supplied as a credential: no header, body, option, secret, environment value or data cell (docs/streaming.md, "run.started").

type Sink

type Sink interface {
	// Name is used in warnings, for example "file run.ndjson".
	Name() string
	Write(line []byte) error
	Close() error
}

Sink receives the serialized events of a run, one call per event, in order. Write is called from a single goroutine per sink and may block: the Recorder puts a bounded queue in front of every sink, so a slow sink cannot delay the run. A Write error marks the sink dead; the rest of the run's events for that sink are discarded.

type Stage

type Stage struct {
	Target   int    `json:"target"`
	Duration string `json:"duration"`
}

Stage is one ramp or step stage.

type Target

type Target struct {
	Protocol string `json:"protocol"`
	BaseURL  string `json:"baseUrl"`
}

Target is the protocol-direct target of a run, without credentials.

type ThresholdSpec

type ThresholdSpec struct {
	Name     string `json:"name"`
	Metric   string `json:"metric"`
	Operator string `json:"operator"`
	Value    string `json:"value"`
	Step     string `json:"step,omitempty"`
	Protocol string `json:"protocol,omitempty"`
	Op       string `json:"op,omitempty"`
}

ThresholdSpec is one pass/fail threshold as the run was given it.

type Warner

type Warner interface{ Warnings() []string }

Warner is implemented by sinks that report problems without failing, for example a batch the receiver rejected.

type Warning

type Warning struct {
	Message string `json:"message"`
}

Warning is the data of warning.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL