journal

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package journal keeps the requests a mock server serves, of any protocol, whatever their outcome, so a test can verify after the fact what the code under test sent: a changelog of JSON lines in segment files, with an index in memory that narrows queries. Queries match with the matchers of mocks, such as those of package github.com/caffeine-projects/vlatte/match. A server embeds a [Verifier], next to its vlatte.Base, to verify its requests.

The servers of github.com/caffeine-projects/vlatte/mockhttp and github.com/caffeine-projects/vlatte/mockgrpc keep a journal when their setup asks for one. A Journal works on its own too, as a changelog of any requests: New opens one.

Entries

An Entry is one request, with the fields every protocol has: its kind, its Outcome, the mock that answered it, its method, path and full target, the status of its answer, its time, headers and body, and, for a request no mock matched, the mocks that came closest. Journal.Append gives each entry a sequence number, which orders them: a query returns the newest first, and Journal.Get finds one by its request id. A server fills in the entry of a request it serves in a Draft, which appends it once the request ends, telling the on_complete events that failed.

Files

The journal never rewrites an entry. Each one is a line of JSON appended to the active segment, a file named journal-NNNNNN.log. A new segment starts once the active one holds a fourth of MaxEntries, 10 000 at most, or 64 MiB, so dropping the oldest entries deletes whole files. In a named directory, a segment that fills gets a sidecar, journal-NNNNNN.idx, with what the index keeps of its entries, as does the active one on Close: the next journal of the directory reads the sidecars instead of the logs, as long as each describes its log as the log is.

A named directory keeps the entries across restarts, and gets a .gitignore that keeps its files out of git. An empty Dir keeps them in a temporary directory, which Close removes. One journal at a time opens a directory, where the system locks files: another fails with ErrInUse.

The index

The journal keeps a row of each entry in memory: its sequence number, request id and time, the fields a query selects by but headers and body, and where its line is on disk. For the indexed fields, kind, outcome, mock id, method, path and status, it also lists, for each value it holds, the entries that have it.

Queries

A Query selects entries by those fields, by the full target, by time, by sequence number, and by headers and body; Filter values, such as the request patterns of a protocol, narrow them further. A query runs in steps, each reading less than the one before:

  1. It copies the rows, and the values of the indexed fields it selects by, then lets go of the journal.
  2. It runs each matcher of an indexed field once per value the journal holds, not once per entry, and takes the entries of the values that match: the fields it selects by intersect.
  3. It checks the rows left against the target and the time, which rows hold too.
  4. Only then, and only when it selects by headers, body or filters, or returns entries, does it read their lines from disk.

So Journal.Count of a query of indexed fields alone reads no file. Journal.Find returns a Page: the entries, the count of all the query selects, and, for a query with a Limit, where the next page starts. A query's After, such as the Journal.Last of a moment before, selects the entries appended since.

Writes and reads

Append is synchronous: an entry is in the journal once Append returns. The mock servers append a request's entry as they end its answer, so a test checks it right after the request returns. Queries match and read the copy they took without holding the journal, so appends don't wait on them. A journal is safe for concurrent use. Journal.Changed wakes a caller waiting for an entry on the next append, clear or close, so it queries again rather than poll.

Retention and recovery

The journal keeps the newest MaxEntries entries, all of them for 0, and MaxBodySize bytes of each body: an entry whose body lost the rest says so. Dropping the oldest entries deletes the segments that hold none of the newest. Journal.Trimmed tells the newest entry dropped so, and Journal.Cleared the last one before the journal's latest start, a clear or the oldest entry it loaded: a count of the entries after a point is whole when neither reaches past it.

A crash may cut a line short, or leave a file half written: the next journal of the directory leaves the line out, logs it, and removes the file. Journal.Clear writes a tombstone, journal.clear, before it deletes the segments, so a journal opened after a crash, or after a delete that failed, drops them all the same.

Verifying

Package github.com/caffeine-projects/vlatte/verify checks the requests of a server's journal in a test: which came, how many, by when and in what order.

Example
package main

import (
	"context"
	"fmt"
	"net/http"

	"github.com/caffeine-projects/vlatte/journal"
	"github.com/caffeine-projects/vlatte/match"
)

func main() {
	// An empty Dir keeps the files in a temporary directory, which Close removes.
	j := journal.MustNew(journal.Options{})
	defer j.Close()

	for _, e := range []*journal.Entry{
		{ID: "1", Kind: "http", Outcome: journal.OutcomeMatched, Method: "POST", Path: "/orders", Status: http.StatusCreated},
		{ID: "2", Kind: "http", Outcome: journal.OutcomeMatched, Method: "GET", Path: "/orders/1", Status: http.StatusOK},
		{ID: "3", Kind: "http", Outcome: journal.OutcomeUnmatched, Method: "GET", Path: "/users", Status: http.StatusTeapot},
	} {
		if err := j.Append(e); err != nil {
			panic(err)
		}
	}

	page, err := j.Find(context.Background(), journal.Query{Path: match.HasPrefix("/orders")})
	if err != nil {
		panic(err)
	}

	// The newest come first.
	for _, e := range page.Entries {
		fmt.Println(e.Seq, e.Method, e.Path, e.Status)
	}

	unmatched, err := j.Count(context.Background(), journal.Query{Outcome: journal.OutcomeUnmatched})
	if err != nil {
		panic(err)
	}

	fmt.Println(unmatched)
}
Output:
2 GET /orders/1 200
1 POST /orders 201
1

Index

Examples

Constants

View Source
const (
	// DefaultMaxEntries is how many entries the journal of a mock server keeps by default, the newest.
	DefaultMaxEntries = 10_000

	// DefaultMaxBodySize is how many bytes of a body the journal of a mock server keeps by default.
	DefaultMaxBodySize = 1 << 20
)

Variables

View Source
var (
	// ErrNoJournal is the error of the methods of a nil [Journal], and of a [Verifier] of a server without a journal.
	ErrNoJournal = errors.New("journal: the server keeps no journal of requests: turn its journal on")

	// ErrClosed is the error of the methods of a [Journal] after [Journal.Close], but Dir and Close.
	ErrClosed = errors.New("journal: closed")

	// ErrInUse is the error of [New] when another journal, of this process or another, keeps its files in the
	// directory, where the system locks files.
	ErrInUse = errors.New("journal: another journal keeps its files in the directory")

	// ErrNotFound is the error of [Journal.Get] when the journal holds no entry with the id, such as one it dropped.
	ErrNotFound = errors.New("journal: request not found")

	// ErrBodyCut is the error of a query, or a pattern, that reads the body of an entry the journal kept part of, as
	// its MaxBodySize says: what was cut may hold what it looks for. It comes with [vlatte.ErrMatching].
	ErrBodyCut = errors.New("journal: the journal kept part of the body")
)

Functions

func ClipBody

func ClipBody(body []byte, n int) []byte

ClipBody returns the first n bytes of body, n less than its length, fewer when n cuts a character of a body whose first n bytes are text: clipped text stays text, as a journal keeps it.

Types

type Draft

type Draft struct {
	// contains filtered or unexported fields
}

Draft is the entry of a request a mock server is serving, which the server fills in as the request goes and appends once the request ends. It keeps the errors of the request's on_complete events that fail, such as webhooks, for the entry to tell. Its methods are safe from many goroutines, as the events of a request run at the same time.

func (*Draft) Append

func (d *Draft) Append(end func(e *Entry)) error

Append runs end on the entry, which must not call the methods of d, and appends it to the journal, as Journal.Append does. The entry's Message tells the events that failed, one line each, unless the entry has a message of its own: the request failed, which its message tells. An error of many lines, such as a panic with its stack, is put on one. Only the first Append counts: another appends nothing and returns nil.

func (*Draft) EventFailed

func (d *Draft) EventFailed(err error)

EventFailed keeps err, the error of an on_complete event of the request. An event that fails once the entry is appended is left out of the journal.

func (*Draft) ID

func (d *Draft) ID() string

ID returns the request id of the entry.

func (*Draft) Update

func (d *Draft) Update(set func(e *Entry))

Update runs set on the entry, which must not call the methods of d. What it sets once the entry is appended is left out of the journal.

type Entry

type Entry struct {
	// Seq orders the entries of a journal: a later entry has a greater one. Append sets it.
	Seq uint64

	// ID is the request's, such as the one the VLatte-Request-Id header names.
	ID string

	// Time is when the request came.
	Time time.Time

	// Kind is the protocol of the request, such as "http".
	Kind vlatte.Kind

	Outcome Outcome

	// MockID is the mock that matched the request, if one did.
	MockID string

	Method string
	Path   string

	// Target is what the request asked for in full, such as its URL.
	Target string

	// Status is the status of the answer, such as an HTTP status or a gRPC code.
	Status int

	// Elapsed is how long the request took to serve.
	Elapsed time.Duration

	// Header holds the request's headers, or its metadata.
	Header map[string][]string

	// Body holds the request's body, up to the journal's limit: BodyTruncated tells it lost the rest.
	Body          []byte
	BodyTruncated bool

	// Message tells what failed: the request, for one that failed, or else its on_complete events that failed, such as
	// webhooks, one line each.
	Message string

	// NearMisses are the mocks that came closest to matching a request no mock matched.
	NearMisses []vlatte.NearMiss

	// Scenarios are the moves of the scenarios of the mock that matched the request. A scenario moves as its mock is
	// picked, so a request that then fails has them too.
	Scenarios []matchscenario.Move
}

Entry is a request a mock server served, as its Journal keeps it. The fields every protocol has are its own; a protocol fills those it has: gRPC has no method, and its path is the full method of the call.

func (*Entry) HeaderValue

func (e *Entry) HeaderValue(name string) string

HeaderValue returns the first value of the entry's header named name, in any case, or "" when it has none: the value a query's Header matcher gets. Of two spellings of a name, the one spelled as name comes first.

func (*Entry) MarshalJSONTo

func (e *Entry) MarshalJSONTo(enc *jsontext.Encoder) error

MarshalJSONTo writes e as its journal's files keep it.

func (*Entry) Messages

func (e *Entry) Messages() []string

Messages returns the lines of the entry's Message, each a failure, trimmed, without the empty ones.

func (*Entry) UnmarshalJSONFrom

func (e *Entry) UnmarshalJSONFrom(dec *jsontext.Decoder) error

UnmarshalJSONFrom reads e as MarshalJSONTo writes it.

type Filter

type Filter func(ctx context.Context, e *Entry) (bool, error)

Filter tells whether an entry of a journal is one a query wants, such as a protocol's request pattern does.

type Journal

type Journal struct {
	// contains filtered or unexported fields
}

Journal keeps the requests a mock server serves, of any protocol, in files: a log of entries, one per line, in segments the journal starts as one fills. It indexes them in memory, so a query reads from disk only the entries its indexed fields leave, and only when it selects by their headers or bodies, or returns them.

Append is synchronous: a request is in the journal once its answer is served, for a test to check right after. A journal is safe for concurrent use. Queries read without holding the journal: appends don't wait on them.

func MustNew

func MustNew(o Options) *Journal

MustNew is like New, but panics on an error.

func New

func New(o Options) (*Journal, error)

New opens the journal of o.Dir, with the entries it holds, or creates it. One journal at a time keeps its files in a directory: another one fails with ErrInUse, where the system locks files.

func (*Journal) Append

func (j *Journal) Append(e *Entry) error

Append adds e to the journal, and gives it its sequence number. A body over the journal's limit loses the rest, and text that isn't UTF-8, which JSON can't hold, gets U+FFFD in place of its bad bytes. A failure is logged, the first and then each time their count doubles, and returned.

func (*Journal) Changed

func (j *Journal) Changed() <-chan struct{}

Changed returns a channel closed once the journal changes: an entry is appended, or the journal is cleared or closed. A caller waiting for an entry queries the journal again once it is closed, and calls Changed again to wait for the next change. A nil or closed journal returns a closed channel: a query tells why.

func (*Journal) Clear

func (j *Journal) Clear() error

Clear removes every entry. Sequence numbers go on from the last one given while the journal is open: a journal opened on an empty directory starts them over. A file it can't remove yet, such as one a query holds open where the system keeps it, it logs and removes later: its entries are gone either way.

func (*Journal) Cleared

func (j *Journal) Cleared() uint64

Cleared returns the sequence number of the last entry before the journal's latest start, of which it holds none: the newest one when Journal.Clear last cleared it, or, for a journal that holds none of the entries of its directory before the oldest it loaded, the one before that. It is 0 for a journal that started empty and was never cleared, and for a nil journal.

func (*Journal) Close

func (j *Journal) Close() error

Close closes the journal, and removes its directory if it is a temporary one. Closing it again does nothing.

func (*Journal) Count

func (j *Journal) Count(ctx context.Context, q Query, filters ...Filter) (int, error)

Count returns how many entries q selects, with filters. It reads no entry when q selects by indexed fields only.

func (*Journal) Dir

func (j *Journal) Dir() string

Dir returns the directory of the journal's files, as an absolute path.

func (*Journal) Draft

func (j *Journal) Draft(entry Entry) *Draft

Draft starts the draft of entry, which Draft.Append appends to j. A nil j keeps no journal: Append returns ErrNoJournal.

func (*Journal) Find

func (j *Journal) Find(ctx context.Context, q Query, filters ...Filter) (Page, error)

Find returns the entries q selects, with filters, the newest first, a page of them when q has a Limit.

Example (Pages)
package main

import (
	"context"
	"fmt"

	"github.com/caffeine-projects/vlatte/journal"
)

func main() {
	j := journal.MustNew(journal.Options{})
	defer j.Close()

	for _, id := range []string{"a", "b", "c", "d", "e"} {
		if err := j.Append(&journal.Entry{ID: id, Kind: "http", Method: "GET", Path: "/" + id}); err != nil {
			panic(err)
		}
	}

	// Set Before to a page's Next to get the next page; Next is 0 on the last one.
	q := journal.Query{Limit: 2}

	for {
		page, err := j.Find(context.Background(), q)
		if err != nil {
			panic(err)
		}

		for _, e := range page.Entries {
			fmt.Print(e.ID, " ")
		}

		fmt.Println("of", page.Count)

		if page.Next == 0 {
			break
		}

		q.Before = page.Next
	}
}
Output:
e d of 5
c b of 5
a of 5

func (*Journal) Get

func (j *Journal) Get(id string) (*Entry, error)

Get returns the entry of the request id.

func (*Journal) Last

func (j *Journal) Last() uint64

Last returns the sequence number of the newest entry appended, which the next entry's exceeds: a Query with it as After selects the entries appended since. It is 0 for a journal that never kept one, and for a nil journal.

Example
package main

import (
	"context"
	"fmt"

	"github.com/caffeine-projects/vlatte/journal"
	"github.com/caffeine-projects/vlatte/match"
)

func main() {
	j := journal.MustNew(journal.Options{})
	defer j.Close()

	// A request from before, such as another test's.
	if err := j.Append(&journal.Entry{ID: "1", Kind: "http", Method: "GET", Path: "/health"}); err != nil {
		panic(err)
	}

	mark := j.Last()

	if err := j.Append(&journal.Entry{ID: "2", Kind: "http", Method: "GET", Path: "/health"}); err != nil {
		panic(err)
	}

	// After selects only the entries appended since the mark.
	n, err := j.Count(context.Background(), journal.Query{Path: match.Equal("/health"), After: mark})
	if err != nil {
		panic(err)
	}

	fmt.Println(n)
}
Output:
1

func (*Journal) Trimmed

func (j *Journal) Trimmed() uint64

Trimmed returns the sequence number of the newest entry the journal dropped to keep MaxEntries since its latest start, as Journal.Cleared tells it, or 0 when it dropped none. A query of the entries after one up to it misses those it dropped. It is 0 for a nil journal.

type Options

type Options struct {
	// Dir is where the journal keeps its files, across restarts. Empty keeps them in a temporary directory, which
	// Close removes.
	Dir string

	// MaxEntries is how many entries the journal keeps, the newest. 0 keeps them all.
	MaxEntries int

	// MaxBodySize is how many bytes of a body an entry keeps. 0 keeps whole bodies.
	MaxBodySize int64

	// Logger logs what the journal fails to keep. Nil logs nothing.
	Logger *zerolog.Logger
}

Options configures a Journal.

type Outcome

type Outcome int8

Outcome is what came of a request a mock server served.

const (
	// OutcomeUnknown is an outcome that could not be parsed.
	OutcomeUnknown Outcome = iota
	// OutcomeMatched is a request a mock answered.
	OutcomeMatched
	// OutcomeUnmatched is a request no mock matched, which the server answered as such.
	OutcomeUnmatched
	// OutcomeProxied is a request another server answered: the host of a forward proxy, or a proxy's target.
	OutcomeProxied
	// OutcomeFailed is a request whose serving failed.
	OutcomeFailed
	// OutcomeAbandoned is a request whose client left before it got its answer.
	OutcomeAbandoned
)

func ParseOutcome

func ParseOutcome(s string) Outcome

ParseOutcome parses an outcome's name in any case, returning OutcomeUnknown for an unrecognized name.

func (Outcome) MarshalText

func (o Outcome) MarshalText() ([]byte, error)

MarshalText encodes o as its name, keeping outcomes readable in journal files and API responses.

func (Outcome) String

func (o Outcome) String() string

String returns the outcome's name, or "unknown" for an unrecognized value.

func (*Outcome) UnmarshalText

func (o *Outcome) UnmarshalText(text []byte) error

UnmarshalText parses an outcome's name as ParseOutcome does.

func (Outcome) Valid

func (o Outcome) Valid() bool

Valid tells whether o is one of the outcomes a journal knows.

type Page

type Page struct {
	Entries []*Entry
	Count   int
	Next    uint64
}

Page is a page of the entries a query selects, the newest first. Count is the number of entries the query selects, its Limit and Before left out. Next, when not 0, is the Before of the next page.

type Query

type Query struct {
	Kind    vlatte.Kind
	Outcome Outcome

	MockID vlatte.Matcher
	Method vlatte.Matcher
	Path   vlatte.Matcher
	Target vlatte.Matcher
	Status vlatte.Matcher

	// Since and Until bound the time of the request: Since includes it, Until doesn't.
	Since time.Time
	Until time.Time

	// Header matches headers by name, in any case: a matcher gets the first value of its header, or "" when the
	// request has none.
	Header map[string]vlatte.Matcher

	// Body gets the body as text. A query the rest of which selects an entry whose body the journal cut fails with
	// [vlatte.ErrMatching]: what was cut may hold what Body looks for.
	Body vlatte.Matcher

	// Limit is the most entries Find returns, the newest first; 0 returns them all. Count leaves it out.
	Limit int

	// Before selects the entries with a lower sequence number, those that follow a page Find returned, which its Next
	// names. Count leaves it out.
	Before uint64

	// After selects the entries with a greater sequence number, those appended since the entry it names, such as the
	// newest one [Journal.Last] returned before a test sent its requests. Count selects by it too.
	After uint64
}

Query selects entries of a Journal. Each field set narrows the entries it selects; the zero query selects them all. A matcher gets the value of its field: text, or an int for Status.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL