index

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package index is the projection a search reads.

The archive is the record: objects under Object Lock, accounted for by a signed digest chain. The index is a derived thing, and the whole design depends on keeping that distinction: it may be lost, rebuilt from the prefixes, or replaced by a different implementation, and none of that touches what the trail says. What it must never do is disagree, which is why the rows are derived here rather than in each implementation.

The interfaces are narrow on purpose. Three implementations ship — memory, an object-storage scan and Postgres — and an interface two of them cannot both satisfy honestly is an interface that has been shaped around one database.

Index

Constants

View Source
const (
	FieldAction     = "action"
	FieldOperation  = "operation"
	FieldOutcome    = "outcome"
	FieldSource     = "source"
	FieldActorKind  = "actor_kind"
	FieldTargetType = "target_type"
	// DataField prefixes an extension property's pointer, so that a data
	// property called action cannot be confused with the record's own.
	DataField = "data:"
)

Core fields every row is counted by. An extension property adds its own, named for its pointer, which is what lets an application's own data become a facet in the viewer without any code.

View Source
const (
	SortOccurredAt = "occurred_at"
	SortRecordedAt = "recorded_at"
	SortID         = "id"
	SortAction     = "action"
	SortTenantID   = "tenant_id"
	SortSource     = "source"
)

The fields a page may be ordered by. Every ordering ends with the identifier, so that a keyset boundary is always total and a page cannot repeat or skip a row whose sort values tie with its neighbour's.

Variables

View Source
var ErrNotFound = errors.New("no such record")

ErrNotFound is a Get for a record the searcher does not hold. Every searcher wraps it, so the service can answer "no such record" rather than mistaking an absence for an outage: an outage tells a client to retry, and retrying will never find a record that is not there.

Functions

func Matches

func Matches(r Row, filter []Conjunction) bool

Matches reports whether a row satisfies a filter: the OR of conjunctions, each of which is an AND of predicates. No filter matches everything.

It is exported because it is the definition of what a query means, and a searcher that worked it out again would be a second opinion on the contract. Anything implementing Searcher over data it holds itself should use this rather than reimplement the operators.

Types

type Boundary

type Boundary struct {
	Values []string
	ID     string
	// RecordedAt and Sequence are what a tail advances on.
	RecordedAt time.Time
	Sequence   uint64
}

Boundary is where a page ended: the sort values of its last row, and that row's identifier.

Keyset rather than an offset. An offset re-reads everything before it, so a deep page costs more than a shallow one, and a row inserted meanwhile shifts every page after it — in a trail that is appended to constantly, that means a poller silently skipping records.

type Capabilities

type Capabilities struct {
	// FreeText is whether the searcher can answer an unstructured query.
	FreeText bool
	// Facets is whether it can count values.
	Facets bool
	// DataPredicates is whether it can filter on extension properties.
	DataPredicates bool
	// MaxConjunctions bounds the OR. Beyond it a query is refused rather than
	// turned into work nobody bounded.
	MaxConjunctions int
	// SortFields are the orderings it can answer.
	SortFields []string
}

Capabilities is what an implementation can do.

type Conjunction

type Conjunction struct {
	ID            []Predicate
	OccurredAt    []TimePredicate
	RecordedAt    []TimePredicate
	Source        []Predicate
	Action        []Predicate
	Operation     []Predicate
	Outcome       []Predicate
	TenantID      []Predicate
	ActorKind     []Predicate
	ActorID       []Predicate
	SubjectKind   []Predicate
	SubjectID     []Predicate
	RequestID     []Predicate
	TraceID       []Predicate
	ClientAddress []Predicate
	ObserverID    []Predicate
	TargetType    []Predicate
	TargetID      []Predicate
	// Data are predicates on the extension properties a catalogue marked
	// filterable, addressed by JSON pointer.
	Data []PathPredicate
}

Conjunction is a set of predicates that must all hold.

type Facet

type Facet struct {
	Field  string
	Values []FacetValue
}

Facet is one field's counted values.

type FacetDelta

type FacetDelta struct {
	TenantID string
	Hour     time.Time
	Field    string
	Value    string
	Count    int64
}

FacetDelta is one increment of the counts table.

func Merge

func Merge(deltas []FacetDelta) []FacetDelta

Merge adds the deltas of several rows together, so that one batch touches each counted value once.

func (FacetDelta) String

func (d FacetDelta) String() string

String describes a delta, for test failures and for the reindex report.

type FacetValue

type FacetValue struct {
	Value string
	Count int64
}

FacetValue is a value and how many rows carry it.

type Fields

type Fields struct {
	// Filter are pointers a query may name in a predicate.
	Filter []string
	// Facet are pointers a searcher may count. Every facet is filterable.
	Facet []string
}

Fields says which extension properties of an action are worth indexing. A catalogue's schema answers this; the index takes the answer rather than the catalogue, so that an implementation of this interface needs neither.

type Indexer

type Indexer interface {
	// Index writes rows and the facet counts they imply. A row already there
	// changes nothing, including the counts.
	Index(ctx context.Context, profile string, rows []Row) error
	// Purge forgets what a profile no longer keeps. The archive's objects are
	// under a lock and are not affected; this is the index catching up with a
	// retention that has already expired, and with the narrower schedule the
	// identifying columns are kept on.
	Purge(ctx context.Context, profile string, before time.Time, what Scope) error
}

Indexer writes the projection.

Index is idempotent by (profile, id), including the facet counts it implies. Counting is not a separate call, and could not be: counts are additive, so a caller that increments them cannot tell a re-delivered record from a new one. Only the implementation holding the transaction can, by counting exactly the rows its insert actually created.

type Kind

type Kind string

Kind is how a value is stored, which decides what a predicate may do with it.

const (
	Text Kind = "text"
	Int  Kind = "int"
	Time Kind = "time"
	Bool Kind = "bool"
)

The kinds an indexed property may have. A schema's type decides which one a property gets, so a predicate over it means the same thing everywhere.

type Memory

type Memory struct {
	Indexed int
	// contains filtered or unexported fields
}

Memory holds the projection in one process.

It is what the tests index into and what a deployment small enough to keep its index in memory uses. It is a real implementation, not a stub: it enforces the same idempotency the interface promises, because a memory implementation that quietly double-counts would let a fault reach Postgres before anybody noticed.

func NewMemory

func NewMemory() *Memory

NewMemory returns an empty index.

func (*Memory) Capabilities

func (m *Memory) Capabilities() Capabilities

Capabilities implements Searcher.

The memory searcher exists so that the interface has two honest implementations from the start. Two is the number that keeps an interface from being a description of one implementation's habits.

func (*Memory) Count

func (m *Memory) Count(profile, tenant, field, value string, hour time.Time) int64

Count returns one facet count.

func (*Memory) Counts

func (m *Memory) Counts(profile string) []FacetDelta

Counts returns every count of a profile, for tests and for the reindex report.

func (*Memory) Facets

func (m *Memory) Facets(_ context.Context, q Query, fields []string, limit int) ([]Facet, error)

Facets implements Searcher.

func (*Memory) Get

func (m *Memory) Get(_ context.Context, profile, id string) (Row, Provenance, error)

Get implements Searcher.

func (*Memory) Index

func (m *Memory) Index(_ context.Context, profile string, rows []Row) error

Index implements Indexer.

func (*Memory) Len

func (m *Memory) Len(profile string) int

Len is how many rows a profile has.

func (*Memory) Purge

func (m *Memory) Purge(_ context.Context, profile string, before time.Time, what Scope) error

Purge implements Indexer.

func (*Memory) Row

func (m *Memory) Row(profile, id string) (Row, bool)

Row returns one row, for tests and for the memory searcher.

func (*Memory) Rows

func (m *Memory) Rows(profile string) []Row

Rows returns a profile's rows in recorded order, which is the order a tail advances in.

func (*Memory) Search

func (m *Memory) Search(_ context.Context, q Query) (Page, error)

Search implements Searcher.

type ObjectAt

type ObjectAt struct {
	Key  string
	Line int
}

ObjectAt is where a copy landed in the archive.

type Op

type Op string

Op is a comparison.

const (
	Equal    Op = "eq"
	NotEqual Op = "ne"
	In       Op = "in"
	NotIn    Op = "nin"
	Prefix   Op = "prefix"
)

The operators a predicate may use. There is no regular expression and no substring search: both are unbounded work on a table that only grows, and an index that cannot answer them quickly would answer them slowly instead.

type Page

type Page struct {
	Rows []Row
	// More says whether the next page would have anything in it. It is answered
	// by asking for one row more than the limit, rather than by counting.
	More bool
	// Next is the boundary for the following page. It is present even on the
	// last page, because a tail keeps polling it and a record recorded after
	// this page must come back through it.
	Next *Boundary
	Prev *Boundary
}

Page is what a search returns.

type PathPredicate

type PathPredicate struct {
	Path string
	Op   Op
	// Text, Int and At are the value, whichever kind the property has.
	Text   string
	Values []string
	Int    int64
	At     time.Time
	Kind   Kind
}

PathPredicate is a comparison on an indexed extension property.

type Predicate

type Predicate struct {
	Op     Op
	Value  string
	Values []string
}

Predicate is one comparison on a text field.

type Provenance

type Provenance struct {
	ObjectKey string
	Line      int
	// Digest is the key of the seal that accounts for the object's hour, when
	// there is one. Empty means no seal covers it yet, which is the ordinary
	// state of the current hour and a finding in any other.
	Digest string
	// VerifiedAt is when that seal was last verified clean. Zero means it
	// has not been, or the last verification found a problem.
	VerifiedAt time.Time
}

Provenance says where a row came from, so that an answer can be checked against the copy the digest chain accounts for rather than believed.

type Query

type Query struct {
	Profile string
	// Tenants is the grant: the caller may see these and no others. Empty means
	// every tenant, which only an operator's grant carries.
	Tenants []string
	// Filter is OR-joined: a row matches if it matches any conjunction.
	Filter []Conjunction
	Sort   []SortBy
	Limit  int
	// After is the keyset boundary a cursor carries.
	After *Boundary
	// Backwards pages towards older results.
	Backwards bool
}

Query is a closed question.

type Row

type Row struct {
	ID         string
	TenantID   string
	OccurredAt time.Time
	RecordedAt time.Time
	Sequence   uint64
	Source     string
	Action     string
	Operation  string
	Outcome    string

	TargetTypes []string
	TargetIDs   []string

	// ObjectKey and Line are where the copy is in the archive, which is what
	// lets a reader check any answer the index gives against the object the
	// digest chain accounts for.
	ObjectKey string
	Line      int

	// The identifying columns. They are kept apart because a profile purges
	// them on a shorter schedule than the event, and because a store that
	// separates them can grant a reader one without the other.
	ActorKind     string
	ActorID       string
	SubjectKind   string
	SubjectID     string
	ClientAddress string
	RequestID     string
	TraceID       string
	ObserverID    string

	// Data are the extension properties the action marked filterable, by
	// pointer into its data slot.
	Data []Value
}

Row is one copy of one record as the index holds it.

It is the copy, not the original: a security copy and a billing copy of one event differ in what they carry, and indexing the original would put back what the split took out.

func RowOf

func RowOf(r *record.Record, at ObjectAt, fields Fields) Row

RowOf derives the row of one copy, as written.

func (Row) Facets

func (r Row) Facets() []FacetDelta

Facets returns the counts one row implies. Every implementation derives them the same way because they derive them here.

type Scope

type Scope int

Scope is what a purge removes.

const (
	// Identifying is the actor, subject, address and request columns, which a
	// profile keeps for less time than the event itself.
	Identifying Scope = iota
	// Everything is the row and its counts, for a profile whose retention has
	// run out.
	Everything
)

type Searcher

type Searcher interface {
	// Search returns a page. The cursor in a request must have come from a
	// response to the same query.
	Search(ctx context.Context, q Query) (Page, error)
	// Facets counts values of the named fields under the same filter.
	Facets(ctx context.Context, q Query, fields []string, limit int) ([]Facet, error)
	// Get returns one row and where it came from.
	Get(ctx context.Context, profile, id string) (Row, Provenance, error)
	// Capabilities says what this implementation can do, so that a caller is
	// refused rather than quietly given a narrower answer than it asked for.
	Capabilities() Capabilities
}

Searcher answers questions about what was written.

It is deliberately narrow, and the query it takes is closed: a caller names fields and operators from a fixed set rather than handing over an expression. That is what lets a grant be AND-ed into every query as one more term, and what lets a second implementation of this interface be honest about what it can and cannot do.

type SortBy

type SortBy struct {
	Field      string
	Descending bool
}

SortBy is one ordering term.

type TimePredicate

type TimePredicate struct {
	From time.Time
	// To is exclusive, so that adjacent ranges neither overlap nor leave a gap.
	To time.Time
}

TimePredicate is a half-open range. Either end may be zero.

type Value

type Value struct {
	Path string
	Kind Kind
	Text string
	Int  int64
	At   time.Time
	// Facet marks a value the index counts as well as stores.
	Facet bool
}

Value is one indexed extension property.

func (Value) String

func (v Value) String() string

String is the value as a facet counts it.

Directories

Path Synopsis
Package indextest is the one corpus every searcher is asked about.
Package indextest is the one corpus every searcher is asked about.
Package postgres is the default index and the shared deduplication store.
Package postgres is the default index and the shared deduplication store.
Package s3scan answers queries by reading the archive, with no index at all.
Package s3scan answers queries by reading the archive, with no index at all.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL