Documentation
¶
Overview ¶
Package index is the projection a search reads.
The archive is the record: objects under Object Lock, accounted for by a signed digest chain. The index is a derived thing, and the whole design depends on keeping that distinction: it may be lost, rebuilt from the prefixes, or replaced by a different implementation, and none of that touches what the trail says. What it must never do is disagree, which is why the rows are derived here rather than in each implementation.
The interfaces are narrow on purpose. Three implementations ship — memory, an object-storage scan and Postgres — and an interface two of them cannot both satisfy honestly is an interface that has been shaped around one database.
Index ¶
- Constants
- Variables
- func Matches(r Row, filter []Conjunction) bool
- type Boundary
- type Capabilities
- type Conjunction
- type Facet
- type FacetDelta
- type FacetValue
- type Fields
- type Indexer
- type Kind
- type Memory
- func (m *Memory) Capabilities() Capabilities
- func (m *Memory) Count(profile, tenant, field, value string, hour time.Time) int64
- func (m *Memory) Counts(profile string) []FacetDelta
- func (m *Memory) Facets(_ context.Context, q Query, fields []string, limit int) ([]Facet, error)
- func (m *Memory) Get(_ context.Context, profile, id string) (Row, Provenance, error)
- func (m *Memory) Index(_ context.Context, profile string, rows []Row) error
- func (m *Memory) Len(profile string) int
- func (m *Memory) Purge(_ context.Context, profile string, before time.Time, what Scope) error
- func (m *Memory) Row(profile, id string) (Row, bool)
- func (m *Memory) Rows(profile string) []Row
- func (m *Memory) Search(_ context.Context, q Query) (Page, error)
- type ObjectAt
- type Op
- type Page
- type PathPredicate
- type Predicate
- type Provenance
- type Query
- type Row
- type Scope
- type Searcher
- type SortBy
- type TimePredicate
- type Value
Constants ¶
const ( FieldAction = "action" FieldOperation = "operation" FieldOutcome = "outcome" FieldSource = "source" FieldActorKind = "actor_kind" FieldTargetType = "target_type" // DataField prefixes an extension property's pointer, so that a data // property called action cannot be confused with the record's own. DataField = "data:" )
Core fields every row is counted by. An extension property adds its own, named for its pointer, which is what lets an application's own data become a facet in the viewer without any code.
const ( SortOccurredAt = "occurred_at" SortRecordedAt = "recorded_at" SortID = "id" SortAction = "action" SortTenantID = "tenant_id" SortSource = "source" )
The fields a page may be ordered by. Every ordering ends with the identifier, so that a keyset boundary is always total and a page cannot repeat or skip a row whose sort values tie with its neighbour's.
Variables ¶
var ErrNotFound = errors.New("no such record")
ErrNotFound is a Get for a record the searcher does not hold. Every searcher wraps it, so the service can answer "no such record" rather than mistaking an absence for an outage: an outage tells a client to retry, and retrying will never find a record that is not there.
Functions ¶
func Matches ¶
func Matches(r Row, filter []Conjunction) bool
Matches reports whether a row satisfies a filter: the OR of conjunctions, each of which is an AND of predicates. No filter matches everything.
It is exported because it is the definition of what a query means, and a searcher that worked it out again would be a second opinion on the contract. Anything implementing Searcher over data it holds itself should use this rather than reimplement the operators.
Types ¶
type Boundary ¶
type Boundary struct {
Values []string
ID string
// RecordedAt and Sequence are what a tail advances on.
RecordedAt time.Time
Sequence uint64
}
Boundary is where a page ended: the sort values of its last row, and that row's identifier.
Keyset rather than an offset. An offset re-reads everything before it, so a deep page costs more than a shallow one, and a row inserted meanwhile shifts every page after it — in a trail that is appended to constantly, that means a poller silently skipping records.
type Capabilities ¶
type Capabilities struct {
// FreeText is whether the searcher can answer an unstructured query.
FreeText bool
// Facets is whether it can count values.
Facets bool
// DataPredicates is whether it can filter on extension properties.
DataPredicates bool
// MaxConjunctions bounds the OR. Beyond it a query is refused rather than
// turned into work nobody bounded.
MaxConjunctions int
// SortFields are the orderings it can answer.
SortFields []string
}
Capabilities is what an implementation can do.
type Conjunction ¶
type Conjunction struct {
ID []Predicate
OccurredAt []TimePredicate
RecordedAt []TimePredicate
Source []Predicate
Action []Predicate
Operation []Predicate
Outcome []Predicate
TenantID []Predicate
ActorKind []Predicate
ActorID []Predicate
SubjectKind []Predicate
SubjectID []Predicate
RequestID []Predicate
TraceID []Predicate
ClientAddress []Predicate
ObserverID []Predicate
TargetType []Predicate
TargetID []Predicate
// Data are predicates on the extension properties a catalogue marked
// filterable, addressed by JSON pointer.
Data []PathPredicate
}
Conjunction is a set of predicates that must all hold.
type Facet ¶
type Facet struct {
Field string
Values []FacetValue
}
Facet is one field's counted values.
type FacetDelta ¶
FacetDelta is one increment of the counts table.
func Merge ¶
func Merge(deltas []FacetDelta) []FacetDelta
Merge adds the deltas of several rows together, so that one batch touches each counted value once.
func (FacetDelta) String ¶
func (d FacetDelta) String() string
String describes a delta, for test failures and for the reindex report.
type FacetValue ¶
FacetValue is a value and how many rows carry it.
type Fields ¶
type Fields struct {
// Filter are pointers a query may name in a predicate.
Filter []string
// Facet are pointers a searcher may count. Every facet is filterable.
Facet []string
}
Fields says which extension properties of an action are worth indexing. A catalogue's schema answers this; the index takes the answer rather than the catalogue, so that an implementation of this interface needs neither.
type Indexer ¶
type Indexer interface {
// Index writes rows and the facet counts they imply. A row already there
// changes nothing, including the counts.
Index(ctx context.Context, profile string, rows []Row) error
// Purge forgets what a profile no longer keeps. The archive's objects are
// under a lock and are not affected; this is the index catching up with a
// retention that has already expired, and with the narrower schedule the
// identifying columns are kept on.
Purge(ctx context.Context, profile string, before time.Time, what Scope) error
}
Indexer writes the projection.
Index is idempotent by (profile, id), including the facet counts it implies. Counting is not a separate call, and could not be: counts are additive, so a caller that increments them cannot tell a re-delivered record from a new one. Only the implementation holding the transaction can, by counting exactly the rows its insert actually created.
type Kind ¶
type Kind string
Kind is how a value is stored, which decides what a predicate may do with it.
type Memory ¶
type Memory struct {
Indexed int
// contains filtered or unexported fields
}
Memory holds the projection in one process.
It is what the tests index into and what a deployment small enough to keep its index in memory uses. It is a real implementation, not a stub: it enforces the same idempotency the interface promises, because a memory implementation that quietly double-counts would let a fault reach Postgres before anybody noticed.
func (*Memory) Capabilities ¶
func (m *Memory) Capabilities() Capabilities
Capabilities implements Searcher.
The memory searcher exists so that the interface has two honest implementations from the start. Two is the number that keeps an interface from being a description of one implementation's habits.
func (*Memory) Counts ¶
func (m *Memory) Counts(profile string) []FacetDelta
Counts returns every count of a profile, for tests and for the reindex report.
type Op ¶
type Op string
Op is a comparison.
The operators a predicate may use. There is no regular expression and no substring search: both are unbounded work on a table that only grows, and an index that cannot answer them quickly would answer them slowly instead.
type Page ¶
type Page struct {
Rows []Row
// More says whether the next page would have anything in it. It is answered
// by asking for one row more than the limit, rather than by counting.
More bool
// Next is the boundary for the following page. It is present even on the
// last page, because a tail keeps polling it and a record recorded after
// this page must come back through it.
Next *Boundary
Prev *Boundary
}
Page is what a search returns.
type PathPredicate ¶
type PathPredicate struct {
Path string
Op Op
// Text, Int and At are the value, whichever kind the property has.
Text string
Values []string
Int int64
At time.Time
Kind Kind
}
PathPredicate is a comparison on an indexed extension property.
type Provenance ¶
type Provenance struct {
ObjectKey string
Line int
// Digest is the key of the seal that accounts for the object's hour, when
// there is one. Empty means no seal covers it yet, which is the ordinary
// state of the current hour and a finding in any other.
Digest string
// VerifiedAt is when that seal was last verified clean. Zero means it
// has not been, or the last verification found a problem.
VerifiedAt time.Time
}
Provenance says where a row came from, so that an answer can be checked against the copy the digest chain accounts for rather than believed.
type Query ¶
type Query struct {
Profile string
// Tenants is the grant: the caller may see these and no others. Empty means
// every tenant, which only an operator's grant carries.
Tenants []string
// Filter is OR-joined: a row matches if it matches any conjunction.
Filter []Conjunction
Sort []SortBy
Limit int
// After is the keyset boundary a cursor carries.
After *Boundary
// Backwards pages towards older results.
Backwards bool
}
Query is a closed question.
type Row ¶
type Row struct {
ID string
TenantID string
OccurredAt time.Time
RecordedAt time.Time
Sequence uint64
Source string
Action string
Operation string
Outcome string
TargetTypes []string
TargetIDs []string
// ObjectKey and Line are where the copy is in the archive, which is what
// lets a reader check any answer the index gives against the object the
// digest chain accounts for.
ObjectKey string
Line int
// The identifying columns. They are kept apart because a profile purges
// them on a shorter schedule than the event, and because a store that
// separates them can grant a reader one without the other.
ActorKind string
ActorID string
SubjectKind string
SubjectID string
ClientAddress string
RequestID string
TraceID string
ObserverID string
// Data are the extension properties the action marked filterable, by
// pointer into its data slot.
Data []Value
}
Row is one copy of one record as the index holds it.
It is the copy, not the original: a security copy and a billing copy of one event differ in what they carry, and indexing the original would put back what the split took out.
func (Row) Facets ¶
func (r Row) Facets() []FacetDelta
Facets returns the counts one row implies. Every implementation derives them the same way because they derive them here.
type Searcher ¶
type Searcher interface {
// Search returns a page. The cursor in a request must have come from a
// response to the same query.
Search(ctx context.Context, q Query) (Page, error)
// Facets counts values of the named fields under the same filter.
Facets(ctx context.Context, q Query, fields []string, limit int) ([]Facet, error)
// Get returns one row and where it came from.
Get(ctx context.Context, profile, id string) (Row, Provenance, error)
// Capabilities says what this implementation can do, so that a caller is
// refused rather than quietly given a narrower answer than it asked for.
Capabilities() Capabilities
}
Searcher answers questions about what was written.
It is deliberately narrow, and the query it takes is closed: a caller names fields and operators from a fixed set rather than handing over an expression. That is what lets a grant be AND-ed into every query as one more term, and what lets a second implementation of this interface be honest about what it can and cannot do.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package indextest is the one corpus every searcher is asked about.
|
Package indextest is the one corpus every searcher is asked about. |
|
Package postgres is the default index and the shared deduplication store.
|
Package postgres is the default index and the shared deduplication store. |
|
Package s3scan answers queries by reading the archive, with no index at all.
|
Package s3scan answers queries by reading the archive, with no index at all. |