datastore

package
v0.2.37 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 20, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package datastore emulates the altengine datastore data plane: a per-namespace JSON document store with a bounded, index-served query/aggregate/transaction engine. It keeps one SQLite database per (instance, namespace) so queries, keyset cursors, and unique-index semantics behave the same as the hosted service.

Index

Constants

View Source
const (
	CollectionsPage = 200
	CollectionsMax  = 1000
)

Collection listing page sizes, as hosted.

Variables

This section is empty.

Functions

func DecodeNs

func DecodeNs(seg string) string

DecodeNs is decodeNs, exported for other packages that route on a {ns} segment.

It is exported because the ADMIN routes need it too and did not have it: they passed the raw segment to Open(), so browsing the default namespace opened a namespace literally named "_default" — a different, always-empty store. The data browser showed no collections, and a stray "_default" appeared in the namespace list beside the real one.

func ToFieldGroups

func ToFieldGroups(m map[string][][]identity.Filter) map[string][][]Filter

ToFieldGroups converts a rule-engine field->groups map (masks / write gates) into datastore form.

func ToGroups

func ToGroups(gs [][]identity.Filter) [][]Filter

ToGroups converts rule-engine OR-of-AND groups into datastore filter groups.

func ValidateAggregateRequest

func ValidateAggregateRequest(v any) error

ValidateAggregateRequest rejects an aggregate body whose shape the engine cannot walk.

func ValidateConfig

func ValidateConfig(cfg map[string]any) error

ValidateConfig is the datastore instance's config check on a write, with the hosted rules.

func ValidateQueryRequest

func ValidateQueryRequest(v any) error

ValidateQueryRequest rejects a query body whose shape the engine cannot walk.

Types

type AggGroup

type AggGroup struct {
	Group   map[string]any     `json:"group"`
	Metrics map[string]float64 `json:"metrics"`
}

type AggregateRequest

type AggregateRequest struct {
	Where   []Filter `json:"where"`
	Group   []string `json:"group"`
	Metrics []Metric `json:"metrics"`
	Order   []Order  `json:"order"`
	Limit   int      `json:"limit"`
}

type AggregateResult

type AggregateResult struct {
	Groups      []AggGroup     `json:"groups"`
	AutoIndexed *AutoIndexInfo `json:"auto_indexed,omitempty"`
}

type AutoIndexInfo

type AutoIndexInfo struct {
	Fields      []string `json:"fields"`
	RowsWritten int      `json:"rows_written"`
}

type Filter

type Filter struct {
	Field string `json:"field"`
	Op    string `json:"op"`
	Value any    `json:"value"`
}

func ToFilters

func ToFilters(fs []identity.Filter) []Filter

ToFilters converts rule-engine filters into datastore filters.

type Handler

type Handler struct {
	Reg  *control.Registry
	Auth *auth.Store
	Mgr  *Manager
	// Ident resolves end-user identity tokens (nil = API keys only).
	Ident *identity.Service
	// Live publishes change events for the datastore→channel live bridge (nil = off).
	Live LivePublisher
	// Triggers receives committed writes for the instance's trigger rules (nil = off).
	Triggers TriggerSink
}

Handler serves the datastore data plane.

func NewHandler

func NewHandler(reg *control.Registry, a *auth.Store, mgr *Manager) *Handler

NewHandler builds a datastore handler.

func (*Handler) Register

func (h *Handler) Register(mux *http.ServeMux)

Register mounts the datastore routes on mux.

func (*Handler) WithIdentity

func (h *Handler) WithIdentity(svc *identity.Service) *Handler

WithIdentity enables end-user identity tokens (and their row rules) on this handler.

func (*Handler) WithLive

func (h *Handler) WithLive(p LivePublisher) *Handler

WithLive enables the datastore→channel live bridge.

func (*Handler) WithTriggers added in v0.2.35

func (h *Handler) WithTriggers(t TriggerSink) *Handler

WithTriggers enables write triggers.

type IndexRow

type IndexRow struct {
	ID         int64    `json:"id"`
	Collection string   `json:"collection"`
	Fields     []string `json:"fields"`
	Unique     bool     `json:"unique"`
}

type Join

type Join struct {
	As         string `json:"as"`
	Collection string `json:"collection"`
	LocalField string `json:"local_field"`
}

type LiveConfig

type LiveConfig struct {
	ChannelInstance string
	Collections     map[string]string // collection -> keyBy field path ("" = no partition)
}

LiveConfig is an instance's `live` config blob.

type LiveDoc

type LiveDoc struct {
	Key  string
	Data json.RawMessage
}

LiveDoc is a committed document change, carried to the live bridge.

type LivePublisher

type LivePublisher interface {
	Publish(instanceID, channel string, data json.RawMessage) (int, error)
}

LivePublisher publishes one message to a channel instance.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager owns the per-(instance,namespace) SQLite databases.

func NewManager

func NewManager(dir string) *Manager

NewManager creates a datastore manager. dir "" (or memory) keeps everything in RAM.

func (*Manager) Drop

func (m *Manager) Drop(instanceID, namespace string) (bool, error)

Drop deletes a namespace: closes its handle, forgets it, and (in file mode) removes the database file. Returns whether the namespace existed.

func (*Manager) DropInstance

func (m *Manager) DropInstance(instanceID string)

DropInstance closes and removes every namespace database an instance owns (used when the instance is deleted).

func (*Manager) NamespaceEntries

func (m *Manager) NamespaceEntries(instanceID string) []NamespaceEntry

NamespaceEntries is Namespaces with each one's creation time.

The time lives in the namespace's own database, so it survives a restart. A database that is on disk and has not been opened this session is opened to read it, which is the only way to answer — and at an emulator's scale costs nothing worth saving.

func (*Manager) Namespaces

func (m *Manager) Namespaces(instanceID string) []string

Namespaces lists namespaces known for an instance (opened this session, plus, in file mode, any *.db already on disk).

func (*Manager) Open

func (m *Manager) Open(instanceID, namespace, autoID string) (*Store, error)

Open returns a Store for (instance, namespace).

type Metric

type Metric struct {
	Fn    string `json:"fn"`
	Field string `json:"field"`
	As    string `json:"as"`
}

type NamespaceEntry

type NamespaceEntry struct {
	Namespace string `json:"namespace"`
	Created   int64  `json:"created_at"`
}

NamespaceEntry is one namespace and when it was created — the hosted listing's row.

type Order

type Order struct {
	Field string `json:"field"`
	Dir   string `json:"dir"`
}

type PutDoc

type PutDoc struct {
	Key  any             `json:"key"`
	Data json.RawMessage `json:"data"`
}

type QueryRequest

type QueryRequest struct {
	Where    []Filter `json:"where"`
	Order    []Order  `json:"order"`
	Limit    int      `json:"limit"`
	Cursor   string   `json:"cursor"`
	KeysOnly bool     `json:"keys_only"`
	Join     []Join   `json:"join"`
}

type QueryResult

type QueryResult struct {
	Documents []StoredDoc `json:"documents,omitempty"`
	Keys      []string    `json:"keys,omitempty"`
	Cursor    *string     `json:"cursor"`
	// Metering plumbing — the rows the query read, behind its billed reads. Never part of a
	// response, as hosted.
	RowsRead    int            `json:"-"`
	AutoIndexed *AutoIndexInfo `json:"auto_indexed,omitempty"`
}

func (*QueryResult) Reads

func (r *QueryResult) Reads() int

Reads is the billed read count behind RowsRead, which the console is shown.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a handle to one namespace's database plus its instance settings.

func (*Store) Aggregate

func (s *Store) Aggregate(collection string, req AggregateRequest, autoIndex bool, orGroups [][]Filter) (*AggregateResult, error)

Aggregate runs a grouped metrics query. Group dimensions are guarded like sort keys. orGroups carries the caller's row-level read rules as OR-of-AND groups: 0/1 group ANDs into the where, ≥2 groups inject an inline `AND ((g1) OR (g2))` — aggregates count each row once, so no merge/de-dup is needed (unlike a query). Each branch is independently index-served.

func (*Store) BatchGet

func (s *Store) BatchGet(collection string, keys []any) ([]StoredDoc, error)

BatchGet returns the found documents for the given keys (order unspecified).

func (*Store) BatchGetScoped

func (s *Store) BatchGetScoped(collection string, keys []any, groups [][]Filter, fieldReads map[string][][]Filter) ([]StoredDoc, error)

BatchGetScoped is BatchGet with the caller's row-level read groups AND per-field read masks applied server-side: a row the caller may not read is omitted (never leaked), and a masked field is projected out of the rows they can read.

func (*Store) Collections

func (s *Store) Collections(after string, limit int) ([]string, *string, error)

Collections lists a page of the collection names in the namespace, in name order, after `after`. The cursor is the last name served, or nil when there are no more.

A skip-scan, as hosted: each step is one seek to the first name after the previous one, so a page costs one seek per collection on it however many documents each holds.

func (*Store) CreateIndex

func (s *Store) CreateIndex(collection string, fields []string, unique bool) (int, error)

CreateIndex declares an index and, when unique, creates a backing SQLite unique index. Returns rows written (the one-time build cost, approximated by the collection size).

func (*Store) Delete

func (s *Store) Delete(collection string, keys []any) (int, error)

Delete removes documents by key, returning the number removed.

func (*Store) DeleteScoped

func (s *Store) DeleteScoped(collection string, keys []any, match [][]Filter) (int, error)

DeleteScoped removes documents whose EXISTING row satisfies the delete rule's match. A row that doesn't match is refused outright (rather than silently skipped) so a client gets a clear 403 instead of a confusing "deleted: 0".

func (*Store) DropIndex

func (s *Store) DropIndex(collection string, id int64) (bool, error)

func (*Store) Get

func (s *Store) Get(collection, key string) (*StoredDoc, error)

Get returns one document by key, or a NOT_FOUND error.

func (*Store) ListIndexes

func (s *Store) ListIndexes(collection string) ([]IndexRow, error)

func (*Store) Put

func (s *Store) Put(collection string, docs []PutDoc) ([]string, int, error)

Put inserts or replaces documents, returning their keys in input order.

func (*Store) PutScoped

func (s *Store) PutScoped(collection string, docs []PutDoc, create, update identity.WritePolicy) ([]string, []LiveDoc, error)

PutScoped inserts or replaces documents under a create/update rule policy, returning the committed keys and their final (post-stamp) bodies.

func (*Store) Query

func (s *Store) Query(collection string, req QueryRequest, autoIndex bool, readGroups [][]Filter) (*QueryResult, error)

Query runs a QueryRequest. autoIndex controls whether an INDEX_REQUIRED failure auto-creates the suggested index and retries. readGroups carries the caller's row-level read rules as OR-of-AND groups: 0/1 group ANDs into the query, ≥2 groups run the multi-query (UNION) merge — each branch independently index-served.

func (*Store) RunTransaction

func (s *Store) RunTransaction(ops []TxnOp) ([]any, int, error)

RunTransaction applies operations atomically within the namespace.

func (*Store) RunTransactionChanges added in v0.2.35

func (s *Store) RunTransactionChanges(ops []TxnOp, collect bool) ([]any, int, []WriteChange, error)

RunTransactionChanges is RunTransaction that, with collect, also reports what each write did to each document — for a trigger watching the collections it touches. A delete of a document that did not exist is no change.

type StoredDoc

type StoredDoc struct {
	Key     string          `json:"key"`
	Data    json.RawMessage `json:"data"`
	Created int64           `json:"created,omitempty"`
	Updated int64           `json:"updated,omitempty"`
	Joins   map[string]any  `json:"joins,omitempty"`
}

StoredDoc is the wire shape of a document.

func MaskFields

func MaskFields(docs []StoredDoc, fieldReads map[string][][]Filter) []StoredDoc

MaskFields projects restricted fields OUT of docs the caller may read at the row level: for each field in fieldReads, remove it from a doc that doesn't satisfy that field's mask groups (nil groups = always visible). Returns NEW StoredDocs only where a field was actually removed — the input docs are never mutated. Fields are top-level; each field's mask is evaluated against the doc's ORIGINAL data (removals are decided before any deletion is applied).

type TriggerSink added in v0.2.35

type TriggerSink interface {
	// DatastoreWatches reports whether any rule could match a write to this collection — asked
	// before the write, so a collection nobody watches pays nothing for the bookkeeping.
	DatastoreWatches(in *control.Instance, namespace, collection string) bool
	// DatastoreWrite hands over a committed write's changes. It must not block.
	DatastoreWrite(in *control.Instance, namespace string, changes []WriteChange, writerDepth int)
}

TriggerSink receives committed writes for an instance's trigger rules.

type TxnOp

type TxnOp struct {
	Op         string             `json:"op"`
	Collection string             `json:"collection"`
	Key        any                `json:"key"`
	Data       json.RawMessage    `json:"data"`
	Set        map[string]any     `json:"set"`
	Increment  map[string]float64 `json:"increment"`
	Remove     []string           `json:"remove"`
	Upsert     bool               `json:"upsert"`
	Exists     bool               `json:"exists"`
}

type WriteChange added in v0.2.35

type WriteChange struct {
	Collection string
	Key        string
	Op         string // create | update | delete
	Doc        json.RawMessage
	Before     json.RawMessage
}

WriteChange is what one write did to one document. Before is the document it replaced, where the write already reads it (row-scoped writes, transactions).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL