docstore

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package docstore defines Backyard's schemaless JSON document database (backyard.db). Implementations live in sub-packages (memory, sqlite, postgres, dynamodb); backends.Open builds one from configuration and the conformance package holds the shared test-suite every backend must pass.

Semantics (filters, ordering, missing fields) are specified in docs/spec/SPEC.md §5.2 and must be identical across backends.

Index

Constants

View Source 
const MaxNestingDepth = 100

MaxNestingDepth is the maximum nesting depth of a stored document: the document body is level 1 and every nested object or array adds a level. Deeper documents are rejected with ErrInvalid by every backend. The limit keeps documents well inside what every engine can evaluate (SQLite's JSON functions, including Cloudflare Durable Object SQLite, fail on JSON nested more than 1000 levels, which would make a single stored document break every query of its collection).

View Source 
const SystemSite = "_system"

SystemSite is the reserved site namespace for Backyard's own data ("_system/_users", "_system/_audit"; SPEC §10). It is exempt from the per-site document quota of WithLimits.

View Source 
const TimePrecision = time.Microsecond

TimePrecision is the precision of document timestamps. Every backend stores and returns CreatedAt/UpdatedAt truncated to this precision (PostgreSQL's timestamptz resolution), so documents look identical whichever backend is configured.

Variables

View Source 
var (
	// ErrNotFound: document does not exist.
	ErrNotFound = errors.New("docstore: not found")
	// ErrConflict: a document with this id already exists (Create), or an
	// optimistic version check failed.
	ErrConflict = errors.New("docstore: conflict")
	// ErrTooLarge: document exceeds the configured size limit.
	ErrTooLarge = errors.New("docstore: document too large")
	// ErrQuota: per-site document quota exceeded.
	ErrQuota = errors.New("docstore: quota exceeded")
	// ErrInvalid: invalid collection, id, field path, operator or value.
	ErrInvalid = errors.New("docstore: invalid request")
)
View Source 
var SystemFields = map[string]bool{
	"id": true, "_meta.created_at": true, "_meta.updated_at": true,
	"_meta.created_by": true, "_meta.updated_by": true, "_meta.version": true,
}

SystemFields are the addressable non-data fields.

Functions

func ApplyPatch 

func ApplyPatch(data, patch map[string]any) map[string]any

ApplyPatch returns the shallow merge of patch into data as a new map: keys of patch overwrite keys of data, and a nil value deletes the key. Neither argument is modified; values are not copied.

func CleanNumber 

func CleanNumber(f float64) (float64, error)

CleanNumber returns f in canonical form: negative zero becomes zero (as PostgreSQL's numeric type stores it) and NaN or infinities, which JSON cannot represent, are rejected with ErrInvalid.

func CleanString 

func CleanString(s string) (string, error)

CleanString returns s in the canonical form every backend stores: invalid UTF-8 is coerced the way encoding/json does (each invalid byte becomes U+FFFD), so a value reads back identically whichever backend stores it. Strings containing U+0000 are rejected with ErrInvalid because PostgreSQL cannot store them in jsonb or text; rejecting them everywhere keeps the backends identical.

func CloneData 

func CloneData(m map[string]any) map[string]any

CloneData returns a deep copy of a document body. A nil map yields an empty, non-nil map.

func CloneValue 

func CloneValue(v any) any

CloneValue returns a deep copy of a JSON-like value (maps and slices are copied recursively; scalars are returned as is).

func CompareDocs 

func CompareDocs(a, b *Doc, order []Order) int

CompareDocs orders two documents by order (which Normalize ends with the id tiebreaker). If every key ties, documents are ordered by id ascending.

func CompareValues 

func CompareValues(a any, aok bool, b any, bok bool) int

CompareValues orders two field values (either may be missing) for one sort key in ascending direction.

func FieldValue 

func FieldValue(d *Doc, path string) (any, bool)

FieldValue returns the value addressed by path in d and whether it exists. path is a dot path into d.Data or one of the system fields: "id" (string), "_meta.created_at" / "_meta.updated_at" (time.Time), "_meta.created_by" / "_meta.updated_by" (string) and "_meta.version" (float64).

func IsDesc 

func IsDesc(o Order) bool

IsDesc reports whether o sorts descending (Desc, or Dir "desc" on a query that was not normalised).

func IsTimeField 

func IsTimeField(field string) bool

IsTimeField reports whether field is a timestamp system field, whose values compare chronologically.

func Less 

func Less(a, b *Doc, order []Order) bool

Less reports whether a sorts before b under order.

func Match 

func Match(d *Doc, filters []Filter) bool

Match reports whether d satisfies every filter (logical AND). Filters should come from a query run through PrepareQuery, which validates them and canonicalises their operands exactly as the backends do.

func MatchFilter 

func MatchFilter(d *Doc, f Filter) bool

MatchFilter reports whether d satisfies one filter.

func NewID 

func NewID() string

NewID returns a 26-character, lexicographically time-ordered ULID. IDs generated within the same millisecond are monotonically increasing.

func NormalizeValue 

func NormalizeValue(v any) any

NormalizeValue converts Go numeric types and json.Number to float64 and []string etc. to []any so that backends compare uniformly.

func Now 

func Now() time.Time

Now returns the current time in UTC, truncated to TimePrecision. Values returned within one process are strictly increasing, which keeps the default "creation order" (created_at, id) deterministic even when many documents are written within the same microsecond: if the wall clock has not advanced (or went backwards) the previous value plus one microsecond is returned.

func ParseTime 

func ParseTime(v any) (t time.Time, ok bool)

ParseTime parses a filter value for a timestamp system field (_meta.created_at / _meta.updated_at). It accepts RFC 3339 strings with optional fractional seconds and returns the time truncated to TimePrecision. ok is false for any other value; such a filter value has a different type than the field and therefore matches nothing.

func PrepareData 

func PrepareData(data map[string]any, maxBytes int) (map[string]any, error)

PrepareData strips reserved keys ("id", "_meta") from a write body, normalises numbers and enforces maxBytes on the JSON encoding.

The size is that of the compact encoding backends store, which does not HTML-escape: '<', '>' and '&' count one byte each (json.Marshal would write six), like the Cloudflare edition counts them.

func SortDocs 

func SortDocs(docs []*Doc, order []Order)

SortDocs sorts docs in place by order (stable).

func TruncTime 

func TruncTime(t time.Time) time.Time

TruncTime normalises t to the stored representation: UTC, truncated to TimePrecision.

func ValidateCollection 

func ValidateCollection(name string, system bool) error

ValidateCollection checks a user collection name (system=false) or a system collection name (system=true, must start with "_").

func ValidateField 

func ValidateField(field string) error

ValidateField checks a dot-path field reference (user data path or system field).

func ValidateID 

func ValidateID(id string) error

ValidateID checks a document id.

func ValidateSite 

func ValidateSite(site string) error

ValidateSite checks a site namespace (a site name or "_system").

Types

type Doc 

type Doc struct {
	Site       string
	Collection string
	ID         string
	Data       map[string]any
	CreatedAt  time.Time
	UpdatedAt  time.Time
	CreatedBy  string
	UpdatedBy  string
	Version    int64
}

Doc is a stored document. Data never contains the keys "id" or "_meta".

func CloneDoc 

func CloneDoc(d *Doc) *Doc

CloneDoc returns a deep copy of d.

func (*Doc) At added in v0.2.0 

func (d *Doc) At(r Rev) bool

At reports whether d is at revision r.

func (*Doc) MarshalJSON 

func (d *Doc) MarshalJSON() ([]byte, error)

MarshalJSON renders the flattened wire shape: {"id", ...data, "_meta": {...}}.

func (*Doc) Rev added in v0.2.0 

func (d *Doc) Rev() Rev

Rev returns the revision of d.

func (*Doc) Wire 

func (d *Doc) Wire() map[string]any

Wire returns the flattened wire shape that MarshalJSON encodes: {"id", ...data, "_meta": Meta}. The data values are shared with d.

type Filter 

type Filter struct {
	Field string `json:"field"`
	Op    Op     `json:"op"`
	Value any    `json:"value"`
}

Filter is one where-clause.

type Kind 

type Kind uint8

Kind is the JSON type of a value as seen by the query engine. 

const (
	KindMissing Kind = iota
	KindNull
	KindBool
	KindNumber
	KindString
	KindArray
	KindObject
	KindTime
	KindInvalid
)

Value kinds. KindMissing marks an absent field; KindTime is used for the timestamp system fields; KindInvalid marks a filter value that can never match (for example an unparseable timestamp).

func FieldKind 

func FieldKind(field string) (Kind, bool)

FieldKind returns the kind of a system field ("id", "_meta.*") and true, or (0, false) for user data paths.

func KindOf 

func KindOf(v any) Kind

KindOf returns the kind of a JSON-like Go value.

func Operand 

func Operand(field string, v any) (any, Kind)

Operand converts a filter value for field into the value and kind used for comparisons. For the timestamp system fields an RFC 3339 string becomes a KindTime time.Time; any other value for those fields is KindInvalid.

func (Kind) Rank 

func (k Kind) Rank() int

Rank is the cross-type sort rank of a kind (see the ordering rules above). SQL backends emit the same ranks.

func (Kind) String 

func (k Kind) String() string

String returns the kind's name.

type Meta 

type Meta struct {
	CreatedAt time.Time `json:"created_at"`
	UpdatedAt time.Time `json:"updated_at"`
	CreatedBy string    `json:"created_by"`
	UpdatedBy string    `json:"updated_by"`
	Version   int64     `json:"version"`
}

Meta is the "_meta" object of the wire representation.

type Op 

type Op string

Op is a filter operator. 

const (
	OpEq       Op = "=="
	OpNe       Op = "!="
	OpLt       Op = "<"
	OpLte      Op = "<="
	OpGt       Op = ">"
	OpGte      Op = ">="
	OpIn       Op = "in"
	OpNotIn    Op = "not-in"
	OpContains Op = "contains"
)

Supported operators.

type Order 

type Order struct {
	Field string `json:"field"`
	Desc  bool   `json:"-"`
	Dir   string `json:"dir,omitempty"` // "asc" | "desc" (wire form; Normalize sets Desc)
}

Order is one sort key.

type Page 

type Page struct {
	Docs    []*Doc
	HasMore bool
}

Page is a query result.

func EvalQuery 

func EvalQuery(docs []*Doc, q Query) *Page

EvalQuery executes a prepared query over an in-memory candidate set (all documents of one collection): it filters with Match, sorts with SortDocs and applies offset/limit. docs is reordered in place; the returned page shares the *Doc pointers.

type Query 

type Query struct {
	Where   []Filter `json:"where,omitempty"`
	OrderBy []Order  `json:"order_by,omitempty"`
	Limit   int      `json:"limit,omitempty"`
	Offset  int      `json:"offset,omitempty"`
}

Query selects documents of one collection.

func PrepareQuery 

func PrepareQuery(q Query) (Query, error)

PrepareQuery returns a validated, normalised deep copy of q for execution by a backend. It runs Normalize (filling defaults and appending the id tiebreaker) without capping the limit any further, so it is idempotent on queries the API layer already normalised with database.max_list. The caller's slices are never modified.

Filter operands are canonicalised like stored data (see CleanString and CleanNumber): invalid UTF-8 is coerced, -0 becomes 0, and NaN, infinities and strings containing U+0000 are rejected with ErrInvalid, so every backend sees identical operands.

func (*Query) Normalize 

func (q *Query) Normalize(maxList int) error

Normalize validates q, applies defaults (limit, default order) and appends the id tiebreaker. maxList caps the limit.

type Rev added in v0.2.0 

type Rev struct {
	Version   int64
	CreatedAt time.Time
}

Rev identifies one state of a document: its incarnation (CreatedAt: a document deleted and created again is another one) and its write count within it (Version). UpdateIf and DeleteIf apply only while a document is still at the Rev that was read.

type Store 

type Store interface {
	// Create inserts a new document. If id is empty a ULID is generated.
	// Returns ErrConflict if the id already exists.
	Create(ctx context.Context, site, collection, id string, data map[string]any, actor string) (*Doc, error)
	// Get returns a document or ErrNotFound.
	Get(ctx context.Context, site, collection, id string) (*Doc, error)
	// Put creates or fully replaces a document (Version incremented; CreatedAt/By preserved).
	Put(ctx context.Context, site, collection, id string, data map[string]any, actor string) (*Doc, error)
	// Update shallow-merges patch into an existing document; nil values delete
	// keys. Returns ErrNotFound if missing. Must be atomic w.r.t. concurrent updates.
	Update(ctx context.Context, site, collection, id string, patch map[string]any, actor string) (*Doc, error)
	// UpdateIf is Update applied only while the document is at rev (a
	// compare-and-set): ErrConflict when it has been written since, or
	// deleted and created again; ErrNotFound when it does not exist.
	UpdateIf(ctx context.Context, site, collection, id string, rev Rev, patch map[string]any, actor string) (*Doc, error)
	// Delete removes a document. Returns (false, nil) if it did not exist.
	Delete(ctx context.Context, site, collection, id string) (bool, error)
	// DeleteIf removes a document only while it is at rev, atomically. It
	// reports whether it removed it: (false, nil) when the document does
	// not exist or is at another revision.
	DeleteIf(ctx context.Context, site, collection, id string, rev Rev) (bool, error)
	// Query returns matching documents (q must be Normalize'd).
	Query(ctx context.Context, site, collection string, q Query) (*Page, error)
	// Collections lists non-empty collection names of a site, sorted
	// (including system collections starting with "_"; callers filter).
	Collections(ctx context.Context, site string) ([]string, error)
	// Count returns the number of documents of a site (all collections).
	Count(ctx context.Context, site string) (int64, error)
	// DeleteSite removes every document of a site.
	DeleteSite(ctx context.Context, site string) error
	// Ping checks connectivity.
	Ping(ctx context.Context) error
	// Close releases resources.
	Close() error
}

Store is a document database namespaced by site and collection. Implementations must be safe for concurrent use. Callers validate names and sizes (Validate* helpers) before calling; implementations may re-check.

func WithLimits 

func WithLimits(s Store, maxDocBytes int, maxDocsPerSite int) Store

WithLimits wraps s so that writes enforce a maximum JSON size per document (ErrTooLarge; maxDocBytes <= 0 disables the check) and a maximum number of documents per site (ErrQuota; maxDocsPerSite <= 0 disables it).

The size limit applies to Create and Put bodies and to the merged result of Update. The quota applies to Create and to a Put that creates a new document; Backyard's own namespace (SystemSite: the user cache and the audit log) is exempt. Per-site counts come from s.Count, are cached for a few seconds and kept current locally: every quota check reserves a slot (released again if the write fails or replaces an existing document), so concurrent writers on one instance cannot overshoot the quota, and the count is always re-read before a write is rejected. Writers on several instances may still overshoot it slightly (a soft limit). If both limits are disabled s is returned unchanged.

Source Files

View all Source files

Directories

Path Synopsis
Package backends constructs a docstore.Store from configuration.
 Package backends constructs a docstore.Store from configuration.
Package conformance is the shared semantic test-suite for docstore.Store implementations.
 Package conformance is the shared semantic test-suite for docstore.Store implementations.
Package dynamodb is the docstore.Store for Amazon DynamoDB (and DynamoDB Local / LocalStack for development).
 Package dynamodb is the docstore.Store for Amazon DynamoDB (and DynamoDB Local / LocalStack for development).
internal
dsutil
Package dsutil holds helpers shared by the docstore backends: argument validation, canonicalisation of document bodies and JSON encoding.
 Package dsutil holds helpers shared by the docstore backends: argument validation, canonicalisation of document bodies and JSON encoding.
Package memory is an in-process docstore.Store backed by maps.
 Package memory is an in-process docstore.Store backed by maps.
Package postgres is the docstore.Store for PostgreSQL (Cloud SQL, RDS, Aurora, Azure Database for PostgreSQL Flexible Server, self-hosted…) using pgx/v5 and JSONB.
 Package postgres is the docstore.Store for PostgreSQL (Cloud SQL, RDS, Aurora, Azure Database for PostgreSQL Flexible Server, self-hosted…) using pgx/v5 and JSONB.
Package sqlite is the embedded, single-node docstore.Store backed by SQLite (modernc.org/sqlite, pure Go, driver name "sqlite").
 Package sqlite is the embedded, single-node docstore.Store backed by SQLite (modernc.org/sqlite, pure Go, driver name "sqlite").

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.