myjson

package module
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 11, 2023 License: Apache-2.0 Imports: 28 Imported by: 0

README

myjson GoDoc

Coverage

███    ███ ██    ██      ██ ███████  ██████  ███    ██ 
████  ████  ██  ██       ██ ██      ██    ██ ████   ██ 
██ ████ ██   ████        ██ ███████ ██    ██ ██ ██  ██ 
██  ██  ██    ██    ██   ██      ██ ██    ██ ██  ██ ██ 
██      ██    ██     █████  ███████  ██████  ██   ████ 
                                                       

MyJSON is an embedded relational document store built on top of pluggable key value storage

go get -u github.com/autom8ter/myjson

Use Case

Build powerful applications on top of simple key value storage.

Features:

Architecture
Feature Description Implemented
Single Node (in-memory) Run the embedded database with no persistance on a single node [x]
Single Node (on-disk) Run the embedded database with persistance on a single node [x]
Distributed (distributed kv-store) Run the embedded database with horizontal scalability using tikv distributed key value storage [x]
Database
Feature Description Implemented
JSON Documents Records are stored as JSON documents [x]
Collections Records are stored in Collections which can hold any number of JSON documents [x]
Collection Schema Collections define a JSON Schema which enforces the schema of JSON documents in a collection [x]
Transactions Cross Collection transactions can be used to persist/rollback changes to the database [x]
Change Streams Built in Change-Data-Capture collection can be queried & streamed for triggering realtime events [x]
Scripting Javascript scripts can be executed with full access to database functionality [x]
Triggers Javascript triggers can be configured at the collection level to add custom business logic based on when events occur [x]
Migrations Built in support for atomic database migrations written in javascript [x]
Relationships Built in support for relationships with foreign keys - Joins and cascade deletes are also supported [x]
Secondary Indexes Multi-field secondary indexes may be used to boost query performance (eq/gt/lt/gte/lte) [x]
Unique Fields Unique fields can be configured which ensure the uniqueness of a field value in a collection [x]
Complex Queries Complex queries can be executed with support for select/where/join/having/orderby/groupby/limit/page clauses [x]
Aggregate Queries Complex aggregate queries can be executed for analytical purposes [x]
Storage Providers
Provider Description Implemented
Badger persistant, embedded LSM database written in Go [x]
Tikv persistant, distributed LSM database written in Rust [x]
RocksDB persistant, embedded LSM database written in C++

Getting Started

go get -u github.com/autom8ter/myjson

Before getting started, take a look at the examples and Godoc

Opening a database instance
Single Node in Memory (badger)
db, err := myjson.Open(context.Background(), "badger", map[string]any{
	"storage_path": "",
})
Single Node w/ Persistance (badger)
db, err := myjson.Open(context.Background(), "badger", map[string]any{
	"storage_path": "./tmp",
})
Multi Node w/ Persistance (tikv)
db, err := myjson.Open(context.Background(), "tikv", map[string]any{
    "pd_addr":    []string{"http://pd0:2379"},
    "redis_addr": "localhost:6379",
    "redis_user": "admin", //change me
    "redis_password": "123232", //change me
})
Configuring a database instance

Collection schemas can be configured at runtime or at startup. Collection schemas are declarative - any changes to indexing or validation happen within the database when ConfigureCollection is called


var (
    //go:embed account.yaml
    accountSchema string
    //go:embed user.yaml
    userSchema string
    //go:embed task.yaml
    taskSchema string
)

if err := db.ConfigureCollection(ctx, []byte(accountSchema)); err != nil {
	panic(err)
}
if err := db.ConfigureCollection(ctx, []byte(userSchema)); err != nil {
	panic(err)
}
if err := db.ConfigureCollection(ctx, []byte(taskSchema)); err != nil {
	panic(err)
}
Working with JSON documents
Creating a JSON document
document, err := myjson.NewDocumentFrom(map[string]any{
    "name": "acme.com",
})
doc := myjson.NewDocument()
doc.Set("name", "acme.com")
Setting JSON values
doc := myjson.NewDocument()
doc.Set("name", "acme.com")

SJSON syntax is supported: https://github.com/tidwall/sjson#path-syntax

doc := myjson.NewDocument()
doc.Set("contact.email", "info@acme.com")
Getting JSON values
doc := myjson.NewDocument()
doc.Set("name", "acme.com")

GJSON syntax is supported: https://github.com/tidwall/sjson#path-syntax

value := doc.Get("contact.email")

additional GJSON modifiers are available:

  • @camelCase - convert a json string field to camel case doc.Get("project|@camelCase")
  • @snakeCase - convert a json string field to snake case doc.Get("project|@snakeCase")
  • @kebabCase - convert a json string field to kebab case doc.Get("project|@kebabCase")
  • @replaceAll - replace a substring within a json string field with another string
  • @unix - get the unix timestamp of the json time field doc.Get("timestamp|@unix")
  • @unixMilli - get the unix millisecond timestamp of the json time field doc.Get("timestamp|@unixMilli")
  • @unixNano - get the unix nanosecond timestamp of the json time field doc.Get("timestamp|@unixNano")
  • @dateTrunc - truncate a date to day, month, or year ex: doc.GetString("timestamp|@dateTrunc:month"), doc.GetString("timestamp|@dateTrunc:year"), doc.GetString("timestamp|@dateTrunc:day")
Transactions

Most database functionality is made available via the Tx interface which has read/write methods across 1-many collections.

Writable
if err := db.Tx(ctx, kv.TxOpts{IsReadOnly: false}, func(ctx context.Context, tx myjson.Tx) error {
	// do stuff ...tx.Set(...)
	// return error to rollback
	// return no error to commit
}
Read Only
if err := db.Tx(ctx, kv.TxOpts{IsReadOnly: true}, func(ctx context.Context, tx myjson.Tx) error {
	// ...tx.Get(...)
}
Adding documents to a collection
if err := db.Tx(ctx, kv.TxOpts{}, func(ctx context.Context, tx myjson.Tx) error {
    doc := myjson.NewDocument()
    doc.Set("name", "acme.com")
	id, err := tx.Create(ctx, "account", document)
	if err != nil {
		return err
    }
}
Queries
results, err := tx.Query(ctx, "user", myjson.Q().
    Select(myjson.Select{Field: "*"}).
	OrderBy(myjson.OrderBy{Field: "age", Direction: myjson.OrderByDirectionDesc}).
    Query())
Joins

1-many joins are

results, err := db.Query(ctx, "user", myjson.Q().
    Select(
        myjson.Select{Field: "acc._id", As: "account_id"},
        myjson.Select{Field: "acc.name", As: "account_name"},
		myjson.Select{Field: "_id", As: "user_id"},
    ).
    Join(myjson.Join{
        Collection: "account",
        On: []myjson.Where{
            {
				Field: "_id",
				Op:    myjson.WhereOpEq,
                Value: "$account_id", //self reference the account_id on the user document
            },
    },
        As: "acc",
    }).
Query())
Iterating through documents in a collection
_, err := tx.ForEach(ctx, "user", myjson.ForEachOpts{}, func(d *myjson.Document) (bool, error) {
    fmt.Println(d)
    return true, nil
})
Reading documents in a collection
doc, err := tx.Get(ctx, "user", "$id")
Change Streams
Stream Changes in a given collection

CDC persistance must be enabled for change streams to work. See the database Options for more info.

ctx, cancel := context.WithCancel(context.Background())
defer cancel()
err := db.ChangeStream(ctx, "user", func(cdc myjson.CDC) (bool, error) {
    fmt.Println(cdc)
    return true, nil
})
Aggregation
query := myjson.Query{
	Select: []myjson.Select{
		{
            Field: "account_id",
		},
		{
            Field:     "age",
            Aggregate: myjson.AggregateFunctionSum,
            As:        "age_sum",
		},
	},
	GroupBy: []string{"account_id"},
}
results, err := db.Query(ctx, "user", query)
Triggers

add a triggers block to your JSON schema ex: update timestamp on set/update/create with a javascript expression

triggers:
  set_timestamp:
    order: 1
    events:
      - on_create
      - on_update
      - on_set
    script: |
      doc.set('timestamp', new Date().toISOString())

javascript variables are injected at runtime:

  • doc - the JSON document that is being changed
  • db - the global database instance(all methods are available lowercased)
  • ctx - the context when the trigger was called
  • metadata - the context metadata when the script is called
  • tx - the current transaction instance
Scripts

Scripts are javascript expressions/functions that can be called in Go - this may be used when embedded/dynamic functionality is required.

getAccountScript:

function setAccount(ctx, db, params) {
    db.tx(ctx, {isReadOnly: false}, (ctx, tx) => {
        tx.set(ctx, "account", params.doc)
    })
}

execute with custom paramaters:

id := ksuid.New().String()
doc, err := myjson.NewDocumentFrom(map[string]any{
    "_id":  id,
    "name": gofakeit.Company(),
})
_, err = db.RunScript(ctx, "setAccount", getAccountScript, map[string]any{
    "doc": doc,
})

javascript variables are injected at runtime:

  • db - the global database instance(all methods are available lowercased)
  • ctx - the context when the script is called
  • metadata - the context metadata when the script is called
  • newDocument - function to intialize a new JSON document
  • newDocumentFrom - function to initialize a new JSON document from a javascript object
Example JSON Schema

MyJSON JSON schemas are a modification of the JSON Schema specification

custom attributes include:

  • x-collection: a root level field for specifying the name of the collection(required)
  • x-foreign: a property level block for specifying a relationship to another collection
    • foreign keys are automatically indexed
  • x-primary: a property level field for specifying the primary key(required)
    • primary key is automatically indexed
  • x-index: a property level block for specifying multi-field secondary indexes
  • x-triggers: a root level block for specifying triggers on document changes
type: object
# x-collection specifies the name of the collection the object will be stored in
x-collection: user
# required specifies the required attributes
required:
  - _id
  - name
  - age
  - contact
  - gender
  - account_id
properties:
  _id:
    type: string
    description: The user's id.
    # x-primary indicates that the property is the primary key for the object - only one primary key may be specified
    x-primary: true
  name:
    type: string
    description: The user's name.
  contact:
    type: object
    properties:
      email:
        type: string
        description: The user's email.
        x-unique: true
  age:
    description: Age in years which must be equal to or greater than zero.
    type: integer
    minimum: 0
  account_id:
    type: string
    # x-foreign indicates that the property is a foreign key - foreign keys are automatically indexed
    x-foreign:
      # foreign key collection
      collection: account
      # foreign key field(must be primary key)
      field: _id
      # automatically delete records when foreign key is deleted
      cascade: true
    # x-index specifies a secondary index which can have 1-many fields
    x-index:
      account_email_idx:
        additional_fields:
          - contact.email
  language:
    type: string
    description: The user's first language.
    x-index:
      language_idx: { }
  gender:
    type: string
    description: The user's gender.
    enum:
      - male
      - female
  timestamp:
    type: string
  annotations:
    type: object

# triggers are javascript functions that execute based on certain events
x-triggers:
  # name of the trigger
  set_timestamp:
    # order determines the order in which the functions are executed - lower ordered triggers are executed first
    order: 1
    # events configures the trigger to execute on certain events
    events:
      - on_create
      - on_update
      - on_set
    # script is the javascript to execute
    script: |
      doc.set('timestamp', new Date().toISOString())

Tikv Setup Guide (full scale)

WIP - see tikv foldeer w/ Makefile for running tikv locally

Contributing

Install Dependencies

go mod download

Checkout Branch

git checkout -b ${issueNumber}

Run Tests

go test -race -covermode=atomic -coverprofile=coverage.out ./...

Run Benchmarks

go test -bench=. -benchmem -run=^#

Lint Repository

golangci-lint run

Check Coverage

go tool cover -func=coverage.out

Documentation

Index

Examples

Constants

View Source
const (
	// Create creates a document
	Create = "create"
	// Set sets a document's values in place
	Set = "set"
	// Update updates a set of fields on a document
	Update = "update"
	// Delete deletes a document
	Delete = "delete"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Action

type Action string

Action is an action that causes a mutation to the database

type AggregateFunction

type AggregateFunction string

AggregateFunction is an agggregate function applied to a list of documents

const AggregateFunctionCount AggregateFunction = "count"

AggregateFunctionCount gets the count of a set of documents

const AggregateFunctionMax AggregateFunction = "max"

AggregateFunctionMax gets the max value in a set of documents

const AggregateFunctionMin AggregateFunction = "min"

AggregateFunctionMin gets the min value in a set of documents

const AggregateFunctionSum AggregateFunction = "sum"

AggregateFunctionSum gets the sum of values in a set of documents

type CDC

type CDC struct {
	// ID is the unique id of the cdc
	ID string `json:"_id" validate:"required"`
	// Collection is the collection the change was applied to
	Collection string `json:"collection" validate:"required"`
	// Action is the action applied to the document
	Action Action `json:"action" validate:"required,oneof='create' 'update' 'delete' 'set'"`
	// DocumentID is the ID of the document that was changed
	DocumentID string `json:"documentID" validate:"required"`
	// Diff is the difference between the previous and new version of the document
	Diff []JSONFieldOp `json:"diff,omitempty"`
	// Timestamp is the nanosecond timestamp the cdc was created at
	Timestamp int64 `json:"timestamp" validate:"required"`
	// Metadata is the context metadata when the change was made
	Metadata *Metadata `json:"metadata" validate:"required"`
}

CDC is a change data capture object used for tracking changes to documents over time

type CollectionSchema

type CollectionSchema interface {
	// Collection is the collection name
	Collection() string
	// ValidateDocument validates the input document against the collection's JSON schema
	ValidateDocument(ctx context.Context, doc *Document) error
	// Indexing returns a copy the schemas indexing
	Indexing() map[string]Index
	// PrimaryIndex returns the collection's primary index
	PrimaryIndex() Index
	// PrimaryKey returns the collection's primary key
	PrimaryKey() string
	// GetPrimaryKey gets the document's primary key
	GetPrimaryKey(doc *Document) string
	// SetPrimaryKey sets the document's primary key
	SetPrimaryKey(doc *Document, id string) error
	// RequireQueryIndex returns whether the collection requires that queries are appropriately indexed
	RequireQueryIndex() bool
	// Properties returns a map of the schema's properties
	Properties() map[string]SchemaProperty
	// PropertyPaths returns a flattened map of the schema's properties - nested properties will be keyed in dot notation
	PropertyPaths() map[string]SchemaProperty
	// Triggers returns a map of triggers keyed by name that are assigned to the collection
	Triggers() []Trigger
	// IsReadOnly returns whether the collection is read only
	IsReadOnly() bool
	// MarshalYAML returns the collection schema as yaml bytes
	MarshalYAML() ([]byte, error)
	// UnmarshalYAML refreshes the collection schema with the given json bytes
	UnmarshalYAML(bytes []byte) error
	json.Marshaler
	json.Unmarshaler
}

CollectionSchema is a database collection configuration

type Command

type Command struct {
	Collection string    `json:"collection" validate:"required"`
	Action     Action    `json:"action" validate:"required,oneof='create' 'update' 'delete' 'set'"`
	Document   *Document `json:"document" validate:"required"`
	Timestamp  int64     `json:"timestamp" validate:"required"`
	Metadata   *Metadata `json:"metadata" validate:"required"`
}

Command is a command executed against the database that causes a change in state

type DBOpt

type DBOpt func(d *defaultDB)

DBOpt is an option for configuring a collection

func WithJavascriptOverrides

func WithJavascriptOverrides(overrides map[string]any) DBOpt

WithJavascriptOverrides adds global variables or methods to the embedded javascript vm(s)

func WithOptimizer

func WithOptimizer(o Optimizer) DBOpt

WithOptimizer overrides the default query optimizer provider

func WithPersistCDC

func WithPersistCDC(persist bool) DBOpt

WithPersistCDC configures the database to persist all change-data-capture entries so that features like time-travel and change streaming are possible

type Database

type Database interface {
	// Collections returns a list of collection names that are registered in the collection
	Collections(ctx context.Context) []string
	// ConfigureCollection overwrites a single database collection configuration
	ConfigureCollection(ctx context.Context, collectionSchemaBytes []byte) error
	// GetSchema gets a collection schema by name (if it exists)
	GetSchema(ctx context.Context, collection string) CollectionSchema
	// HasCollection reports whether a collection exists in the database
	HasCollection(ctx context.Context, collection string) bool
	// DropCollection drops the collection and it's indexes from the database
	DropCollection(ctx context.Context, collection string) error
	// Tx executes the given function against a new transaction.
	// if the function returns an error, all changes will be rolled back.
	// otherwise, the changes will be commited to the database
	Tx(ctx context.Context, opts kv.TxOpts, fn TxFunc) error
	// NewTx returns a new transaction. a transaction must call Commit method in order to persist changes
	NewTx(opts kv.TxOpts) (Txn, error)
	// ChangeStream streams changes to documents in the given collection. CDC Persistence must be enabled to use this method.
	ChangeStream(ctx context.Context, collection string, fn func(cdc CDC) (bool, error)) error
	// Get gets a single document by id
	Get(ctx context.Context, collection, id string) (*Document, error)
	// ForEach scans the optimal index for a collection's documents passing its filters.
	// results will not be ordered unless an index supporting the order by(s) was found by the optimizer
	// Query should be used when order is more important than performance/resource-usage
	ForEach(ctx context.Context, collection string, opts ForEachOpts, fn ForEachFunc) (Explain, error)
	// Query queries a list of documents
	Query(ctx context.Context, collection string, query Query) (Page, error)
	// Get gets 1-many document by id(s)
	BatchGet(ctx context.Context, collection string, ids []string) (Documents, error)
	// RunScript executes a javascript function within the script
	// The following global variables will be injected: 'db' - a database instance, 'ctx' - the context passed to RunScript, and 'params' - the params passed to RunScript
	RunScript(ctx context.Context, function string, script string, params map[string]any) (any, error)
	// RunMigrations runs migration scripts against the database. If a migration(id) has already successfully run, it will do nothing.
	// Migrations are run sequentially in the order they are provided
	// Migration executions will if an error is encountered
	// The following global variables will be injected into the script: 'db' - a Database instance, 'ctx' - the context passed to RunMigrations
	RunMigrations(ctx context.Context, migrations ...Migration) error
	// RawKV returns the database key value provider - it should be used with caution and only when standard database functionality is insufficient.
	RawKV() kv.DB
	// Serve serves the database over the given transport
	Serve(ctx context.Context, t Transport) error
	// NewDoc creates a new document builder instance
	NewDoc() *DocBuilder
	// Close closes the database
	Close(ctx context.Context) error
}

Database is a NoSQL database built on top of key value storage

func Open

func Open(ctx context.Context, provider string, providerParams map[string]any, opts ...DBOpt) (Database, error)

Open opens a new database instance from the given config

type DocBuilder added in v0.1.0

type DocBuilder struct {
	// contains filtered or unexported fields
}

DocBuilder is a builder for creating a document

func D added in v0.1.0

func D() *DocBuilder

D creates a new document builder

func (*DocBuilder) Doc added in v0.1.0

func (b *DocBuilder) Doc() *Document

Doc returns the document

func (*DocBuilder) Err added in v0.1.0

func (b *DocBuilder) Err() error

Err returns an error if one exists

func (*DocBuilder) From added in v0.1.0

func (b *DocBuilder) From(d *Document) *DocBuilder

From creates a new document builder from the input document

func (*DocBuilder) Set added in v0.1.0

func (b *DocBuilder) Set(values map[string]any) *DocBuilder

DocBuilder sets the key value pairs on the document

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document is a concurrency safe JSON document

func NewDocument

func NewDocument() *Document

NewDocument creates a new json document

func NewDocumentFrom

func NewDocumentFrom(value any) (*Document, error)

NewDocumentFrom creates a new document from the given value - the value must be json compatible

Example
package main

import (
	"fmt"

	"github.com/autom8ter/myjson"
)

func main() {
	doc, _ := myjson.NewDocumentFrom(map[string]any{
		"name": "autom8ter",
	})
	fmt.Println(doc.String())
}
Output:
{"name":"autom8ter"}

func NewDocumentFromBytes

func NewDocumentFromBytes(json []byte) (*Document, error)

NewDocumentFromBytes creates a new document from the given json bytes

func (*Document) Bytes

func (d *Document) Bytes() []byte

Bytes returns the document as json bytes

func (*Document) Clone

func (d *Document) Clone() *Document

Clone allocates a new document with identical values

func (*Document) Del

func (d *Document) Del(field string) error

Del deletes a field from the document. SJSON systax is supported For information on sjson syntax, check out https://github.com/tidwall/sjson#path-syntax

func (*Document) DelAll

func (d *Document) DelAll(fields ...string) error

DelAll deletes a field from the document. SJSON systax is supported For information on sjson syntax, check out https://github.com/tidwall/sjson#path-syntax

func (*Document) Diff

func (d *Document) Diff(before *Document) []JSONFieldOp

Diff calculates a json diff between the document and the input document

func (*Document) Encode

func (d *Document) Encode(w io.Writer) error

Encode encodes the json document to the io writer

func (*Document) Exists

func (d *Document) Exists(field string) bool

Exists returns true if the fieldPath has a value in the json document. Exists has GJSON syntax support // For information on gjson syntax, check out https://github.com/tidwall/gjson/blob/master/SYNTAX.md

func (*Document) FieldPaths

func (d *Document) FieldPaths() []string

FieldPaths returns the paths to fields & nested fields in dot notation format

func (*Document) Get

func (d *Document) Get(field string) any

Get gets a field on the document. Get has GJSON syntax support For information on gjson syntax, check out https://github.com/tidwall/gjson/blob/master/SYNTAX.md

func (*Document) GetArray

func (d *Document) GetArray(field string) []any

GetArray gets an array field on the document. Get has GJSON syntax support For information on gjson syntax, check out https://github.com/tidwall/gjson/blob/master/SYNTAX.md

func (*Document) GetBool

func (d *Document) GetBool(field string) bool

GetBool gets a bool field value on the document. GetBool has GJSON syntax support For information on gjson syntax, check out https://github.com/tidwall/gjson/blob/master/SYNTAX.md

func (*Document) GetFloat

func (d *Document) GetFloat(field string) float64

GetFloat gets a float64 field value on the document. GetFloat has GJSON syntax support For information on gjson syntax, check out https://github.com/tidwall/gjson/blob/master/SYNTAX.md

func (*Document) GetString

func (d *Document) GetString(field string) string

GetString gets a string field value on the document. GetString has GJSON syntax support For information on gjson syntax, check out https://github.com/tidwall/gjson/blob/master/SYNTAX.md

func (*Document) GetTime

func (d *Document) GetTime(field string) time.Time

GetTime gets a time.Time field value on the document. GetTime has GJSON syntax support For information on gjson syntax, check out https://github.com/tidwall/gjson/blob/master/SYNTAX.md

func (*Document) MarshalJSON

func (d *Document) MarshalJSON() ([]byte, error)

MarshalJSON satisfies the json Marshaler interface

func (*Document) Merge

func (d *Document) Merge(with *Document) error

Merge merges the document with the provided document.

func (*Document) MergeJoin

func (d *Document) MergeJoin(with *Document, alias string) error

MergeJoin merges the document with the input document with each key from the input document prefixed with the given alias

func (*Document) Overwrite

func (d *Document) Overwrite(values map[string]any) error

Overwrite resets the document with the given values. SJSON systax is supported For information on sjson syntax, check out https://github.com/tidwall/sjson#path-syntax

func (*Document) Scan

func (d *Document) Scan(value any) error

Scan scans the json document into the value

func (*Document) Set

func (d *Document) Set(field string, val any) error

Set sets a field on the document. Set has SJSON syntax support (dot notation) For information on sjson syntax, check out https://github.com/tidwall/sjson#path-syntax

func (*Document) SetAll

func (d *Document) SetAll(values map[string]any) error

SetAll sets all fields on the document. SJSON systax is supported For information on sjson syntax, check out https://github.com/tidwall/sjson#path-syntax

func (*Document) String

func (d *Document) String() string

String returns the document as a json string

func (*Document) UnmarshalJSON

func (d *Document) UnmarshalJSON(bytes []byte) error

UnmarshalJSON satisfies the json Unmarshaler interface

func (*Document) Valid

func (d *Document) Valid() bool

Valid returns whether the document is valid

func (*Document) Value

func (d *Document) Value() map[string]any

Value returns the document as a map

func (*Document) Where

func (d *Document) Where(wheres []Where) (bool, error)

Where executes the where clauses against the document and returns true if it passes the clauses. If the value of a where clause is prefixed with $. it will compare where.field to the same document's $.{field}.

type Documents

type Documents []*Document

Documents is an array of documents

func (Documents) Filter

func (documents Documents) Filter(predicate func(document *Document, i int) bool) Documents

Filter applies the filter function against the documents

func (Documents) ForEach

func (documents Documents) ForEach(fn func(next *Document, i int))

ForEach applies the function to each document in the documents

func (Documents) Map

func (documents Documents) Map(mapper func(t *Document, i int) *Document) Documents

Map applies the mapper function against the documents

func (Documents) Slice

func (documents Documents) Slice(start, end int) Documents

Slice slices the documents into a subarray of documents

type EventType

type EventType string
const (
	OnSet    EventType = "on_set"
	OnDelete EventType = "on_delete"
	OnUpdate EventType = "on_update"
	OnCreate EventType = "on_create"
)

type Explain added in v0.1.0

type Explain struct {
	// Collection
	Collection string `json:"collection"`
	// Index is the index the query optimizer chose
	Index Index `json:"index"`
	// MatchedFields are the fields that matched the index
	MatchedFields []string `json:"matched_fields"`
	// MatchedValues are the values that were matched to the index
	MatchedValues map[string]any `json:"matched_values,omitempty"`
	// SeekFields indicates that the given fields will be seeked
	SeekFields []string `json:"seek,omitempty"`
	// SeekValues are the values to seek
	SeekValues map[string]any `json:"seek_values,omitempty"`
	// Reverse indicates that the index should be scanned in reverse
	Reverse bool `json:"reverse,omitempty"`
}

Explain is the optimizer's output for a query

type ForEachFunc

type ForEachFunc func(d *Document) (bool, error)

ForEachFunc returns false to stop scanning and an error if one occurred

type ForEachOpts

type ForEachOpts struct {
	Where []Where `json:"where,omitempty"`
	Join  []Join  `json:"join,omitempty"`
}

ForEachOpts are options when executing db.ForEach against a collection

type ForeignKey

type ForeignKey struct {
	// Collection is the foreign collection
	Collection string `json:"collection"`
	// Cascade indicates that the document should be deleted when the foreign key is deleted
	Cascade bool `json:"cascade"`
}

ForeignKey is a reference/relationship to another collection by primary key

type Index

type Index struct {
	// Name is the indexes unique name in the collection
	Name string `json:"name" validate:"required,min=1"`
	// Fields to index - order matters
	Fields []string `json:"fields" validate:"required,min=1"`
	// Unique indicates that it's a unique index which will enforce uniqueness
	Unique bool `json:"unique"`
	// Unique indicates that it's a primary index
	Primary bool `json:"primary"`
	// ForeignKey indecates that it's an index for a foreign key
	ForeignKey *ForeignKey `json:"foreign_key,omitempty"`
}

Index is a database index used to optimize queries against a collection

type JSONFieldOp

type JSONFieldOp struct {
	// Path is the path to the field within the document
	Path string `json:"path"`
	// Op is the operation applied
	Op JSONOp `json:"op"`
	// Value is the value applied with the operation
	Value any `json:"value,omitempty"`
	// BeforeValue is the value before the operation was applied
	BeforeValue any `json:"beforeValue,omitempty"`
}

JSONFieldOp is an operation against a JSON field

type JSONOp

type JSONOp string

JSONOp is an json field operation type

const (
	// JSONOpRemove removes a field from a json document
	JSONOpRemove JSONOp = "remove"
	// JSONOpAdd adds a json field to a json document
	JSONOpAdd JSONOp = "add"
	// JSONOpReplace replaces an existing field in a json document
	JSONOpReplace JSONOp = "replace"
)

type Join

type Join struct {
	Collection string  `json:"collection" validate:"required"`
	On         []Where `json:"on" validate:"required,min=1"`
	As         string  `json:"as,omitempty"`
}

Join is a join against another collection

type Metadata

type Metadata struct {
	// contains filtered or unexported fields
}

Metadata holds key value pairs associated with a go Context

func GetMetadata

func GetMetadata(ctx context.Context) (*Metadata, bool)

GetMetadata gets metadata from the context if it exists

func NewMetadata

func NewMetadata(tags map[string]any) *Metadata

NewMetadata creates a Metadata with the given tags

Example
package main

import (
	"fmt"

	"github.com/autom8ter/myjson"
)

func main() {
	var orgID = "123"
	md := myjson.NewMetadata(map[string]any{})
	md.SetNamespace(orgID)
	bytes, _ := md.MarshalJSON()
	fmt.Println(string(bytes))
}
Output:
{"namespace":"123"}

func (*Metadata) Del

func (m *Metadata) Del(key string)

Del deletes a key from the metadata

func (*Metadata) Exists

func (m *Metadata) Exists(key string) bool

Exists returns true if the key exists in the metadata

func (*Metadata) Get

func (m *Metadata) Get(key string) (any, bool)

Get gets a key from the metadata if it exists

func (*Metadata) GetNamespace

func (m *Metadata) GetNamespace() string

GetNamespace gets the namespace from the metadata, or 'default' if it does not exist Data belonging to different namespaces are stored/indexed separately, though collections exist across namespaces

func (*Metadata) Map

func (m *Metadata) Map() map[string]any

Map returns the metadata keyvalues as a map

func (*Metadata) MarshalJSON

func (m *Metadata) MarshalJSON() ([]byte, error)

MarshalJSON returns the metadata values as json bytes

func (*Metadata) Set

func (m *Metadata) Set(key string, value any)

Set sets a key value pair on the metadata

func (*Metadata) SetAll

func (m *Metadata) SetAll(data map[string]any)

SetAll sets the key value fields on the metadata

func (*Metadata) SetNamespace

func (m *Metadata) SetNamespace(value string)

SetNamespace sets the namespace on the context metadata Data belonging to different namespaces are stored/indexed separately, though collections exist across namespaces

func (*Metadata) String

func (m *Metadata) String() string

String return a json string of the context

func (*Metadata) ToContext

func (m *Metadata) ToContext(ctx context.Context) context.Context

ToContext adds the metadata to the input go context

func (*Metadata) UnmarshalJSON

func (m *Metadata) UnmarshalJSON(bytes []byte) error

UnmarshalJSON decodes the metadata from json bytes

type Migration

type Migration struct {
	// ID is the unique id of the cdc
	ID string `json:"_id" validate:"required"`
	// Timestamp is the nanosecond timestamp the cdc was created at
	Timestamp int64 `json:"timestamp"`
	// Script is the script's content
	Script string `json:"script" validate:"required"`
	// Dirty indicates whether the migration failed or not
	Dirty bool `json:"dirty"`
	// An error message if one was encountered
	Error string `json:"error"`
}

Migration is an atomic database migration

type Optimizer

type Optimizer interface {
	// Optimize selects the optimal index to use based on the given where clauses
	Optimize(c CollectionSchema, where []Where) (Explain, error)
}

Optimizer selects the best index from a set of indexes based on where clauses

type OrderBy

type OrderBy struct {
	Direction OrderByDirection `json:"direction" validate:"oneof='desc' 'asc'"`
	Field     string           `json:"field"`
}

OrderBy indicates how to order results returned from a query

type OrderByDirection

type OrderByDirection string

OrderByDirection is the direction of an order by clause

const OrderByDirectionAsc OrderByDirection = "asc"

OrderByDirectionAsc is ascending order

const OrderByDirectionDesc OrderByDirection = "desc"

OrderByDirectionDesc is descending order

type Page

type Page struct {
	// Documents are the documents that make up the page
	Documents Documents `json:"documents"`
	// Next page
	NextPage int `json:"next_page"`
	// Document count
	Count int `json:"count"`
	// Stats are statistics collected from a document aggregation query
	Stats PageStats `json:"stats,omitempty"`
}

Page is a page of documents

type PageStats

type PageStats struct {
	// ExecutionTime is the execution time to get the page
	ExecutionTime time.Duration `json:"execution_time,omitempty"`
	// Explain is the optimizer's output for the query that returned a page
	Explain *Explain `json:"explain,omitempty"`
}

PageStats are statistics collected from a query returning a page

type PropertyIndex

type PropertyIndex struct {
	AdditionalFields []string `json:"additional_fields,omitempty"`
}

PropertyIndex is an index attached to a json schema property

type Query

type Query struct {
	// Select selects fields - at least 1 select is required.
	// 1 select with Field: * gets all fields
	Select []Select `json:"select" validate:"min=1,required"`
	// Join joins the results to another collection
	Join []Join `json:"join,omitempty" validate:"dive"`
	// Where filters results. The optimizer will select the appropriate index based on where clauses
	Where []Where `json:"where,omitempty" validate:"dive"`
	// GroupBy groups results by a given list of fields
	GroupBy []string `json:"groupBy,omitempty"`
	// Page is the page of results - it is used with Limit for pagination
	Page int `json:"page" validate:"min=0"`
	// Limit is used to limit the number of results returned
	Limit int `json:"limit,omitempty" validate:"min=0"`
	// OrderBy orders results
	OrderBy []OrderBy `json:"orderBy,omitempty" validate:"dive"`
	// Having applies a final filter after any aggregations have occured
	Having []Where `json:"having,omitempty" validate:"dive"`
}

Query is a query against the NOSQL database

func (Query) String

func (q Query) String() string

String returns the query as a json string

func (Query) Validate

func (q Query) Validate(ctx context.Context) error

Validate validates the query and returns a validation error if one exists

type QueryBuilder

type QueryBuilder struct {
	// contains filtered or unexported fields
}

QueryBuilder is a utility for creating queries via chainable methods

func Q

func Q() *QueryBuilder

Q creates a new QueryBuilder instance

Example
package main

import (
	"fmt"

	"github.com/autom8ter/myjson"
)

func main() {
	query := myjson.Q().
		Select(myjson.Select{
			Field: "*",
		}).
		Where(myjson.Where{
			Field: "description",
			Op:    myjson.WhereOpContains,
			Value: "testing",
		}).Query()
	fmt.Println(query.String())
}
Output:
{"select":[{"field":"*"}],"where":[{"field":"description","op":"contains","value":"testing"}],"page":0}

func (*QueryBuilder) GroupBy

func (q *QueryBuilder) GroupBy(groups ...string) *QueryBuilder

GroupBy adds the GroupBy clause(s) to the query

func (*QueryBuilder) Having

func (q *QueryBuilder) Having(where ...Where) *QueryBuilder

Having adds the Where clause(s) to the query - they execute after all other clauses have resolved

func (*QueryBuilder) Join

func (q *QueryBuilder) Join(join ...Join) *QueryBuilder

Join adds the Join clause(s) to the query

func (*QueryBuilder) Limit

func (q *QueryBuilder) Limit(limit int) *QueryBuilder

Limit adds the Limit clause(s) to the query

func (*QueryBuilder) OrderBy

func (q *QueryBuilder) OrderBy(ob ...OrderBy) *QueryBuilder

OrderBy adds the OrderBy clause(s) to the query

func (*QueryBuilder) Page

func (q *QueryBuilder) Page(page int) *QueryBuilder

Page adds the Page clause(s) to the query which controls pagination results

func (*QueryBuilder) Query

func (q *QueryBuilder) Query() Query

Query returns the built query

func (*QueryBuilder) Select

func (q *QueryBuilder) Select(fields ...Select) *QueryBuilder

Select adds the SelectFiel(s) to the query

func (*QueryBuilder) Where

func (q *QueryBuilder) Where(where ...Where) *QueryBuilder

Where adds the Where clause(s) to the query

type SchemaProperty

type SchemaProperty struct {
	// Primary indicates the property is the primary key
	Primary bool `json:"x-primary,omitempty"`
	// Name is the name of the property
	Name string `json:"name" validate:"required"`
	// Description is the description of the property
	Description string `json:"description,omitempty"`
	// Type is the type of the property
	Type string `json:"type" validate:"required"`
	// Path is a dot notation path to the property
	Path string `json:"path" validate:"required"`
	// Unique indicates the field value is unique
	Unique bool `json:"x-unique,omitempty"`
	// ForeignKey is a relationship to another collection
	ForeignKey *ForeignKey `json:"x-foreign,omitempty"`
	// Index is a secondary index mapped by index name
	Index map[string]PropertyIndex `json:"x-index,omitempty"`
	// Properties are object properties
	Properties map[string]SchemaProperty `json:"properties,omitempty"`
}

SchemaProperty is a property belonging to a JSON Schema

type Select

type Select struct {
	Aggregate AggregateFunction `json:"aggregate,omitempty" validate:"oneof='count' 'max' 'min' 'sum'"`
	As        string            `json:"as,omitempty"`
	Field     string            `json:"field"`
}

Select is a field to select

type Stream

type Stream[T any] interface {
	// Broadcast broadcasts the entity to the channel
	Broadcast(ctx context.Context, channel string, msg T)
	// Pull pulls entities off of the given channel as they are broadcast
	Pull(ctx context.Context, channel string, fn func(T) (bool, error)) error
}

Stream broadcasts and subscribes to entities. A Stream can be implemented in memory, or with message-queue services like sqs/pubsub/rabbitmq/nats/splunk/etc

type Transport added in v0.1.0

type Transport interface {
	Serve(ctx context.Context, db Database) error
}

Transport serves the database over a network (optional for integration with different transport mechanisms)

type Trigger

type Trigger struct {
	// Name is the unique name of the trigger
	Name string `json:"name" validate:"required"`
	// Order is used to sort triggers into an array where the lower order #s are executed before larger order #s
	Order int `json:"order"`
	// Events is an array of events that the trigger executes on
	Events []EventType `json:"events" validate:"min=1,required"`
	// Script is the javascript script to execute
	Script string `json:"script" validate:"required"`
}

Trigger is a javasript function executed after a database event occurs

type Tx

type Tx interface {
	// Query executes a query against the database
	Query(ctx context.Context, collection string, query Query) (Page, error)
	// Get returns a document by id
	Get(ctx context.Context, collection string, id string) (*Document, error)
	// Create creates a new document - if the documents primary key is unset, it will be set as a sortable unique id
	Create(ctx context.Context, collection string, document *Document) (string, error)
	// Update updates a value in the database
	Update(ctx context.Context, collection, id string, document map[string]any) error
	// Set sets the specified key/value in the database
	Set(ctx context.Context, collection string, document *Document) error
	// Delete deletes the specified key from the database
	Delete(ctx context.Context, collection string, id string) error
	// ForEach scans the optimal index for a collection's documents passing its filters.
	// results will not be ordered unless an index supporting the order by(s) was found by the optimizer
	// Query should be used when order is more important than performance/resource-usage
	ForEach(ctx context.Context, collection string, opts ForEachOpts, fn ForEachFunc) (Explain, error)
	// CDC returns the change data capture array associated with the transaction.
	// CDC's are persisted to the cdc collection when the transaction is commited.
	CDC() []CDC
	// DB returns the transactions underlying database
	DB() Database
}

Tx is a database transaction interface - it holds the methods used while using a transaction

type TxFunc

type TxFunc func(ctx context.Context, tx Tx) error

TxFunc is a function executed against a transaction - if the function returns an error, all changes will be rolled back. Otherwise, the changes will be commited to the database

type Txn

type Txn interface {
	// Commit commits the transaction to the database
	Commit(ctx context.Context) error
	// Rollback rollsback all changes made to the datbase
	Rollback(ctx context.Context) error
	// Close closes the transaction - it should be deferred after
	Close(ctx context.Context)
	Tx
}

Txn is a database transaction interface - it holds the methods used while using a transaction + commit,rollback,and close functionality

type Where

type Where struct {
	Field string      `json:"field" validate:"required"`
	Op    WhereOp     `json:"op" validate:"oneof='eq' 'neq' 'gt' 'gte' 'lt' 'lte' 'contains' 'containsAny' 'containsAll' 'in'"`
	Value interface{} `json:"value" validate:"required"`
}

Where is a filter against documents returned from a query

type WhereOp

type WhereOp string

WhereOp is an operation belonging to a where clause

const WhereOpContains WhereOp = "contains"

WhereOpContains checks if a string field contains subtext

const WhereOpContainsAll WhereOp = "containsAll"

WhereOpContainsAll is a check whether an array contains all of the given array values

const WhereOpContainsAny WhereOp = "containsAny"

WhereOpContainsAny is a check whether an array contains any of the given array values

const WhereOpEq WhereOp = "eq"

WhereOpEq is an equality check.

const WhereOpGt WhereOp = "gt"

WhereOpGt is a check whether a value is greater than another

const WhereOpGte WhereOp = "gte"

WhereOpGte is a check whether a value is greater than or equal to another

const WhereOpHasPrefix WhereOp = "hasPrefix"

WhereOpHasPrefix is a check whether a string value has a prefix

const WhereOpHasSuffix WhereOp = "hasSuffix"

WhereOpHasSuffix is a check whether a string value has a suffix

const WhereOpIn WhereOp = "in"

WhereOpIn is a check whether a value is one of a list of values

const WhereOpLt WhereOp = "lt"

WhereOpLt is a check whether a value is less than another

const WhereOpLte WhereOp = "lte"

WhereOpLte is a check whether a values is less than or equal to another

const WhereOpNeq WhereOp = "neq"

WhereOpNeq is a non-equality check

const WhereOpRegex WhereOp = "regex"

WhereOpRegex is a check whtether a string value matches a regex expression

Directories

Path Synopsis
kv

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL