Documentation
¶
Overview ¶
Package datastore is the provider-agnostic interface to a NoSQL document store: DynamoDB, Firestore, Cosmos DB, and the engines a self-hosted deployment runs.
A handle is taken by naming the blueprint resource, and the vocabulary for addressing, narrowing and conditioning a write lives here:
orders := resources.Datastore(app, "ordersTable")
rev, err := orders.Get(ctx, datastore.Key{Partition: customerID, Sort: orderID}, &order)
Implementations are per provider and live in their own modules, such as resources/aws.
Index ¶
- Constants
- Variables
- func DeliverRevisions(out any, revisions []Revision)
- func Items[T any](ctx context.Context, store Client, q Query) iter.Seq2[T, error]
- func Scanned[T any](ctx context.Context, store Client, s Scan) iter.Seq2[T, error]
- type AtomicOp
- type BatchOp
- type Client
- type Condition
- func All(conditions ...Condition) Condition
- func Any(conditions ...Condition) Condition
- func Between(name string, low, high any) Condition
- func Contains(name string, value any) Condition
- func Eq(name string, value any) Condition
- func Exists(name string) Condition
- func Ge(name string, value any) Condition
- func Gt(name string, value any) Condition
- func Le(name string, value any) Condition
- func Lt(name string, value any) Condition
- func Ne(name string, value any) Condition
- func NotExists(name string) Condition
- func StartsWith(name, prefix string) Condition
- type Cursor
- type Key
- type Operator
- type Query
- type Revision
- type RevisionReceiver
- type Scan
- type SortCondition
- func SortBetween(low, high string) *SortCondition
- func SortEqual(value string) *SortCondition
- func SortGreater(value string) *SortCondition
- func SortGreaterOrEqual(value string) *SortCondition
- func SortLess(value string) *SortCondition
- func SortLessOrEqual(value string) *SortCondition
- func SortStartsWith(prefix string) *SortCondition
- type Update
- type UpdateKind
- type WriteOption
- type WriteOptions
Constants ¶
const MaxAtomicOps = 100
MaxAtomicOps is the most operations one atomic write may carry.
Enforced by every provider, so a write that works on one store is not refused on another. Exceeding it is ErrTooManyOperations.
const MaxUpdates = 10
MaxUpdates is the most mutations one update may carry.
Enforced by every provider, so an update that works on one store is not refused on another. Exceeding it reports ErrTooManyOperations.
const RevisionField = "_celerity_rev"
RevisionField is the item attribute a provider keeps an item's revision in, on a store that has no revision of its own.
This is reserved, an application must not read, write or index it, and a schema must not declare it. Providers on a store that maintains its own version, such as Firestore's update time or Cosmos DB's ETag, do not use it.
Variables ¶
var ErrConditionFailed = errors.New("celerity: condition not met")
ErrConditionFailed reports that a conditional write was refused because the item was not in the state the condition described.
Often an answer rather than a failure, it is what a handler doing a read, a decision and a write is told when something else got there first, and the ordinary response is to read again and retry.
if errors.Is(err, datastore.ErrConditionFailed) { ... }
var ErrInvalidCondition = errors.New("celerity: invalid condition")
ErrInvalidCondition is a condition the store cannot check, which is what a Condition or a SortCondition built as a struct literal rather than by one of the functions here produces.
An application error rather than a runtime condition, so it is reported before a request is made:
if errors.Is(err, datastore.ErrInvalidCondition) { ... }
Sibling of ErrInvalidUpdate, which is the same mistake made with an update.
var ErrInvalidCursor = errors.New("celerity: cursor did not come from this store")
ErrInvalidCursor is a cursor the store did not produce, passed to Query.Cursor or Scan.Cursor.
A cursor travels out to a client and back, so this reports what a client sent rather than something the application got wrong: a handler mapping errors to a response should answer it as it answers any other bad request.
if errors.Is(err, datastore.ErrInvalidCursor) { ... }
Reported whether the cursor is malformed or merely does not belong to the query it was given to.
var ErrInvalidRevision = errors.New("celerity: revision did not come from a read")
ErrInvalidRevision is a revision that did not come from a read, passed to IfUnchanged. Reported before a request is made, so no write happens without the precondition that was asked for.
var ErrInvalidUpdate = errors.New("celerity: invalid update")
ErrInvalidUpdate is a mutation that doesn't name any fields, or names one in an invalid format. Reported before a request is made.
var ErrNotFound = errors.New("celerity: not found")
ErrNotFound reports that the store doesn't hold an item under the key asked for.
Providers translate to this one and a handler checks for it with errors.Is:
if errors.Is(err, datastore.ErrNotFound) { ... }
Not used by a delete, which is idempotent on every backing store: a key that holds nothing is not a failure.
var ErrNotSupported = errors.New("celerity: not supported by this data store")
ErrNotSupported is a capability the configured store does not have.
Rare, since the contract is the intersection of what these stores can do. Client.Atomically reports it on a store with single-item conditional writes and no multi-item transactions, such as ScyllaDB Alternator.
var ErrTooManyOperations = errors.New("celerity: too many operations in one request")
ErrTooManyOperations is a request carrying more operations than the portable ceiling allows. Reported before a request is made.
var ErrWrongPartition = errors.New("celerity: operation outside the write's partition")
ErrWrongPartition is an operation in an atomic write addressing a partition other than the one the write named. Reported before a request is made.
var Unrevisioned = Revision{/* contains filtered or unexported fields */}
Unrevisioned is the revision of an item that carries none.
An item written before the application adopted a Celerity SDK, or by a data script, a console edit or another service, has no stored revision. A read of one returns this, and it is used like any other revision:
var order Order rev, err := orders.Get(ctx, key, &order) // ... _, err = orders.Put(ctx, key, order, resources.IfUnchanged(rev))
IfUnchanged with it requires that the item still carries no revision, so read-modify-write is correct on items that predate revisions without any migration or back-fill. Every write sets a revision, so an item passes through this state at most once.
Functions ¶
func DeliverRevisions ¶
DeliverRevisions hands each item its own revision, for a provider that has just filled a destination from a read of many items. Revisions are in the order the items were read.
Does nothing unless the item type implements RevisionReceiver.
func Items ¶
Items returns every item a query matches, fetching a page at a time.
for order, err := range datastore.Items[Order](ctx, orders, datastore.Query{
Partition: customerID,
}) {
if err != nil {
return err
}
total += order.Total
}
This is lazy, a page is fetched when the one before it has been read through, and stopping early stops the fetching, so a loop that breaks on the first match pays for one page rather than for the whole partition.
A failure is yielded once, with the zero item, and ends the iteration.
Where a handler is paging for a caller rather than reading to the end, it wants Client.Query itself, which returns one page and the cursor to hand back.
func Scanned ¶
Scanned returns every item in the store, fetching a page at a time.
for order, err := range datastore.Scanned[Order](ctx, orders, datastore.Scan{}) {
if err != nil {
return err
}
...
}
The scan equivalent of Items, and lazy for the same reason. A scan reads every item in the store, so reading one through is the expensive thing this package can be asked to do; stopping early stops the fetching.
Types ¶
type AtomicOp ¶
type AtomicOp struct {
// Key addresses the item the operation applies to.
Key Key
// Item is what a put writes.
Item any
// Updates are the mutations an update applies.
Updates []Update
// Options are the preconditions the operation is subject to.
Options WriteOptions
// contains filtered or unexported fields
}
AtomicOp is one operation in an atomic write, built with AtomicPut, AtomicUpdate or AtomicDelete.
Unlike a BatchOp it carries preconditions, which is why the two are separate types. The exported fields are there for a provider translating one to read.
func AtomicDelete ¶
func AtomicDelete(key Key, opts ...WriteOption) AtomicOp
AtomicDelete removes an item as part of an atomic write.
func AtomicPut ¶
func AtomicPut(key Key, item any, opts ...WriteOption) AtomicOp
AtomicPut writes a whole item as part of an atomic write.
func AtomicUpdate ¶
func AtomicUpdate(key Key, updates []Update, opts ...WriteOption) AtomicOp
AtomicUpdate mutates part of an item as part of an atomic write. As with Client.Update, the item has to already exist.
type BatchOp ¶
type BatchOp struct {
// Key addresses the item the operation applies to.
Key Key
// Item is what a put writes, and is nil for a delete.
Item any
// contains filtered or unexported fields
}
BatchOp is one operation in a batch write, built with PutOp or DeleteOp.
The exported fields are there for a provider translating one to read. Which of the two an operation is cannot be set from outside.
type Client ¶
type Client interface {
// Get fills out with the item under the key and returns its revision, for
// a later write that must require the item has not changed since.
Get(ctx context.Context, key Key, out any) (Revision, error)
// Put writes an item, replacing whatever was under the key.
//
// [If] makes it conditional, which a handler that read, decided and is
// writing back needs. Without one, two invocations running at the same time
// silently lose an update.
//
// Returns the item's new revision, so a sequence of writes needs no read
// between them.
Put(ctx context.Context, key Key, item any, opts ...WriteOption) (Revision, error)
// Delete removes an item, and is idempotent. [If] makes it conditional.
Delete(ctx context.Context, key Key, opts ...WriteOption) error
// Query fills out with one page of the items under a partition, and
// returns the cursor to resume from. Use [Items] to read a query to the
// end.
Query(ctx context.Context, q Query, out any) (Cursor, error)
// Scan fills out with one page of the whole store, and returns the cursor
// to resume from. Use [Scanned] to read a scan to the end.
//
// A scan reads every item; a query reads one partition. Reach for this only
// for genuinely whole-store work, such as an export or a backfill.
Scan(ctx context.Context, s Scan, out any) (Cursor, error)
// BatchGet fills out with the items under the keys, in as few requests as
// the store allows, and returns the keys it could not fetch.
//
// Items come back in no particular order, and a key that holds nothing is
// simply absent rather than an error, so match items to keys by key and
// never by position.
BatchGet(ctx context.Context, keys []Key, out any) ([]Key, error)
// Update mutates parts of an item that already exists, leaving the rest
// alone, and returns the item's new revision.
//
// Built with [Set], [Remove] and [Increment], at most [MaxUpdates] of them.
// The item has to be there: an update of a key that holds nothing reports
// [ErrNotFound] rather than creating it. Use [Client.Put] to create.
//
// Preconditions apply as they do to a put, so an update can be partial and
// conditional in the same request.
Update(ctx context.Context, key Key, updates []Update, opts ...WriteOption) (Revision, error)
// BatchWrite applies puts and deletes, in as few requests as the store
// allows, and returns the operations it could not apply.
//
// This is not atomic. Some operations may succeed while others fail, on every
// store, so a caller that needs all or none cannot build it out of this.
//
// This only supports Puts and deletes. A batch write does not carry update on the stores this
// has to work on. To change part of each of many items, call [Client.Update]
// per item, or [Client.Atomically] where they share a partition and
// all-or-none is wanted as well.
//
// A [BatchOp] carries no preconditions, for the same reason: no store takes
// one on a batch write.
BatchWrite(ctx context.Context, ops []BatchOp) ([]BatchOp, error)
// Atomically applies every operation or none of them.
//
// Built with [AtomicPut], [AtomicUpdate] and [AtomicDelete], at most
// [MaxAtomicOps] of them, each able to carry its own preconditions. A
// precondition that does not hold refuses the whole write with
// [ErrConditionFailed] and nothing is applied.
//
// Every operation has to address the partition named here, which is a
// parameter rather than inferred from the operations so that the
// one-partition rule cannot be broken by accident.
//
// There is no reading inside. For read-modify-write, take the revision from
// an earlier read and pass it as [IfUnchanged] on the operation that needs
// it.
//
// Reports [ErrNotSupported] on a store with no multi-item transactions.
Atomically(ctx context.Context, partition string, ops []AtomicOp) error
}
Client is a NoSQL document store (e.g. DynamoDB, Firestore, Cosmos DB).
type Condition ¶
type Condition struct {
// Op is the test. Always set; a zero Condition is refused by the provider
// rather than quietly matching everything.
Op Operator
// Name is the attribute tested, and is empty for [All] and [Any].
Name string
// Value is what the attribute is tested against, and is unset for
// [Exists], [NotExists], [All] and [Any].
Value any
// High is the upper bound of [Between], whose lower bound is Value.
High any
// Of are the conditions [All] and [Any] combine, and nest as deeply as
// they are written.
Of []Condition
}
Condition is a test on an item, for a write that should only happen if the item is in a particular state as well as queries.
Built with the functions below rather than by hand:
resources.All(
resources.Eq("status", "open"),
resources.Gt("version", 3),
)
The exported fields are there for a provider translating one to read. Only operators every supported store can do are here; a store's own are reached through its provider package.
What a condition costs ¶
The meaning of these ports everywhere. What they cost does not, which is worth knowing before a hot path depends on one.
Exists and NotExists are a precondition every store takes as part of the write itself, so they are one request everywhere. Testing an attribute's value is one request on DynamoDB, which evaluates it server-side, and costs an extra read on Firestore and Cosmos DB, neither of which has a condition on properties.
So an insert that must not overwrite is portable and free. An optimistic update is not: prefer IfUnchanged, which is one request per step everywhere.
func NotExists ¶
NotExists matches an item that does not have the attribute.
What an insert that must not overwrite is written as, on the store's own key:
err := orders.Put(ctx, key, order, resources.If(resources.NotExists("orderId")))
The write is refused with ErrConditionFailed where the item is already there, which is the difference between creating and replacing.
func StartsWith ¶
StartsWith matches a string attribute beginning with a prefix.
type Cursor ¶
type Cursor string
Cursor is a position in a listing, to resume it from.
Thi is opaque, what is in one belongs to the store that produced it, and only that store can read it back. A string, so a handler can hand it to a client as the token for the next page and take it back on the next request. Empty means a listing reached its end.
type Operator ¶
type Operator string
Operator is a test that a condition applies.
const ( OpEqual Operator = "eq" OpNotEqual Operator = "ne" OpLess Operator = "lt" OpLessOrEqual Operator = "le" OpGreater Operator = "gt" OpGreaterOrEqual Operator = "ge" OpBetween Operator = "between" OpStartsWith Operator = "startsWith" OpContains Operator = "contains" OpExists Operator = "exists" // OpNotExists is what an insert that must not overwrite is written as. OpNotExists Operator = "notExists" // The two that combine conditions rather than test an attribute. OpAnd Operator = "and" OpOr Operator = "or" )
type Query ¶
type Query struct {
// Partition is the value every matching item shares.
Partition string
// Sort narrows the query to part of the partition. Nil reads all of it.
//
// Built with [SortBetween] and the other Sort functions. This is what the
// store uses to read less, rather than reading the partition and discarding
// what does not match, so a partition that grows can still be queried
// efficiently.
Sort *SortCondition
// Filter narrows the page to items matching a condition on their own
// fields. Nil applies none.
//
// Applied by the store after the items are read, so it reduces what crosses
// the network and not what the query costs. [Query.Sort] is what reduces the
// cost, so prefer it where the data model allows.
Filter *Condition
// Index names a secondary index to query rather than the table itself.
Index string
// Project lists the fields to return. Empty returns whole items.
//
// The key fields are always returned whether or not they are listed. Like a
// filter, this reduces what crosses the network rather than what the query
// costs.
Project []string
// Descending reads the partition from the end of the sort order rather than
// the start, which with a timestamp sort key is how the newest items are
// read without reading the partition through.
Descending bool
// Limit is the most items one page holds. Zero leaves it to the store.
Limit int
// Cursor resumes a query where an earlier one stopped. See [Cursor].
Cursor string
}
Query describes a data store query.
type Revision ¶
type Revision struct {
// contains filtered or unexported fields
}
Revision identifies the version of an item, so that a write can require that nothing has changed the item since it was read.
This is opaque, what is in one belongs to the store that produced it. Revisions are not ordered and not comparable, and one is only ever handed back to the store it came from, for the item it was read from. Do not persist one, pass one between handlers, or use one obtained from a different data store.
The zero Revision does not come from a read, and is refused by IfUnchanged rather than writing without the precondition that was asked for.
func NewRevision ¶
NewRevision is how a provider reports the revision it read off an item. The empty string means the item carried none, which is Unrevisioned.
type RevisionReceiver ¶
type RevisionReceiver interface {
SetRevision(Revision)
}
RevisionReceiver is implemented by an item type that wants the revision of the item a read produced.
Client.Get returns a revision directly. A query, a scan or a batch get read many items into a slice, which has nowhere to put a second value per element, so an item type that intends to write back what it read says so:
type Order struct {
Total int `json:"total"`
rev datastore.Revision
}
func (o *Order) SetRevision(r datastore.Revision) { o.rev = r }
func (o Order) Revision() datastore.Revision { return o.rev }
Types generated by `celerity schema codegen` implement this, so an application using generated item types has it already. Hand-written types implement it where they need it, which is the case above.
An item type that does not implement this is read exactly as before. The revision is still taken off the item either way, since it is reserved.
type Scan ¶
type Scan struct {
// Filter drops items that do not match a condition on their own fields.
// Nil applies none.
//
// Applied by the store after the items are read, so every item is still
// read and paid for. A filter makes a scan easier to receive, but not
// cheaper to run.
Filter *Condition
// Project lists the fields to return. Empty returns whole items. The key
// fields are always returned whether or not they are listed.
Project []string
// Limit is the most items one page holds. Zero leaves it to the store.
Limit int
// Cursor resumes a scan where an earlier one stopped. See [Cursor].
Cursor string
}
Scan describes a read of the whole data store.
Not a Query without a partition: a scan does not take a sort condition, since there is no partition to narrow, and doesn't name a index. What is left is a filter, a projection and where to resume.
type SortCondition ¶
type SortCondition struct {
// Op is the test, and is one of equal, the four comparisons, between and
// starts-with.
Op Operator
// Value is what the sort key is tested against, and the lower bound of
// [SortBetween].
Value string
// High is the upper bound of [SortBetween].
High string
}
SortCondition narrows a query to part of a partition.
Every store can do all of these, so a query that reads part of a partition reads part of one everywhere. This is the cheaper of the two condition types.
Built by the Sort functions, and carries no attribute name: the attribute is the store's own sort key. The operators are fewer than Condition's because a sort key is ordered, and a test that is not about order cannot use that order to read less.
Without one, a query reads a whole partition. With one it reads the part that was asked for, which is the difference between a cost that grows with the partition and one that grows with the answer.
func SortBetween ¶
func SortBetween(low, high string) *SortCondition
SortBetween matches sort values within a range, ends included.
func SortEqual ¶
func SortEqual(value string) *SortCondition
SortEqual matches one sort value, which addresses part of a composite key without naming the item.
func SortGreater ¶
func SortGreater(value string) *SortCondition
SortGreater matches sort values above a value.
func SortGreaterOrEqual ¶
func SortGreaterOrEqual(value string) *SortCondition
SortGreaterOrEqual matches sort values at or above a value.
func SortLess ¶
func SortLess(value string) *SortCondition
SortLess matches sort values below a value.
func SortLessOrEqual ¶
func SortLessOrEqual(value string) *SortCondition
SortLessOrEqual matches sort values at or below a value.
func SortStartsWith ¶
func SortStartsWith(prefix string) *SortCondition
SortStartsWith matches sort values beginning with a prefix, which is how a hierarchy held in a sort key is read one level at a time.
type Update ¶
type Update struct {
Kind UpdateKind
// Path names the field, with a dot between the segments of a nested one:
// "status", or "profile.theme".
//
// Object fields only. An array element cannot be addressed, since not every
// store can change one by index. To change an array, read the item and
// write it back with [IfUnchanged].
//
// A field whose own name contains a dot is not addressable either: it
// cannot be told apart from a path.
Path string
// Value is what [Set] writes. Unused by the other two.
Value any
// By is what [Increment] adds. Unused by the other two.
By float64
}
Update is one mutation of an item, built with Set, Remove or Increment.
The exported fields are there for a provider translating one to read.
func Increment ¶
Increment adds to the number at a path, and subtracts where the amount is negative. A field the item does not have counts as zero.
Applied by the store, so two handlers incrementing the same field do not overwrite one another the way a read, an addition and a write would.
type UpdateKind ¶
type UpdateKind string
UpdateKind is which of the three supported mutations an Update is.
const ( UpdateSet UpdateKind = "set" UpdateRemove UpdateKind = "remove" UpdateIncrement UpdateKind = "increment" )
type WriteOption ¶
type WriteOption func(*WriteOptions)
WriteOption configures a write.
func If ¶
func If(condition Condition) WriteOption
If refuses a write unless the item is in the state described.
This is what makes a read-then-write safe. A handler that reads an item, decides and writes has no way of knowing the item did not change in between, and a condition on the value it read turns the lost update into ErrConditionFailed, which it can retry.
err := orders.Put(ctx, key, updated, resources.If(resources.Eq("version", was)))
func IfUnchanged ¶
func IfUnchanged(revision Revision) WriteOption
IfUnchanged refuses a write unless the item still carries the revision a read returned, which is the portable way to do read-modify-write: one request per step on every store, where a condition on the item's own fields costs an extra read on some of them.
var order Order
rev, err := orders.Get(ctx, key, &order)
if err != nil {
return err
}
order.Status = "paid"
if _, err := orders.Put(ctx, key, order, resources.IfUnchanged(rev)); err != nil {
if errors.Is(err, resources.ErrConditionFailed) {
return retry // something got there first
}
return err
}
A revision that did not come from a read is refused with ErrInvalidRevision before a request is made, rather than quietly writing without a precondition.
What this detects depends on the store. Where the store maintains the revision itself, it catches every writer. Where a provider maintains it in RevisionField, it catches only writers that go through a Celerity SDK, so a write made with a provider's own SDK is invisible to it.
type WriteOptions ¶
type WriteOptions struct {
// Condition that has to be true of the item already there, or the write is refused.
Condition *Condition
// Revision is the revision the item has to still carry, or the write is
// refused. Nil when the write is not conditional on one.
Revision *Revision
}
WriteOptions is the resolved configuration for a write.
func ResolveWriteOptions ¶
func ResolveWriteOptions(opts []WriteOption) WriteOptions
ResolveWriteOptions applies write options, for a provider implementing a write.