crdt

package module
v0.34.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2026 License: BSD-3-Clause Imports: 6 Imported by: 0

README

go-crdt/crdt

crdt — collaborative text editing in pure Go

CI Go Reference coverage license

github.com/go-crdt/crdt is a pure-Go, CGO=0 conflict-free replicated data type for plain text: any number of replicas may edit the same document at once, offline, over an unreliable transport, and every replica ends up with the same text. No server arbitrates, and no operation is ever transformed.

It compiles to js/wasm, so the same merge logic runs on the server and in every browser tab — one implementation, one source of truth, no second codebase in JavaScript to keep in step. The whole test suite, convergence properties included, runs under js/wasm in CI.

Zero dependencies.

Packages

Package Purpose
crdt the replicated text document (Doc), a replicated sequence of values (List), a replicated map (Map) and a document of named parts (Composite) — operations, version vectors, snapshots
crdt/awareness ephemeral presence — who is here and where their cursor is
crdt/structured a shared substrate for co-editing structured documents over one core — a block document (Blocks), formatted text (RichText), a value replicas may disagree about (MultiRegister), a set of names (Set), changes put up for review (Proposals), a spreadsheet (Sheet), an isometric diagram (Diagram), a movable tree (Tree), a movable sequence (Sequence), a counter (Counter), chunked files (Blobs), handwriting (Ink) and undo (Undo), plus the Register, RecordMap and Cell pieces they compose

Using it

ada, grace := crdt.New(1), crdt.New(2)

opening, _ := ada.Insert(0, "the quick fox")
grace.Apply(opening...)

// Both edit at once, neither having seen the other.
fromAda, _ := ada.Insert(10, "brown ")
fromGrace, _ := grace.Insert(13, " jumps")

ada.Apply(fromGrace...)
grace.Apply(fromAda...)

fmt.Println(ada)   // the quick brown fox jumps
fmt.Println(grace) // the quick brown fox jumps

A replica joining late loads a snapshot instead of the whole history; one coming back from offline hands over its version vector and is sent only what it missed:

client, err := crdt.Load(siteID, snapshot)   // join
missed := server.OpsSince(client.Version())  // catch up

A list, for what sits beside the text

The things built around a document are sequences too — the comments on it, the record of who changed what, the messages beside it — and they need the same guarantees. List is the same algorithm over values the caller encodes:

comments := crdt.NewList(site)
ops, _ := comments.Insert(0, encoded)   // values are opaque []byte
comments.Apply(fromPeers...)
anchor, _ := comments.Anchor(pos)       // a handle that survives other edits

It is a separate type rather than a generic one, deliberately: a document holds hundreds of thousands of characters and earns run-length storage and an index over runs, while a list holds tens or hundreds of values, where a slice is both faster and obviously correct.

A map, for the cells beside it

Not everything beside a document is a sequence. A spreadsheet is a map of cells, a table of settings is a map of settings: written and cleared, never woven in between their neighbours. Map merges those by last writer wins per key, under the same (clock, site) order the text uses, over values the caller encodes:

sheet := crdt.NewMap(site)
op, _ := sheet.Set("B7", []byte("42"))  // send op to every peer
value, ok := sheet.Get("B7")            // a copy; writing to it changes nothing
op, _ = sheet.Delete("B7")
sheet.Keys()                            // sorted, so every replica agrees

A deleted key keeps its clock. Dropping it would let an older write arriving afterwards bring the key back on the replica that heard it late and not on the one that heard it early, permanently — the classic mistake in a last-writer-wins map, and the thing here most worth reading docs/design.md for.

One document, many parts

An editor does not hold one structure. It holds the text, the comments on it, the record of who changed what, the messages beside it and a sheet of cells — and persisting those separately means five snapshots saved at five moments, five things to authorize, and no instant at which the set of them is consistent. Composite is those parts under one name:

doc := crdt.NewComposite(site)

text, _ := doc.Text("file:src/main.tex")   // a *Doc,  created on first use
chat, _ := doc.List("chat")                // a *List
cells, _ := doc.Map("cells")               // a *Map

snapshot := doc.Snapshot()                 // one thing to persist
missed := server.OpsSince(client.Version())

A part is identified by its name and its kind together, and it exists because operations for it exist — so there is no operation to create one, and two replicas that reach for "chat" at the same moment are already holding the same part. It follows that a part which exists and holds nothing is indistinguishable from one that was never created, and that has to be true rather than convenient: an empty part is in no snapshot and no version, or two replicas holding the same operations would disagree about their state.

Each part keeps its own site counter, clock and version vector, so a Doc inside a composite behaves exactly as one standing alone, and the version is per part. On a document of three hundred parts — one small map per comment, which is how a resolved flag flips with one write instead of a delete and a reinsert — the version encodes to 16 KB with the site identities shared in one table rather than repeated in every part, a third less than writing them out.

Keeping a view in step

A view of the text — an editor, a preview — has to be told what changed, not what the text now is. Handed only the new text it would have to replace everything, and replacing everything throws away the selection, the scroll position, the folded regions and the decorations, on every keystroke anybody else makes.

ApplyChanges is Apply and also reports the edits, in the order they have to be made, coalesced: a peer typing a word is one change, not one per letter. Apply does not pay for that.

Anchoring, and who wrote what

An offset names a place; an anchor names a character, and keeps naming it however the document moves around it. That is what a comment, a mark or a stored selection should be:

changes, _ := doc.ApplyChanges(ops...) // what a view has to do to catch up
anchor, _ := doc.Anchor(pos)     // the identity of the character there
pos, ok := doc.Position(anchor)  // where it is now — or where it was, if deleted
doc.Visible(anchor)              // whether it is still in the text
doc.AuthorRuns()                 // the visible text split by who wrote each stretch

Every character already carried the identity of the operation that created it, and the site is part of that identity, so none of this costs the document anything to store.

Counting the way a browser counts

The document counts characters. CodeMirror, the DOM, the Language Server Protocol and every index into a JavaScript string count UTF-16 code units, in which an emoji, an extended CJK ideograph or a 𝔸 is two units and one character. An editor handing its cursor offset to Insert therefore edits in the wrong place as soon as the document holds one of them — silently, and with nothing left behind that a later read could notice.

So the same operations are addressed both ways, and a caller who counts in UTF-16 never converts by hand:

doc.LenUTF16()                 // what JavaScript's String.length would report
doc.InsertUTF16(pos, "text")   // pos in code units
doc.DeleteUTF16(pos, n)        // pos and n in code units
doc.UTF16Offset(runePos)       // and the conversion, both ways
doc.RuneOffset(utf16Pos)

An offset landing between the two units of one character is refused rather than rounded; docs/design.md says why, and how to round it in one step if that is what you want. A document holding no such character converts in constant time, so nothing pays for this until it has an emoji in it. The answers are checked against node's, not against ours.

What it guarantees

  • Convergence. Replicas holding the same operations hold the same document, whatever order those operations arrived in. Proven by randomised sessions with late, reordered and duplicated delivery, and by enumerating every permutation of small histories.
  • Idempotence and reordering. Duplicates are ignored; an operation arriving before what it depends on is buffered until that lands. A transport needs to be eventually complete, not ordered.
  • Determinism. No wall clock, no randomness. Replica identity is injected by the caller, so a document behaves identically on a server and in a browser.
  • Intent. A run of characters typed one after another is never chopped up by someone else's concurrent insertion.
  • Nothing is trusted. Every decoder is fuzzed; a snapshot that no replica could have produced is rejected rather than loaded; and operations arranged to make integration walk the whole document cost a descent of an index instead.

Status

Version 0.27: the text, list and map CRDTs, a composite document that holds them as named parts, the wire and snapshot formats, awareness, and the surface an editor needs — reported changes, anchors, authorship, UTF-16 addressing, undo, and reading a document as it stood at any version. Pure Go, CGO=0, 100% statement coverage on both packages, race-clean, six-arch CI, and the full suite green under js/wasm.

A real editing history — 259 778 edits from the trace text CRDTs are commonly measured on — replays in 18.4 ms and matches the recorded text exactly; the same history delivered back to front, nothing applicable until the last operation, settles in 0.25 s, and the document encodes to 620 KB. See docs/performance.md.

See docs/design.md for how it works and why, and docs/performance.md for what it costs and what changes next.

License

BSD-3-Clause — see LICENSE. Copyright the go-crdt authors.

Documentation

Overview

Package crdt implements a conflict-free replicated data type for plain text: a replicated character sequence that any number of replicas may edit concurrently, offline, and in any delivery order, and that is guaranteed to converge to the same text on every replica.

The sequence is an RGA (Replicated Growable Array). Every character carries a unique ID and the ID of the character it was inserted after, so insertions are placed relative to content rather than to an index that concurrent edits would invalidate. Deletions are tombstones, so an insertion may still refer to a character another replica has already removed.

Determinism

The package never reads the wall clock and never draws random numbers, so the same Doc compiled to js/wasm behaves exactly as it does on a server. Replica identity is injected by the caller as a SiteID; see DeriveSiteID for a deterministic way to obtain one from bytes the caller already has.

Two counters

Each operation carries two numbers, and they are not the same thing:

  • ID.Seq is a per-site counter that increases by exactly one per operation the site issues. It gives the operation its identity and lets a VersionVector describe, exactly, which operations a replica holds.
  • Op.Clock is a Lamport timestamp, bumped past every clock a replica has seen. It orders concurrent insertions at the same position, and it is what makes RGA integration convergent.

Folding the two into one counter would create gaps in a site's own sequence, and a version vector cannot describe a sequence with gaps.

Delivery

Doc.Apply tolerates duplicates, and it tolerates operations that arrive before the operations they depend on: an operation that is not yet ready is buffered and integrated as soon as its dependencies land. Callers therefore do not need an ordered transport, only an eventually-complete one.

A replicated map

Map is the same machinery applied to a key-value map, for what an editor keeps beside its text — a spreadsheet of cells, a table of settings. The last write to a key wins, under the same (clock, site) order, and a deleted key keeps its clock so that an older write arriving later cannot resurrect it. It shares ID, VersionVector, ErrMalformed and the wire conventions with Doc and nothing else, so neither structure can disturb the other.

One document of many parts

Composite holds named parts, each a Doc, a List or a Map, so that a text, the comments on it and a sheet of cells are one snapshot, one version and one thing to authorize rather than five. It adds no merge rule: each part keeps its own counters and converges exactly as it does standing alone. A part is identified by its name and its kind together, and exists because operations for it exist, so two replicas that reach for the same part are already holding it and nothing is exchanged to create one.

Example

Two people edit the same document at the same time, neither having seen the other's change. Once the two operations have crossed, both replicas hold the same text — with no server deciding anything. 

package main

import (
	"fmt"

	"github.com/go-crdt/crdt"
)

func main() {
	ada, grace := crdt.New(1), crdt.New(2)

	opening, err := ada.Insert(0, "the quick fox")
	if err != nil {
		panic(err)
	}
	if err := grace.Apply(opening...); err != nil {
		panic(err)
	}

	// Both edit at once, neither seeing the other.
	fromAda, err := ada.Insert(10, "brown ")
	if err != nil {
		panic(err)
	}
	fromGrace, err := grace.Insert(13, " jumps")
	if err != nil {
		panic(err)
	}

	if err := ada.Apply(fromGrace...); err != nil {
		panic(err)
	}
	if err := grace.Apply(fromAda...); err != nil {
		panic(err)
	}

	fmt.Println(ada)
	fmt.Println(grace)
}

Output:
the quick brown fox jumps
the quick brown fox jumps

Index

Examples

Constants

View Source 
const MaxClock = 1 << 62

MaxClock is the highest Lamport timestamp an operation may carry, and so also the highest sequence number, since a clock is never below the sequence number beside it.

A clock counts operations: to reach this one, a session would have to issue four quintillion of them, one per nanosecond for a century and a half. The ceiling exists for what arrives from elsewhere, not for what is issued here. A replica raises its clock past every clock it is told about, so without a ceiling one operation from one peer — a corrupted varint is enough, no malice required — leaves the receiver's clock at the top of the range, and its next edit wraps to zero. That edit is then an operation every replica rejects as invalid, its own author included, and one that loses every tie it takes part in: the peer is silently and permanently unable to write. Refusing the clock on arrival is what keeps that from being reachable at all.

Variables

View Source 
var ErrCollected = errors.New("crdt: history below the collection floor")

ErrCollected reports a question about a version this replica can no longer answer, because Doc.Collect has dropped the operations that would decide it. The alternative to refusing is a past text with characters missing from it, which is worse than no answer: nothing downstream could tell the two apart.

View Source 
var ErrEmptyValue = errors.New("crdt: a list value must not be empty")

ErrEmptyValue reports an attempt to insert a value of no bytes. A list of nothings is almost always a caller encoding badly, and allowing it would make "absent" and "empty" the same on the wire.

View Source 
var ErrExhausted = errors.New("crdt: the site has no clock left")

ErrExhausted reports a replica that can issue no further operations because its Lamport clock has reached MaxClock. Reaching it honestly is not something a running program does; see MaxClock.

View Source 
var ErrInvalidOp = errors.New("crdt: invalid operation")

ErrInvalidOp reports an operation that cannot be applied because it is not well formed — an unknown kind, a missing identity, an unusable character, or a field set that does not belong to its kind.

View Source 
var ErrInvalidPart = errors.New("crdt: invalid part")

ErrInvalidPart reports a part that cannot name anything: an unknown kind, a name that is not valid UTF-8, or no name at all.

View Source 
var ErrInvalidText = errors.New("crdt: invalid UTF-8")

ErrInvalidText reports text that is not valid UTF-8. The package refuses it rather than substituting replacement characters, which would silently corrupt a document that no later edit could repair.

View Source 
var ErrMalformed = errors.New("crdt: malformed encoding")

ErrMalformed reports bytes that are not a valid encoding.

View Source 
var ErrOutOfRange = errors.New("crdt: position out of range")

ErrOutOfRange reports a position or length outside the document.

View Source 
var ErrStranded = errors.New("crdt: operation names a collected character")

ErrStranded reports an operation that can never be applied, because the character it names was dropped by Doc.Collect.

Parking it instead would be the silent version of the same failure: the operation would wait for something that is never coming, and the work it carries would be lost with nothing said. It means collection was given a version some replica had not reached — the one precondition Doc.Collect asks for — and the replica this operation came from is the one that was left behind.

View Source 
var ErrSurrogateBoundary = errors.New("crdt: UTF-16 offset splits a surrogate pair")

ErrSurrogateBoundary reports a UTF-16 offset that falls between the two code units of one character.

Such an offset names a position that does not exist: half of an emoji is not a place a cursor can be, and no editor's user ever put it there. It is refused rather than rounded because rounding would move an edit somewhere the caller did not ask for and leave nothing behind to say so — the same reasoning that has Doc.Insert refuse invalid UTF-8 rather than substitute replacement characters.

It is not a hypothetical. JavaScript will happily do the operation, and `"a😀b".slice(0, 2) + "x"` is a string containing a lone high surrogate: not text, not valid UTF-8, and not anything this package can hold. An offset that splits a character has already lost the information needed to honour it.

A caller who must tolerate such an offset can round it down in one step, without a second API: an offset that splits a character is always exactly one past that character's first unit, so Doc.RuneOffset of pos-1 is the position of the character it landed inside.

Functions

func AppendListOps added in v0.12.0 

func AppendListOps(dst []byte, ops []ListOp) ([]byte, error)

AppendListOps encodes a batch of operations onto dst — the form a transport sends. The batch is length-prefixed, so ParseListOps can reject a truncated message instead of silently returning the operations that happened to survive.

func AppendMapOps added in v0.10.0 

func AppendMapOps(dst []byte, ops []MapOp) ([]byte, error)

AppendMapOps encodes a batch of operations onto dst — the form a transport sends. The batch is length-prefixed, so ParseMapOps can reject a truncated message instead of silently returning the operations that happened to survive.

func AppendOps 

func AppendOps(dst []byte, ops []Op) ([]byte, error)

AppendOps encodes a batch of operations onto dst — the form a transport sends. The batch is length-prefixed, so ParseOps can reject a truncated message instead of silently returning the operations that happened to survive.

func AppendPartOps added in v0.12.0 

func AppendPartOps(dst []byte, batches []PartOps) ([]byte, error)

AppendPartOps encodes batches of operations onto dst — the form a transport sends, and what Composite.OpsSince returns handed over whole. The batches are length-prefixed, and so are the operations inside each of them, so ParsePartOps can reject a truncated message instead of silently returning the batches that happened to survive.

Every batch is validated before a byte is written, so what this produces is what Composite.Apply accepts, and what it refuses it refuses with the error Apply would have given: a batch that cannot be sent is a batch that could not have been applied either.

Unlike a snapshot, this is a message rather than a state, and the canonical claim it makes is the one a message can make: re-encoding what was decoded gives back the same bytes. It does not claim that one set of operations has one encoding — batches may repeat a part or arrive in any order, because Composite.Apply accepts them that way and applying an operation twice changes nothing.

Types

type AuthorRun added in v0.7.0 

type AuthorRun struct {
	// Pos is the visible offset the stretch starts at.
	Pos int
	// Len is how many characters it covers.
	Len int
	// Site is the replica that wrote them.
	Site SiteID
}

An AuthorRun is a stretch of the visible text one replica wrote.

type Change added in v0.7.0 

type Change struct {
	Pos     int
	Removed int
	Text    string
}

A Change is one contiguous edit to the visible text: remove Removed characters at Pos, then put Text there. Either part may be empty.

Offsets are in runes, and each change is expressed against the text as it stands after the changes before it. Applying them in order to a copy of the text is what brings the copy up to date.

func ChangesFrom added in v0.7.0 

func ChangesFrom(text, want string) []Change

ChangesFrom returns the edits that turn text into the document's current text, for a caller holding a copy it cannot otherwise reconcile — a view that has just been reconnected, say. It is a convenience over Doc.String, not a cheaper path: it compares the two.

type Composite added in v0.11.0 

type Composite struct {
	// contains filtered or unexported fields
}

A Composite is a document made of named parts, each a Doc, a List or a Map. It is one replica of the whole: one snapshot to persist, one version to hand a peer, one thing to authorize.

Parts are created by being used

Composite.Text, Composite.List and Composite.Map return the part with that name, creating it on first use. There is no operation for creating one and none is needed: two replicas that each reach for the same name are already holding the same part, because a part is identified by nothing but its name and its kind. Nothing has to be exchanged, so nothing can be lost, arrive late, or conflict.

The consequence is that a part which exists and holds nothing is indistinguishable from one that was never created — and that had better be true rather than merely convenient, because it is: a replica that has reached for "chat" and typed nothing must produce the same snapshot bytes as one that has never heard the word, or two replicas holding exactly the same operations would disagree about their state. So an empty part is written to no snapshot, carried in no version, and not returned by Composite.Parts. It costs its creator a map entry and every other replica nothing at all.

Each part keeps its own counters

A part is an ordinary Doc, List or Map, with its own site counter, Lamport clock and version vector, and it is edited through its own methods. The alternative — one counter for the whole composite — was considered and rejected twice over. It would make Doc.Version describe operations a standalone Doc knows nothing about, damaging three clean types for a container's convenience; and it would not buy cross-part causality anyway, since contiguous sequence numbers only order operations issued by the same site, so a comment Bob writes on text Ada typed is unprotected either way.

What follows from that is worth stating: operations are addressed to a part by the caller, in a PartOps, and nothing here can check that the address is the one the operations came from. Editing the text and sending its operations labelled as a list is a caller bug this type cannot catch.

A Composite is not safe for concurrent use. The zero Composite is unusable — construct one with NewComposite or LoadComposite.

Example

A composite holds a text, the lists beside it and a map of cells as one document. Nothing is exchanged to create a part: both replicas reach for "chat" and are already holding the same one. 

package main

import (
	"bytes"
	"fmt"

	"github.com/go-crdt/crdt"
)

func main() {
	ada, grace := crdt.NewComposite(1), crdt.NewComposite(2)

	text, err := ada.Text("file:main.tex")
	if err != nil {
		panic(err)
	}
	typed, err := text.Insert(0, "\\section{Results}")
	if err != nil {
		panic(err)
	}

	chat, err := grace.List("chat")
	if err != nil {
		panic(err)
	}
	said, err := chat.Insert(0, []byte("looks good"))
	if err != nil {
		panic(err)
	}

	// Operations are addressed to a part by the caller; only the caller knows.
	err = grace.Apply(crdt.PartOps{
		Part: crdt.Part{Kind: crdt.PartText, Name: "file:main.tex"},
		Text: typed,
	})
	if err != nil {
		panic(err)
	}
	err = ada.Apply(crdt.PartOps{
		Part: crdt.Part{Kind: crdt.PartList, Name: "chat"},
		List: said,
	})
	if err != nil {
		panic(err)
	}

	fmt.Println(ada.Parts())
	fmt.Println(bytes.Equal(ada.Snapshot(), grace.Snapshot()))

	// A part reached for and left empty is in no snapshot: it is
	// indistinguishable from one nobody ever named.
	if _, err := ada.Map("cells"); err != nil {
		panic(err)
	}
	fmt.Println(bytes.Equal(ada.Snapshot(), grace.Snapshot()))

}

Output:
[{text file:main.tex} {list chat}]
true
true

func LoadComposite added in v0.11.0 

func LoadComposite(site SiteID, snapshot []byte) (*Composite, error)

LoadComposite rebuilds a document from a snapshot, to be edited as site. The site need not be one that appears in the snapshot — a client joining brings its own.

Each part's bytes are handed to that part's own loader, which is where the clock ceiling, the version vector's promise and the rest of what Load, LoadList and LoadMap refuse are enforced; nothing is re-checked here and nothing is waived. What this loader adds is what only it can see: that the parts are the ones a replica could have written, in the order it would have written them, and that none of them is empty — an empty part is one no snapshot carries, so a snapshot carrying one is not a snapshot this package produced, and accepting it would mean re-encoding gave back different bytes.

What it deliberately does not insist on is that a part's bytes are themselves the current encoding. A text part written by an older build is still read by Load, and is written back in the current form — so a snapshot this package accepts is normalised on load, while one it produced reloads to itself byte for byte. Refusing the older form would buy an exact fixed point on arbitrary input at the price of the migration the older form exists for.

func NewComposite added in v0.11.0 

func NewComposite(site SiteID) *Composite

NewComposite returns a document with no parts, whose parts issue operations as site. Every replica editing a composite concurrently must pass a distinct site; see SiteID.

func (*Composite) Apply added in v0.11.0 

func (c *Composite) Apply(batches ...PartOps) error

Apply integrates batches of operations from peers, creating any part they name and do not find. Duplicates are ignored and an operation arriving before what it depends on waits, exactly as it does in the part standing alone.

A malformed batch is rejected and nothing in the call is applied, parts included: a call that fails creates no part. That is why every batch is checked before any is applied.

func (*Composite) ApplyAbsorbed added in v0.32.0 

func (c *Composite) ApplyAbsorbed(batches ...PartOps) ([]PartOps, error)

ApplyAbsorbed is Composite.Apply, and also reports what it integrated, per part, including operations that had been parked waiting for them.

A part that integrated nothing is not in the result, so an empty result means this replica learned nothing — which is exactly when a relay has nothing to pass on and a loop between two replicas has to stop.

func (*Composite) ApplyChanges added in v0.13.0 

func (c *Composite) ApplyChanges(batches ...PartOps) ([]PartChange, error)

ApplyChanges is Composite.Apply, and also reports what each part did, in the canonical part order so that two replicas given the same batches report the same thing.

Only what actually happened is reported: an operation already applied, or one still waiting for the operation its site issued before it, changes nothing and says nothing; when a waiting one lands, the change is reported then. A batch naming a part that ends up doing nothing produces no PartChange.

Finding where each text edit landed costs a walk up the index per operation, which Composite.Apply does not pay. Use that one when nothing is watching.

func (*Composite) CanReplay added in v0.33.0 

func (c *Composite) CanReplay(v CompositeVersion) bool

CanReplay reports whether Composite.OpsSince would hand back a complete history from v, for every part of this document.

It is false when any text or list part has collected below what v holds; see Doc.CanReplay. A peer it is false for has to be sent a snapshot rather than a difference.

A map part never makes it false. A map gives back a sequence number without its operation as a matter of course — a second write to a key overwrites the first — so the span that stands in for one collected tombstone is the same span that already stood in for an overwritten value, and a peer applying it catches up either way.

func (*Composite) Collect added in v0.33.0 

func (c *Composite) Collect(stable CompositeVersion) int

Collect drops what every part of this document can spare, given what every replica has delivered, and reports how many characters, elements and tombstones went.

A part not named in stable is left alone. That is not a nicety: collecting a part against a version nobody vouched for is exactly the mistake the other Doc.Collect guards against, and the guard cannot see a part the caller forgot.

Why this exists here and a rewrite does not

There is deliberately no rewrite for a Composite — see Doc.Rewritten — because rewriting mints new identities, and the structured layer keeps rich text marks, tree parents and sequence positions against the identities of the characters they describe. A rewrite would silently empty every one of those.

Collection is the opposite: it keeps every identity it does not drop, and drops only what is already invisible and already agreed to be gone. A mark on a character that is collected was a mark on text nobody could see, and it stays exactly as inert as it was. So a composite may be collected where it may not be rewritten, and that difference is the whole reason this one is offered.

func (*Composite) DropPending added in v0.31.0 

func (c *Composite) DropPending() int

DropPending forgets what every part is holding back, and returns how many operations that was. See Doc.DropPending for why it is safe.

func (*Composite) List added in v0.11.0 

func (c *Composite) List(name string) (*List, error)

List returns the list part called name, creating an empty one on first use.

func (*Composite) Map added in v0.11.0 

func (c *Composite) Map(name string) (*Map, error)

Map returns the map part called name, creating an empty one on first use.

func (*Composite) OpsSince added in v0.11.0 

func (c *Composite) OpsSince(v CompositeVersion) ([]PartOps, error)

OpsSince returns the operations this replica holds that v does not, batched by part and ready to send to the peer that produced v. Pass a nil version for everything.

The version is compared before anything else is done, so a part the peer is up to date on costs a walk of its version vector rather than of its history. That is what a document of hundreds of parts needs: a peer that has missed one comment must not pay for the other two hundred and ninety-nine.

No batch it returns is empty, which is the same statement: a part with nothing to send is a part whose version the peer already covers. A part that has been reached for and left empty needs no case of its own — the empty vector is covered by everything, this one included.

func (*Composite) Parts added in v0.11.0 

func (c *Composite) Parts() []Part

Parts returns every part holding at least one operation, ordered by kind and then by name. Two replicas that have applied the same operations return the same slice, which is what makes anything a caller derives from it — a list of files, a count of comments — the same everywhere. Parts a caller has reached for and left empty are not among them; see Composite.

func (*Composite) Pending added in v0.11.0 

func (c *Composite) Pending() int

Pending reports how many operations, across every part, are still waiting for operations they depend on.

func (*Composite) Site added in v0.11.0 

func (c *Composite) Site() SiteID

Site returns the replica identity this document's parts issue operations as.

func (*Composite) Snapshot added in v0.11.0 

func (c *Composite) Snapshot() []byte

Snapshot encodes the whole document: every part that holds anything, in the canonical order, each as its own snapshot with its name and kind in front. It is what a server sends a client joining an existing session, and what it persists — one file rather than one per part, saved at one moment rather than at five.

A part's bytes are its own snapshot, verbatim, including that snapshot's magic and format version. Stripping the six bytes back off was the obvious economy and was not taken: it would tie this format to the internal layout of the other three, so a part's format could not be revised without revising this one, and a document a previous build wrote could not be opened by wrapping its bytes. Six bytes per part is a twentieth of what a part's name costs.

The encoding is canonical. Two replicas that have applied the same operations produce identical bytes, whatever order those operations arrived in, because each part's own encoding is canonical and the order the parts are written in is fixed. That is what lets the test suite compare snapshots rather than values, which is the stronger claim: two replicas can agree on every value and still disagree about which write produced it.

func (*Composite) Text added in v0.11.0 

func (c *Composite) Text(name string) (*Doc, error)

Text returns the text part called name, creating an empty one on first use. The name is rejected rather than corrected; see Part.

func (*Composite) Version added in v0.11.0 

func (c *Composite) Version() CompositeVersion

Version returns what this replica holds, to be handed to a peer that will send back what it is missing; see Composite.OpsSince.

Every vector in it is a copy. Handing out the live ones would put the thing a caller measures against under the control of the thing being measured: a server that took a client's version, then applied an operation, and then asked what the client was missing would find the question had answered itself.

A map has no order, so this is one of the two places that walk the parts without asking Composite.Parts for them in order — which on a document of three hundred parts is most of the cost of the answer.

type CompositeVersion added in v0.11.0 

type CompositeVersion map[Part]VersionVector

A CompositeVersion records what a replica holds, part by part. There is no single vector for a composite, because there is no single sequence of operations: each part counts its own, so what a peer is missing is a question asked once per part.

The nil version is valid and reads as "nothing at all", and a part mapped to a vector that promises nothing is the same as a part not mentioned.

func (CompositeVersion) Clone added in v0.11.0 

func (v CompositeVersion) Clone() CompositeVersion

Clone returns an independent copy, vectors included.

func (CompositeVersion) Equal added in v0.11.0 

func (v CompositeVersion) Equal(other CompositeVersion) bool

Equal reports whether v and other describe the same operations. A part promising nothing counts as absent, so a nil version equals one whose every part is empty.

func (CompositeVersion) MarshalBinary added in v0.11.0 

func (v CompositeVersion) MarshalBinary() ([]byte, error)

MarshalBinary encodes the version. A peer sends this on every join, and a document with one map part per comment has hundreds of parts and a handful of sites, so the sites are written once in a table and each part's entries name an index into it.

That is not a micro-economy. A SiteID is a whole uint64 — DeriveSiteID hashes one, so it uses the range — and a uint64 is ten bytes as a varint, while an index into a table of three is one. Repeating the identity in every part would put nine tenths of the message in the same three numbers written three hundred times.

Like VersionVector.MarshalBinary it carries no magic: it travels in a field whose type is already known.

A part that could not name anything is refused rather than encoded, as is a sequence number above MaxClock, which no replica could have issued.

func (*CompositeVersion) UnmarshalBinary added in v0.11.0 

func (v *CompositeVersion) UnmarshalBinary(data []byte) error

UnmarshalBinary decodes a version written by MarshalBinary.

It arrives from a peer, so it is a trust boundary, and what it is held to is that no two *structures* describe one version: the site table ascends and is used in full, the parts ascend, each part's entries ascend, and no part is written that promises nothing. Each of those would otherwise let a peer state the same thing two ways, and one of the two would be a shape nothing here produces.

What that does not reach is the varint layer underneath it. binary.Uvarint accepts an overlong encoding — 0x80 0x00 is a zero written in two bytes — so bytes this decoder accepts may still re-encode shorter. Every decoder in this package inherits that from the reader they share, and the guarantee is therefore the one they all make: what this package encodes reloads to itself byte for byte, and what it accepts is normalised. A caller wanting to compare two peers by their bytes must compare what MarshalBinary gave back, not what arrived.

A sequence number above MaxClock names an operation no replica could have issued, and is refused here rather than in the parts, because a version never passes through a part's loader; see MaxClock.

type Doc 

type Doc struct {
	// contains filtered or unexported fields
}

A Doc is one replica of a text document. It is not safe for concurrent use; serialize access from the outside, as an editor naturally does.

The zero Doc is unusable — construct one with New or Load.

func Load 

func Load(site SiteID, snapshot []byte) (*Doc, error)

Load rebuilds a document from a snapshot, to be edited as site. The site need not be one that appears in the snapshot — a client joining an existing document brings its own.

Example

A replica joining an existing session is given a snapshot rather than the whole history, and can take part immediately. 

package main

import (
	"fmt"

	"github.com/go-crdt/crdt"
)

func main() {
	server := crdt.New(1)
	if _, err := server.Insert(0, "shared draft"); err != nil {
		panic(err)
	}

	client, err := crdt.Load(2, server.Snapshot())
	if err != nil {
		panic(err)
	}
	edit, err := client.Insert(client.Len(), " — revised")
	if err != nil {
		panic(err)
	}
	if err := server.Apply(edit...); err != nil {
		panic(err)
	}

	fmt.Println(server)
}

Output:
shared draft — revised

func New 

func New(site SiteID) *Doc

New returns an empty document that issues operations as site. Every replica editing a document concurrently must pass a distinct site; see SiteID.

func (*Doc) Anchor added in v0.7.0 

func (d *Doc) Anchor(pos int) (ID, error)

Anchor returns the identity of the character at visible offset pos.

The identity does not move. Insertions and deletions elsewhere change which offset the character sits at, and never change what it is, so an anchor is what a comment, a mark or a selection should be stored as — an offset stored instead would point somewhere else the moment anyone edits above it.

pos may equal Doc.Len, which anchors to the end of the document and returns the zero ID: the position after every character there is, and the one thing insertions at the end do not move.

func (*Doc) Apply 

func (d *Doc) Apply(ops ...Op) error

Apply integrates operations from peers. Duplicates are ignored, and an operation that arrives before the operations it depends on is buffered until they do, so the caller needs no ordered delivery.

A malformed operation is rejected and nothing in the batch is applied.

func (*Doc) ApplyAbsorbed added in v0.32.0 

func (d *Doc) ApplyAbsorbed(ops ...Op) ([]Op, error)

ApplyAbsorbed is Doc.Apply, and also reports the operations it integrated, including any that had been parked waiting for them.

The order is the order they were integrated in, which is a causal order: an operation appears after the one it was waiting for.

func (*Doc) ApplyChanges added in v0.7.0 

func (d *Doc) ApplyChanges(ops ...Op) ([]Change, error)

ApplyChanges is Doc.Apply, and also reports what the document did: the edits a view of the text has to make to catch up, in the order it has to make them.

Only what actually happened is reported. An operation already applied, or one still waiting for the operations it depends on, changes nothing and says nothing; when it does land, the change is reported then.

Finding where each edit landed costs a walk up the index per operation, which Doc.Apply does not pay. Use that one when nothing is watching.

func (*Doc) Author added in v0.7.0 

func (d *Doc) Author(pos int) (SiteID, error)

Author returns the replica that wrote the character at visible offset pos.

func (*Doc) AuthorRuns added in v0.7.0 

func (d *Doc) AuthorRuns() []AuthorRun

AuthorRuns splits the visible text into stretches by who wrote them, in order. It is what colouring a document by author needs, and it costs one pass rather than one lookup per character.

Adjacent stretches by the same replica are joined, so the result depends on the text rather than on how the document happens to be stored: two replicas holding the same document return the same runs.

func (*Doc) CanReplay added in v0.33.0 

func (d *Doc) CanReplay(v VersionVector) bool

CanReplay reports whether Doc.OpsSince would hand back a complete history from v.

It is false when v is below Doc.Floor: collection dropped operations that v has not seen, so what OpsSince returns has holes in its sequence numbers, and a replica applying them parks everything after the first hole instead of catching up — silently, since nothing in the batch is wrong on its own. A peer below the floor has to be sent a snapshot rather than a difference.

It is true for every version of a replica that has never collected, which is every replica until somebody asks.

func (*Doc) ChangesSince added in v0.26.0 

func (d *Doc) ChangesSince(v VersionVector) ([]Change, error)

ChangesSince returns the edits that turn the text as it stood at v into the text as it stands now, in order, with offsets into the text being edited as each is applied — the same shape Doc.ApplyChanges reports, so a caller that can replay one can replay the other.

It is not a diff. Two texts can be turned into one another in many ways and a diff picks one; this reports what actually happened, because every character says whether it arrived since v and every deletion says whether it did. Text that was written and then removed, both since v, is in neither the old text nor the new one and is reported in neither.

func (*Doc) Collect added in v0.33.0 

func (d *Doc) Collect(stable VersionVector) int

Collecting the tombstones nothing can still name.

A deletion hides a character and does not forget it, because a replica that forgot could not tell a character arriving late from one it had already seen. That is why a document's snapshot grows with every edit and never shrinks, and it is the cost Doc.Collect exists to bound.

A tombstone may go when three things are true at once:

  1. Every character of the run is deleted. Runs are collected whole, never in part: a character's origin is the character before it in its own run, so collecting a prefix would leave the survivor behind it naming something that is gone.
  2. Every one of those deletions is dominated by stable — a version every replica has delivered. No replica still has the character visible, and an insertion names as its origin a character that was visible to whoever issued it, so no operation that could name this run can still be written. Anything already written naming it is, by the same argument, already here.
  3. Every survivor that named a character of it is re-pointed at the nearest character still alive before it. Without this nothing is ever collected: a run appended to a document names the last character of the run before it, so an entirely deleted run is essentially always named by its successor. Measured on a document written and revised the ordinary way, 332 runs of 667 were entirely deleted and stable, and every one of them was named by a survivor.

Re-pointing is what makes the result representable as well: an origin left pointing at a collected character is one [Doc.Load] must reject, since placing a character needs the character it follows.

What it costs

The operations are gone, and with them the ability to say what the document said before they were applied. Doc.TextAt and its neighbours refuse below Doc.Floor rather than answer with characters missing — a wrong answer about the past being worse than none. The version vector is untouched: the replica still knows it has seen these operations, so a duplicate arriving late is still recognised and dropped rather than applied a second time.

Who may call it

stable has to be a version every replica has delivered *and* whose operations this replica holds. A server that fans operations out and collects acknowledgements knows one; a replica on its own does not, and passing a version that some replica has not reached will strand that replica's work — its operations will name origins that are gone, and it will be told so by ErrStranded rather than left to park them for ever.

Collect reports how many characters it dropped.

func (*Doc) Delete 

func (d *Doc) Delete(pos, length int) ([]Op, error)

Delete removes length runes starting at rune offset pos and returns the operations that describe it. Deleting nothing is a no-op.

func (*Doc) DeleteUTF16 added in v0.8.0 

func (d *Doc) DeleteUTF16(pos, length int) ([]Op, error)

DeleteUTF16 is Doc.Delete with pos and length counted in UTF-16 code units rather than in runes.

Both ends of the range are converted, so length is a number of code units and the number of characters removed may be fewer — deleting the four units of two emoji removes two characters. A range whose either end splits a character is refused; see ErrSurrogateBoundary.

func (*Doc) DropPending added in v0.31.0 

func (d *Doc) DropPending() int

DropPending forgets the operations this replica is holding back, and returns how many there were.

An operation that arrives before the one it depends on is parked, which is right: it may become applicable a moment later, and dropping it silently would lose an edit. What is not right is that nothing bounds the pile. A peer sending operations that can never apply — each waiting on a sequence number that site never issues — costs about 140 bytes apiece, forever, for a document that stays empty.

This is the lever for that, and it is safe for one reason: a parked operation has had no effect on the state, so it is not in the version vector. A peer asked what this replica is missing sends it again. Dropping and re-syncing therefore loses nothing and diverges from nobody — which is asserted in the tests rather than argued here.

It is deliberately the caller's decision. A cap inside this package would have to choose what to do when it is reached, and the only answers are to drop — which is a policy, not a merge rule — or to refuse an Apply for reasons that have nothing to do with what it was handed.

func (*Doc) Floor added in v0.33.0 

func (d *Doc) Floor() VersionVector

Floor reports the version below which this replica can no longer say what the document held: everything Doc.Collect has been given, joined. It is empty for a document that has never collected, which is every document until somebody asks.

func (*Doc) Insert 

func (d *Doc) Insert(pos int, text string) ([]Op, error)

Insert adds text at rune offset pos and returns the operations that describe it. The operations are already applied here; send them to every peer.

pos may equal Doc.Len, which appends. Inserting the empty string is a no-op that returns no operations.

func (*Doc) InsertUTF16 added in v0.8.0 

func (d *Doc) InsertUTF16(pos int, text string) ([]Op, error)

InsertUTF16 is Doc.Insert with pos counted in UTF-16 code units rather than in runes. The text itself is a Go string, and so is measured in neither.

Example

A browser's cursor offset counts UTF-16 code units, and an emoji is two of them. Handing that offset to Insert would put the text one place to the left of where the user asked for it, without an error and without a trace. 

package main

import (
	"fmt"

	"github.com/go-crdt/crdt"
)

func main() {
	doc := crdt.New(1)
	if _, err := doc.Insert(0, "ship \U0001F680 it"); err != nil {
		panic(err)
	}

	// Nine characters, ten code units: the rocket is one and two.
	fmt.Println(doc.Len(), doc.LenUTF16())

	// The editor reports its caret just after the rocket, which is code unit
	// seven and character six.
	if _, err := doc.InsertUTF16(7, " now"); err != nil {
		panic(err)
	}
	fmt.Println(doc)

	// The offset between the rocket's two units names no position at all.
	_, err := doc.InsertUTF16(6, "x")
	fmt.Println(err)

}

Output:
9 10
ship 🚀 now it
crdt: UTF-16 offset splits a surrogate pair

func (*Doc) Len 

func (d *Doc) Len() int

Len returns the number of visible characters, counted in runes.

func (*Doc) LenAt added in v0.26.0 

func (d *Doc) LenAt(v VersionVector) (int, error)

LenAt returns how many characters the text held at version v, without building it.

func (*Doc) LenUTF16 added in v0.8.0 

func (d *Doc) LenUTF16() int

LenUTF16 returns the length of the document in UTF-16 code units — the number JavaScript's String.prototype.length reports for Doc.String.

It is a counter, not a walk: the count of visible supplementary characters is maintained beside the count of visible characters, so this reads the document no more than Doc.Len does.

func (*Doc) OpsSince 

func (d *Doc) OpsSince(vv VersionVector) ([]Op, error)

OpsSince returns the operations this replica holds that vv does not, ready to be sent to the peer that produced vv. Pass a nil vector for the whole history.

The result is in document order, which for insertions is a causal order: a character always follows its origin. Deletions may arrive before the insertions they refer to, which the receiving Doc.Apply buffers.

A deletion's Lamport timestamp is not retained — it never affects ordering — so replayed deletions carry their sequence number as their clock.

Example

A replica that has been away asks for what it missed by handing over its version vector; it is sent those operations and nothing else. 

package main

import (
	"fmt"

	"github.com/go-crdt/crdt"
)

func main() {
	online, offline := crdt.New(1), crdt.New(2)
	start, err := online.Insert(0, "notes")
	if err != nil {
		panic(err)
	}
	if err := offline.Apply(start...); err != nil {
		panic(err)
	}

	// The offline replica misses everything that follows.
	if _, err := online.Insert(5, ": chapter one"); err != nil {
		panic(err)
	}
	if _, err := online.Delete(0, 1); err != nil {
		panic(err)
	}

	// OpsSince refuses a version below what this replica has collected, because
	// what it gave back is not in the difference any more. Nothing here has
	// collected, so this cannot happen; a caller that does collect sends a
	// snapshot instead. See [crdt.Doc.CanReplay].
	missed, err := online.OpsSince(offline.Version())
	if err != nil {
		panic(err)
	}
	fmt.Println(len(missed), "operations missed")
	if err := offline.Apply(missed...); err != nil {
		panic(err)
	}
	fmt.Println(offline)
}

Output:
14 operations missed
otes: chapter one

func (*Doc) Pending 

func (d *Doc) Pending() int

Pending returns the number of operations buffered awaiting their dependencies. A healthy replica returns to zero once delivery catches up; a number that only grows means a peer is withholding operations.

func (*Doc) Position added in v0.7.0 

func (d *Doc) Position(anchor ID) (pos int, ok bool)

Position returns where the character an anchor names sits now.

A deleted character still has a place — the offset it would occupy, which is where the text around it closed up — and that is returned too, because a comment on a deleted sentence belongs where the sentence was rather than nowhere. Use Doc.Visible to tell the two apart.

The zero ID anchors to the end of the document. ok is false only for an identity this document has never seen, which means the anchor came from somewhere else or the operations that would explain it have not arrived.

func (*Doc) Rewritten added in v0.33.0 

func (d *Doc) Rewritten(site SiteID) (*Doc, error)

Rewriting a replica trades its past for its size.

A replica remembers every operation ever applied to it, including the ones that removed things: a deletion hides a character, it does not forget it, because a replica that forgot could not tell a late arrival apart from a character it had already seen. That memory is what makes merging work without a server, and it is also what makes a heavily revised document larger than its text.

A rewrite builds a new replica holding the same content and none of the history. What that buys is exactly what was deleted and nothing else: a document nothing was ever removed from rewrites to the same size, while one that was emptied rewrites to nothing. Measured on a text of 40 000 edits: no deletions, 1.0x; a third of it deleted, 1.6x; all of it, four orders of magnitude.

What it costs is every identity. The new replica mints its own, so:

  • Operations from the old replica no longer apply to the new one. They are not rejected and they do not corrupt it; they anchor to characters it has never heard of, so they park as pending and stay there. Any replica still holding the old identities has to be replaced by the rewrite, not merged with it.
  • Anything anchored to a character is left pointing at nothing. This is the same trap [Proposals] exists to avoid, and it is why there is no rewrite for a Composite: rich text marks, tree parents and sequence positions are stored against the identities of the characters they describe, and a composite cannot tell a part that carries such anchors from one that does not. Rewrite the parts you know are plain, or rebuild the anchors yourself.

So a rewrite belongs where a document is quiescent and about to be archived, or where a single writer is compacting its own copy. It does not belong in a live session.

Rewritten returns a new document with this one's text, minted at site. Pass a site the old replica never used: reusing one would let two different characters carry the same identity, which is the one thing a replica may not allow.

func (*Doc) RuneOffset added in v0.8.0 

func (d *Doc) RuneOffset(pos int) (int, error)

RuneOffset converts a UTF-16 offset into the rune offset naming the same position. pos may equal Doc.LenUTF16, which converts to Doc.Len.

An offset falling between the two code units of one character is refused with ErrSurrogateBoundary; see there for why, and for the one-line way to round it down instead.

func (*Doc) Site 

func (d *Doc) Site() SiteID

Site returns the replica identity this document issues operations as.

func (*Doc) Snapshot 

func (d *Doc) Snapshot() []byte

Snapshot encodes the whole document — every character, alive or tombstoned, in document order, plus the version vector. It is what a server sends a client joining an existing session, and what it persists.

Characters are written in runs: one header for a stretch one site typed consecutively, then its text, then the stretches of it that have been deleted. Writing one record per character instead, as version 1 did, cost twenty-five bytes for every character of a real document — measured against other implementations, between eight and twenty-four times what they need.

The runs written are maximal, whatever boundaries the document happens to hold in memory. That is what keeps the encoding canonical: two replicas that have applied the same operations produce identical bytes even if the operations arrived in different orders, so a snapshot doubles as a convergence check. It also keeps the format independent of the layout a replica stores, which is what let that layout change twice without a flag day.

The full history is recoverable from a snapshot: Doc.OpsSince on a loaded document returns the same operations it would have on the original.

func (*Doc) String 

func (d *Doc) String() string

String returns the document text.

func (*Doc) TextAt added in v0.26.0 

func (d *Doc) TextAt(v VersionVector) (string, error)

TextAt returns the text as it stood at version v: every character whose insertion v had seen, less every character whose deletion v had seen.

A version this replica has not reached is not refused. Operations it has not seen simply are not in the document to be counted, so the answer is the text as of everything the two have in common — which is what a replica can honestly say about a version it does not hold.

func (*Doc) Tombstones 

func (d *Doc) Tombstones() int

Tombstones returns the number of deleted characters still held in memory. They cannot be dropped, because a concurrent insertion may name one as its origin; see docs/performance.md.

func (*Doc) UTF16Offset added in v0.8.0 

func (d *Doc) UTF16Offset(pos int) (int, error)

UTF16Offset converts a rune offset into the UTF-16 offset naming the same position. pos may equal Doc.Len, which converts to Doc.LenUTF16.

This is the direction an editor needs to place someone else's cursor, or to report where an edit of its own landed.

Example

Awareness offsets are rune positions, because both peers have to agree what an offset means and an update has nowhere to say. A peer whose editor counts UTF-16 converts at its own edge — where it has to clamp in any case, since a cursor may describe a document longer than the one it now has. 

package main

import (
	"fmt"

	"github.com/go-crdt/crdt"
	"github.com/go-crdt/crdt/awareness"
)

func main() {
	doc := crdt.New(1)
	if _, err := doc.Insert(0, "\U0001F600 hello"); err != nil {
		panic(err)
	}

	peers := awareness.New()
	peers.Apply(awareness.Update{Site: 2, Clock: 1, Cursor: awareness.Cursor{Anchor: 2, Head: 99}})

	for _, peer := range peers.Peers() {
		head := min(max(peer.Cursor.Head, 0), doc.Len())
		at, err := doc.UTF16Offset(head)
		if err != nil {
			panic(err)
		}
		fmt.Println("caret at code unit", at)
	}
}

Output:
caret at code unit 8

func (*Doc) Version 

func (d *Doc) Version() VersionVector

Version returns a copy of the version vector describing which operations this replica holds. Pass it to a peer's Doc.OpsSince to be sent exactly what is missing.

func (*Doc) Visible added in v0.7.0 

func (d *Doc) Visible(anchor ID) bool

Visible reports whether the character an anchor names is still in the text.

type ID 

type ID struct {
	Site SiteID
	Seq  uint64
}

ID names a single operation, and — for an insertion — the character that operation created. It is unique across replicas because Site is unique and Seq counts that site's own operations.

The zero ID is the document root: the virtual character that precedes all content. It is a valid insertion origin and is never the ID of a real operation, because Seq starts at one.

func (ID) IsRoot 

func (id ID) IsRoot() bool

IsRoot reports whether id names the virtual character at the start of every document rather than a real operation.

The test is on Seq alone, because Seq counts from one: no operation ever carries zero, whatever its site. A decoder that only compared against the zero ID would let a sequence number of zero paired with a non-zero site through as if it named something real.

func (ID) String 

func (id ID) String() string

String renders the ID as "seq@site", the notation used in the CRDT literature. The root prints as "root".

type List added in v0.9.0 

type List struct {
	// contains filtered or unexported fields
}

A List is one replica of a sequence of values. It is not safe for concurrent use; serialize access from the outside.

The zero List is unusable — construct one with NewList or LoadList.

func LoadList added in v0.9.0 

func LoadList(site SiteID, snapshot []byte) (*List, error)

LoadList rebuilds a list from a snapshot, to be edited as site.

func NewList added in v0.9.0 

func NewList(site SiteID) *List

NewList returns an empty list that issues operations as site. Every replica editing a list concurrently must pass a distinct site; see SiteID.

func (*List) Anchor added in v0.9.0 

func (l *List) Anchor(pos int) (ID, error)

Anchor returns the identity of the value at index pos, which keeps naming that value however the list moves around it — what a reference to a comment should hold. pos may equal List.Len, which anchors to the end and returns the zero ID.

func (*List) Apply added in v0.9.0 

func (l *List) Apply(ops ...ListOp) error

Apply integrates operations from peers. Duplicates are ignored, and an operation arriving before what it depends on waits until that lands.

A malformed operation is rejected and nothing in the batch is applied.

func (*List) ApplyAbsorbed added in v0.32.0 

func (l *List) ApplyAbsorbed(ops ...ListOp) ([]ListOp, error)

ApplyAbsorbed is List.Apply, and also reports the operations it integrated, including any that had been parked waiting for them.

func (*List) ApplyChanges added in v0.13.0 

func (l *List) ApplyChanges(ops ...ListOp) (bool, error)

ApplyChanges is List.Apply, and also reports whether the list is not what it was. An operation already applied, or one still waiting for the operation its site issued before it, changes nothing and reports false; when a waiting one lands, that call reports true.

It reports that something changed rather than what, which is a deliberate stop. Naming the positions would be a second protocol to keep correct, and the consumers this was written for read the whole list back when they are told — a list here holds tens or hundreds of values, not the hundreds of thousands a document holds, which is the same reason a list is a slice and a document is not. A caller that needs the positions can be given them without breaking anyone; none has needed them yet.

func (*List) CanReplay added in v0.33.0 

func (l *List) CanReplay(v VersionVector) bool

CanReplay reports whether List.OpsSince would hand back a complete history from v.

It is false when v is below List.Floor: collection dropped operations that v has not seen, so what OpsSince returns has holes in its sequence numbers, and a replica applying them parks everything after the first hole instead of catching up — silently, since nothing in the batch is wrong on its own. A peer below the floor has to be sent a snapshot rather than a difference.

It is true for every version of a replica that has never collected, which is every replica until somebody asks.

func (*List) Collect added in v0.33.0 

func (l *List) Collect(stable VersionVector) int

Collect drops the tombstones nothing can name any more, on the same terms as Doc.Collect and for the same reasons — see that method for what the version handed in has to be, and what is lost by handing in the wrong one.

A list is simpler than a text to collect, because it is simpler to begin with: every element carries its own origin, where a character's is implicit in the run it belongs to. There is no rule here that an element goes only together with its neighbours, so an element goes when its own deletion is one every replica has, and the survivors that named it are re-pointed at the nearest element still alive before it.

A deletion that duplicates another's — two replicas removing the same element concurrently — is recorded against that element, and goes with it. It has to be stable as well, or a replica could still be about to send one, and would find nothing to record it against.

Collect reports how many elements it dropped.

func (*List) Delete added in v0.9.0 

func (l *List) Delete(pos, count int) ([]ListOp, error)

Delete removes count values from index pos and returns the operations describing it. Removing nothing is a no-op.

func (*List) DropPending added in v0.31.0 

func (l *List) DropPending() int

DropPending forgets the operations this replica is holding back, and returns how many there were.

An operation that arrives before the one it depends on is parked, which is right: it may become applicable a moment later, and dropping it silently would lose an edit. What is not right is that nothing bounds the pile. A peer sending operations that can never apply — each waiting on a sequence number that site never issues — costs about 140 bytes apiece, forever, for a document that stays empty.

This is the lever for that, and it is safe for one reason: a parked operation has had no effect on the state, so it is not in the version vector. A peer asked what this replica is missing sends it again. Dropping and re-syncing therefore loses nothing and diverges from nobody — which is asserted in the tests rather than argued here.

It is deliberately the caller's decision. A cap inside this package would have to choose what to do when it is reached, and the only answers are to drop — which is a policy, not a merge rule — or to refuse an Apply for reasons that have nothing to do with what it was handed.

func (*List) Floor added in v0.33.0 

func (l *List) Floor() VersionVector

Floor reports the version below which this replica can no longer say what the list held. It is empty for a list that has never collected.

func (*List) Get added in v0.9.0 

func (l *List) Get(pos int) ([]byte, error)

Get returns a copy of the value at index pos.

func (*List) Insert added in v0.9.0 

func (l *List) Insert(pos int, values ...[]byte) ([]ListOp, error)

Insert adds values at index pos and returns the operations describing it. The operations are already applied here; send them to every peer.

pos may equal List.Len, which appends. Inserting nothing is a no-op.

func (*List) Len added in v0.9.0 

func (l *List) Len() int

Len returns the number of values present.

func (*List) LenAt added in v0.26.0 

func (l *List) LenAt(v VersionVector) (int, error)

LenAt returns how many elements the list held at version v, without building them.

func (*List) OpsSince added in v0.9.0 

func (l *List) OpsSince(vv VersionVector) ([]ListOp, error)

OpsSince returns the operations this replica holds that vv does not, ready to send to the peer that produced vv. Pass a nil vector for the whole history.

The result is in list order, which for insertions is a causal order: an element always follows its origin.

func (*List) Pending added in v0.9.0 

func (l *List) Pending() int

Pending returns the number of operations waiting for the operations they depend on.

func (*List) Position added in v0.9.0 

func (l *List) Position(anchor ID) (pos int, ok bool)

Position returns where the value an anchor names sits now, or where it was if it has been removed. ok is false for an identity this list has never seen.

func (*List) Rewritten added in v0.33.0 

func (l *List) Rewritten(site SiteID) (*List, error)

Rewritten returns a new list with this one's values, minted at site. It trades the list's past for its size on the terms described on Doc.Rewritten.

func (*List) Site added in v0.9.0 

func (l *List) Site() SiteID

Site returns the replica identity this list issues operations as.

func (*List) Snapshot added in v0.9.0 

func (l *List) Snapshot() []byte

Snapshot encodes the whole list — every value, present or removed, in order, plus the version vector. It is what a server sends a client joining, and what it persists.

The encoding is deterministic: two replicas holding the same operations produce identical bytes, so a snapshot doubles as a convergence check. The full history is recoverable: List.OpsSince on a loaded list returns what it would have on the original.

func (*List) Tombstones added in v0.9.0 

func (l *List) Tombstones() int

Tombstones returns the number of removed values still held. They cannot be dropped: a concurrent insertion may still name one as its origin.

func (*List) Values added in v0.9.0 

func (l *List) Values() [][]byte

Values returns copies of every value present, in order.

func (*List) ValuesAt added in v0.26.0 

func (l *List) ValuesAt(v VersionVector) ([][]byte, error)

ValuesAt returns the elements the list held at version v, in order: every element whose insertion v had seen, less every element whose deletion v had seen. It reads the list the way Doc.TextAt reads the text, and for the same reason — an element carries the identity of what made it and of what removed it, so the list is its own history.

func (*List) Version added in v0.9.0 

func (l *List) Version() VersionVector

Version returns a copy of the version vector describing which operations this replica holds.

func (*List) Visible added in v0.9.0 

func (l *List) Visible(anchor ID) bool

Visible reports whether the value an anchor names is still in the list.

type ListOp added in v0.9.0 

type ListOp struct {
	// Kind selects insertion or deletion.
	Kind OpKind
	// ID names this operation, its Seq counting the issuing site's operations.
	ID ID
	// Clock is the Lamport timestamp ordering this against concurrent
	// operations; see the package documentation on the two counters.
	Clock uint64
	// Origin is the element the new one is inserted after; the zero ID means
	// the start of the list. OpInsert only.
	Origin ID
	// Value is the inserted element. OpInsert only.
	Value []byte
	// Target is the element to remove. OpDelete only.
	Target ID
}

A ListOp is one indivisible change to a list. Like Op it is self-describing and applying it is idempotent, so a transport may duplicate or reorder freely.

func ParseListOps added in v0.12.0 

func ParseListOps(data []byte) ([]ListOp, error)

ParseListOps decodes a batch written by AppendListOps.

func (ListOp) MarshalBinary added in v0.12.0 

func (o ListOp) MarshalBinary() ([]byte, error)

MarshalBinary encodes the operation. It reports ErrInvalidOp rather than producing bytes that would be rejected on arrival.

func (*ListOp) UnmarshalBinary added in v0.12.0 

func (o *ListOp) UnmarshalBinary(data []byte) error

UnmarshalBinary decodes an operation written by MarshalBinary. Trailing bytes are an error: an operation is decoded from exactly its own encoding.

type Map added in v0.10.0 

type Map struct {
	// contains filtered or unexported fields
}

A Map is a replicated key-value map: any number of replicas may write to it at once, offline, in any delivery order, and every replica ends up holding the same keys and the same values. Values are opaque bytes — the caller encodes whatever it likes — and they are copied in and out, so no slice is ever shared between a caller and the map.

Writes to one key are ordered by the same (clock, site) total order the text uses: a Lamport clock raised past everything the replica has seen, ties broken by site. The highest write wins, and it wins everywhere whatever order the writes arrived in, because a maximum does not depend on the order it is taken in. Writes to different keys never interact at all.

A deleted key keeps its clock

A deletion leaves a record behind rather than dropping the key. Dropping it would leave nothing for an older write arriving later to lose against, so that write would take effect — resurrecting the key on the replica that heard it late and not on the one that heard it early, permanently. Keeping the clock is also what makes a delete and a concurrent set to the same key resolve identically everywhere. Map.Len and Map.Keys do not count a deleted key; Map.Snapshot writes it.

What a replica may forget, and what it may not

A map keeps one record per key, so the value a write put there is gone the moment a later write replaces it. The operation is not gone. Sequence numbers are contiguous per site — that is what lets a VersionVector describe a replica exactly — and Map.Apply never skips one: an operation that arrives before its predecessor waits for it rather than being dropped. A peer catching up therefore has to be told that the number was used, and Map.OpsSince tells it with a MapSuperseded operation, which names no key, carries no value, and covers a whole run of numbers at once. Catching up a peer from nothing therefore costs one operation per key it does not hold plus one per stretch it does not need, never one per write ever made.

That is sound only because the operation which superseded it travels in the same batch: it is either the record now held for that key, which OpsSince sends whenever the peer lacks it, or something the peer already has. A caller that filters what OpsSince returns breaks the map.

A Map is not safe for concurrent use. The zero Map is unusable — construct one with NewMap or LoadMap.

Example

Two replicas write to the same cell at the same time, and one of them then deletes it. The later write wins wherever the operations arrive, and the deleted cell keeps the clock that beat them: an older write turning up afterwards cannot bring it back. 

package main

import (
	"fmt"

	"github.com/go-crdt/crdt"
)

func main() {
	ada, grace := crdt.NewMap(1), crdt.NewMap(2)

	fromAda, err := ada.Set("B7", []byte("41"))
	if err != nil {
		panic(err)
	}
	fromGrace, err := grace.Set("B7", []byte("42"))
	if err != nil {
		panic(err)
	}
	if err := ada.Apply(fromGrace); err != nil {
		panic(err)
	}
	if err := grace.Apply(fromAda); err != nil {
		panic(err)
	}

	value, _ := ada.Get("B7")
	fmt.Printf("%s %s\n", value, ada.Keys())

	// Grace clears the cell, having seen both writes.
	cleared, err := grace.Delete("B7")
	if err != nil {
		panic(err)
	}
	if err := ada.Apply(cleared); err != nil {
		panic(err)
	}
	_, present := ada.Get("B7")
	fmt.Println(present, ada.Len())

}

Output:
42 [B7]
false 0

func LoadMap added in v0.10.0 

func LoadMap(site SiteID, snapshot []byte) (*Map, error)

LoadMap rebuilds a map from a snapshot, to be written as site. The site need not be one that appears in the snapshot — a client joining an existing map brings its own.

A snapshot arrives over a network, so most of what follows is refusing states no replica could have reached. The version vector is what the rest is measured against: a record written by an operation the vector does not promise, or by an operation another record already claims, describes a map that could not reproduce its own history.

func NewMap added in v0.10.0 

func NewMap(site SiteID) *Map

NewMap returns an empty map that issues operations as site. Every replica writing to a map concurrently must pass a distinct site; see SiteID.

func (*Map) Apply added in v0.10.0 

func (m *Map) Apply(ops ...MapOp) error

Apply integrates operations from peers. Duplicates are ignored, and an operation that arrives before the operation its site issued before it is buffered until that one lands, so the caller needs no ordered delivery.

A malformed operation is rejected and nothing in the batch is applied.

func (*Map) ApplyAbsorbed added in v0.32.0 

func (m *Map) ApplyAbsorbed(ops ...MapOp) ([]MapOp, error)

ApplyAbsorbed is Map.Apply, and also reports the operations it integrated, including any that had been parked waiting for them.

func (*Map) ApplyChanges added in v0.13.0 

func (m *Map) ApplyChanges(ops ...MapOp) ([]string, error)

ApplyChanges is Map.Apply, and also reports which keys it changed: those whose value or presence is not what it was, in ascending order so that two replicas given the same batch report the same thing.

Only what actually happened is reported. An operation already applied, one still waiting for the operation its site issued before it, and one that lost to a write already held all change nothing and say nothing; when a waiting one lands, its key is reported then.

A key is named, not its new value. A caller reads what it wants of the key it is told about, which is also what keeps this honest when a batch writes one key twice: the key appears once, and reading it gives the winner.

func (*Map) Collect added in v0.33.0 

func (m *Map) Collect(stable VersionVector) int

Collect drops the tombstones nothing can still be confused by.

A map keeps one record per key rather than one per edit, so it has far less past to give back than a text or a list: what accumulates is not the history of a key but the keys that were deleted. A diagram whose nodes come and go carries every node it ever held, as a record saying that node is gone.

A tombstone is kept for one reason, which [Map.integrate] states: it is what stops an older set resurrecting a key somebody has since deleted. So it may go once no replica can still send a write that would lose to it — which is when the deletion is one every replica has delivered, exactly as elsewhere. Pass a version every replica has; see Doc.Collect for what that means and who can know it.

Why this one needs no format version and no floor

A map already gives back a sequence number without its operation: a second write to a key overwrites the first, so the first is gone and Map.OpsSince reports the gap as MapSuperseded rather than pretending the operation is still here. Collecting a tombstone frees a sequence number the same way, and the same span covers it. Nothing on the wire changes, no peer has to be re-seeded, and a map's loader has no completeness ledger to relax because a map never had a complete history to hold it to.

What it costs

Map.Stamp can no longer say when a collected key was deleted; it answers as it does for a key that never existed. That is the whole of what is lost from reading.

What is lost from *misuse* is worse than elsewhere, which is why the guard below exists. Given a version some replica has not delivered, that replica's write arrives naming a key whose tombstone is gone, finds no record to lose to, and the key comes back — silently, and on that replica only. So a map remembers the highest clock it collected under and refuses a write at or below it for a key it does not hold, with ErrStranded. Under correct use no such operation can arrive at all, so the guard never fires; when it does, it is naming the mistake.

Collect reports how many tombstones it dropped.

func (*Map) CollectedBelow added in v0.33.0 

func (m *Map) CollectedBelow() uint64

CollectedBelow reports the highest clock this replica has collected a tombstone under, and zero if it has collected none.

It is a clock rather than a version, unlike Doc.Floor and List.Floor, because what a map has to recognise is not an operation it dropped but a write that would have lost to one. That comparison is by clock, so the guard is too.

func (*Map) Delete added in v0.10.0 

func (m *Map) Delete(key string) (MapOp, error)

Delete removes key and returns the operation that describes it.

It writes a tombstone whether or not this replica holds the key. What a deletion means cannot depend on what the deleting replica happens to have heard: a peer's write still in flight has to lose to a later deletion on every replica, including one that has not yet seen the write it is beating.

func (*Map) DropPending added in v0.31.0 

func (m *Map) DropPending() int

DropPending forgets the operations this replica is holding back, and returns how many there were.

An operation that arrives before the one it depends on is parked, which is right: it may become applicable a moment later, and dropping it silently would lose an edit. What is not right is that nothing bounds the pile. A peer sending operations that can never apply — each waiting on a sequence number that site never issues — costs about 140 bytes apiece, forever, for a document that stays empty.

This is the lever for that, and it is safe for one reason: a parked operation has had no effect on the state, so it is not in the version vector. A peer asked what this replica is missing sends it again. Dropping and re-syncing therefore loses nothing and diverges from nobody — which is asserted in the tests rather than argued here.

It is deliberately the caller's decision. A cap inside this package would have to choose what to do when it is reached, and the only answers are to drop — which is a policy, not a merge rule — or to refuse an Apply for reasons that have nothing to do with what it was handed.

func (*Map) Get added in v0.10.0 

func (m *Map) Get(key string) ([]byte, bool)

Get returns the value stored at key, and whether the key is present. The value is a copy: writing to it does not change what the map holds.

A value stored as empty reads back as nil — the two are one value here, and the encoding cannot tell them apart either.

func (*Map) Keys added in v0.10.0 

func (m *Map) Keys() []string

Keys returns the keys present, in ascending order, so that anything a caller derives from them is the same on every replica. Deleted keys are not among them.

func (*Map) Len added in v0.10.0 

func (m *Map) Len() int

Len returns how many keys are present, not counting deleted ones.

func (*Map) OpsSince added in v0.10.0 

func (m *Map) OpsSince(vv VersionVector) []MapOp

OpsSince returns the operations this replica holds that vv does not, ready to be sent to the peer that produced vv. Pass a nil vector for everything.

The result is ordered by site and then by sequence number, so the peer can apply it as it arrives without ever having to buffer. Operations whose values have since been overwritten come back as MapSuperseded runs, carrying nothing but the sequence numbers they stand for; the writes that overwrote them are in the same result, which is what makes that safe. See Map.

func (*Map) Pending added in v0.10.0 

func (m *Map) Pending() int

Pending reports how many received operations are still waiting for the operation their own site issued before them.

func (*Map) Rewritten added in v0.33.0 

func (m *Map) Rewritten(site SiteID) (*Map, error)

Rewritten returns a new map with this one's entries, minted at site. A map keeps one record per key rather than a record per edit, so it has far less past to trade than a text does; this exists so a composite's parts can all be rewritten the same way, on the terms described on Doc.Rewritten.

func (*Map) Set added in v0.10.0 

func (m *Map) Set(key string, value []byte) (MapOp, error)

Set stores value at key and returns the operation that describes it. The operation is already applied here; send it to every peer.

The map copies value, so the caller may reuse the slice, and the returned operation carries a copy of its own, so writing to it cannot reach the map. A key that is not valid UTF-8 is refused; see MapOp.

func (*Map) Site added in v0.20.0 

func (m *Map) Site() SiteID

Site returns the replica this map writes as. Doc and List answer the same question the same way.

func (*Map) Snapshot added in v0.10.0 

func (m *Map) Snapshot() []byte

Snapshot encodes the whole map — every key ever written, deleted ones included, each with the identity and Lamport timestamp of the write it holds — plus the version vector. It is what a server sends a client joining an existing session, and what it persists.

Keys are written in ascending order, so two replicas that have applied the same operations produce identical bytes whatever order those operations arrived in. That makes a snapshot a convergence check: agreeing on the keys and values is weaker than agreeing on the state, because two replicas can agree on every value while disagreeing about which write produced it, and would then resolve the next concurrent write differently.

What a snapshot does not keep is the values of writes that have already lost; see Map for what Map.OpsSince sends in their place.

func (*Map) Stamp added in v0.20.0 

func (m *Map) Stamp(key string) (clock uint64, site SiteID, ok bool)

Stamp returns the (clock, site) the current value of key was written at, and whether the key is live. It is the total order the map resolves concurrent writes by, made readable so that something built on the map can order two writes the same way the map did — see structured.Tree, which has to decide which of two concurrent moves happened later.

func (*Map) Tombstones added in v0.10.0 

func (m *Map) Tombstones() int

Tombstones returns how many keys are held only to keep their clock. They are the one thing here that grows without bound, so a caller measuring what a long session costs measures this.

func (*Map) Version added in v0.10.0 

func (m *Map) Version() VersionVector

Version returns what this replica holds, to be handed to a peer that will send back what it is missing; see Map.OpsSince.

type MapOp added in v0.10.0 

type MapOp struct {
	// Kind selects writing, removal, or values the sender has forgotten.
	Kind MapOpKind
	// ID names this operation. Its Seq is the issuing site's own counter and
	// increases by exactly one per operation. For MapSuperseded it is the last
	// of the sequence numbers the operation accounts for.
	ID ID
	// Clock is the Lamport timestamp that orders this operation against
	// concurrent writes to the same key. See the package documentation.
	Clock uint64
	// Key is the key written or removed. MapSet and MapDelete only.
	Key string
	// Value is the bytes written. MapSet only; it may be empty.
	Value []byte
	// Span is how many consecutive sequence numbers this operation accounts for,
	// ending at ID.Seq. MapSuperseded only, where it is at least one.
	Span uint64
}

A MapOp is one indivisible change to a Map. It is the only thing replicas exchange, it is self-describing, and applying it is idempotent, so a transport may duplicate or reorder operations freely.

Which fields carry meaning depends on Kind: Key belongs to MapSet and MapDelete, Value to MapSet alone, Span to MapSuperseded alone. The unused fields must be zero, and are checked, so a garbled operation is rejected rather than silently reinterpreted.

func ParseMapOps added in v0.10.0 

func ParseMapOps(data []byte) ([]MapOp, error)

ParseMapOps decodes a batch written by AppendMapOps.

func (MapOp) MarshalBinary added in v0.10.0 

func (o MapOp) MarshalBinary() ([]byte, error)

MarshalBinary encodes the operation. It reports ErrInvalidOp rather than producing bytes that would be rejected on arrival.

func (*MapOp) UnmarshalBinary added in v0.10.0 

func (o *MapOp) UnmarshalBinary(data []byte) error

UnmarshalBinary decodes an operation written by MarshalBinary. Trailing bytes are an error: an operation is decoded from exactly its own encoding.

type MapOpKind added in v0.10.0 

type MapOpKind uint8

MapOpKind distinguishes the operations a replicated map exchanges. 

const (
	// MapSet writes a value at a key.
	MapSet MapOpKind = 1
	// MapDelete removes a key, and leaves its clock behind. See [Map].
	MapDelete MapOpKind = 2
	// MapSuperseded stands in for operations whose values the sending replica no
	// longer holds, because later writes to the same keys replaced them. It names
	// no key and carries no value: it exists so that a peer catching up can
	// account for the sequence numbers and move on. Only [Map.OpsSince] produces
	// one.
	//
	// It covers a run of consecutive sequence numbers rather than one, because
	// the numbers a replica has forgotten are exactly the gaps between the ones
	// it still holds: a key written a million times leaves one record and one
	// run, not a million operations to send.
	MapSuperseded MapOpKind = 3
)

func (MapOpKind) String added in v0.10.0 

func (k MapOpKind) String() string

String renders the kind for diagnostics.

type Op 

type Op struct {
	// Kind selects insertion or deletion.
	Kind OpKind
	// ID names this operation. Its Seq is the issuing site's own counter and
	// increases by exactly one per operation.
	ID ID
	// Clock is the Lamport timestamp that orders this operation against
	// concurrent ones. See the package documentation.
	Clock uint64
	// Origin is the character the new one is inserted after; the zero ID means
	// the start of the document. OpInsert only.
	Origin ID
	// Char is the inserted character. OpInsert only.
	Char rune
	// Target is the character to tombstone. OpDelete only.
	Target ID
}

An Op is one indivisible change to a document. It is the only thing replicas exchange, it is self-describing, and applying it is idempotent, so a transport may duplicate or reorder operations freely.

Which fields carry meaning depends on Kind: Origin and Char belong to OpInsert, Target to OpDelete. The unused fields must be zero, and are checked, so a garbled operation is rejected rather than silently reinterpreted.

func ParseOps 

func ParseOps(data []byte) ([]Op, error)

ParseOps decodes a batch written by AppendOps.

func (Op) MarshalBinary 

func (o Op) MarshalBinary() ([]byte, error)

MarshalBinary encodes the operation. It reports ErrInvalidOp rather than producing bytes that would be rejected on arrival.

func (*Op) UnmarshalBinary 

func (o *Op) UnmarshalBinary(data []byte) error

UnmarshalBinary decodes an operation written by MarshalBinary. Trailing bytes are an error: an operation is decoded from exactly its own encoding.

type OpKind 

type OpKind uint8

OpKind distinguishes the two operations a text CRDT needs. 

const (
	// OpInsert adds one character after an existing one.
	OpInsert OpKind = 1
	// OpDelete tombstones an existing character.
	OpDelete OpKind = 2
)

func (OpKind) String 

func (k OpKind) String() string

String renders the kind for diagnostics.

type Part added in v0.11.0 

type Part struct {
	Kind PartKind
	Name string
}

A Part names one structure inside a Composite.

The kind is part of the name, not a property of it. Two replicas that have never spoken may each reach for "notes", one as a list and one as a map, and there is no operation to exchange that would tell them so — a part exists because operations for it exist, and those operations were addressed to different things. Identifying a part by name alone would make that pair a conflict needing a tie-break, and whichever way the tie-break fell one replica would find its writes gone. Identifying it by both makes it two parts, which is a convergent answer needing no arbitration and no rule anybody has to know.

A name is arbitrary UTF-8 and is expected to carry structure — "file:src/main.tex", "comment:9f3c…", "chat". It must be valid UTF-8 because a name crosses into JavaScript, where a string is UTF-16 and bytes that are not text cannot survive the trip; it is the same rule Map holds its keys to, for the same reason. It must not be empty, which is a decision rather than an oversight: the name is the only thing telling one part from another, and "" is what a caller passes when the name it meant to compute never got computed. Two unrelated bugs would then share one part, silently, and nothing downstream could notice.

type PartChange added in v0.13.0 

type PartChange struct {
	// Part names the part that changed.
	Part Part
	// Text is the edits to make, in order. PartText only.
	Text []Change
	// Keys names the keys that changed, ascending. PartMap only.
	Keys []string
}

A PartChange is what one part did when a batch was applied. A part that did nothing is not reported at all, so a PartChange existing is itself the news that its part is not what it was — which for a list is the whole of the news.

Which field carries it depends on the kind, and the three differ because what a view has to do with them differs:

  • PartText fills Text with the edits a view of the text has to make, in the order it has to make them. A text editor cannot re-read a document per keystroke and keep a cursor, so it needs the edits themselves.
  • PartMap fills Keys with the keys whose value or presence changed, ascending. A view reads back the keys it is told about.
  • PartList fills neither. A list here holds tens or hundreds of values and the views written against one read it back whole; naming positions would be a second protocol to keep correct for nobody. It can be added later without breaking a caller, which is why it is a field left empty rather than a kind left out.

type PartKind added in v0.11.0 

type PartKind uint8

PartKind names which of the three replicated structures a part is. 

const (
	// PartText is a [Doc], a replicated sequence of characters.
	PartText PartKind = 1
	// PartList is a [List], a replicated sequence of values.
	PartList PartKind = 2
	// PartMap is a [Map], a replicated key-value map.
	PartMap PartKind = 3
)

func (PartKind) String added in v0.11.0 

func (k PartKind) String() string

String renders the kind for diagnostics.

type PartOps added in v0.11.0 

type PartOps struct {
	// Part names what the operations are addressed to.
	Part Part
	// Text carries the operations of a PartText batch.
	Text []Op
	// List carries the operations of a PartList batch.
	List []ListOp
	// Map carries the operations of a PartMap batch.
	Map []MapOp
}

PartOps is a batch of operations addressed to one part. Operations travel grouped rather than one by one because a composite has many parts and few operations each: naming the part once per batch keeps a document of three hundred comment parts from paying its part names three hundred times, and it keeps a batch's operations the one kind its part can hold.

Exactly the field matching the part's kind may be set, and the other two are checked to be empty, so a batch built with the wrong field is refused rather than silently read as an empty one.

func ParsePartOps added in v0.12.0 

func ParsePartOps(data []byte) ([]PartOps, error)

ParsePartOps decodes batches written by AppendPartOps.

There is deliberately no MarshalBinary on PartOps to go with the ones Op, ListOp and MapOp carry. Those three are operations: indivisible, the unit a replica exchanges, and a caller may reasonably want one on its own in a field. A PartOps is not an operation but the envelope addressing a batch of them to a part, and the unit that crosses a wire is the whole set of them — which is what Composite.OpsSince returns and what Composite.Apply takes. Encoding one alone would advertise a message neither end of this package ever produces or consumes, and it would be a fourth format to keep canonical, fuzzed and held to its coverage for no caller. A caller that really has one batch writes AppendPartOps(nil, batches[:1]).

type SiteID 

type SiteID uint64

SiteID identifies a replica. It is chosen by the caller and must be distinct for every replica that concurrently edits a document; two replicas sharing a SiteID can mint the same ID for different characters, which breaks convergence.

The package never generates one itself: a random or clock-derived identifier is unavailable, or not reproducible, under js/wasm. See DeriveSiteID.

func DeriveSiteID 

func DeriveSiteID(b []byte) SiteID

DeriveSiteID hashes b into a SiteID with FNV-1a. It gives callers a deterministic way to turn an identifier they already hold — a session token, a user ID, a tab identifier — into a replica identity, on any platform, including js/wasm where the usual sources of randomness are absent.

Distinctness is the caller's responsibility: distinct b almost always yields distinct SiteIDs, but a hash cannot promise it.

Across instances that have never spoken

That responsibility has a sharp edge the moment more than one instance is involved. A site identity has to be unique across every replica that will ever meet, not merely across the ones one server hands out — two operations claiming one identity is the thing this package rests on not happening, and no merge can recover from it.

So b must carry the instance. Derive from something SCOPED — an eduPersonPrincipalName, a subject-id, an OIDC issuer and subject together, a URL — and never from a bare local identifier: two instances that each have a user "42" would derive the same site from it, on purpose, because a hash is a function and that is what a function does. There is a test of both.

Measured on scoped identifiers of the shape a SAML assertion carries: twenty million of them over four thousand scopes, no collisions, which is what a uniform 64-bit hash gives at that size. See federation_test.go.

Example

Site identities have to be distinct and cannot be drawn at random under js/wasm, so they are derived from something the caller already holds. 

package main

import (
	"fmt"

	"github.com/go-crdt/crdt"
)

func main() {
	first := crdt.DeriveSiteID([]byte("session-8f2c"))
	second := crdt.DeriveSiteID([]byte("session-8f2c"))
	fmt.Println(first == second, first == crdt.DeriveSiteID([]byte("session-91ab")))
}

Output:
true false

type VersionVector 

type VersionVector map[SiteID]uint64

A VersionVector records, per site, the highest sequence number a replica has applied. Because a site's sequence numbers have no gaps and Doc refuses to apply an operation until its predecessor has landed, the vector describes a replica's state exactly: it holds operation Seq from Site if and only if Seq <= v[Site].

The nil vector is valid and reads as "nothing applied".

func (VersionVector) Clone 

func (v VersionVector) Clone() VersionVector

Clone returns an independent copy. The clone of a nil vector is empty but non-nil, so it can be written to.

func (VersionVector) Equal 

func (v VersionVector) Equal(other VersionVector) bool

Equal reports whether v and other describe the same set of operations. Sites recorded with a zero sequence number count as absent, so a nil vector equals an empty one.

func (VersionVector) Get 

func (v VersionVector) Get(site SiteID) uint64

Get returns the highest sequence number applied for site, or zero.

func (VersionVector) Includes 

func (v VersionVector) Includes(id ID) bool

Includes reports whether the operation named by id has been applied.

func (VersionVector) MarshalBinary added in v0.1.1 

func (v VersionVector) MarshalBinary() ([]byte, error)

MarshalBinary encodes the vector. Entries are written in ascending site order, so the same state always produces the same bytes and a caller may compare or cache them.

A replica sends this to be told what it has missed; see Doc.OpsSince.

func (*VersionVector) UnmarshalBinary added in v0.1.1 

func (v *VersionVector) UnmarshalBinary(data []byte) error

UnmarshalBinary decodes a vector written by MarshalBinary. A site listed twice, a sequence number of zero, and trailing bytes are all rejected: each would leave what the vector means dependent on decoding order.

Source Files

View all Source files

Directories

Path Synopsis
Package awareness tracks who else has a document open and where their cursor is — the coloured carets and name labels a collaborative editor shows.
 Package awareness tracks who else has a document open and where their cursor is — the coloured carets and name labels a collaborative editor shows.
Package structured turns the replicated primitives of the crdt package into a shared substrate for co-editing structured documents.
 Package structured turns the replicated primitives of the crdt package into a shared substrate for co-editing structured documents.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.