Documentation
¶
Overview ¶
Package coedit is filex's relay for editing a file together (task #189): the server half of the app SDK's `coedit.*` bridge methods, for an app granted `files:co-edit` (docs/APP-PLUGINS-API.md → Editing together).
The model ¶
Every member's editor applies the same log entries in the same order. The filex page (AppFrame, packages/core lib/coeditSession) seals each entry an app appends with the session's key (AES-256-GCM, lib/e2eoffice) and hands the sealed bytes here. The relay:
- gives every entry its place: an entry is written against the head the writer has seen (AppendRequest.Base) and lands right after it or not at all (ConflictError). The writer binds the place into the entry's authenticated data, so the order the relay hands out is the order the writers sealed, and a reader notices a gap, a swap or a replay;
- keeps the "save lock" promise a Document Server makes with a lease: changes are appended only by the one member holding it, and the lease goes only to a member that has seen every change already in the log;
- writes the entries everybody has to agree on itself: who joined (with the editor's per-session user index, never given twice), who left (and when a member's page went away: dropped after MemberTTL), and which part of the log a save holds;
- keeps the session's sealed blobs (the base document a member who joins late opens, the images added while editing) as files under its directory.
The relay never opens an entry, a cursor or a blob. Who a member is and whether it may write are the server's answer - the account signed in on the request and the file's rules, asked by the HTTP layer - never something the joining connection says about itself.
Keys ¶
A session's key is made once, when it starts, and how it is kept depends on the folder (OpenRequest.KeyMode):
- KeyServer, an ordinary folder: the server makes the key and keeps it sealed with FILEX_SECRET_KEY (Options.Keys); each member's filex page is handed the key when it joins. The server holds the file in the clear anyway, so this is not end-to-end encryption: it is the SAME path as an encrypted folder's, and it keeps the unsaved work in the database and the data directory sealed at rest. Without a secret key an ordinary folder is not edited together (ErrNoServerKey).
- KeyFolder, an encrypted folder: the members' browsers seal the key with the folder key (SetKey), and the server cannot open it.
State ¶
Everything is in the database (migration 00107): sessions move by compare-and-set on their `rev`, so two filex processes on one database can serve one session. Live delivery is per process: a member's socket waits on this process's notice of a new entry (Watcher.Notify) and reads the entries from the database, so a member connected to another process still gets every entry - on that process's poll (Options.PollEvery). Cursor frames are not kept: they reach the members connected to the same process.
Index ¶
- Constants
- Variables
- func DocKey(storageID int64, path string) string
- func MemberKind(k Kind) bool
- func NewToken() (string, error)
- func TokenHash(token string) string
- func ValidBlobName(name string) bool
- func ValidSID(s string) bool
- type AppendRequest
- type ConflictError
- type Departure
- type Entry
- type Eph
- type Hello
- type Identity
- type KeyBox
- type Kind
- type LeaseOp
- type Member
- type OpenRequest
- type Options
- type Relay
- func (r *Relay) Append(ctx context.Context, sid, client string, req AppendRequest) (uint64, error)
- func (r *Relay) Authorise(ctx context.Context, sid, client, token string, userID int64) (*model.CoEditMember, error)
- func (r *Relay) Blob(ctx context.Context, sid, client, name string) (io.ReadCloser, int64, error)
- func (r *Relay) Cursors(ctx context.Context, sid string, sinceMs int64) ([]Eph, error)
- func (r *Relay) Entries(ctx context.Context, sid string, after uint64, limit int) ([]Entry, error)
- func (r *Relay) Lease(ctx context.Context, sid, client string, op LeaseOp, changesSeen uint64) (bool, error)
- func (r *Relay) Leave(ctx context.Context, sid, client, reason string) error
- func (r *Relay) Member(ctx context.Context, sid, client string) (*model.CoEditMember, error)
- func (r *Relay) MemberTTL() time.Duration
- func (r *Relay) Open(ctx context.Context, req OpenRequest, who Identity) (Hello, error)
- func (r *Relay) Options() Options
- func (r *Relay) PollEvery() time.Duration
- func (r *Relay) PutBlob(ctx context.Context, sid, client, name string, body io.Reader) (int64, error)
- func (r *Relay) Run(ctx context.Context)
- func (r *Relay) Saved(ctx context.Context, sid, client string, through uint64, sig string) (uint64, error)
- func (r *Relay) SendEph(sid string, e Eph) error
- func (r *Relay) ServerKeys() bool
- func (r *Relay) Session(ctx context.Context, sid string) (*model.CoEditSession, error)
- func (r *Relay) SetKey(ctx context.Context, sid, client, sealed string) error
- func (r *Relay) Sweep(ctx context.Context) []string
- func (r *Relay) Touch(ctx context.Context, sid, client string) error
- func (r *Relay) Watch(sid, client string) *Watcher
- type Store
- type Watcher
Constants ¶
const ( // KeyServer: the server makes the session key and keeps it sealed with // its secret key - an ordinary folder. KeyServer = "server" // KeyFolder: the members' browsers seal the key with the folder key - an // encrypted folder. The server cannot open it. KeyFolder = "folder" )
Key modes (OpenRequest.KeyMode).
const ( DefaultLeaseTTL = 60 * time.Second DefaultMemberTTL = 45 * time.Second DefaultMaxEntryBytes = 2 << 20 DefaultMaxLogBytes = 64 << 20 DefaultMaxBlobBytes = 256 << 20 DefaultMaxMembers = 20 DefaultSavedIdle = time.Hour DefaultUnsavedRetention = 30 * 24 * time.Hour DefaultNotReadyTTL = 10 * time.Minute DefaultPollEvery = 2 * time.Second // MaxEphBytes caps one sealed cursor frame. MaxEphBytes = 64 << 10 // MaxSealedKeyBytes caps a sealed key a member hands over (KeyFolder). MaxSealedKeyBytes = 4096 )
Defaults.
const BaseBlob = "base"
BaseBlob is the name of the sealed base document every member opens.
const CursorEvery = 500 * time.Millisecond
CursorEvery is the least time between two writes of one member's cursor to the database.
const KeyBytes = 32
KeyBytes is the size of a session key.
const SweepEvery = 15 * time.Second
SweepEvery is how often Run sweeps.
Variables ¶
var ( ErrNoSession = errors.New("coedit: no such session") ErrNotMember = errors.New("coedit: not a member of this session") ErrReadOnly = errors.New("coedit: this member may not write") ErrKind = errors.New("coedit: a member may not append this kind") ErrNoLease = errors.New("coedit: changes need the lease") ErrEmpty = errors.New("coedit: an entry needs a sealed payload") ErrTooLarge = errors.New("coedit: too large") ErrLogFull = errors.New("coedit: the session log is full; save, and the next opening starts a new session") ErrCounter = errors.New("coedit: the counter must grow") ErrBuild = errors.New("coedit: the session runs another build of the app") ErrFileChanged = errors.New("coedit: the file changed outside the session") ErrKeySet = errors.New("coedit: the session key is already set") ErrBlobSet = errors.New("coedit: that blob is already stored") ErrBlobName = errors.New("coedit: bad blob name") ErrNoBlob = errors.New("coedit: no such blob") ErrNotReady = errors.New("coedit: the session has no key or no base yet") ErrSaveBehind = errors.New("coedit: a save of a later point is already recorded") ErrSaveAhead = errors.New("coedit: a save cannot hold entries the log does not have") ErrBadRequest = errors.New("coedit: bad request") ErrNoIdentity = errors.New("coedit: who is joining is not known") ErrFull = errors.New("coedit: the session has as many members as it may") ErrNoServerKey = errors.New("coedit: no secret key to keep a session key with (FILEX_SECRET_KEY)") ErrBusy = errors.New("coedit: the session kept changing under this request") // ErrNothingToWatch: a person who may only read the file opened it while // no session is there to join. A watcher never starts one - its starter // stores the base document every member builds on, which a person who // may not change the file must not choose. ErrNothingToWatch = errors.New("coedit: nobody is editing this file; a watcher starts no session") )
Errors. The HTTP layer turns each into a code and a sentence of its own.
Functions ¶
func ValidBlobName ¶
ValidBlobName reports whether name may name a blob.
Types ¶
type AppendRequest ¶
type AppendRequest struct {
Kind Kind
// Base is the head the writer has seen; the entry lands at Base+1 or not
// at all.
Base uint64
Ctr uint64
CT []byte
}
AppendRequest is a member's sealed entry.
type ConflictError ¶
type ConflictError struct {
Head uint64
}
ConflictError says the entry was written against another head than the log's. The writer catches up to Head, seals again and retries.
func (*ConflictError) Error ¶
func (e *ConflictError) Error() string
type Departure ¶
type Departure struct {
SID string
StorageID int64
Path string
App string
Member model.CoEditMember
// Reason: "left" (it said so), "timeout" (its page went away), "closed"
// (the session ended).
Reason string
}
Departure is a member that left, for the audit trail.
type Entry ¶
type Entry struct {
Seq uint64 `json:"seq"`
Kind Kind `json:"kind"`
Client string `json:"client"`
// At is the relay's clock when the entry landed, Unix milliseconds - the
// same for every member, where a Document Server would use its own.
At int64 `json:"at"`
// Ctr is the writer's counter, bound into the sealed entry.
Ctr uint64 `json:"ctr,omitempty"`
// CT is a member's sealed payload (base64 on the wire).
CT []byte `json:"ct,omitempty"`
// Member is who joined (KindJoin).
Member *Member `json:"member,omitempty"`
// Through is the last entry a save holds (KindSaved).
Through uint64 `json:"through,omitempty"`
}
Entry is one place in a session's log, as the members receive it.
type Eph ¶
type Eph struct {
Client string `json:"client"`
Ctr uint64 `json:"ctr"`
CT []byte `json:"ct"`
// At is when the relay took it (Unix ms) - for the sockets of other
// processes, which read the latest cursors from the database
// (cursors.go).
At int64 `json:"-"`
}
Eph is a frame that is not kept: a cursor, sealed, for the other members.
type Hello ¶
type Hello struct {
SID string
// Created: this member started the session and must store the base
// document (and, KeyFolder, the sealed key) before anybody else joins.
Created bool
Me Member
Head uint64
ChangesHead uint64
SavedThrough uint64
KeyMode string
// Key is the raw session key (KeyServer).
Key []byte
// SealedKey is the sealed session key (KeyFolder), "" until set.
SealedKey string
}
Hello is what a new member needs to start.
type Identity ¶
type Identity struct {
User int64
// Person is the person's presence identity (the account, and the end
// user a shared proxy token names): two of them behind one proxy token
// are two people. Empty = the account alone.
Person string
Name string
CanEdit bool
// TokenHash is hex SHA-256 of the token the member is handed: the
// requests that name the member carry it (MemberToken checks it).
TokenHash string
}
Identity is who a joining member is, as the SERVER knows it: the account signed in on the request and the file's rules - never anything the caller sent.
type KeyBox ¶
type KeyBox interface {
Enabled() bool
SealFor(plaintext, aad string) (string, error)
OpenFor(stored, aad string) (string, error)
}
KeyBox seals and opens the keys the server makes (internal/secretbox.Box).
type Kind ¶
type Kind string
Kind names what an entry is. The first three are written by members and carry a sealed payload the relay cannot read; the last three are the relay's own and carry plain metadata.
const ( // KindChanges is one chunk of an editor's changes. KindChanges Kind = "changes" // KindLock is an editor's request for locks. KindLock Kind = "lock" // KindRelease releases an editor's locks. KindRelease Kind = "release" // KindJoin records a member joining, with the indexUser it was given. KindJoin Kind = "join" // KindLeave records a member leaving (or dropped). KindLeave Kind = "leave" // KindSaved records a save: the file now holds the log up to Through. KindSaved Kind = "saved" )
type Member ¶
type Member struct {
Client string `json:"client"`
// User is an opaque id of the person in THIS session (the same for each
// of their connections in it), never an account id.
User string `json:"user"`
Name string `json:"name"`
// IndexUser is the editor's per-session user index: given once, never
// twice in a session, not even after its member left.
IndexUser int `json:"index_user"`
CanEdit bool `json:"can_edit"`
}
Member is one connection in a session, as the members are told.
type OpenRequest ¶
type OpenRequest struct {
// Doc is the file's key (DocKey: a digest of the storage and the path).
Doc string
StorageID int64
Path string
// App and Build: the app and the build its members run. Members of one
// session run one build: another build may write entries this one cannot
// read.
App string
Build string
// FileSig is the file as the opener's server read it (size:mtime, etag).
// A session goes on from the file it started from or from its last save;
// a file that changed outside it is not the document its log applies to.
FileSig string
// KeyMode is KeyServer (an ordinary folder) or KeyFolder (an encrypted
// one).
KeyMode string
}
OpenRequest asks for the session of one file.
type Options ¶
type Options struct {
Store Store
// BlobDir is where the sealed blobs are kept, one folder per session
// (<data_dir>/coedit). Several processes on one database must share it,
// as they share the upload staging directory.
BlobDir string
// Keys seals the keys of KeyServer sessions. nil or not enabled: an
// ordinary folder is not edited together (ErrNoServerKey).
Keys KeyBox
// LeaseTTL is how long the changes lease lasts without a renewal (the
// Document Server's save lock expires after 60 s).
LeaseTTL time.Duration
// MemberTTL is how long a member whose page gives no sign of life stays.
MemberTTL time.Duration
// MaxEntryBytes caps one sealed payload.
MaxEntryBytes int
// MaxLogBytes caps a session's log: past it a member has to save, and
// the next opening starts a new session.
MaxLogBytes int64
// MaxBlobBytes caps a session's blobs together.
MaxBlobBytes int64
// MaxMembers caps the members of one session.
MaxMembers int
// SavedIdle is how long an empty session whose log is all saved is kept.
SavedIdle time.Duration
// UnsavedRetention is how long an empty session with unsaved changes is
// kept, for the next person to open the file to see and save them.
UnsavedRetention time.Duration
// NotReadyTTL is how long a session may wait for its key and its base.
NotReadyTTL time.Duration
// PollEvery is how often a member's socket reads the log without a
// notice from this process (another process wrote the entry).
PollEvery time.Duration
// OnDeparture is told about every member that left, after it is recorded.
OnDeparture func(ctx context.Context, d Departure)
// Now is the clock (tests).
Now func() time.Time
}
Options tune a Relay. The zero value of each duration and size takes its default.
type Relay ¶
type Relay struct {
// contains filtered or unexported fields
}
Relay is the editing-together relay. One per process; safe for concurrent use, and for several processes on one database.
func (*Relay) Authorise ¶
func (r *Relay) Authorise(ctx context.Context, sid, client, token string, userID int64) (*model.CoEditMember, error)
Authorise returns the member a request names, when the request is that member's: the same account, and the member's token. Anything else is ErrNotMember - a request that names somebody else's connection learns nothing about it.
func (*Relay) Cursors ¶
Cursors returns the latest cursors of a session's members written after sinceMs (Unix ms), oldest first: what a socket reads on its poll. The caller checked the reader is a member, and skips its own.
func (*Relay) Entries ¶
Entries returns up to limit entries after `after`, in order. The caller checked the reader is a member.
func (*Relay) Lease ¶
func (r *Relay) Lease(ctx context.Context, sid, client string, op LeaseOp, changesSeen uint64) (bool, error)
Lease acquires (or renews) the changes lease, or releases it. It is given only to a writer that has seen every change in the log (changesSeen is the seq of the last changes entry it applied) while nobody else holds it: the Document Server's isSaveLock, which refuses an editor that is behind and one whose turn it is not.
func (*Relay) Leave ¶
Leave removes a member: its lease goes, its socket is told, and the log records it (every bridge then drops its locks). reason is "left" or "timeout".
func (*Relay) Member ¶
Member returns a member of a session (ErrNotMember when it is not one). The HTTP layer checks with it that a request naming a member comes from that member's account.
func (*Relay) Open ¶
Open joins the caller to the file's session, starting it when there is none (or the one there makes way: nobody is in it and everything in it is saved). A session nobody may join yet - its creator is still storing the base document - answers ErrNotReady; ask again in a moment.
func (*Relay) Options ¶
Options returns the relay's options with their defaults (the HTTP layer reads the limits).
func (*Relay) PutBlob ¶
func (r *Relay) PutBlob(ctx context.Context, sid, client, name string, body io.Reader) (int64, error)
PutBlob stores a sealed blob of a session, once per name, and returns its size. The base document may be stored by any member while the session has none (its creator, in practice: nobody else joins before); any other blob only by a member that may write. The session's blobs together stay under Options.MaxBlobBytes.
func (*Relay) Saved ¶
func (r *Relay) Saved(ctx context.Context, sid, client string, through uint64, sig string) (uint64, error)
Saved records that a member saved the file with the log up to `through` in it, and the file's signature after the write - the server's own reading of the file, never the member's word. Saves only move forward.
func (*Relay) SendEph ¶
SendEph passes a sealed cursor frame to the other members connected to this process. It is not kept. The caller checked the member may write.
func (*Relay) ServerKeys ¶
ServerKeys reports whether an ordinary folder can be edited together here.
func (*Relay) SetKey ¶
SetKey stores the sealed session key of a KeyFolder session, once - by a member, before anybody else may join.
func (*Relay) Sweep ¶
Sweep drops the members whose page went away (unseen for MemberTTL: a leave entry, the lease goes, every bridge drops their locks) and forgets the sessions nobody is in any more: a saved one after SavedIdle, one with unsaved changes after UnsavedRetention (30 days - for the next person to open the file to see and save them), one that never got its key or base after NotReadyTTL. It returns the ids of the sessions it forgot.
type Store ¶
type Store interface {
WithTx(ctx context.Context, fn func(ctx context.Context) error) error
db.CoEditStore
}
Store is what the relay keeps its state in: db.Store's editing-together half (db.CoEditStore) and its transactions.
type Watcher ¶
type Watcher struct {
SID string
Client string
// Notify gets a signal (never more than one waiting) when an entry lands
// in this process: read the log from where the socket is.
Notify chan struct{}
// Eph gets the other members' cursors. A full buffer drops them: a
// cursor is worth nothing late.
Eph chan Eph
// Ended gets why the membership ended ("left", "timeout", "closed").
Ended chan string
// contains filtered or unexported fields
}
Watcher is one member's open socket in this process.