Documentation
¶
Overview ¶
Package knowledge is the registry's store of knowledge packs: the packs a signed-in person, or an organization, keeps in Agentry, each held as its tree sealed under a data key of its own (pkg/packseal) with that key wrapped for its owner by a KeyWrapper, scanned when it arrives, and destroyed with its owner. The core carries all of it without certificates: its wrapper is the deployment's knowledge sealing key; an extension (a certificate wrapper) replaces the wrapper and nothing else.
Index ¶
- Constants
- Variables
- func DecodeSealingKey(b64 string) ([]byte, error)
- func EraseOwner(ctx context.Context, q *storegen.Queries, owner Owner) ([]string, error)
- func MaxUpload() int64
- func ValidName(name string) bool
- func WriteTree(w io.Writer, files []scan.PackFile) error
- type Admit
- type AllocationError
- type Also
- type DeploymentWrapper
- func (d *DeploymentWrapper) Destroy(ctx context.Context, owner Owner) error
- func (d *DeploymentWrapper) Recipient() string
- func (d *DeploymentWrapper) Unwrap(ctx context.Context, owner Owner, w WrappedKey) ([]byte, error)
- func (d *DeploymentWrapper) Wrap(ctx context.Context, owner Owner, dataKey []byte) (WrappedKey, error)
- type Deps
- type EraseArgs
- type Info
- type KeyWrapper
- type Owner
- type OwnerKind
- type OwnerSalts
- type Pool
- type Pools
- type ScanArgs
- type Scanner
- type Service
- func (s *Service) Configured() bool
- func (s *Service) Create(ctx context.Context, owner Owner, name string, w WrappedKey, also Also) (storegen.KnowledgePack, error)
- func (s *Service) CreateSealed(ctx context.Context, owner Owner, name string) (Info, error)
- func (s *Service) Delete(ctx context.Context, owner Owner, id uuid.UUID, also Also) error
- func (s *Service) Export(ctx context.Context, owner Owner, id uuid.UUID) (storegen.KnowledgePack, *Tree, error)
- func (s *Service) ExportIf(ctx context.Context, owner Owner, id uuid.UUID, ...) (storegen.KnowledgePack, *Tree, error)
- func (s *Service) FinishErasure(ctx context.Context, owner Owner, keys []string)
- func (s *Service) Get(ctx context.Context, owner Owner, id uuid.UUID) (storegen.KnowledgePack, error)
- func (s *Service) Info(ctx context.Context, owner Owner, id uuid.UUID) (Info, error)
- func (s *Service) List(ctx context.Context, owner Owner) ([]storegen.KnowledgePack, error)
- func (s *Service) RegisterHandlers(q queue.Client)
- func (s *Service) Replace(ctx context.Context, owner Owner, id uuid.UUID, from string, ...) (info Info, changed bool, err error)
- func (s *Service) SealedTree(ctx context.Context, owner Owner, id uuid.UUID) (storegen.KnowledgePack, io.ReadCloser, error)
- func (s *Service) SetExportable(ctx context.Context, owner Owner, id uuid.UUID, exportable bool, also Also) (storegen.KnowledgePack, error)
- func (s *Service) SetWrapper(w KeyWrapper)
- func (s *Service) Sizes(ctx context.Context, owner Owner) ([]storegen.ListKnowledgePackSizesRow, error)
- func (s *Service) SweepScans(ctx context.Context, cutoff time.Time) (int, error)
- func (s *Service) TreeOf(ctx context.Context, owner Owner, id uuid.UUID) (Info, []byte, error)
- func (s *Service) Unwrap(ctx context.Context, owner Owner, w WrappedKey) ([]byte, error)
- func (s *Service) Upload(ctx context.Context, owner Owner, id uuid.UUID, sealed []byte) (storegen.KnowledgePack, error)
- func (s *Service) UploadAdmitted(ctx context.Context, owner Owner, id uuid.UUID, sealed []byte, admit Admit) (storegen.KnowledgePack, error)
- func (s *Service) Usage(ctx context.Context, owner Owner) (Usage, error)
- func (s *Service) Wrap(ctx context.Context, owner Owner, dataKey []byte) (WrappedKey, error)
- func (s *Service) Wrapper() KeyWrapper
- type State
- type StoreSalts
- type Tree
- type UploadAdmission
- type Usage
- type WrappedKey
Constants ¶
const ( JobScan = "knowledge.scan" JobErase = "knowledge.erase" JobScanSweep = "knowledge.scan.sweep" )
The job kinds: the pack scan runs on the sandbox queue, as a version's scan does, because it reads what an account uploaded; the retry of an erasure's last step and the sweep that queues a lost scan again on the core queue, since they read no content.
const ( WikiDir = "wiki" SourcesDir = "sources" )
The two directories a pack's tree holds (agentry-cli pkg/knowledge).
const DeploymentKind = "deployment"
DeploymentKind is the core wrapper's kind.
const MaxBytes = manifest.MaxKnowledgePackBytes
MaxBytes caps a pack: every file of its tree, in MiB as the CLI counts it (agentry-cli pkg/knowledge.MaxBytes).
const MaxFiles = manifest.MaxKnowledgePackFiles
MaxFiles caps the files of one pack's tree.
const MaxTreeBytes = MaxBytes + MaxFiles*2048 + 1024
MaxTreeBytes bounds a tree's tar: its files, and for each at most a PAX header, a ustar header and a block of padding, and the end blocks.
const SaltSize = 32
SaltSize is an owner salt's length.
const ScanStallAfter = 2 * time.Hour
ScanStallAfter is how long a pack waits scanning since its tree was stored, or since the sweep last queued its scan, before the sweep queues the scan again: past a scan job's whole retry budget (three attempts at the queue's job timeout, about 75 minutes at worst), so a scan still being tried is not doubled, with room for a sandbox queue that is behind.
Variables ¶
var ( // ErrNotOwner is a wrapped key that does not open for the owner who // presents it: wrapped for someone else, under a key since destroyed, // or altered. Which of those is not the caller's to learn. ErrNotOwner = errors.New("knowledge: the wrapped key is not this owner's") // ErrUnconfigured is a deployment with no knowledge sealing key and no // extension wrapper: it stores no packs. ErrUnconfigured = errors.New("knowledge: no key wrapper is configured") )
var ( // ErrNotFound is a pack the caller does not own, or that does not // exist: the two are one answer. ErrNotFound = errors.New("knowledge: no such pack") // ErrNameTaken is a second pack of one name for one owner. ErrNameTaken = errors.New("knowledge: the owner already has a pack of this name") // ErrName is a name that is not a pack's. ErrName = errors.New("knowledge: a pack name is 1-64 lowercase letters, digits, '.', '-' or '_', starting with a letter or a digit") // ErrScanning is a pack whose tree is being scanned: a pack is // unusable only then. ErrScanning = errors.New("knowledge: the pack is being scanned") )
var ( // ErrTooLarge is a tree past MaxBytes or MaxFiles. ErrTooLarge = fmt.Errorf("knowledge: a pack is at most %d MB and %d files", MaxBytes>>20, MaxFiles) // ErrDigest is a tree whose bytes are not the digest it names. ErrDigest = errors.New("knowledge: the tree does not match its digest") // ErrTree is a tree that is not a pack's canonical tar. ErrTree = errors.New("knowledge: not a knowledge pack's tree") )
var ErrChanged = errors.New("knowledge: the pack's tree changed since it was read")
ErrChanged is a replace whose pack no longer holds the tree it was read with: another upload or build landed in between.
var ErrNotExportable = errors.New("knowledge: the pack is not marked exportable")
ErrNotExportable is an organization's shared pack its admins have not marked exportable, exported by a member.
Functions ¶
func DecodeSealingKey ¶
DecodeSealingKey decodes a base64 knowledge sealing key: 32 bytes.
func EraseOwner ¶
EraseOwner deletes every pack of owner on q, the caller's transaction — the account erasure, a data clear, an organization's decommission — and the salt the core's wrapper derives the owner's keys from, so that the crypto-erasure commits with the erasure itself. It returns the blob keys, which FinishErasure takes once the transaction has committed. It runs whether or not the store is on, so that turning it off strands nothing an erasure should remove.
func MaxUpload ¶
func MaxUpload() int64
MaxUpload bounds the body of a tree upload: a sealed tree of the largest tar a pack can be.
func ValidName ¶
ValidName is the CLI's rule for a pack name (agentry-cli pkg/knowledge.ValidName): the name is the pack's directory.
Types ¶
type Admit ¶
type Admit func(ctx context.Context, a UploadAdmission) (ticket string, release func(stored bool), err error)
Admit admits an upload that will queue a scan. It is asked once, after the tree is read and checked against the allocation, and before anything is stored; an error refuses the upload with nothing stored. ticket, when not empty, is recorded as stored (admitted_stores, kind "knowledge") in the transaction that stores the tree. release is called once with whether that transaction committed, so a ticket for an upload the core then refused (the allocation cap, a lost race, a panic) can be voided.
type AllocationError ¶
AllocationError is an upload past the owner's allocation: what they would keep and the cap.
func (*AllocationError) Error ¶
func (e *AllocationError) Error() string
type Also ¶
type Also func(q *storegen.Queries, p storegen.KnowledgePack) error
Also is what a caller records in the same transaction as a change to a pack — an organization's audit event — so the change and its record commit or roll back together. It sees the pack as the change left it.
type DeploymentWrapper ¶
type DeploymentWrapper struct {
// contains filtered or unexported fields
}
DeploymentWrapper is the core's KeyWrapper: AES-256-GCM under a key derived, per owner, from the deployment's knowledge sealing key (AGENTRY_KNOWLEDGE_SEALING_KEY, a key of its own) and the owner's random salt, with the owner named in the additional data. A core-only registry so stores trees it can open only for their owners, and an owner's erasure deletes the salt, after which no copy of their wrapped keys — not the pack rows', not one on a laptop — opens again.
func NewDeploymentWrapper ¶
func NewDeploymentWrapper(key []byte, salts OwnerSalts) (*DeploymentWrapper, error)
NewDeploymentWrapper builds the core wrapper over a 32-byte key.
func (*DeploymentWrapper) Destroy ¶
func (d *DeploymentWrapper) Destroy(ctx context.Context, owner Owner) error
Destroy implements KeyWrapper: the owner's salt goes, and with it every key-encryption key their wrapped keys need.
func (*DeploymentWrapper) Recipient ¶
func (d *DeploymentWrapper) Recipient() string
Recipient is the id the wrapper names its wrapped keys with: a digest of the key, never the key.
func (*DeploymentWrapper) Unwrap ¶
func (d *DeploymentWrapper) Unwrap(ctx context.Context, owner Owner, w WrappedKey) ([]byte, error)
Unwrap implements KeyWrapper.
func (*DeploymentWrapper) Wrap ¶
func (d *DeploymentWrapper) Wrap(ctx context.Context, owner Owner, dataKey []byte) (WrappedKey, error)
Wrap implements KeyWrapper.
type Deps ¶
type Deps struct {
Store *store.Store
Blobs registry.BlobStore
Wrapper KeyWrapper
Scanner Scanner
Jobs queue.Client
// Entitlements answers the owner's allocation; read on every upload,
// so a resolver swapped in later (an extension's) is the one asked.
Entitlements func() entitlements.Resolver
Log *slog.Logger
}
Deps is what the service is built from.
type EraseArgs ¶
type EraseArgs struct {
Kind OwnerKind `json:"kind"`
ID uuid.UUID `json:"id"`
Keys []string `json:"keys,omitempty"`
}
EraseArgs names an erasure's last step: the owner whose keys the wrapper destroys, and the sealed blobs to delete.
type Info ¶
type Info struct {
ID uuid.UUID
Name string
State State
Digest string // "" for an empty pack
Bytes int64
Files int
Pages int
}
Info is what an extension reads of a pack: never a key.
type KeyWrapper ¶
type KeyWrapper interface {
// Wrap seals dataKey for owner.
Wrap(ctx context.Context, owner Owner, dataKey []byte) (WrappedKey, error)
// Unwrap opens a wrapped key for owner, or refuses with ErrNotOwner.
Unwrap(ctx context.Context, owner Owner, wrapped WrappedKey) ([]byte, error)
// Destroy makes every key ever wrapped for owner unopenable, wherever
// a copy of it is kept. It is the owner's erasure, a data clear, or an
// organization's decommission; it is idempotent.
Destroy(ctx context.Context, owner Owner) error
}
KeyWrapper wraps a pack's data key for its owner and unwraps it again, and destroys what makes an owner's wrapped keys open. The registry is the only party that unwraps: a data key leaves it over TLS to its owner, and no route ever returns what wraps it.
func NewCoreWrapper ¶
func NewCoreWrapper(sealingKey string, devMode bool, salts OwnerSalts) (KeyWrapper, error)
NewCoreWrapper is the core's wrapper from configuration: the knowledge sealing key (base64), or in dev mode without one a fixed insecure key, as the auth sealer has. With neither it is nil, which leaves the store off unless an extension wraps keys.
type Owner ¶
Owner is a pack's owner, by the id of its account or its organization.
type OwnerSalts ¶
type OwnerSalts interface {
// Salt returns the owner's salt, creating it first when create is set;
// with create unset, an owner with none gets ErrNotOwner.
Salt(ctx context.Context, owner Owner, create bool) ([]byte, error)
// DeleteSalt removes the owner's salt, if any.
DeleteSalt(ctx context.Context, owner Owner) error
}
OwnerSalts keeps the random salt each owner's key-encryption key is derived with. Deleting an owner's salt is what destroys their keys.
type Pool ¶
type Pool struct {
// Key names the pool: every upload into it is reserved under one
// transaction-scoped lock on the key, so two owners who share it
// cannot both pass its cap.
Key string
// Owners are every owner whose packs count against the pool, the one
// asking included.
Owners []Owner
// Cap is the pool's allocation; a cap never deletes.
Cap entitlements.Cap
}
Pool is the allocation an owner's knowledge packs draw on: the owner alone in the core, or, where an extension pools owners, a paid organization's seats shared by its packs and its members'.
type Pools ¶
Pools is the optional interface of an entitlements resolver that pools owners' knowledge. The core asks the resolver it holds, so an extension's resolver answers; an error is not permission. The contract the core cannot check: every owner a pool lists must be answered with that same pool (the same Key), or two of them are not serialised on one lock. Pooling discloses the pool's total to every owner in it — the meter and the refusal name it — which a shared cap cannot avoid.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service is the registry's knowledge store.
func NewService ¶
NewService builds the store. A nil Wrapper leaves it off: every call answers ErrUnconfigured.
func (*Service) Configured ¶
Configured says whether the store is on.
func (*Service) Create ¶
func (s *Service) Create(ctx context.Context, owner Owner, name string, w WrappedKey, also Also) (storegen.KnowledgePack, error)
Create makes an empty pack whose data key is w, which must open for owner: a pack is never created with a key its owner cannot use. also, when given, runs in the same transaction.
func (*Service) CreateSealed ¶
CreateSealed makes an empty pack for owner under a data key the registry makes and its wrapper wraps for owner; the key is kept only wrapped.
func (*Service) Delete ¶
Delete removes one of owner's packs, its objects and their blobs. The pack row is locked first, so an upload racing the delete either lands before it (and its object goes with the pack) or finds no pack. also, when given, runs in the same transaction with the pack as it stood before it went.
func (*Service) Export ¶
func (s *Service) Export(ctx context.Context, owner Owner, id uuid.UUID) (storegen.KnowledgePack, *Tree, error)
Export is one of owner's packs' tree, decrypted: the canonical tar whose digest is the pack's. It is opened and checked in full before a byte is written, so a tree that does not open writes nothing.
func (*Service) ExportIf ¶
func (s *Service) ExportIf(ctx context.Context, owner Owner, id uuid.UUID, allow func(storegen.KnowledgePack) error) (storegen.KnowledgePack, *Tree, error)
ExportIf is Export, refused when allow answers an error for the pack as the export read it (nil: always allowed), so that the rule and the tree come from one read of the row.
func (*Service) FinishErasure ¶
FinishErasure is an erasure's last step, after its transaction committed: the wrapper destroys what opens the owner's keys (an extension's wrapper holds its own; the core's salt is already gone), then the sealed blobs go. A step that fails is handed to a retrying job and logged, never hidden, and undoes nothing: a blob left behind is sealed under a data key no one can unwrap any more.
func (*Service) Get ¶
func (s *Service) Get(ctx context.Context, owner Owner, id uuid.UUID) (storegen.KnowledgePack, error)
Get is one of owner's packs.
func (*Service) RegisterHandlers ¶
RegisterHandlers wires the scan job, the sweep of stalled scans and the erasure's retry.
func (*Service) Replace ¶
func (s *Service) Replace(ctx context.Context, owner Owner, id uuid.UUID, from string, files []scan.PackFile) (info Info, changed bool, err error)
Replace makes files the tree of one of owner's packs, provided the pack still holds the tree whose digest is from ("" for an empty pack): the files written as the canonical tar, sealed under the pack's own data key with the stored header's pack id kept, and stored as an upload is — opened and checked, counted against its pool under the pool's lock, the tree it replaces deleted, its scan queued. It answers ErrChanged when the pack moved on, the upload's errors otherwise (ErrTree, ErrTooLarge, *AllocationError). A tree the pack already holds is not stored again, and changed is false, whatever from says: the pack is already what the caller asked for, so a replace retried after it landed is not a conflict.
func (*Service) SealedTree ¶
func (s *Service) SealedTree(ctx context.Context, owner Owner, id uuid.UUID) (storegen.KnowledgePack, io.ReadCloser, error)
SealedTree is the stored sealed tree of one of owner's packs.
func (*Service) SetExportable ¶
func (s *Service) SetExportable(ctx context.Context, owner Owner, id uuid.UUID, exportable bool, also Also) (storegen.KnowledgePack, error)
SetExportable marks whether one of an organization's packs may be exported by its members, with also run in its transaction. A person's own pack has no such mark: its owner exports it.
func (*Service) SetWrapper ¶
func (s *Service) SetWrapper(w KeyWrapper)
SetWrapper replaces the key wrapper: an extension's (pkg/extension WrapsKeys) in place of the deployment key.
func (*Service) Sizes ¶
func (s *Service) Sizes(ctx context.Context, owner Owner) ([]storegen.ListKnowledgePackSizesRow, error)
Sizes is what platform staff see of owner's packs: names and sizes.
func (*Service) SweepScans ¶
SweepScans queues the scan again of every pack still scanning since before cutoff, its tree stored or its scan last queued then, and reports how many. It is how a pack whose scan was never queued, or whose job was lost, reaches ready or failed: a scan's own last attempt fails the pack rather than leave it scanning.
func (*Service) TreeOf ¶
TreeOf is one of owner's packs' tree, decrypted, as the canonical tar whose SHA-256 is the pack's digest. A pack being scanned is ErrScanning, an empty one ErrNotFound, as the export answers.
func (*Service) Upload ¶
func (s *Service) Upload(ctx context.Context, owner Owner, id uuid.UUID, sealed []byte) (storegen.KnowledgePack, error)
Upload stores a pack's new sealed tree and queues its scan. The registry opens it first — the data key unwrapped for owner, the tree read and its digest checked — so that what it keeps is always a pack, and counts it against the allocation owner's packs draw on, under a lock on that pool, then stores it sealed as it came.
func (*Service) UploadAdmitted ¶
func (s *Service) UploadAdmitted(ctx context.Context, owner Owner, id uuid.UUID, sealed []byte, admit Admit) (storegen.KnowledgePack, error)
UploadAdmitted is Upload with admit asked first. An upload of the tree the pack already holds, scanned or being scanned, stores nothing and queues no scan, so it is never admitted; one that failed to scan is scanned again unadmitted, since the failure was the registry's.
func (*Service) Wrapper ¶
func (s *Service) Wrapper() KeyWrapper
Wrapper is the key wrapper in use, nil when the store is off.
type State ¶
type State string
State is where a pack is in its life.
const ( StateEmpty State = "empty" StateScanning State = "scanning" StateReady State = "ready" StateFailed State = "failed" )
The states a pack is in: empty until a tree is uploaded, scanning from each upload until its scan ends, ready after it whatever it found, failed when the stored tree could not be read to scan.
type StoreSalts ¶
type StoreSalts struct {
// contains filtered or unexported fields
}
StoreSalts keeps owner salts in knowledge_owner_keys.
func NewStoreSalts ¶
func NewStoreSalts(st *store.Store) *StoreSalts
NewStoreSalts is the database's OwnerSalts.
func (*StoreSalts) DeleteSalt ¶
func (s *StoreSalts) DeleteSalt(ctx context.Context, owner Owner) error
DeleteSalt implements OwnerSalts.
type Tree ¶
type Tree struct {
Files []scan.PackFile
// Bytes is the files' size, Pages the wiki's pages (the index and the
// log aside), as the CLI counts them.
Bytes int64
Pages int
// Digest is the SHA-256 of the tar, "sha256:<hex>".
Digest string
}
Tree is a pack's tree as the registry read it.
func ReadTree ¶
ReadTree reads a pack's tree from the canonical tar its digest is taken over (agentry-cli pkg/knowledge.TreeDigest: regular files under wiki/ and sources/, in byte order of their paths) and checks that its bytes are digest. It refuses anything else a tar can carry — a link, a device, a directory entry, a path that climbs or is not the pack's, a name twice — and a tree past the pack's caps.
type UploadAdmission ¶
UploadAdmission is what an upload that will queue a scan is admitted on: the pack and the tree read from the body.
type Usage ¶
type Usage struct {
// Bytes and Packs are the owner's own packs.
Bytes int64
Packs int
// Used is what counts against the allocation: Bytes, or the pool's
// total when Pooled.
Used int64
// Max is the allocation; 0 is no cap.
Max int64
Pooled bool
}
Usage is an owner's meter: what the owner keeps, and the allocation it draws on — the owner's own, or a pool an extension shares among owners, whose total every owner in it sees.
type WrappedKey ¶
WrappedKey is a pack's data key as its owner's wrapper sealed it: which wrapper (Kind), to which key (Recipient: the deployment key's id, or, with an extension's wrapper, a certificate's serial), and the sealed bytes. It is what a client keeps beside a sealed pack and what a pack's row holds.
func WrappedKeyOf ¶
func WrappedKeyOf(p storegen.KnowledgePack) WrappedKey
WrappedKeyOf is a pack's data key as its row holds it.