Documentation
¶
Overview ¶
Package adhocdata implements the ad-hoc dataset store of ADR-0134: ephemeral tabular data an app publishes for a SQL applet to query, held as chunk-encrypted Arrow files whose keys live only in process memory. This file carries the on-disk format — a segmented-AEAD stream (the STREAM construction) that authenticates incrementally at constant memory and detects truncation, so a crash leaves ciphertext whose key no longer exists rather than readable data.
Format (little-endian lengths, big-endian nonce fields):
header = magic "BXAD" (4) | version u8 (1) | chunk-size u32 (4) // 9 bytes chunk = ct-len u32 (4) | ciphertext (ct-len, plaintext+GCM tag) stream = header chunk* // ≥1 chunk
Each chunk is sealed with AES-256-GCM under a 12-byte nonce built as an 8-byte big-endian chunk counter followed by a 4-byte flags word whose bit 0 marks the final chunk. The header bytes are the AAD, so the version and chunk size are authenticated on every chunk. A non-final chunk always carries exactly chunk-size plaintext; the final chunk carries 0..chunk-size and is always present (an empty dataset is a single empty final chunk). Because finality is bound into the nonce, truncating the stream — dropping the final chunk, or cutting a chunk short — makes the last readable chunk fail authentication, so no truncated prefix is ever accepted as complete.
Index ¶
- Constants
- Variables
- func DisableCoreDumps(log zerolog.Logger)
- func RetractRequest(bus app.BusI, handle string) (err error)
- func StructureFor(schema *arrow.Schema) (structure string, err error)
- func SubscribeEvents(bus app.BusI, handler func(ev Event)) (unsubscribe func(), err error)
- type Config
- type DecryptorI
- type Event
- type EventOpE
- type GrantResult
- type KeyRegistrarI
- type PlaintextI
- type PublishInput
- type PublishResult
- type Reader
- type Ref
- type ResolveResult
- type SeekableReader
- type Service
- func (inst *Service) Close(context.Context) (err error)
- func (inst *Service) FlushRetracts()
- func (inst *Service) Grant(handle string) (res GrantResult, err error)
- func (inst *Service) IsLive(handle string) (live bool)
- func (inst *Service) Publish(in PublishInput) (res PublishResult, err error)
- func (inst *Service) Resolve(alias string) (res ResolveResult, err error)
- func (inst *Service) Retract(handle string) (err error)
- type Writer
Constants ¶
const ( // KeySize is the AES-256 key length in bytes. KeySize = 32 // ChunkSize is the plaintext bytes per non-final chunk (ADR-0134 // SD1). 64 KiB keeps per-chunk overhead negligible while bounding // the reader's working set. ChunkSize = 64 * 1024 )
const ( // PerDatasetMaxBytes caps one dataset's ciphertext. PerDatasetMaxBytes = 256 << 20 // 256 MiB // StoreMaxBytes caps the whole store. StoreMaxBytes = 1 << 30 // 1 GiB // MaxDatasets caps how many datasets may coexist. MaxDatasets = 64 )
Quotas bound the store (ADR-0134 SD1). A publish that would breach one is refused with a named error, never discovered at query time.
const ( SubjectPublish = "adhoc.publish" SubjectGrant = "adhoc.grant" SubjectRetract = "adhoc.retract" SubjectResolve = "adhoc.resolve" )
Capability subjects (request/reply, CBOR, audited — ADR-0134 SD2, the ADR-0026 taxonomy).
const ( SubjectEventPublished = "adhoc.event.published" SubjectEventRetracted = "adhoc.event.retracted" SubjectEventAll = "adhoc.event.>" )
Event subjects (fire-and-forget, CBOR — ADR-0188 §SD3). The service publishes one event per dataset transition so consumers react in a frame instead of polling: `published` on every publish and republish (the push notification ADR-0134 deferred), `retracted` at the LEAVE step of a two-phase withdrawal — the dataset has already stopped resolving when the event goes out, and its provider stays queryable for RetractGrace so a query that had already resolved the handle completes. Consumers declare `Sub adhoc.event.>` (SubjectEventAll) to receive them.
const CatalogTableName = "adhoc"
CatalogTableName is the keelson table that lists the live datasets (ADR-0134 SD6): the operator's window onto what ad-hoc data exists now.
const DefaultRetractGrace = inprocbus.DefaultRequestTimeout
DefaultRetractGrace is the RetractGrace applied when Config leaves it zero: one bus request timeout, the longest a consumer's already-issued query can be waiting on the transport (ADR-0188 §SD3, its F1).
const ServiceAppId app.AppIdT = "runtime.adhoc"
ServiceAppId is the synthetic identity the capability service speaks under on the bus; audit rows attribute publishes/grants/retracts to it.
Variables ¶
var ErrNoLiveDataset = errors.New("no live dataset under alias")
ErrNoLiveDataset is Resolve's answer when nothing is published under the alias (or everything under it has been retracted). It travels the wire as its own flag so a bus caller can tell it from a transport failure with errors.Is — the difference between "wait" and "retry".
var StoreDir = env.NewString(env.Spec{ Name: "BOXER_ADHOC_DIR", Default: "", Description: "directory for the ad-hoc dataset store (ADR-0134); empty resolves to <user cache dir>/boxer/adhoc", Category: env.CategorySystem, })
StoreDir names the directory holding the encrypted dataset files. Empty resolves to <user cache dir>/boxer/adhoc; on the appliance the host sets it beneath /perm (ADR-0134 SD1/SD8).
Functions ¶
func DisableCoreDumps ¶
DisableCoreDumps sets RLIMIT_CORE to zero so a crash cannot spill process memory — decrypted buffers and AES round-key schedules included — to disk via a core dump (ADR-0134 SD8). It closes one of the two RAM→disk bridges the ephemerality guarantee must not leak through (swap is the other, a box-level concern). Plain Go panics do not dump core; the exposure this closes is an FFI or `unsafe` fault. It is unconditional and best-effort: a failure is logged, not fatal. The shell calls this once at startup.
func RetractRequest ¶
RetractRequest retracts a dataset via the adhoc.retract subject.
func StructureFor ¶
StructureFor renders the ClickHouse structure string — a comma-joined list of backtick-quoted `name Type` columns — that a `file(fifo,'ArrowStream',<structure>)` read requires, because schema inference over a pipe is impossible (ADR-0134 SD3). Every column name is backtick-quoted, so leeway-encoded / nested columnar schemas — whose physical names carry colons and whose repeated sections are Array-typed — survive the round trip. The type mapping is recursive and total over the bounded set the publish gate admits; it rejects everything else, naming the offending column, so an unsupported type is refused at publish, never discovered at query time (ADR-0134 SD1).
Supported Arrow types → ClickHouse, applied recursively to list elements, struct fields, and map values:
Utf8, Binary → String FixedSizeBinary(N) → FixedString(N) Bool → Bool Int8/16/32/64 → Int8/16/32/64 Uint8/16/32/64 → UInt8/16/32/64 Float32/64 → Float32/64 Date32 → Date32 Timestamp(µs|ns,"UTC") → DateTime64(6|9,'UTC') Timestamp(µs|ns,"") → DateTime64(6|9) (timezone-naive) List/LargeList/FixedSizeList(T) → Array(<T>) Struct(f T, …) → Tuple(`f` <T>, …) Map(K,V) → Map(<K>, <V>)
A nullable Arrow field maps to Nullable(T), but only for a scalar leaf: ClickHouse forbids Nullable(Array)/Nullable(Tuple)/Nullable(Map), and its ArrowStream reader coerces a null container to an empty/default one, so container-level nullability is dropped (verified against clickhouse-local 26.6). Dictionaries, unions, large/view string variants, and a timestamp with a non-UTC/non-empty zone or a coarser-than-µs unit are rejected.
func SubscribeEvents ¶ added in v0.0.20
SubscribeEvents delivers every dataset transition the service publishes (ADR-0188 §SD3) to handler, decoded. The caller's bus must carry `Sub adhoc.event.>`; the returned unsubscribe releases the subscription (the host releases it at the closing edge as well). Payloads that fail to decode are dropped with a log line by the caller's own choosing — handler is only ever invoked with a well-formed Event.
Types ¶
type Config ¶
type Config struct {
// Bus, when non-nil, backs the adhoc.publish/grant/retract
// request/reply subjects. Nil leaves only the in-process Go methods.
Bus *inprocbus.Inst
// Registry is where dataset handles register as EncryptedEntry
// providers; defaults to introspect.Default.
Registry *introspect.Registry
// Keys is the broker key store; required.
Keys KeyRegistrarI
// Dir overrides the store directory; empty resolves from StoreDir.
Dir string
// Log is the service logger.
Log zerolog.Logger
// RetractGrace is how long a retracted dataset stays queryable after it
// stops resolving (ADR-0188 §SD3); zero means DefaultRetractGrace.
RetractGrace time.Duration
}
Config parameterises the capability Service.
type DecryptorI ¶ added in v0.0.17
type DecryptorI interface {
OpenDatasetPlaintext(ref Ref) (rc PlaintextI, err error)
}
DecryptorI serves a sealed dataset's plaintext.
The implementation resolves the key in-process by Ref.Handle and returns a seekable reader over the decrypted Arrow stream; the caller closes it. An authentication or truncation failure surfaces as a read error rather than as a short result, which is the property the whole scheme rests on — a truncated dataset must fail the query, not shorten it.
Taking an interface here is what keeps the loopback endpoint from importing the broker.
type Event ¶ added in v0.0.20
type Event struct {
// Op is EventPublished or EventRetracted.
Op EventOpE
// Handle is the dataset handle the transition concerns.
Handle string
// Alias is the stable alias the dataset was published under.
Alias string
// Publisher is the app that published it (bus sender or embedder stamp).
Publisher string
// Revision is the dataset revision after a publish, or the last live
// revision at a retract.
Revision uint64
}
Event is one dataset transition as consumers see it (decoded from the wire by DecodeEvent / SubscribeEvents).
type EventOpE ¶ added in v0.0.20
type EventOpE uint8
EventOpE names the dataset transition an Event carries.
type GrantResult ¶
GrantResult is the metadata a grant hands back.
type KeyRegistrarI ¶ added in v0.0.17
type KeyRegistrarI interface {
RegisterDatasetKey(name string, key []byte)
DeregisterDatasetKey(name string)
}
KeyRegistrarI is the broker-side key custody the capability service drives (ADR-0134 K2). *chlocalbroker.KeyStore satisfies it; taking an interface keeps this package from importing the broker (the broker imports the AEAD stream from here).
It is deliberately register-and-forget, with no lookup. ADR-0134 §SD2 splits key roles — this service is the POLICY OWNER, the broker is the DECRYPT EXECUTOR — and a custody interface spanning both halves would hand the policy owner exactly the ability that split exists to deny it. See DecryptorI for the executor's half.
type PlaintextI ¶ added in v0.0.20
type PlaintextI interface {
io.ReadSeeker
io.Closer
}
PlaintextI is a sealed dataset's decrypted read seam: a seekable reader over the plaintext Arrow stream. Seek is load-bearing — it is what lets the /table endpoint honor HTTP range requests, which ClickHouse's Arrow reader issues to skip column buffers a query does not touch (ADR-0134 update 2026-08-01). SeekableReader provides it from the chunk geometry.
type PublishInput ¶
type PublishInput struct {
Alias string
Handle string
ArrowIPCStream []byte
// Publisher attributes the dataset in the catalog. Over the bus it is
// the authenticated sender; an in-process embedder passes a composed
// stamp (embedder id carrying the applet slug, ADR-0134 SD7). It is
// recorded on first publish and kept across republishes.
Publisher string
}
PublishInput is the in-process shape of a publish (the bus wire mirrors it). Handle empty mints a new dataset; a known Handle republishes it.
type PublishResult ¶
PublishResult reports the minted (or reused) handle and dataset stats.
func PublishRequest ¶
func PublishRequest(bus app.BusI, in PublishInput) (res PublishResult, err error)
PublishRequest publishes — or, with in.Handle set, republishes — a dataset via the adhoc.publish capability subject and returns the minted or reused handle (ADR-0134 SD2). It is how an in-process app (e.g. an embedder) drives the capability without holding a Service reference; the caller's bus client needs Pub on adhoc.publish. The publisher is attributed to the authenticated sender by the service, so in.Publisher is ignored on this path.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader decrypts a BXAD stream produced by Writer. It implements io.Reader; a truncated stream, a wrong key, or any tampering surfaces as an error rather than a short or silently-wrong read. A Reader is not safe for concurrent use.
type Ref ¶ added in v0.0.17
type Ref struct {
// Handle is the dataset's unguessable name, and the key under which
// custody holds its key. It is also a valid keelson table name, which
// is what lets a query name it.
Handle string
// Path locates the chunk-encrypted Arrow file. Ciphertext, so its
// exposure is the disk exposure the scheme already assumes.
Path string
// Structure is the explicit ClickHouse structure string. Schema
// inference cannot be used on this read — it consumes the stream and
// cannot re-read it — so the publish gate computes this once and every
// reader is handed it.
Structure string
// Revision increments on republish. It is what makes a same-revision
// requery a legitimate cache hit and a republish a miss.
Revision uint64
}
Ref names one sealed dataset for reading. It carries what a decryptor needs and, deliberately, nothing else: the key is resolved on the executor's side by handle and never travels — not on a bus, not on a wire, and not in this struct.
type ResolveResult ¶ added in v0.0.20
type ResolveResult struct {
Handle string
Revision uint64
Rows uint64
Bytes uint64
CreatedAtUnixUs int64
}
ResolveResult is what an alias resolves to: the newest live dataset published under it.
func ResolveRequest ¶ added in v0.0.20
func ResolveRequest(bus app.BusI, alias string) (res ResolveResult, err error)
ResolveRequest maps a stable alias to the newest live dataset published under it via the adhoc.resolve subject (ADR-0134 §SD4, update 2026-08-01). It is how a standalone applet binds its declared `datasets:` aliases at open; the caller's bus client needs Pub on adhoc.resolve.
func ResolveVerifyRequest ¶ added in v0.0.20
func ResolveVerifyRequest(bus app.BusI, alias string, boundHandle string) (res ResolveResult, boundLive bool, err error)
ResolveVerifyRequest is ResolveRequest with a second question in the same round trip: is boundHandle still live? A consumer bound to boundHandle reconciles its binding with it (ADR-0188 §SD3): boundLive true means keep the binding whatever the alias's newest dataset is (an open applet does not re-resolve to a newer sibling, ADR-0134); false means the handle has left, and res — when err is nil — is the successor to bind, or when err is set there is nothing live under the alias yet. err is set only for transport failures and for "no live dataset under alias"; boundLive is meaningful in both cases.
type SeekableReader ¶ added in v0.0.20
type SeekableReader struct {
// contains filtered or unexported fields
}
SeekableReader is a random-access reader over a BXAD stream's plaintext. It exists so the /table endpoint can honor HTTP range requests: ClickHouse's Arrow reader skips column buffers it does not need by re-requesting the source from a later offset, and a stream-only reader cannot serve that (ADR-0134 update 2026-08-01).
Random access is sound because the format pins the geometry: every non-final chunk carries exactly chunk-size plaintext, so a plaintext offset maps arithmetically to a chunk index and the chunk's ciphertext location. Each chunk authenticates independently under its counter nonce, and the total plaintext size falls out of the ciphertext size — so a corrupt or truncated file fails construction or the first touched chunk, never returns wrong bytes.
A SeekableReader is not safe for concurrent use.
func NewSeekableReader ¶ added in v0.0.20
NewSeekableReader opens a BXAD stream held in ra (size ciphertext bytes) for random access under key. The header is read and the geometry validated eagerly; chunks authenticate lazily as they are first touched.
func (*SeekableReader) PlaintextSize ¶ added in v0.0.20
func (inst *SeekableReader) PlaintextSize() int64
PlaintextSize reports the stream's total plaintext length.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service owns the encrypted dataset store: it validates and encrypts published data, mints ephemeral handles, custodies keys with the broker, registers handles as queryable providers, and retracts on request (ADR-0134 SD2). It is safe for concurrent use.
func NewService ¶
NewService builds the Service, creates and sweeps the store directory (ADR-0134 SD1: crash residue is ciphertext without a key, but the sweep removes it anyway), and, when a bus is supplied, subscribes to the capability subjects.
func (*Service) Close ¶
Close unsubscribes, deregisters every key and provider, and deletes all store files (best-effort; the ephemerality guarantee does not rest on this — after a crash the files are ciphertext whose key is gone).
func (*Service) FlushRetracts ¶ added in v0.0.20
func (inst *Service) FlushRetracts()
FlushRetracts runs the UNLOAD step now for every dataset that has left but whose grace has not elapsed. Close calls it; tests call it to make the two-phase withdrawal synchronous.
func (*Service) Grant ¶
func (inst *Service) Grant(handle string) (res GrantResult, err error)
Grant returns a dataset's binding metadata and records the audit event that is the grant (ADR-0134 SD2: audited, not enforced).
func (*Service) IsLive ¶ added in v0.0.20
IsLive reports whether handle names a dataset in the live set — published and not retracted. A dataset in its retract grace (left, not yet unloaded) is not live: it still answers queries but no longer resolves, and a consumer verifying its binding should rebind (ADR-0188 §SD3).
func (*Service) Publish ¶
func (inst *Service) Publish(in PublishInput) (res PublishResult, err error)
Publish validates and encrypts a dataset, mints or reuses a handle, registers the key and a queryable provider, and returns the handle and stats (ADR-0134 SD1/SD2). A republish (known Handle) bumps the revision and swaps the file/key in place under the same handle.
func (*Service) Resolve ¶ added in v0.0.20
func (inst *Service) Resolve(alias string) (res ResolveResult, err error)
Resolve maps a stable alias to the newest live dataset published under it — newest by creation instant, ties broken on handle for determinism. It is what lets a committed applet declare `datasets: [pprof_cpu]` and bind at open time without ever learning a handle ahead of time (ADR-0134 §SD4 for standalone applets, update 2026-08-01). Republishing onto a handle keeps its creation instant, so a producer that reuses one handle per alias — the imzrt Profiles pattern — stays the resolution target across re-captures.
func (*Service) Retract ¶
Retract withdraws a dataset in two phases (ADR-0134 SD2 as revised by ADR-0188 §SD3). LEAVE, now: the record leaves the live set, so the catalog and Resolve stop naming it, the quota is released, and an adhoc.event.retracted goes out to consumers. UNLOAD, after RetractGrace: the key, the provider and the file go, so a query that had already resolved the handle completes instead of failing mid-flight. A republish onto a retracted handle is refused as unknown from the leave step on; a producer that wants the data back publishes afresh.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer encrypts a plaintext stream into the BXAD chunk format. It implements io.WriteCloser: Write buffers plaintext and emits full non-final chunks as they accumulate, and Close seals the trailing bytes as the final chunk. Close MUST be called to produce a valid stream. A Writer is not safe for concurrent use.
func NewWriter ¶
NewWriter returns a Writer that encrypts to w under key (32 bytes) using the default ChunkSize. It writes the header immediately, so a failure to write the header surfaces here.
Source Files