wire

package
v0.30.11 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 26, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package wire defines a compact TLV binary form of a ClassAd with attribute-key interning and a hot-attribute header, plus the encoder/decoder that convert to and from the fully-public ast.ClassAd representation.

The package depends only on the ast package (and the standard library): Encode takes an *ast.ClassAd and Decode returns one, so wire stays decoupled from the higher-level classad.ClassAd wrapper. Bridging classad.ClassAd <-> ast.ClassAd lives in the store layer.

Index

Constants

This section is empty.

Variables

View Source 
var ErrMalformed = errors.New("wire: malformed encoding")

ErrMalformed is returned when the input is not a well-formed wire ad.

View Source 
var ErrSealed = errors.New("attribute is sealed and no decryption key is available")

ErrSealed reports that an attribute is sealed and this decoder has no key for it. It wraps into the error DecodeResolveEnc returns, so a caller distinguishes "not entitled to this value" from "this ad is corrupt" with errors.Is -- the two used to be the same ErrMalformed.

View Source 
var ErrUnsupported = errors.New("wire: expression construct not handled by the native parser")

ErrUnsupported signals the wire-native parser hit a construct it does not handle (a record literal, an int64-min magnitude, a computed select, ...). The caller should fall back to the reference parser, which is the source of truth.

Functions

func AppendAdMergedInline added in v0.30.0 

func AppendAdMergedInline(dst []byte, base Ad, overlays []Ad, sc *MergeScratch) ([]byte, bool)

AppendAdMergedInline appends to dst the ad formed by applying overlays, in order, over base, and returns the extended buffer. A later overlay's attribute wins over an earlier one's, and any overlay's wins over base's; an attribute no overlay mentions is copied from base byte for byte.

Every participant must be an inline-names ad that is not standalone (no embedded intern table): entries are copied between ads, so the names have to be IN the entries. It reports false when that does not hold, or when any participant is malformed, so the caller can fall back to decoding -- which is the source of truth.

The result carries NO hot header. The hot header stores absolute offsets into the entries region, which a merge invalidates, and every reader treats it as a guarded fast path that falls back to a linear walk when it comes up short (see rawprojected.go) or refuses a merged ad outright (hotClosureMatch requires flagHotClosure, which this does not set). AppendAdSubsetInline emits none for the same reason.

func AppendAdSubsetInline added in v0.16.4 

func AppendAdSubsetInline(dst []byte, src Ad, keep func(name, node []byte) bool, open Sealer) ([]byte, bool)

AppendAdSubsetInline assembles a self-contained inline-names ad holding the subset of src's attribute entries keep selects, appending it to dst and returning the extended slice. It is the relay primitive for shipping stored ads across a trust/process boundary in wire form: an inline entry is a contiguous (name, node) byte range, so the subset is assembled by slice copies -- no value is decoded, rendered or re-encoded.

An entry whose node is at-rest encrypted (nEncrypted) is emitted with its value OPENED via open when non-nil -- the consumer holds no data key -- and skipped entirely when open is nil or fails (matching the fast paths, which treat an unopenable value as absent). The emitted ad carries no hot header (its consumer renders linearly). Returns ok=false when src is not a well-formed inline-names ad; keep=nil keeps every entry.

func AppendAdSubsetInlineHotFirst added in v0.16.4 

func AppendAdSubsetInlineHotFirst(dst []byte, src Ad, keep func(name, node []byte) bool, needed int, open Sealer, sc *SubsetScratch) ([]byte, bool)

AppendAdSubsetInlineHotFirst is AppendAdSubsetInline with the hot-header shortcut: entries reachable through the ad's hot pairs are copied first via their stored offsets, and when that satisfies the whole selection (kept == needed) the linear walk is skipped entirely -- the projected relay's fast path, mirroring the projected text scan's. Anything short of full satisfaction falls back to the walk, which skips the entries the hot phase already copied. needed <= 0 disables the shortcut (a whole-ad selection must walk everything).

func AppendBoolNode added in v0.22.0 

func AppendBoolNode(dst []byte, b bool) []byte

AppendBoolNode, AppendIntNode, AppendRealNode, and AppendStringNode append a scalar literal wire node to dst and return the extended slice -- the inverse of LiteralValue. They let a caller reconstruct a wire node from a typed value (e.g. the per-segment schema decoder rebuilding a node from a fixed-layout field), producing bytes LiteralValue reads back exactly.

func AppendIntNode added in v0.22.0 

func AppendIntNode(dst []byte, v int64) []byte

func AppendInternedFromInline added in v0.30.0 

func AppendInternedFromInline(dst []byte, src Ad, t *InternTable, hot map[uint32]struct{}) ([]byte, bool)

AppendInternedFromInline appends to dst the INTERNED form of src, an inline-names ad, and returns the extended buffer. Names are interned into t; ids present in hot get a hot-header entry, matching EncodeWithHotEnc (a sealed value is never hot).

It reports false when src is not a plain inline-names ad -- standalone (carrying its own intern table) or already interned -- or when it is malformed. The caller falls back to decoding, which is the source of truth.

func AppendKey added in v0.28.0 

func AppendKey(dst []byte, inline bool, id uint32, name string) []byte

AppendKey writes an attribute key in the encoding the ad's flags declare: the interned id, or the name inline.

func AppendNodeRefIDs added in v0.16.3 

func AppendNodeRefIDs(node []byte, dst []uint32) []uint32

AppendNodeRefIDs appends to dst the interned id of every attribute reference inside node that could resolve against the enclosing ad -- unscoped and MY.-scoped references (TARGET./PARENT. refer to a different ad) -- returning the extended slice. It walks the node's TLV structure directly (the same walk as skipNode), so collecting a node's references costs no decode, no name resolution and no allocation beyond dst's growth; a caller building a projection's reference closure recycles dst across nodes and passes.

The collection is a superset: references inside nested record literals are included even though record-member resolution could shadow them, and duplicate ids are appended as-is (the caller's wanted-set absorbs both). An encrypted node is opaque and contributes nothing. Returns dst unchanged (beyond any partial appends) with no error indication on a malformed node -- the caller's subsequent render of the same bytes surfaces the corruption.

func AppendNodeRefNames added in v0.24.1 

func AppendNodeRefNames(node []byte, dst [][]byte) [][]byte

AppendNodeRefNames is AppendNodeRefIDs for an INLINE-NAME ad: it appends the inline name of every attribute reference inside node that could resolve against the enclosing ad -- unscoped and MY.-scoped -- returning the extended slice.

A persistent (inline-name) collection stores references as names rather than interned ids, so the id collector finds nothing in one; this is what lets a projection over such a collection build the same reference closure. The appended names are sub-slices of node, valid for as long as node is, and are NOT copied -- a caller that retains them past the record's buffer must copy.

Same superset semantics as AppendNodeRefIDs: nested record members are included even though member resolution could shadow them, duplicates are appended as-is, and an encrypted node contributes nothing.

func AppendNodeText 

func AppendNodeText(dst, node []byte, t *InternTable) ([]byte, error)

AppendNodeText appends the canonical ClassAd text of a wire node to dst and returns the extended slice. It is byte-for-byte identical to DecodeNode(node, t).String(), but renders straight from the wire with no ast.Expr and no per-node string: the AST-free query-result path uses it for the non-scalar values (lists, records, computed expressions) that LiteralValue cannot format directly. Interned name ids resolve via t.

func AppendNodeTextInline 

func AppendNodeTextInline(dst, node []byte) ([]byte, error)

AppendNodeTextInline is AppendNodeText for inline-names nodes (self-contained, no intern table) — the form stored by the persistent store.

func AppendNodeTextInlineOld added in v0.24.1 

func AppendNodeTextInlineOld(dst, node []byte) ([]byte, error)

AppendNodeTextInlineOld is AppendNodeTextInline with old-ClassAd string quoting; see AppendNodeTextOld for why the dialect has to be chosen explicitly.

func AppendNodeTextInlineOldEnc added in v0.28.0 

func AppendNodeTextInlineOldEnc(dst, node []byte, open Sealer, redactSealed bool) ([]byte, error)

AppendNodeTextInlineOldEnc is AppendNodeTextInlineOld for wire that may carry SEALED values.

open renders the real value; a nil open with redactSealed renders `undefined` instead -- by node, so it covers any sealed attribute rather than only the ones a private-name list knows about. A nil open without redactSealed keeps the old behaviour and errors, which a caller must want deliberately: every raw caller treats a render error as "skip this ad".

func AppendNodeTextOld added in v0.24.1 

func AppendNodeTextOld(dst, node []byte, t *InternTable) ([]byte, error)

AppendNodeTextOld is AppendNodeText rendering string literals for OLD-ClassAd text, whose tokenizer does no escape processing (C++ Lexer::tokenizeStringOld: only a backslash before the closing quote is a sequence).

The distinction is not cosmetic. AppendNodeText mirrors ast.Expr.String(), which escapes backslashes, tabs and newlines for the new-ClassAd reader that unescapes them. Feeding that text to an old-ClassAd reader, as every raw-projection consumer does, doubles a backslash on each round trip -- so a caller emitting old text must use this.

func AppendNodeTextOldEnc added in v0.28.0 

func AppendNodeTextOldEnc(dst, node []byte, t *InternTable, open Sealer, redactSealed bool) ([]byte, error)

AppendNodeTextOldEnc is AppendNodeTextOld for wire that may carry sealed values; see AppendNodeTextInlineOldEnc.

func AppendRealNode added in v0.22.0 

func AppendRealNode(dst []byte, f float64) []byte

func AppendStringNode added in v0.22.0 

func AppendStringNode(dst []byte, s string) []byte

func BuildAd added in v0.28.0 

func BuildAd(dst, header []byte, n int, entries []byte) []byte

BuildAd assembles an ad from a header (as SplitBody returns) and a sequence of already-encoded attribute entries, writing an empty hot header. entries must be key+node pairs in the same key encoding the header's flags declare.

The hot header is left empty deliberately: it indexes offsets into the entry region, and a rebuilt region invalidates them. Nothing is lost -- it is regenerated where it matters.

func Decode 

func Decode(b []byte, t *InternTable) (*ast.ClassAd, error)

Decode parses an ad encoded with Encode or EncodeInline. Interned ads resolve names via t; inline-names ads (flagInlineNames) are self-contained and t may be nil.

func DecodeInline 

func DecodeInline(b []byte) (*ast.ClassAd, error)

DecodeInline parses a self-contained inline-names ad (EncodeInline), requiring no intern table.

func DecodeInlineEnc added in v0.7.0 

func DecodeInlineEnc(b []byte, open Sealer) (*ast.ClassAd, error)

DecodeInlineEnc is DecodeInline for an ad that may contain nEncrypted attributes: open supplies the segment's key so encrypted values decrypt to their real nodes. A nil open leaves encrypted attributes opaque and DecodeInline errors on them -- use DecodeInlineEnc only on the DAEMON read path that holds the key.

func DecodeInlineRedact added in v0.28.0 

func DecodeInlineRedact(b []byte) (*ast.ClassAd, error)

DecodeInlineRedact is DecodeInlineEnc with no key that substitutes undefined for a sealed attribute instead of failing the ad -- the inline-names counterpart of DecodeResolveEncRedact, for a reader that is not entitled to sealed values.

func DecodeNode 

func DecodeNode(node []byte, t *InternTable) (ast.Expr, error)

DecodeNode decodes raw node bytes (as returned by Lookup/ForEach) into an ast.Expr, resolving interned ids via t.

func DecodeNodeInline 

func DecodeNodeInline(node []byte) (ast.Expr, error)

DecodeNodeInline decodes raw node bytes from an inline-names ad (as returned by LookupByName), which need no intern table.

func DecodeNodeInlineEnc added in v0.7.0 

func DecodeNodeInlineEnc(node []byte, open Sealer) (ast.Expr, error)

DecodeNodeInlineEnc decodes raw node bytes from an inline-names ad (as returned by LookupByName), opening an nEncrypted node with open. A nil open leaves an encrypted node an error (the fast path treats that as absent).

func DecodeNodeResolve added in v0.23.0 

func DecodeNodeResolve(node []byte, resolve func(uint32) (string, bool)) (ast.Expr, error)

DecodeNodeResolve decodes raw node bytes from an interned ad (as returned by Lookup), resolving interned ids via `resolve` instead of an *InternTable -- e.g. backed by a sealed segment's mmap dictionary. Counterpart to DecodeNode for the per-segment interned path.

func DecodeNodeResolveEnc added in v0.23.0 

func DecodeNodeResolveEnc(node []byte, resolve func(uint32) (string, bool), open Sealer) (ast.Expr, error)

DecodeNodeResolveEnc is DecodeNodeResolve that opens an nEncrypted node with open (the per-segment interned analog of DecodeNodeInlineEnc). A nil open leaves an encrypted node an error.

func DecodeResolve added in v0.23.0 

func DecodeResolve(b []byte, resolve func(uint32) (string, bool)) (*ast.ClassAd, error)

DecodeResolve parses an interned ad (Encode/EncodeWithHot), resolving attribute ids to names via `resolve` instead of an *InternTable -- e.g. backed by a sealed segment's mmap dictionary (id -> name), so no in-memory table is materialized to read the segment. The ad must be interned (not inline -- those are self-contained, use DecodeInline; not standalone). The resolver is inherited by nested ad decoders, so it covers sub-ads too.

func DecodeResolveEnc added in v0.23.0 

func DecodeResolveEnc(b []byte, resolve func(uint32) (string, bool), open Sealer) (*ast.ClassAd, error)

DecodeResolveEnc is DecodeResolve for an interned ad that may contain nEncrypted attributes: open supplies the segment's key so sealed values decrypt to their real (interned) nodes, which then resolve their ids via resolve. A nil open leaves encrypted attributes opaque and decode errors on them -- so pass the sealer only on the daemon read path that holds the key.

func DecodeResolveEncRedact added in v0.28.0 

func DecodeResolveEncRedact(b []byte, resolve func(uint32) (string, bool), open Sealer) (*ast.ClassAd, error)

DecodeResolveEncRedact is DecodeResolveEnc that SUBSTITUTES undefined for any attribute it cannot open, instead of failing the ad. For a read path serving a client not entitled to a value: the ad is readable, one attribute of it is not, and undefined is what an absent attribute yields.

It must never be used where the decoded ad is re-encoded (compaction, dict retraining): substituting undefined there would write the substitution back and destroy the secret.

func DecodeStandalone 

func DecodeStandalone(b []byte) (*ast.ClassAd, error)

DecodeStandalone parses a self-contained ad written by EncodeStandalone.

func Encode 

func Encode(dst []byte, ad *ast.ClassAd, t *InternTable) []byte

Encode appends the binary form of ad to dst and returns the extended slice, using the shared intern table t (mutating it to add any new names). The resulting bytes are NOT self-contained: decoding requires the same table.

func EncodeInline 

func EncodeInline(dst []byte, ad *ast.ClassAd) []byte

EncodeInline encodes ad with inline attribute names (no interning), producing a fully self-contained ad that DecodeInline reads with no InternTable. Used by the persistent store so on-disk records are recoverable without a shared table.

func EncodeInlineDelta added in v0.30.0 

func EncodeInlineDelta(dst []byte, ad *ast.ClassAd, hot map[string]struct{}, encrypt func(name string) bool, seal Sealer) []byte

EncodeInlineDelta encodes ad as a DELTA record: byte-identical to an ordinary inline ad except that flagDelta is set, so a reader knows these are only the attributes one write changed and that the rest must be merged in from older versions of the key. The caller is responsible for having built ad from exactly the changed attributes.

func EncodeInlineWithHot 

func EncodeInlineWithHot(dst []byte, ad *ast.ClassAd, hot map[string]struct{}) []byte

func EncodeInlineWithHotEnc added in v0.7.0 

func EncodeInlineWithHotEnc(dst []byte, ad *ast.ClassAd, hot map[string]struct{}, encrypt func(name string) bool, seal Sealer) []byte

EncodeInlineWithHotEnc is EncodeInlineWithHot with at-rest encryption: the value of any attribute for which encrypt(name) is true is sealed with seal and stored as an nEncrypted node (decodable only with the matching Opener). A predicate (rather than a set) lets the caller encrypt by rule -- e.g. HTCondor's private-attribute prefix -- not just an enumerable list. An encrypted attribute is never hot: it is opaque to the index/match fast path, so it is excluded from the hot header even if listed in hot. encrypt and seal must both be non-nil to encrypt anything.

func EncodeStandalone 

func EncodeStandalone(ad *ast.ClassAd) []byte

EncodeStandalone returns a self-contained encoding that embeds a minimal intern table, so it can be decoded with DecodeStandalone alone (e.g. for transport out of a collection).

func EncodeWithHot 

func EncodeWithHot(dst []byte, ad *ast.ClassAd, t *InternTable, hot map[uint32]struct{}) []byte

EncodeWithHot is Encode with a populated hot header: attributes whose interned id is in hot are indexed by a (id, entries-relative offset) pair written before the body, so Ad.Lookup finds them in O(1) instead of scanning. The body layout is otherwise identical to Encode, so the result decodes with Decode and the hot header is a pure read-time accelerator. hot may be nil (equivalent to Encode).

func EncodeWithHotClosure added in v0.6.0 

func EncodeWithHotClosure(dst []byte, ad *ast.ClassAd, t *InternTable, hot map[uint32]struct{}, closureComplete bool) []byte

EncodeWithHotClosure is EncodeWithHot that, when closureComplete, marks the ad with flagHotClosure -- a promise that hot holds the complete match closure, so a matcher can read it via ForEachHot without scanning. Set closureComplete only when hot truly contains every attribute the match reads from ad (see collections' astClosure).

func EncodeWithHotEnc added in v0.23.0 

func EncodeWithHotEnc(dst []byte, ad *ast.ClassAd, t *InternTable, hot map[uint32]struct{}, encrypt func(name string) bool, seal Sealer) []byte

EncodeInlineWithHot is EncodeInline with a populated hot header: attributes whose (case-folded) name is in hot are indexed by a (nameHash32, entries-relative offset-to-entry) pair, so Ad.LookupByName finds them without scanning. hot keys must be lower-cased. hot may be nil. EncodeWithHotEnc is EncodeWithHot for at-rest encryption: any attribute for which encrypt(name) is true is sealed with seal and stored as an nEncrypted node (its value's interned node encoding, sealed), exactly as EncodeInlineWithHotEnc does for inline ads. Encrypted attributes are never hot (never indexed), so they never enter the hot header. encrypt and seal must both be non-nil to encrypt anything; with seal nil this is EncodeWithHot.

func FoldEqualBytes added in v0.16.3 

func FoldEqualBytes(b []byte, name string) bool

FoldEqualBytes reports whether the raw attribute-name bytes equal name case-insensitively (ASCII fold, matching ClassAd attribute-name semantics).

func IsDelta added in v0.30.0 

func IsDelta(rec []byte) bool

IsDelta reports whether a decoded (decompressed) record holds only changed attributes rather than a whole ad. A record too short to carry a header is not a delta -- callers treat anything unrecognized as a full record, which is the safe direction: a full record read as a delta would send the reader hunting for a base that does not exist.

func IsEncryptedNode added in v0.16.4 

func IsEncryptedNode(node []byte) bool

IsEncryptedNode reports whether node is an nEncrypted (at-rest sealed) node.

func NameHash32 added in v0.16.3 

func NameHash32(name string) uint32

NameHash32 exposes the case-insensitive 32-bit hash used by inline hot-header pairs, so a projected reader can precompute its wanted-name hashes.

func NameHash32Bytes added in v0.16.3 

func NameHash32Bytes(name []byte) uint32

NameHash32Bytes is NameHash32 over raw name bytes (no string conversion).

func NodeLen added in v0.22.0 

func NodeLen(b []byte) (int, bool)

skipNode advances c past exactly one node without allocating. NodeLen returns the byte length of the single wire node at the start of b -- a scalar literal (int/real/bool/string/undefined/error) or a computed expression (op/ref/list/ record/func). It lets a caller that stores concatenated uvarint(id)+node entries (e.g. the per-segment schema's cold tail) split them without a separate length prefix. ok is false if b does not begin with a well-formed node.

func OpenEncryptedNode added in v0.16.4 

func OpenEncryptedNode(node []byte, open Sealer) ([]byte, bool)

OpenEncryptedNode returns the plain node bytes sealed inside an nEncrypted node -- the sealed payload IS the value's ordinary node encoding, so opening it needs no re-encode. Returns (nil, false) if node is not an nEncrypted node, open is nil, or authentication fails. A relay that ships wire-form rows to a consumer without the data key uses this to emit the entry with its value in the clear (the consumer cannot open it).

func ParseExprToWire added in v0.3.0 

func ParseExprToWire(input string, t *InternTable, dst []byte) ([]byte, error)

ParseExprToWire parses a standalone ClassAd expression from input and appends its wire node bytes to dst (interning names into t), returning the extended buffer. Output is byte-identical to encoding the reference-parsed ast.Expr. It returns ErrUnsupported for constructs it does not handle (so the caller can fall back to parser.ParseExpr), or a syntax error for malformed input.

func StringLiteralValue 

func StringLiteralValue(node []byte) ([]byte, bool)

StringLiteralValue returns the raw value bytes of a string-literal node as a subslice of node (no copy), with ok=true, or (nil, false) if node is not a string literal. The returned bytes alias node and are valid only while node is. It lets a caller quote a string value straight from the wire (via ast.AppendQuoteStringBytes) without the string allocation LiteralValue's Str field would incur.

Types 

type Ad []byte

Ad is a zero-copy view over an encoded ad (the non-standalone in-store form: magic/version/flags header followed by the ad body). Lookup and ForEach walk the bytes without allocating ast nodes, resolving attributes by interned id.

func (Ad) AttrCount added in v0.6.0 

func (a Ad) AttrCount() int

AttrCount returns the number of attributes stored in the ad (0 if malformed). It reads only the header (past the hot index), so it is cheap enough to gate width- dependent decode strategies.

func (Ad) ForEach 

func (a Ad) ForEach(fn func(id uint32, node []byte) bool) bool

ForEach calls fn with each attribute's interned id and raw node bytes, in stored order, until fn returns false or the ad ends. It skips the hot header (which only duplicates body entries). Returns false if the ad is malformed.

func (Ad) ForEachHot added in v0.6.0 

func (a Ad) ForEachHot(fn func(id uint32, node []byte) bool) bool

func (Ad) ForEachHotBuf added in v0.16.3 

func (a Ad) ForEachHotBuf(scratch []uint32, fn func(id uint32, node []byte) bool) ([]uint32, bool)

ForEachHot calls fn with the interned id and raw node bytes of each attribute recorded in the hot header, in header order, resolving each via its stored entries-relative offset (no scan). Returns false if fn stopped early or the ad is malformed. Cost is O(hotCount), independent of the total attribute count -- so a collection whose hot set is the match closure can read exactly the match-relevant attributes of a very wide ad without touching the cold ones. ForEachHotBuf is ForEachHot with caller-provided scratch for the header's (id, offset) pairs, grown as needed and returned for reuse -- a scan calling it once per ad performs no per-ad allocation (ForEachHot allocates two slices per call). scratch holds the pairs interleaved: id, offset, id, offset, ... Interned ads only: an inline ad's hot pairs are (nameHash32, offset-to-ENTRY) -- see ForEachHotInlineBuf -- so it reports false rather than mis-parse.

func (Ad) ForEachHotInlineBuf added in v0.16.3 

func (a Ad) ForEachHotInlineBuf(scratch []uint32, fn func(hash uint32, name, node []byte) bool) ([]uint32, bool)

ForEachHotInlineBuf is ForEachHotBuf's inline-names counterpart: an inline ad's hot pairs are (nameHash32, offset) where the offset points at the whole attribute ENTRY (inline name, then node). It yields the folded name hash, the raw name bytes and the node bytes; the caller must verify the name (hashes can collide) with foldEqualBytes before trusting a match. scratch as in ForEachHotBuf. Reports false for an interned ad.

func (Ad) ForEachIDNode added in v0.16.3 

func (a Ad) ForEachIDNode(fn func(id uint32, node []byte) bool) bool

ForEachIDNode iterates the ad's attribute entries as raw (interned id, node bytes) pairs, resolving nothing: the caller filters by id -- a projection's wanted-set test, a privacy flag -- before paying for name resolution or value rendering. Skipped entries cost one uvarint plus a TLV length hop. Returns false (calling fn for nothing) for an inline-names ad, whose entries carry no interned ids; fn returning false stops the walk early.

func (Ad) ForEachNameNode added in v0.16.3 

func (a Ad) ForEachNameNode(fn func(name, node []byte) bool) bool

ForEachNameNode iterates an inline-names ad's attribute entries as raw (name bytes, node bytes) pairs, allocating nothing -- the projection walk for a persistent collection filters on the name bytes before rendering anything. Reports false (calling fn for nothing) for an interned ad.

func (Ad) ForEachNamed added in v0.6.0 

func (a Ad) ForEachNamed(t *InternTable, fn func(name string, node []byte) bool) bool

ForEachNamed calls fn with each attribute's name and raw node bytes. Inline-name ads (flagInlineNames, written by a persistent collection) yield their stored names directly; interned ads resolve each id to a name via t (an id t cannot resolve is skipped). Unlike ForEach -- which always reads the key as an interned id and so is wrong for inline ads -- this works for both encodings. Returns false if fn stopped early or the ad is malformed.

func (Ad) ForEachNamedRedact added in v0.16.3 

func (a Ad) ForEachNamedRedact(t *InternTable, fn func(name string, node []byte) bool) bool

ForEachNamedRedact is ForEachNamed except attributes whose name the table classified as private (see NewInternTableWithPrivacy) are skipped -- their nodes are never decoded and fn never sees them. Interned attributes are filtered by their precomputed per-id flag (O(1), no name resolution); inline attributes fall back to the table's privacy predicate on the stored name.

func (Ad) ForEachRaw added in v0.28.0 

func (a Ad) ForEachRaw(fn func(id uint32, name string, node []byte) bool) bool

ForEachRaw calls fn with each attribute's interned id (0 for an inline ad), its name (empty for an interned ad), and its raw node bytes.

The two encodings key attributes differently, and a caller REWRITING an ad has to preserve whichever it was given -- ForEach reads the key as an id and is wrong for inline ads, while ForEachNamed resolves names through a table the rewriter may not want to touch.

func (Ad) HotClosureComplete added in v0.6.0 

func (a Ad) HotClosureComplete() bool

HotClosureComplete reports whether the ad's hot header holds the complete match closure (flagHotClosure): ForEachHot then yields every attribute the match reads, so the matcher can trust it without scanning the ad body.

func (Ad) Lookup 

func (a Ad) Lookup(id uint32) ([]byte, bool)

Lookup returns the raw node bytes for the attribute with the given interned id, or (nil, false) if absent or malformed. The returned slice aliases a; it can be decoded with DecodeNode or handed to the vm.

A popular ("hot") attribute is found in O(1) via the hot header: its node offset (relative to the start of the attribute-entries region) is read directly, skipping the linear scan of the body. Non-hot attributes fall back to a linear skip-scan.

func (Ad) LookupByName 

func (a Ad) LookupByName(name string) ([]byte, bool)

LookupByName returns the raw node bytes for the named attribute in an inline-names ad, or (nil, false) if absent, not an inline ad, or malformed. Attribute names are compared case-insensitively. A hot attribute is found in O(1) via the hot header (keyed by a case-folded name hash, verified against the stored name); others fall back to a linear scan.

func (Ad) SplitBody added in v0.28.0 

func (a Ad) SplitBody() (header, entries []byte, n int, inline, ok bool)

SplitBody returns an ad's attribute-entry region and how many entries it holds, plus the header bytes that precede the body (magic, version, flags, and a standalone name table if present).

For a caller that needs to REBUILD an ad from parts rather than read one. The hot header is not returned: it is an index of (id, offset) pairs into the entry region, so any change to that region invalidates it, and it is a pure read accelerator that the store repopulates.

type InternTable 

type InternTable struct {
	// contains filtered or unexported fields
}

InternTable maps attribute (and function) names to small integer ids so the wire form can reference names by uvarint id instead of repeating bytes.

Names are case-insensitive but case-preserving, matching ClassAd attribute semantics: "Owner", "owner" and "OWNER" all intern to the same id, and the id resolves back to the first-seen casing. The table is append-only: an id is stable for the life of the table (a collection compaction may build a fresh table with a remap, but never mutates ids in place).

func NewInternTable 

func NewInternTable() *InternTable

NewInternTable returns an empty table. Id 0 is a valid id (the first name interned); callers that need a sentinel should track presence separately.

func NewInternTableWithPrivacy added in v0.16.3 

func NewInternTableWithPrivacy(privacy func(string) bool) *InternTable

NewInternTableWithPrivacy returns an empty table whose entries are flagged private when privacy(name) is true, evaluated once per unique name at intern time. Redacting iterators (Ad.ForEachNamedRedact) skip flagged ids, so a serialization path can strip secrets in O(1) per attribute instead of re-classifying every attribute name of every ad. A nil privacy flags nothing.

func (*InternTable) Intern 

func (t *InternTable) Intern(name string) uint32

Intern returns the id for name, allocating a new id (and recording name's casing) the first time a fold-equal name is seen.

func (*InternTable) InternBytes added in v0.30.0 

func (t *InternTable) InternBytes(name []byte) uint32

InternBytes is Intern for a caller holding the name as bytes -- a wire-level transcode, which never has it as a string. Taking the string conversion at the call site would allocate per attribute per record; indexing the map with string(name) directly does not, because the compiler elides the copy for a map index. Only a name never seen before pays for one, in Intern.

func (*InternTable) IsPrivate added in v0.16.3 

func (t *InternTable) IsPrivate(id uint32) bool

IsPrivate reports whether id's name was classified private by the table's privacy predicate when it was interned. Lock-free; an unallocated id is not private. O(1) -- this is the point of precomputing the flag: a redacting serializer checks one bool per attribute instead of re-classifying its name.

func (*InternTable) Len 

func (t *InternTable) Len() int

Len returns the number of interned names (== the next id to be allocated).

func (*InternTable) LookupID 

func (t *InternTable) LookupID(name string) (uint32, bool)

LookupID returns the id for name if it has already been interned, without allocating a new one (a read-only counterpart to Intern). Used by the query fast path to resolve an attribute name to its id without polluting the table with names that are not attributes.

func (*InternTable) Name 

func (t *InternTable) Name(id uint32) (string, bool)

Name returns the canonical (first-seen) casing for id, or ("", false) if id was never allocated.

func (*InternTable) Names added in v0.23.0 

func (t *InternTable) Names() []string

snapshotNames returns a copy of the id->name slice, for embedding a standalone intern table into a self-contained ad. Names returns the canonical (first-seen casing) names in id order, so Names()[id] is the name for id. Re-interning these in order into an empty table reproduces the exact id assignment -- the basis for serializing/reloading the table.

type LitKind 

type LitKind uint8

LitKind identifies the kind of a scalar literal node. 

const (
	LitUndef LitKind = iota
	LitError
	LitBool
	LitInt
	LitReal
	LitString
)

type Literal 

type Literal struct {
	Kind LitKind
	Bool bool
	Int  int64
	Real float64
	Str  string
}

Literal is a decoded scalar literal value, described without depending on the classad package so wire stays decoupled from it. Non-scalar nodes (lists, records, and computed expressions) are not literals.

func LiteralValue 

func LiteralValue(node []byte) (Literal, bool)

LiteralValue decodes node as a scalar literal, returning (lit, true) if node is one, or (_, false) if it is a list, record, or computed expression (which must be decoded via DecodeNode and evaluated). It is allocation-free except for the string case, which copies the string bytes.

type MergeScratch added in v0.30.0 

type MergeScratch struct {
	// contains filtered or unexported fields
}

MergeScratch holds AppendAdMergedInline's per-call scratch: the overlay attributes' names and entry byte ranges. Reused across merges so a merge allocates only its output.

func (*MergeScratch) Release added in v0.30.0 

func (s *MergeScratch) Release()

Release is Reset plus dropping the references the backing arrays still hold. The names and entries are slices INTO the caller's ads, so a pooled scratch that keeps them keeps those ads alive; a caller returning a scratch to a pool must Release, not Reset.

func (*MergeScratch) Reset added in v0.30.0 

func (s *MergeScratch) Reset()

Reset clears the scratch, keeping its backing arrays.

type NodeEmitter added in v0.3.0 

type NodeEmitter struct {
	// contains filtered or unexported fields
}

NodeEmitter builds one wire expression node into a reusable buffer, for a parser that emits wire directly from text rather than via an ast.Expr. The wire form is prefix (a node's tag precedes its children), but expression parsing is infix, so a caller records Mark() before parsing an operand and, once the enclosing operator is known, calls a Wrap* method to insert the operator's tag/header before that already-emitted operand. Names are interned as they are emitted.

Byte-for-byte identical to encoder.node for the same expression, so a node it builds decodes with DecodeNode.

func NewInlineNodeEmitter added in v0.3.0 

func NewInlineNodeEmitter() *NodeEmitter

NewInlineNodeEmitter returns an emitter writing inline-name node variants (no interning), matching the persistent-store encoding.

func NewNodeEmitter added in v0.3.0 

func NewNodeEmitter(t *InternTable) *NodeEmitter

NewNodeEmitter returns an emitter interning names into t.

func (*NodeEmitter) AttrRef added in v0.3.0 

func (e *NodeEmitter) AttrRef(scope byte, name string)

AttrRef emits an attribute reference with the given scope (ast.AttributeScope).

func (*NodeEmitter) Bool added in v0.3.0 

func (e *NodeEmitter) Bool(v bool)

func (*NodeEmitter) Bytes added in v0.3.0 

func (e *NodeEmitter) Bytes() []byte

Bytes returns the emitted node bytes (valid until the next Reset).

func (*NodeEmitter) Err added in v0.3.0 

func (e *NodeEmitter) Err()

func (*NodeEmitter) FuncNameID added in v0.3.0 

func (e *NodeEmitter) FuncNameID(name string) uint32

FuncNameID interns a function name and returns its id. A caller emits a call by interning the name with FuncNameID *before* parsing/emitting the argument nodes (so the name's id is assigned ahead of the args, matching the reference encoder's pre-order), then calls WrapFuncID after the args. Returns 0 in inline mode, where the name is written verbatim and no interning happens.

func (*NodeEmitter) Int added in v0.3.0 

func (e *NodeEmitter) Int(v int64)

func (*NodeEmitter) Mark added in v0.3.0 

func (e *NodeEmitter) Mark() int

Mark returns the current buffer position, to pass to a later Wrap* call.

func (*NodeEmitter) Real added in v0.3.0 

func (e *NodeEmitter) Real(v float64)

func (*NodeEmitter) RecordKey added in v0.3.0 

func (e *NodeEmitter) RecordKey(name string)

RecordKey emits a record attribute key -- an interned id, or the inline name -- matching the reference encoder's putKey. Call before emitting the attribute's value node.

func (*NodeEmitter) Reset added in v0.3.0 

func (e *NodeEmitter) Reset()

Reset clears the emitter for another node, keeping the buffer.

func (*NodeEmitter) Str added in v0.3.0 

func (e *NodeEmitter) Str(v string)

func (*NodeEmitter) UnOp added in v0.3.0 

func (e *NodeEmitter) UnOp(op string)

UnOp emits a unary-operator tag; the operand is emitted after (source-prefix, so no insertion needed).

func (*NodeEmitter) Undef added in v0.3.0 

func (e *NodeEmitter) Undef()

func (*NodeEmitter) WrapBinOp added in v0.3.0 

func (e *NodeEmitter) WrapBinOp(mark int, op string)

WrapBinOp turns the operand at [mark:end] plus the operand appended after it into a binary-op node by inserting the op tag at mark.

func (*NodeEmitter) WrapCond added in v0.3.0 

func (e *NodeEmitter) WrapCond(mark int)

WrapCond wraps cond+true+false (cond at mark, true/false appended after) into a conditional node.

func (*NodeEmitter) WrapElvis added in v0.3.0 

func (e *NodeEmitter) WrapElvis(mark int)

WrapElvis wraps left+right (left at mark, right appended after) into an elvis node.

func (*NodeEmitter) WrapFuncID added in v0.3.0 

func (e *NodeEmitter) WrapFuncID(mark int, name string, id uint32, argc int)

WrapFuncID wraps the argc argument nodes at [mark:end] into a function call, using the name id pre-interned by FuncNameID (interned mode) or name verbatim (inline mode).

func (*NodeEmitter) WrapList added in v0.3.0 

func (e *NodeEmitter) WrapList(mark, count int)

WrapList wraps the count elements at [mark:end] into a list node.

func (*NodeEmitter) WrapParen added in v0.3.0 

func (e *NodeEmitter) WrapParen(mark int)

WrapParen wraps the operand at [mark:end] in an explicit-parenthesis node.

func (*NodeEmitter) WrapRecord added in v0.3.0 

func (e *NodeEmitter) WrapRecord(mark, attrCount int)

WrapRecord wraps the attrCount (key, node) pairs at [mark:end] into a record node by inserting the record tag and ad-body header (hotCount 0, attrCount) at mark. A record carries no hot header.

func (*NodeEmitter) WrapSelect added in v0.3.0 

func (e *NodeEmitter) WrapSelect(mark int, attr string)

WrapSelect wraps the record at [mark:end] into a select of attr: the select tag is inserted at mark and the attribute id/name appended after the record (matching the nSelect <record> <attr> layout).

func (*NodeEmitter) WrapSubscript added in v0.3.0 

func (e *NodeEmitter) WrapSubscript(mark int)

WrapSubscript wraps container+index (container at mark, index appended after).

type Sealer added in v0.7.0 

type Sealer interface {
	Seal(plaintext []byte) (nonce, ciphertext []byte, err error)
	Open(nonce, ciphertext []byte) (plaintext []byte, err error)
}

Sealer encrypts and decrypts an attribute value's encoded bytes for at-rest encryption (the nEncrypted node). A collection supplies one backed by a segment's data-encryption key; the wire layer stays crypto-agnostic. Seal returns a fresh nonce and ciphertext; Open reverses it and authenticates (errors on tampering or a wrong key). It is used from a single goroutine per encode/decode pass. Implementations must be safe for concurrent use: both the parallel query scan and the parallel wire scan open sealed values from several goroutines at once, and nothing serializes them. The built-in dataKeySealer satisfies this by holding only an immutable key and building a fresh GCM per call.

type StreamEncoder 

type StreamEncoder struct {
	// contains filtered or unexported fields
}

StreamEncoder builds a wire ad one attribute at a time, so a caller decoding an external serialization (e.g. old-ClassAd lines from a socket) can emit wire form directly without first materializing an ast.ClassAd. Scalar literals are written with no AST at all (Int/Real/String/Bool/Undefined/Error); only genuinely computed values need Expr (an ast.Expr subtree).

A StreamEncoder is reusable: Reset() clears it for the next ad, retaining the backing buffers.

func NewInlineStreamEncoder 

func NewInlineStreamEncoder(hotNames map[string]struct{}) *StreamEncoder

NewInlineStreamEncoder returns a StreamEncoder that writes inline names (no interning), for the persistent store. hotNames (may be nil) are the case-folded names to front-load in the hot header.

func NewStreamEncoder 

func NewStreamEncoder(t *InternTable, hot map[uint32]struct{}) *StreamEncoder

NewStreamEncoder returns an encoder interning names into t. hot (may be nil) is the set of interned ids to front-load in the hot header.

func (*StreamEncoder) Bool 

func (s *StreamEncoder) Bool(name string, v bool)

Bool writes a boolean attribute.

func (*StreamEncoder) Bytes 

func (s *StreamEncoder) Bytes(dst []byte) []byte

Bytes assembles the encoded ad (appending to dst) from the attributes written since the last Reset. The result is byte-identical to EncodeWithHot of the same attributes and decodes with Decode.

func (*StreamEncoder) Count 

func (s *StreamEncoder) Count() int

Count returns the number of attributes written so far.

func (*StreamEncoder) Error 

func (s *StreamEncoder) Error(name string)

Error writes an error attribute.

func (*StreamEncoder) Expr 

func (s *StreamEncoder) Expr(name string, e ast.Expr)

Expr writes a computed (non-literal) attribute from an ast.Expr subtree.

func (*StreamEncoder) ExprWire added in v0.3.0 

func (s *StreamEncoder) ExprWire(name, val string) error

ExprWire writes a computed (non-literal) attribute by parsing its expression text straight to wire via the native parser -- no ast.Expr. It leaves no partial entry if the native parser cannot handle the expression (a record, an int64-min magnitude, ...): it rolls the buffer back and returns the error, so the caller can fall back to Expr(name, parser.ParseExpr(val)). Interned names added before a rollback are harmless (unused ids in a shared table).

func (*StreamEncoder) Int 

func (s *StreamEncoder) Int(name string, v int64)

Int writes an integer attribute.

func (*StreamEncoder) Real 

func (s *StreamEncoder) Real(name string, v float64)

Real writes a real attribute.

func (*StreamEncoder) Reset 

func (s *StreamEncoder) Reset()

Reset clears the encoder for another ad, keeping its buffers.

func (*StreamEncoder) SetHot 

func (s *StreamEncoder) SetHot(hot map[uint32]struct{})

SetHot replaces the hot-id set (e.g. after the store refreshes popularity).

func (*StreamEncoder) String 

func (s *StreamEncoder) String(name, v string)

String writes a string attribute (v is the unescaped value).

func (*StreamEncoder) StringBytes 

func (s *StreamEncoder) StringBytes(name string, v []byte)

StringBytes writes a string attribute whose (already unescaped) value is in v. It is String for a caller holding the value in a reused byte buffer, so no intermediate string is allocated.

func (*StreamEncoder) Undefined 

func (s *StreamEncoder) Undefined(name string)

Undefined writes an undefined attribute.

type SubsetScratch added in v0.16.4 

type SubsetScratch struct {
	// contains filtered or unexported fields
}

SubsetScratch holds AppendAdSubsetInlineHotFirst's per-scan scratch (hot pairs, copied-entry offsets), reused across ads so assembly allocates nothing per ad.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.