Documentation
¶
Overview ¶
Package db is an embedded ClassAd log: a persistent key->ClassAd store with optimistic multi-writer transactions, mirroring HTCondor's ClassAdLog (src/condor_utils/classad_log.h). It is the Go core that the cgo layer (package capi) exposes as C symbols for a C++ interface to sit on top of, and that the client/server module serves over CEDAR.
It maps directly onto the collections store: the key->ClassAd table is a Collection, and each transaction is a collections.Txn (snapshot isolation, write-write conflicts, per-ad commit). Unlike classad_log.h -- which allows only one active transaction -- this supports any number of independent concurrent transactions, each a distinct *Txn.
Index ¶
- func MatchSignature(ad *classad.ClassAd, significantAttrs []string) uint64
- func ValidTableName(name string) bool
- type Catalog
- type CodecStats
- type Config
- type ConflictError
- type Constraint
- type DB
- func (db *DB) AddHotAttrs(names ...string) []string
- func (db *DB) AddIndex(categorical, value []string) bool
- func (db *DB) Begin() *Txn
- func (db *DB) Close() error
- func (db *DB) CodecStats(sampleMax int) CodecStats
- func (db *DB) Compact() int
- func (db *DB) DropIndex(names ...string) bool
- func (db *DB) Explain(constraint string) (QueryExplain, error)
- func (db *DB) ExplainMatch(job *classad.ClassAd, targetConstraint string) MatchExplain
- func (db *DB) ForEach(fn func(ad *classad.ClassAd) bool)
- func (db *DB) HotAttrs() []string
- func (db *DB) ID() string
- func (db *DB) IndexSizes() IndexSizes
- func (db *DB) IndexedAttrs() (categorical, value []string)
- func (db *DB) InstanceID() string
- func (db *DB) Keys() []string
- func (db *DB) Len() int
- func (db *DB) LookupClassAd(key string) (*classad.ClassAd, bool)
- func (db *DB) Maintain(opts MaintainOptions)
- func (db *DB) Match(job *classad.ClassAd) iter.Seq[*classad.ClassAd]
- func (db *DB) MatchSorted(job *classad.ClassAd, limit int) []*classad.ClassAd
- func (db *DB) MatchSortedRanked(job *classad.ClassAd, limit int) []RankedMatch
- func (db *DB) MatchSortedRankedFiltered(job *classad.ClassAd, targetConstraint string, limit int) ([]RankedMatch, error)
- func (db *DB) Ordered(index int, partition string, resume OrderCursor) iter.Seq[OrderedAd]
- func (db *DB) Query(constraint string) (iter.Seq[*classad.ClassAd], error)
- func (db *DB) QueryProject(constraint string, attrs []string) (iter.Seq[[]classad.Value], error)
- func (db *DB) RecordDemand(constraint string)
- func (db *DB) RefreshHotSet(sampleMax, topN int) int
- func (db *DB) Reindex()
- func (db *DB) RetrainDict(sampleMax int) (int, error)
- func (db *DB) Rewrite() int
- func (db *DB) StartMaintenance(interval time.Duration, opts MaintainOptions) (stop func())
- func (db *DB) Stats() Stats
- func (db *DB) SuggestDrops(sampleMax int) []DropSuggestion
- func (db *DB) SuggestIndexes(sampleMax int) []collections.IndexSuggestion
- func (db *DB) Watch(ctx context.Context, cursor []byte) (iter.Seq[WatchEvent], error)
- type DropSuggestion
- type IndexSize
- type IndexSizes
- type IndexSuggestion
- type MaintainOptions
- type MatchExplain
- type OrderCursor
- type OrderSpec
- type OrderedAd
- type QueryExplain
- type RankedMatch
- type SortKey
- type Stats
- type Txn
- func (t *Txn) Abort()
- func (t *Txn) Commit() error
- func (t *Txn) CommitNondurable() error
- func (t *Txn) DeleteAttribute(key, name string)
- func (t *Txn) DestroyClassAd(key string)
- func (t *Txn) LookupAttr(key, name string) (string, bool)
- func (t *Txn) LookupClassAd(key string) (*classad.ClassAd, bool)
- func (t *Txn) NewClassAd(key string, ad *classad.ClassAd)
- func (t *Txn) SetAttribute(key, name, expr string) error
- type WatchEvent
- type WatchKind
- type Watcher
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MatchSignature ¶
MatchSignature is HTCondor's autocluster key: a 64-bit checksum over the given significant attributes' expression text in ad. Two ads with textually identical significant attributes (same Requirements, same RequestCpus literal, ...) hash equal, so a matchmaker can compute one candidate list per distinct signature and reuse it for every identical request.
func ValidTableName ¶
ValidTableName reports whether name is usable as a table (and a directory): it must start with a letter or underscore and contain only letters, digits, underscores, and hyphens.
Types ¶
type Catalog ¶
type Catalog struct {
// contains filtered or unexported fields
}
A Catalog is a set of named tables, each an independent ClassAd store (its own keyspace, indexes, hot set, and persisted index config). Tables give the database more than one collection to work with -- e.g. separate machine and job ads -- without joins. A persistent catalog keeps each table in its own subdirectory (<dir>/tables/<name>), so tables are isolated on disk and the set is recovered by enumerating those subdirectories on open.
func OpenCatalog ¶
OpenCatalog opens the catalog rooted at dir, recovering any tables from <dir>/tables/. dir == "" makes an in-memory catalog whose tables do not persist.
func (*Catalog) CreateTable ¶
CreateTable creates (or returns the existing) table named name. Its data persists under <dir>/tables/<name> for a persistent catalog.
func (*Catalog) DropTable ¶
DropTable closes and removes the table named name, deleting its on-disk data.
func (*Catalog) EnsureTable ¶
EnsureTable creates the table if it does not exist, returning it.
type CodecStats ¶
type CodecStats = collections.CodecStats
Diagnostic and management types, re-exported from collections for callers that only import db.
type Config ¶
type Config struct {
Dir string
// Ordered configures maintained, filtered, sorted indexes -- e.g. the negotiator's
// resource-request lists (partition by Owner, sort by JobPrio then QDate), iterated
// in order via Ordered. Optional.
Ordered []OrderSpec
// HotAttrs / CategoricalAttrs / ValueAttrs / MatchClosureRoots tune storage and
// query/match push-down (see collections.Options). Optional.
HotAttrs []string
CategoricalAttrs, ValueAttrs []string
MatchClosureRoots []string
}
Config opens a DB with indexing and ordered-index configuration. Dir empty is in-memory; a non-empty Dir is persistent.
type ConflictError ¶
type ConflictError struct{ Keys []string }
ConflictError reports the keys whose writes lost an optimistic write-write race at commit. The other writes in the transaction committed; the caller re-reads and retries the conflicted keys.
func (*ConflictError) Error ¶
func (e *ConflictError) Error() string
type Constraint ¶
type Constraint struct {
// contains filtered or unexported fields
}
Constraint is a compiled ClassAd boolean expression, for evaluating the same filter against many ads without re-parsing.
func ParseConstraint ¶
func ParseConstraint(expr string) (*Constraint, error)
ParseConstraint compiles a ClassAd boolean expression.
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB is an embedded ClassAd log. Safe for concurrent use.
func Open ¶
Open opens a ClassAd log with default configuration. A non-empty dir makes it persistent (memory-mapped arenas under dir, recovered on reopen); an empty dir is in-memory. See OpenConfig for indexing / ordered-index configuration.
func OpenConfig ¶
OpenConfig opens a ClassAd log with the given configuration. Every DB is stamped with a stable DB id (persisted alongside a persistent store, so it survives reopen) and a fresh instance id for this open. Until high-availability DB servers exist they are effectively the same identity, but a follower/replica shares the DB id while carrying its own instance id.
func (*DB) AddHotAttrs ¶
AddHotAttrs pins the named attributes into the hot set and returns the resulting hot attributes. The hot set is persisted.
func (*DB) AddIndex ¶
AddIndex adds categorical and/or value indexes at runtime, returning whether the configuration changed. The spec is updated immediately; call Reindex to build the new index over existing ads. A change is persisted so it survives a restart.
func (*DB) CodecStats ¶
func (db *DB) CodecStats(sampleMax int) CodecStats
CodecStats reports the storage codec's state and effectiveness (name, dictionary size, last-retrain time, and a sampled compression ratio).
func (*DB) Compact ¶
Compact reclaims dead space in shards whose dead-byte ratio warrants it, returning the number of shards compacted.
func (*DB) DropIndex ¶
DropIndex removes the named attributes from the configured indexes, returning whether the configuration changed. A change is persisted.
func (*DB) Explain ¶
func (db *DB) Explain(constraint string) (QueryExplain, error)
Explain reports how the store would execute a constraint query -- which conjuncts are index-usable and the resulting access path.
func (*DB) ExplainMatch ¶
func (db *DB) ExplainMatch(job *classad.ClassAd, targetConstraint string) MatchExplain
ExplainMatch reports how matchmaking job against this (resource) collection would execute: the job's Requirements rewritten over the slot with the job's attributes baked to constants, and which of the resulting probes prune via a configured index. targetConstraint, if non-empty, is the MATCH resource-side filter (WHERE TARGET / NOPREEMPT) melded into the explanation.
func (*DB) ForEach ¶
ForEach calls fn for every committed ad, in no particular order, until fn returns false. It reads a consistent snapshot (concurrent writers do not block it).
func (*DB) HotAttrs ¶
HotAttrs returns the current hot attributes (front-loaded in each ad's hot header for cheap access).
func (*DB) IndexSizes ¶
func (db *DB) IndexSizes() IndexSizes
IndexSizes reports each configured index's resident bytes (with human/auto provenance) against the live data bytes -- the memory cost of indexing.
func (*DB) IndexedAttrs ¶
IndexedAttrs returns the currently-indexed attribute names, split into categorical (string equality/membership) and value (numeric + range) indexes.
func (*DB) InstanceID ¶
InstanceID is this open's identity (fresh each Open). Equal in spirit to ID until HA replicas exist, when replicas of one DB share ID but differ by InstanceID.
func (*DB) Keys ¶
Keys returns every committed key at a consistent snapshot, in no particular order. Useful for administrative enumeration and for a replica that must clear its keyspace before a full re-sync (see the leader-follower replicator).
func (*DB) LookupClassAd ¶
LookupClassAd returns the committed ad for key (the hash table, outside any transaction), or (nil, false).
func (*DB) Maintain ¶
func (db *DB) Maintain(opts MaintainOptions)
Maintain runs one self-tuning pass: it auto-tunes indexes (adds demand-driven ones, trims auto indexes over the memory budget, never touches human-created ones), refreshes the hot-attribute set, and optionally retrains the compression dictionary. Index/hot changes are persisted. Synchronous; a server drives it on a schedule (see dbrpc.Server.StartMaintenance).
func (*DB) Match ¶
Match returns the ads that symmetrically match job (bilateral Requirements), pushed down to the store. For the negotiator's pick-best pattern, prefer MatchSorted.
func (*DB) MatchSorted ¶
MatchSorted returns job's matches ranked by the job's Rank, best first, at most limit (<=0 = all) -- the negotiator resource-request path, with the store's deferred materialization so only the returned top-N are built.
func (*DB) MatchSortedRanked ¶
func (db *DB) MatchSortedRanked(job *classad.ClassAd, limit int) []RankedMatch
MatchSortedRanked is MatchSorted that also returns each match's Rank. The job's Requirements prunes candidate slots via any covering index (the matchmaking pushdown) rather than bilaterally evaluating every slot.
func (*DB) MatchSortedRankedFiltered ¶
func (db *DB) MatchSortedRankedFiltered(job *classad.ClassAd, targetConstraint string, limit int) ([]RankedMatch, error)
MatchSortedRankedFiltered is MatchSortedRanked restricted to slots that also satisfy targetConstraint (a MATCH's WHERE TARGET / NOPREEMPT filter over the resource ad). The constraint's index probes narrow the candidate scan (pushdown) and it is re-checked on each matched slot. An empty constraint is exactly MatchSortedRanked.
func (*DB) Ordered ¶
Ordered iterates one partition of the index-th configured ordered index in sort order (Config.Ordered), yielding each member ad with a resume cursor and its cluster signature (for run-length folding into resource-request lists). partition selects the run (e.g. an Owner); it is ignored for an index with no Partition. A zero resume starts at the beginning. The snapshot is O(1) and stable under concurrent churn.
func (*DB) Query ¶
Query returns the committed ads matching the constraint expression (an "old ClassAd" boolean expression over each ad, e.g. `JobStatus == 2 && Owner == "alice"`). The store pushes the filter down -- indexed constraints visit only candidates -- so this is far cheaper than ForEach + client-side filtering. Errors only on a malformed constraint.
func (*DB) QueryProject ¶
QueryProject returns, for each ad matching the constraint, just the named attributes' values (aligned with attrs), read wire-native where possible so an aggregate or projection does not pay the full-ad decode Query costs. The yielded slice is reused across iterations; copy any value to retain it past the next step. Errors only on a malformed constraint.
func (*DB) RecordDemand ¶
RecordDemand notes the attributes constraint filters on (for index suggestions) without scanning. Use it for filters applied outside the normal Query path -- e.g. a MATCH's resource-side WHERE TARGET constraint -- so those attributes still surface in SuggestIndexes. A malformed constraint is ignored.
func (*DB) RefreshHotSet ¶
RefreshHotSet recomputes the hot set as the topN most frequent attributes from a sample of up to sampleMax live ads, returning how many were chosen. The resulting hot set is persisted.
func (*DB) Reindex ¶
func (db *DB) Reindex()
Reindex rebuilds all configured indexes from the live ads.
func (*DB) RetrainDict ¶
RetrainDict trains a fresh ZSTD dictionary from a sample of up to sampleMax live ads, switches new writes to it, and recompresses existing records under it. It returns the new dictionary's size in bytes. This is what turns on (or refreshes) compression for a collection that started with the identity codec. See collections.Collection.RetrainDict.
func (*DB) Rewrite ¶
Rewrite re-encodes every live ad with the current hot set (so a changed hot set applies to existing ads) and force-compacts, returning the number of ads rewritten. A maintenance operation -- see collections.Collection.Rewrite.
func (*DB) StartMaintenance ¶
func (db *DB) StartMaintenance(interval time.Duration, opts MaintainOptions) (stop func())
StartMaintenance starts a background goroutine that runs Maintain with the given options every interval, returning a stop function. Prefer the catalog-wide dbrpc.Server.StartMaintenance, which also covers tables created later.
func (*DB) Stats ¶
Stats returns a snapshot of the store's storage (ad count, segment/arena/dead bytes) for observability.
func (*DB) SuggestDrops ¶
func (db *DB) SuggestDrops(sampleMax int) []DropSuggestion
SuggestDrops recommends indexes to drop (unused or low-cardinality) from observed demand and a sample of up to sampleMax live ads.
func (*DB) SuggestIndexes ¶
func (db *DB) SuggestIndexes(sampleMax int) []collections.IndexSuggestion
SuggestIndexes samples the store and returns attributes that queries filter on but are not yet indexed (advisory; a server may log or auto-apply them).
type DropSuggestion ¶
type DropSuggestion = collections.DropSuggestion
Diagnostic and management types, re-exported from collections for callers that only import db.
type IndexSize ¶
type IndexSize = collections.IndexSize
Diagnostic and management types, re-exported from collections for callers that only import db.
type IndexSizes ¶
type IndexSizes = collections.IndexSizes
Diagnostic and management types, re-exported from collections for callers that only import db.
type IndexSuggestion ¶
type IndexSuggestion = collections.IndexSuggestion
Diagnostic and management types, re-exported from collections for callers that only import db.
type MaintainOptions ¶
type MaintainOptions struct {
// SampleMax caps the ads sampled for index tuning, hot-set frequency, and dictionary
// training. Default 4096.
SampleMax int
// HotTopN refreshes the hot set to the topN most common attributes; 0 disables it.
HotTopN int
// Retrain retrains the ZSTD dictionary (recompacting + reindexing). Expensive on a
// large store, so a server may run it on a longer cadence than the rest.
Retrain bool
// MinIndexDemand is the minimum observed query count for the auto-tuner to add an
// index; 0 leaves index auto-tune off (no demand-driven adds).
MinIndexDemand int64
// IndexBudgetHighFrac / IndexBudgetLowFrac / IndexBudgetSlackBytes bound auto-created
// index memory as a fraction of the live data bytes (see collections.AutoTuneOptions);
// 0 high frac disables the budget (auto indexes grow unbounded).
IndexBudgetHighFrac float64
IndexBudgetLowFrac float64
IndexBudgetSlackBytes int64
}
MaintainOptions configures one maintenance pass (DB.Maintain).
type MatchExplain ¶
type MatchExplain = collections.MatchExplain
Diagnostic and management types, re-exported from collections for callers that only import db.
type OrderCursor ¶
type OrderCursor = collections.OrderCursor
OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.
type OrderSpec ¶
type OrderSpec = collections.OrderSpec
OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.
type OrderedAd ¶
type OrderedAd = collections.OrderedAd
OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.
type QueryExplain ¶
type QueryExplain = collections.QueryExplain
Diagnostic and management types, re-exported from collections for callers that only import db.
type RankedMatch ¶
type RankedMatch = collections.RankedMatch
RankedMatch is a matched ad with the job's Rank of it.
type SortKey ¶
type SortKey = collections.SortKey
OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.
type Stats ¶
type Stats = collections.Stats
Diagnostic and management types, re-exported from collections for callers that only import db.
type Txn ¶
type Txn struct {
// contains filtered or unexported fields
}
Txn is an independent optimistic transaction. Operations are buffered and applied at Commit under snapshot-isolation OCC. A *Txn is not safe for concurrent use by multiple goroutines; independent transactions are.
func (*Txn) Abort ¶
func (t *Txn) Abort()
Abort discards the transaction's buffered operations. Nothing is written.
func (*Txn) Commit ¶
Commit applies the buffered operations. It returns a *ConflictError if any key was modified by another committer since this transaction's snapshot (the non-conflicted operations still committed), or nil on full success.
func (*Txn) CommitNondurable ¶
CommitNondurable is Commit that defers the disk durability sync (classad_log.h CommitNondurableTransaction): the writes are visible immediately but their flush is batched to a later durable commit. On an in-memory DB it is identical to Commit.
func (*Txn) DeleteAttribute ¶
DeleteAttribute removes one attribute of key (classad_log.h LogDeleteAttribute). A no-op if key or the attribute is absent.
func (*Txn) DestroyClassAd ¶
DestroyClassAd removes key (classad_log.h LogDestroyClassAd).
func (*Txn) LookupAttr ¶
LookupAttr returns the unparsed expression of one attribute as the transaction sees it (classad_log.h LookupInTransaction), or ("", false).
func (*Txn) LookupClassAd ¶
LookupClassAd returns key's ad as the transaction sees it: its own buffered writes (read-your-writes) merged over the snapshot (classad_log.h Lookup + the LookupInTransaction overlay in one call).
func (*Txn) NewClassAd ¶
NewClassAd stores ad under key (classad_log.h LogNewClassAd). An existing ad at key is replaced.
func (*Txn) SetAttribute ¶
SetAttribute sets one attribute of key to the expression parsed from expr (classad_log.h LogSetAttribute) -- a read-modify-write within the transaction, so it composes with the transaction's own earlier writes to key. The ad is created if absent.
type WatchEvent ¶
type WatchEvent struct {
Kind WatchKind
Key string
Ad *classad.ClassAd // nil for a delete
Cursor []byte
}
WatchEvent is one change to the log. Cursor resumes a watch just after this event (survives reconnect / restart, so no change is missed).
type WatchKind ¶
type WatchKind uint8
WatchKind classifies a watch event.
const ( // WatchUpsert: key was added or updated; Ad holds its new value. WatchUpsert WatchKind = iota // WatchDelete: key was removed; Ad is nil. WatchDelete // WatchReset: discard prior state and rebuild from the upserts that follow (the // initial full replay, or a re-sync after the cursor fell out of retention). Key // and Ad are empty. WatchReset )
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher is a watch whose readiness is signalled on a file descriptor -- for the C (DaemonCore) side, which registers the fd in its poll loop and, on wakeup, drains events with Next. Go writes a single wakeup byte when the queue goes non-empty (coalesced: the reader drains fully per wakeup). The fd is a pipe or eventfd the caller created and owns; Watcher never closes it.
func NewWatcher ¶
NewWatcher starts watching db, signalling notifyFD when events are queued. cursor nil starts from now.
func (*Watcher) Next ¶
func (w *Watcher) Next() (WatchEvent, bool)
Next dequeues one event without blocking. ok is false when the queue is empty; the caller (having been woken via the fd) drains until ok is false.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package main builds the embedded ClassAd log as a C archive: it exports C symbols (cadb_*) mirroring HTCondor's classad_log.h, so a C++ interface (built on libcondor_utils) can sit on top.
|
Package main builds the embedded ClassAd log as a C archive: it exports C symbols (cadb_*) mirroring HTCondor's classad_log.h, so a C++ interface (built on libcondor_utils) can sit on top. |