Documentation
¶
Overview ¶
Package clustering groups entities by entity-ID prefix and publishes one bounded record per group to the COMMUNITY_INDEX bucket.
A group (a "community" in the record) is every entity whose ID shares one prefix: level 0 is org.platform.system.domain, level 1 is org.platform.system and level 2 is org.platform (ADR-099, with the levels ADR-102 gives). Nothing is detected: an entity's groups follow from its ID alone.
A Maintainer keeps the groups current from entity changes. The caller feeds it one ordered watch of ENTITY_STATES: Upsert or Delete for each delivery, with the revision of the exact value decoded, then BootstrapDone when the watch's initial replay ends. Tick, called on an interval, writes the record of each group that changed since its last publication (Community) through a NATSCommunityStorage, and removes nothing until every write of that tick has succeeded. GetAllCommunities reads every record back.
The package starts no goroutine and holds no ENTITY_STATES handle; the watch and the ticker belong to the caller (processor/graph-clustering). Records are bounded by construction (MaxRecordBytes) and hold counts and samples, never a member list. The README lists what is not here: label propagation, and the language-model summaries.
Index ¶
Constants ¶
const ( // MaxRepEntities is the most representative entity IDs a record holds. MaxRepEntities = 5 // MaxKeywords is the most keywords, and the most entity types, a record holds. MaxKeywords = 10 // MaxEntityTerms is the most words one entity contributes to each of its groups. MaxEntityTerms = 32 // MaxWordBytes is the longest word. A longer run of letters and digits is not a word. MaxWordBytes = 32 // MaxRecordBytes bounds a record's JSON encoding whatever its members' values: a // 256-byte ID, MaxKeywords types of at most 246 bytes, MaxKeywords words, // MaxRepEntities 256-byte IDs and a summary naming three types and five keywords // encode to under 6 KiB (TestRecordIsBounded). MaxRecordBytes = 8 << 10 )
The bounds of a published group record (design D4). Each holds by construction, so a record is never truncated at run time.
Variables ¶
var ErrBootstrapIncomplete = errors.New("clustering: bootstrap incomplete")
ErrBootstrapIncomplete is returned by Tick before BootstrapDone: a record built from a partial replay would be published as if it were the graph.
Functions ¶
func GroupKey ¶
GroupKey returns the COMMUNITY_INDEX key of entityID's group at level: the level, a dot, and the ID's first four (level 0), three (1) or two (2) positions. It refuses, with an invalid error, a level outside 0 to 2 and an ID the canonical parser refuses.
func LevelPrefix ¶
LevelPrefix returns the prefix every COMMUNITY_INDEX key at level starts with: the level and a dot. It refuses, with an invalid error, a level outside 0 to 2.
Types ¶
type Community ¶
type Community struct {
// ID is the group's entity-ID prefix.
ID string `json:"id"`
// Level is the group's level: 0, 1 or 2.
Level int `json:"level"`
// MemberCount is the number of entities in the group.
MemberCount int `json:"member_count"`
// EntityRevision is the highest ENTITY_STATES revision among the Upsert and Delete
// calls the maintainer applied for entities whose IDs parse into the group, held,
// refused or not; a delivery at or below an entity's last-applied revision is not
// applied. The mark ends with the group: a recreated group starts at the write that
// recreated it. With the caller feeding ascending revisions across keys, every write at
// or below it is reflected in the record. It is never 0 (DecodeCommunity).
EntityRevision uint64 `json:"entity_revision"`
// TypeCounts counts members by entity type (the ID's fifth position), for the
// MaxKeywords most common types, ties broken lexically.
TypeCounts map[string]int `json:"type_counts"`
// Keywords are the group's MaxKeywords most frequent words, most frequent first,
// ties broken lexically. Words come only from members' string values.
Keywords []string `json:"keywords,omitempty"`
// StatisticalSummary is a one-line text built from the counts, three types and five
// keywords.
StatisticalSummary string `json:"statistical_summary,omitempty"`
// RepEntities are the lexically smallest IDs of members with words, at most
// MaxRepEntities.
RepEntities []string `json:"rep_entities,omitempty"`
}
Community is the published record of one group: every entity whose ID shares one prefix. Level 0 groups by org.platform.system.domain, level 1 by the source org.platform.system, level 2 by the deployment org.platform (ADR-102 d6); ID is that prefix. The record holds counts and samples, never the member list, so its size does not grow with the group.
func DecodeCommunity ¶
DecodeCommunity decodes value, the COMMUNITY_INDEX entry at key, as storage does. It refuses, with an invalid error naming key, a value that is not JSON, a record whose level is not 0, 1 or 2, a record whose level and ID do not give key, and a record whose EntityRevision is 0, which no maintainer writes.
type IncompleteError ¶
type IncompleteError struct {
// Refused are the refused entity IDs, sorted.
Refused []string
}
IncompleteError is returned by a Tick that published every group while some entities stay refused: the records leave those entities out.
func (*IncompleteError) Error ¶
func (e *IncompleteError) Error() string
type Maintainer ¶
type Maintainer struct {
// contains filtered or unexported fields
}
Maintainer keeps one group per entity-ID prefix at each level current from entity changes, and publishes the changed groups' records on each Tick. A caller feeds it one ordered watch of the entity bucket: Upsert and Delete for each delivery, in increasing revision order per key, then BootstrapDone when the initial replay ends. Its methods may be called from the watch goroutine and a ticker goroutine at once.
func NewMaintainer ¶
func NewMaintainer(storage *NATSCommunityStorage) (*Maintainer, error)
NewMaintainer returns a Maintainer publishing to storage. It refuses a nil storage.
func (*Maintainer) BootstrapDone ¶
func (m *Maintainer) BootstrapDone()
BootstrapDone records that the initial replay has been applied; Tick publishes only after it.
func (*Maintainer) Delete ¶
func (m *Maintainer) Delete(entityID string, revision uint64) error
Delete removes entityID's contribution at revision. A revision at or below the one last applied changes nothing. An entity it does not hold has nothing to remove, but the Delete is applied all the same: like any applied Delete it raises its groups' Community.EntityRevision to revision. Revision 0 is refused, as in Upsert.
func (*Maintainer) Tick ¶
func (m *Maintainer) Tick(ctx context.Context) error
Tick publishes every group changed since its last successful publication: it writes each record that differs from the one last written and, only after every write succeeded, removes the records of groups that became empty and, after the first tick's one key listing, the keys of groups that do not exist. A failure, or the context ending before the pending work is done, returns an error and leaves the work pending for the next Tick; with nothing pending, an ended context is not consulted. It returns ErrBootstrapIncomplete before BootstrapDone, an *IncompleteError when all is published but some entities are refused, and nil otherwise.
func (*Maintainer) Upsert ¶
func (m *Maintainer) Upsert(entityID string, revision uint64, state *graph.EntityState) error
Upsert applies entityID's value at revision, the revision of the exact value state was decoded from. A revision at or below the one last applied for entityID changes nothing. Revision 0 is refused and changes nothing: a KV entry's revision is its stream sequence, which starts at 1 (TestKVRevisionStartsAtOne), so 0 is a caller that passed no revision. It refuses, with an invalid error naming the entity, a nil state, an ID the canonical parser refuses, or a state whose ID is not entityID; a refused entity contributes nothing until a valid Upsert or a Delete for it. An applied Upsert, refused or not, raises its groups' Community.EntityRevision to revision.
type NATSCommunityStorage ¶
type NATSCommunityStorage struct {
// contains filtered or unexported fields
}
NATSCommunityStorage reads and writes group records in the COMMUNITY_INDEX bucket (graph.BucketCommunityIndex), one key per group: {level}.{group ID}.
func NewNATSCommunityStorage ¶
func NewNATSCommunityStorage(kv *natsclient.KVStore) (*NATSCommunityStorage, error)
NewNATSCommunityStorage returns a storage over kv, the COMMUNITY_INDEX bucket. It refuses a nil kv.
func (*NATSCommunityStorage) GetAllCommunities ¶
func (s *NATSCommunityStorage) GetAllCommunities(ctx context.Context) ([]*Community, error)
GetAllCommunities returns every record in the bucket, in key order. An entry that cannot be read, or is not the record of the group its key names, fails the call with an error naming the key; an ended context returns its error. An empty bucket gives no records and no error.