Documentation
¶
Overview ¶
Package efficiency offers utilities for generating read-only channels, merging and splitting streams, concurrent processing of channel items, and partitioning key-value stores using shards for scalability and performance.
Index ¶
- func Generate[T any](in ...T) <-chan T
- func Merge[T any](in ...<-chan T) chan T
- func Process[IN, OUT any](in <-chan IN, fn service.Function[IN, OUT]) (<-chan OUT, <-chan error)
- func SearchContext(ctx context.Context) *atomic.Bool
- func Split[T any](in <-chan T, num int) []<-chan T
- func WithCompression(next http.Handler) http.Handler
- type Result
- type SearchOptions
- type Shard
- type Sharding
- type SparseSet
- type SparseSetProvider
- type SparseVectorProvider
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Generate ¶
func Generate[T any](in ...T) <-chan T
Generate takes a variadic input of any type T and returns a read-only channel of type T. It sends each input value into the returned channel in a separate goroutine.
func Merge ¶
func Merge[T any](in ...<-chan T) chan T
Merge combines multiple input channels into a single output channel. It starts a goroutine for each input channel to forward its values to the output channel. Once all input channels are processed, the output channel is closed.
func Process ¶
Process concurrently processes items from the input channel using the provided function `fn`. It spawns a number of worker goroutines equal to the number of available CPU cores.
func SearchContext ¶ added in v0.5.10
SearchContext creates a stopped flag that respects context cancellation. Use this to integrate with context-based cancellation since ForEach does not accept context.
Types ¶
type Result ¶ added in v0.5.10
type Result[K comparable, V any] struct { Key K Value *V Score float64 }
Result represents a similarity search result with the key, value, and similarity score.
func FindSimilarCosine ¶ added in v0.5.10
func FindSimilarCosine[K comparable, V SparseVectorProvider]( store interface{ ForEach(fn func(K, V) bool) }, query V, opts SearchOptions, stopped *atomic.Bool, ) []Result[K, V]
FindSimilarCosine searches for items similar to the query using cosine similarity. V must implement SparseVectorProvider to extract TF-IDF sparse vectors. Returns results sorted by descending similarity score.
The function iterates over the store using ForEach, computing similarity for each item and maintaining a top-K heap for efficient result collection.
Use stopped flag (from SearchContext) to enable cancellation since ForEach does not accept context. Pass nil if cancellation is not needed.
func FindSimilarJaccard ¶ added in v0.5.10
func FindSimilarJaccard[K comparable, V SparseSetProvider]( store interface{ ForEach(fn func(K, V) bool) }, query V, opts SearchOptions, stopped *atomic.Bool, ) []Result[K, V]
FindSimilarJaccard searches for items similar to the query using Jaccard similarity. V must implement SparseSetProvider to extract term sets. Returns results sorted by descending similarity score.
The function iterates over the store using ForEach, computing similarity for each item and maintaining a top-K heap for efficient result collection.
Use stopped flag (from SearchContext) to enable cancellation since ForEach does not accept context. Pass nil if cancellation is not needed.
type SearchOptions ¶ added in v0.5.10
type SearchOptions struct {
// TopK limits results to the K most similar items. Default: 10.
TopK int
// Threshold filters results below this similarity score. Default: 0.0.
Threshold float64
// EarlyStopCount stops after finding this many results above threshold.
// Zero means no early stopping. Default: 0.
EarlyStopCount int
}
SearchOptions configures similarity search behavior.
func DefaultSearchOptions ¶ added in v0.5.10
func DefaultSearchOptions() SearchOptions
DefaultSearchOptions returns sensible defaults.
type Shard ¶
type Shard[K comparable, V any] struct { // contains filtered or unexported fields }
Shard represents a single partition of a sharded key-value store. It holds a map of keys (of type K) to values (of type V), and a mutex for thread-safety.
type Sharding ¶
type Sharding[K comparable, V any] []Shard[K, V]
Sharding is a collection of Shard objects, each representing a separate partition of the key-value store. This allows for distributing keys across multiple shards.
func NewSharding ¶
func NewSharding[K comparable, V any](num int) Sharding[K, V]
NewSharding creates an array of Shard objects with the given number of shards.
func (Sharding[K, V]) Delete ¶
func (a Sharding[K, V]) Delete(key K)
Delete removes a key-value pair from the appropriate shard.
type SparseSet ¶ added in v0.1.62
type SparseSet[T comparable] struct { Dense []T Sparse []int Size int }
SparseSet is a data structure that provides efficient storage and retrieval of elements.
func NewSparseSet ¶ added in v0.1.62
func NewSparseSet[T comparable](initialSize int) *SparseSet[T]
NewSparseSet creates a new SparseSet with the given initial Size.
func (*SparseSet[T]) Densed ¶ added in v0.1.65
func (a *SparseSet[T]) Densed() []T
Densed returns the Dense representation of the SparseSet.
type SparseSetProvider ¶ added in v0.5.10
type SparseSetProvider interface {
// SparseSet returns sorted term indices representing set membership.
// Indices MUST be sorted in ascending order for merge-loop efficiency.
SparseSet() []int
}
SparseSetProvider extracts a sparse set (binary term presence) from a value. Used for Jaccard similarity on token sets.
type SparseVectorProvider ¶ added in v0.5.10
type SparseVectorProvider interface {
// SparseVector returns sorted term indices, corresponding TF-IDF values, and pre-computed L2 norm.
// Indices MUST be sorted in ascending order for merge-loop efficiency.
SparseVector() (indices []int, values []float64, norm float64)
}
SparseVectorProvider extracts a sparse vector representation from a value. Used for cosine similarity on TF-IDF vectors.
Source Files