Documentation
¶
Overview ¶
Package efficiency offers utilities for generating read-only channels, merging and splitting streams, concurrent processing of channel items, and partitioning key-value stores using shards for scalability and performance.
Index ¶
- func CosineSimilarity(indicesA, indicesB []int, valuesA, valuesB []float64, normA, normB float64) float64
- func Generate[T any](in ...T) <-chan T
- func JaccardSimilarity(setA, setB []int) float64
- func Merge[T any](in ...<-chan T) chan T
- func Process[IN, OUT any](in <-chan IN, fn service.Function[IN, OUT]) (<-chan OUT, <-chan error)
- func Split[T any](in <-chan T, num int) []<-chan T
- func WithCompression(next http.Handler) http.Handler
- type KeyedSparseSet
- func (s *KeyedSparseSet[K, V]) Clear()
- func (s *KeyedSparseSet[K, V]) Delete(key K) bool
- func (s *KeyedSparseSet[K, V]) ForEach(fn func(K, V) bool)
- func (s *KeyedSparseSet[K, V]) Get(key K) *V
- func (s *KeyedSparseSet[K, V]) Has(key K) bool
- func (s *KeyedSparseSet[K, V]) Len() int
- func (s *KeyedSparseSet[K, V]) Put(key K, value V) bool
- func (s *KeyedSparseSet[K, V]) Values() []V
- type Shard
- type Sharding
- type SparseSet
- type SparseShard
- type SparseSharding
- func (s *SparseSharding[K, V]) Clear()
- func (s *SparseSharding[K, V]) Delete(key K) bool
- func (s *SparseSharding[K, V]) ForEach(fn func(K, V) bool)
- func (s *SparseSharding[K, V]) ForEachShard(fn func(shardIdx int, iterate func(fn func(K, V) bool)))
- func (s *SparseSharding[K, V]) Get(key K) *V
- func (s *SparseSharding[K, V]) Has(key K) bool
- func (s *SparseSharding[K, V]) Len() int
- func (s *SparseSharding[K, V]) Put(key K, value V) bool
- func (s *SparseSharding[K, V]) Values() []V
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CosineSimilarity ¶ added in v0.5.11
func CosineSimilarity(indicesA, indicesB []int, valuesA, valuesB []float64, normA, normB float64) float64
CosineSimilarity computes cosine similarity between two sparse vectors using merge-loop. Both index slices MUST be sorted in ascending order. Formula: cosine(A, B) = dot(A, B) / (norm(A) * norm(B)).
Example usage with ShardedSparseAccess.SearchSimilar:
results := store.SearchSimilar(ctx, func(doc Document) float64 {
return efficiency.CosineSimilarity(
query.Indices, doc.Indices,
query.Values, doc.Values,
query.Norm, doc.Norm,
)
}, resource.SearchOptions{TopK: 10})
func Generate ¶
func Generate[T any](in ...T) <-chan T
Generate takes a variadic input of any type T and returns a read-only channel of type T. It sends each input value into the returned channel in a separate goroutine.
func JaccardSimilarity ¶ added in v0.5.11
JaccardSimilarity computes Jaccard similarity between two sparse sets using merge-loop. Both index slices MUST be sorted in ascending order. Formula: jaccard(A, B) = |A ∩ B| / |A ∪ B|.
Example usage with ShardedSparseAccess.SearchSimilar:
results := store.SearchSimilar(ctx, func(article Article) float64 {
return efficiency.JaccardSimilarity(queryTags, article.Tags)
}, resource.SearchOptions{TopK: 10})
func Merge ¶
func Merge[T any](in ...<-chan T) chan T
Merge combines multiple input channels into a single output channel. It starts a goroutine for each input channel to forward its values to the output channel. Once all input channels are processed, the output channel is closed.
func Process ¶
Process concurrently processes items from the input channel using the provided function `fn`. It spawns a number of worker goroutines equal to the number of available CPU cores.
Types ¶
type KeyedSparseSet ¶ added in v0.5.11
type KeyedSparseSet[K comparable, V any] struct { // contains filtered or unexported fields }
KeyedSparseSet is a generic key-value sparse set with O(1) operations. It uses bidirectional mapping (key->index, index->key) for O(1) swap-remove.
func NewKeyedSparseSet ¶ added in v0.5.11
func NewKeyedSparseSet[K comparable, V any](capacity int) *KeyedSparseSet[K, V]
NewKeyedSparseSet creates a new KeyedSparseSet with the given initial capacity.
func (*KeyedSparseSet[K, V]) Clear ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) Clear()
Clear removes all elements.
func (*KeyedSparseSet[K, V]) Delete ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) Delete(key K) bool
Delete removes a key-value pair. Returns true if the key existed. Uses O(1) swap-remove: swaps the deleted element with the last element.
func (*KeyedSparseSet[K, V]) ForEach ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) ForEach(fn func(K, V) bool)
ForEach iterates over all elements. Stops if fn returns false.
func (*KeyedSparseSet[K, V]) Get ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) Get(key K) *V
Get returns the value for a key, or nil if not found.
func (*KeyedSparseSet[K, V]) Has ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) Has(key K) bool
Has returns true if the key exists.
func (*KeyedSparseSet[K, V]) Len ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) Len() int
Len returns the number of elements.
func (*KeyedSparseSet[K, V]) Put ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) Put(key K, value V) bool
Put adds or updates a key-value pair. Returns true if the key was new.
func (*KeyedSparseSet[K, V]) Values ¶ added in v0.5.11
func (s *KeyedSparseSet[K, V]) Values() []V
Values returns a slice of all values (dense array).
type Shard ¶
type Shard[K comparable, V any] struct { // contains filtered or unexported fields }
Shard represents a single partition of a sharded key-value store. It holds a map of keys (of type K) to values (of type V), and a mutex for thread-safety.
type Sharding ¶
type Sharding[K comparable, V any] []Shard[K, V]
Sharding is a collection of Shard objects, each representing a separate partition of the key-value store. This allows for distributing keys across multiple shards.
func NewSharding ¶
func NewSharding[K comparable, V any](num int) Sharding[K, V]
NewSharding creates an array of Shard objects with the given number of shards.
func (Sharding[K, V]) Delete ¶
func (a Sharding[K, V]) Delete(key K)
Delete removes a key-value pair from the appropriate shard.
type SparseSet ¶ added in v0.1.62
type SparseSet[T comparable] struct { Dense []T Sparse []int Size int }
SparseSet is a data structure that provides efficient storage and retrieval of elements.
func NewSparseSet ¶ added in v0.1.62
func NewSparseSet[T comparable](initialSize int) *SparseSet[T]
NewSparseSet creates a new SparseSet with the given initial Size.
func (*SparseSet[T]) Densed ¶ added in v0.1.65
func (a *SparseSet[T]) Densed() []T
Densed returns the Dense representation of the SparseSet.
type SparseShard ¶ added in v0.5.11
type SparseShard[K comparable, V any] struct { // contains filtered or unexported fields }
SparseShard is a single partition using KeyedSparseSet for O(1) operations.
type SparseSharding ¶ added in v0.5.11
type SparseSharding[K comparable, V any] struct { // contains filtered or unexported fields }
SparseSharding distributes key-value pairs across multiple KeyedSparseSets. It provides per-shard locking for high-concurrency workloads and O(1) delete.
func NewSparseSharding ¶ added in v0.5.11
func NewSparseSharding[K comparable, V any](numShards int) *SparseSharding[K, V]
NewSparseSharding creates a new SparseSharding with the given number of shards. A typical value is 32 or runtime.NumCPU() * 4.
func NewSparseShardingWithCapacity ¶ added in v0.5.11
func NewSparseShardingWithCapacity[K comparable, V any](numShards, capacityPerShard int) *SparseSharding[K, V]
NewSparseShardingWithCapacity creates a new SparseSharding with pre-allocated capacity per shard.
func (*SparseSharding[K, V]) Clear ¶ added in v0.5.11
func (s *SparseSharding[K, V]) Clear()
Clear removes all elements from all shards.
func (*SparseSharding[K, V]) Delete ¶ added in v0.5.11
func (s *SparseSharding[K, V]) Delete(key K) bool
Delete removes a key-value pair. Returns true if the key existed.
func (*SparseSharding[K, V]) ForEach ¶ added in v0.5.11
func (s *SparseSharding[K, V]) ForEach(fn func(K, V) bool)
ForEach iterates over all elements across all shards. Stops if fn returns false. Iteration order is not guaranteed.
func (*SparseSharding[K, V]) ForEachShard ¶ added in v0.5.11
func (s *SparseSharding[K, V]) ForEachShard(fn func(shardIdx int, iterate func(fn func(K, V) bool)))
ForEachShard iterates over each shard with the shard index. The callback receives the shard index and a function to iterate within that shard. Useful for checking cancellation between shards.
func (*SparseSharding[K, V]) Get ¶ added in v0.5.11
func (s *SparseSharding[K, V]) Get(key K) *V
Get retrieves a value by key. Returns nil if not found.
func (*SparseSharding[K, V]) Has ¶ added in v0.5.11
func (s *SparseSharding[K, V]) Has(key K) bool
Has returns true if the key exists.
func (*SparseSharding[K, V]) Len ¶ added in v0.5.11
func (s *SparseSharding[K, V]) Len() int
Len returns the total number of elements across all shards.
func (*SparseSharding[K, V]) Put ¶ added in v0.5.11
func (s *SparseSharding[K, V]) Put(key K, value V) bool
Put adds or updates a key-value pair. Returns true if the key was new.
func (*SparseSharding[K, V]) Values ¶ added in v0.5.11
func (s *SparseSharding[K, V]) Values() []V
Values returns all values across all shards.
Source Files