storage

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 15, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Copyright 2026 Teradata

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const (
	// DefaultMaxMemoryBytes is 1GB
	DefaultMaxMemoryBytes = 1 * 1024 * 1024 * 1024
	// DefaultSharedMemoryThreshold is 2.5KB (balanced for multi-agent communication)
	// Payloads larger than this will be stored as references instead of inline values,
	// providing 20-30% token savings on typical workloads.
	DefaultSharedMemoryThreshold = 2560 // 2.5KB
	// DefaultCompressionThreshold is 1MB
	DefaultCompressionThreshold = 1 * 1024 * 1024
	// DefaultTTLSeconds is 1 hour
	DefaultTTLSeconds = 3600
)
View Source 
const (
	// DefaultMaxDiskBytes is 10GB
	DefaultMaxDiskBytes = 10 * 1024 * 1024 * 1024
)

Variables

This section is empty.

Functions

func GenerateID 

func GenerateID() string

GenerateID generates a unique ID for a data reference.

func GetDefaultLoomDBPath 

func GetDefaultLoomDBPath() string

GetDefaultLoomDBPath returns the default path to loom.db

func IsLargeData 

func IsLargeData(data []byte, threshold int64) bool

IsLargeData checks if data exceeds the threshold for shared memory.

func IsSQLResult 

func IsSQLResult(data interface{}) bool

IsSQLResult checks if data looks like a SQL result (has rows and columns).

func RefToString 

func RefToString(ref *loomv1.DataReference) string

RefToString converts a DataReference to a human-readable string.

func ResetGlobalSharedMemory 

func ResetGlobalSharedMemory()

ResetGlobalSharedMemory resets the global singleton (for testing only). This should NOT be used in production code.

func ShouldUseSharedMemory 

func ShouldUseSharedMemory(dataSize int64, threshold int64) bool

ShouldUseSharedMemory determines if data should go to shared memory.

func ValidateReference 

func ValidateReference(ref *loomv1.DataReference) error

ValidateReference validates a data reference.

Types

type Config 

type Config struct {
	MaxMemoryBytes       int64
	CompressionThreshold int64
	TTLSeconds           int64
	OverflowHandler      OverflowHandler
}

Config configures the shared memory store.

type DataMetadata 

type DataMetadata struct {
	ID              string
	ContentType     string // "application/json", "text/csv", "text/plain"
	DataType        string // "json_array", "json_object", "csv", "text"
	SizeBytes       int64
	EstimatedTokens int64
	Schema          *SchemaInfo
	Preview         *PreviewData
	CreatedAt       time.Time
	ExpiresAt       time.Time
	Location        loomv1.StorageLocation
}

DataMetadata contains metadata about stored data for progressive disclosure. This enables agents to inspect data structure and size before retrieving full content.

type DiskMetadata 

type DiskMetadata struct {
	ID          string
	FilePath    string
	Size        int64
	StoredAt    time.Time
	AccessedAt  time.Time
	Checksum    string
	ContentType string
	Compressed  bool
}

DiskMetadata tracks metadata for disk-stored data.

type DiskOverflowConfig 

type DiskOverflowConfig struct {
	CachePath   string
	MaxDiskSize int64
	TTLSeconds  int64
}

DiskOverflowConfig configures the disk overflow manager.

type DiskOverflowManager 

type DiskOverflowManager struct {
	// contains filtered or unexported fields
}

DiskOverflowManager manages overflow data on disk.

func NewDiskOverflowManager 

func NewDiskOverflowManager(config *DiskOverflowConfig) (*DiskOverflowManager, error)

NewDiskOverflowManager creates a new disk overflow manager.

func (*DiskOverflowManager) Delete 

func (d *DiskOverflowManager) Delete(id string) error

Delete removes data from disk.

func (*DiskOverflowManager) Promote 

func (d *DiskOverflowManager) Promote(id string) ([]byte, error)

Promote retrieves data from disk and removes it (for promoting back to memory).

func (*DiskOverflowManager) Retrieve 

func (d *DiskOverflowManager) Retrieve(id string) ([]byte, error)

Retrieve retrieves data from disk.

func (*DiskOverflowManager) Stats 

func (d *DiskOverflowManager) Stats() DiskStats

Stats returns statistics about the disk overflow manager.

func (*DiskOverflowManager) Store 

func (d *DiskOverflowManager) Store(id string, data []byte, ref *loomv1.DataReference) error

Store stores data to disk.

type DiskStats 

type DiskStats struct {
	CurrentSize int64
	MaxSize     int64
	ItemCount   int
}

DiskStats holds statistics about the disk overflow manager.

type FieldInfo 

type FieldInfo struct {
	Name string
	Type string // "string", "number", "boolean", "object", "array", "null"
}

FieldInfo describes a field in JSON data.

type OverflowHandler 

type OverflowHandler interface {
	Store(id string, data []byte, metadata *loomv1.DataReference) error
	Retrieve(id string) ([]byte, error)
	Delete(id string) error
}

OverflowHandler handles data that doesn't fit in memory.

type PreviewData 

type PreviewData struct {
	First5 []any
	Last5  []any
}

PreviewData contains sample data for quick inspection.

type SQLResultMetadata 

type SQLResultMetadata struct {
	ID          string
	TableName   string
	RowCount    int64
	ColumnCount int
	Columns     []string
	Preview     *PreviewData // Preview data (first 5 + last 5 rows)
	StoredAt    time.Time
	AccessedAt  time.Time
	SizeBytes   int64
}

SQLResultMetadata tracks metadata for stored SQL results.

type SQLResultStore 

type SQLResultStore struct {
	// contains filtered or unexported fields
}

SQLResultStore stores SQL query results in queryable SQLite tables. This allows the LLM to filter/analyze large result sets without loading everything into context.

CRITICAL FIX: Uses regular tables (not TEMP) and persists metadata in database to ensure results are accessible across all database connections and survive restarts.

func NewSQLResultStore 

func NewSQLResultStore(config *SQLResultStoreConfig) (*SQLResultStore, error)

NewSQLResultStore creates a new SQL result store.

func (*SQLResultStore) Close 

func (s *SQLResultStore) Close() error

Close closes the database connection.

func (*SQLResultStore) Delete 

func (s *SQLResultStore) Delete(id string) error

Delete removes a stored result.

func (*SQLResultStore) GetMetadata 

func (s *SQLResultStore) GetMetadata(id string) (*SQLResultMetadata, error)

GetMetadata returns metadata about a stored result.

func (*SQLResultStore) Query 

func (s *SQLResultStore) Query(id, query string) (interface{}, error)

Query executes a SQL query against a stored result.

func (*SQLResultStore) Store 

func (s *SQLResultStore) Store(id string, data interface{}) (*loomv1.DataReference, error)

Store stores SQL result data in a queryable table.

type SQLResultStoreConfig 

type SQLResultStoreConfig struct {
	DBPath     string // Path to SQLite database (defaults to ~/.loom/loom.db)
	TTLSeconds int64
}

SQLResultStoreConfig configures the SQL result store.

type SchemaInfo 

type SchemaInfo struct {
	Type       string      // "array", "object", "table"
	ItemCount  int64       // Number of items in array or rows in table
	Fields     []FieldInfo // For JSON objects
	Columns    []string    // For CSV/tabular data
	SampleItem any         // Representative sample
}

SchemaInfo describes the structure of the data.

type SessionReferenceTracker 

type SessionReferenceTracker struct {
	// contains filtered or unexported fields
}

SessionReferenceTracker tracks which references belong to which sessions for automatic cleanup when sessions end. Thread-safe for concurrent access.

Design principles: - Lightweight: In-memory tracking only (ephemeral tool results don't need persistence) - Idempotent: Safe to call Pin/Unpin multiple times - Defensive: Nil/empty inputs are no-ops (prevents crashes) - Observable: Provides stats for monitoring

Usage: 

tracker := NewSessionReferenceTracker(sharedMemory)
// When storing tool result:
tracker.PinForSession(sessionID, refID)
// When session ends:
tracker.UnpinSession(sessionID)  // Releases all refs for session

func NewSessionReferenceTracker 

func NewSessionReferenceTracker(sharedMemory *SharedMemoryStore) *SessionReferenceTracker

NewSessionReferenceTracker creates a new reference tracker. The sharedMemory store is used to call Release() on references during cleanup.

func (*SessionReferenceTracker) GetSessionReferences 

func (t *SessionReferenceTracker) GetSessionReferences(sessionID string) []string

GetSessionReferences returns a copy of all references for a session. Returns empty slice if session has no references or doesn't exist. Used primarily for testing and debugging.

Thread-safe: Can be called concurrently from multiple goroutines.

func (*SessionReferenceTracker) PinForSession 

func (t *SessionReferenceTracker) PinForSession(sessionID, refID string)

PinForSession tracks a reference for a session (idempotent). If the reference is already pinned for this session, this is a no-op. Empty sessionID or refID are silently ignored (defensive).

This increments the RefCount to prevent LRU eviction while the session is active. When the session ends, UnpinSession() will call Release() to decrement RefCount.

Thread-safe: Can be called concurrently from multiple goroutines.

func (*SessionReferenceTracker) Stats 

func (t *SessionReferenceTracker) Stats() TrackerStats

Stats returns tracker statistics for monitoring and observability. Provides insight into reference lifecycle and helps detect leaks.

Thread-safe: Can be called concurrently from multiple goroutines.

func (*SessionReferenceTracker) UnpinSession 

func (t *SessionReferenceTracker) UnpinSession(sessionID string) int

UnpinSession releases all references for a session (idempotent). Returns the number of references released. If session has no references or doesn't exist, returns 0.

This method calls Release() on the SharedMemoryStore for each reference, decrementing the RefCount. When RefCount reaches 0, the reference becomes eligible for LRU eviction or TTL-based cleanup.

Thread-safe: Can be called concurrently from multiple goroutines.

type SharedData 

type SharedData struct {
	ID          string
	Data        []byte
	Compressed  bool
	Size        int64
	Checksum    string
	ContentType string
	Metadata    map[string]string
	StoredAt    time.Time
	AccessedAt  time.Time
	RefCount    int32
	// contains filtered or unexported fields
}

SharedData represents a data chunk in shared memory.

type SharedMemoryStore 

type SharedMemoryStore struct {
	// contains filtered or unexported fields
}

SharedMemoryStore manages shared memory with LRU eviction.

func GetGlobalSharedMemory 

func GetGlobalSharedMemory(config *Config) *SharedMemoryStore

GetGlobalSharedMemory returns a singleton shared memory store with disk persistence. This ensures that tool result references work across agent instances and survive restarts.

The global store uses a two-tier architecture: 1. Hot data (frequently accessed) stays in memory (fast, up to MaxMemoryBytes) 2. Cold data (LRU evicted) overflows to disk (persistent, up to MaxDiskSize)

This design solves the reference-not-found problem where: - Multiple agent instances share the same reference store - References survive agent restarts (disk-backed) - References survive memory pressure (disk overflow)

Observability: The store automatically tracks hits, misses, evictions, and compressions. Use the Stats() method to retrieve metrics for monitoring.

func NewSharedMemoryStore 

func NewSharedMemoryStore(config *Config) *SharedMemoryStore

NewSharedMemoryStore creates a new shared memory store.

func (*SharedMemoryStore) Delete 

func (s *SharedMemoryStore) Delete(id string) error

Delete removes data from shared memory.

func (*SharedMemoryStore) Get 

func (s *SharedMemoryStore) Get(ref *loomv1.DataReference) ([]byte, error)

Get retrieves data from shared memory.

func (*SharedMemoryStore) GetMetadata 

func (s *SharedMemoryStore) GetMetadata(ref *loomv1.DataReference) (*DataMetadata, error)

GetMetadata returns metadata about stored data without loading full content. This enables progressive disclosure - agents can inspect data before retrieving.

func (*SharedMemoryStore) IncrementRefCount 

func (s *SharedMemoryStore) IncrementRefCount(id string)

IncrementRefCount increments the reference count for a data chunk. Used by SessionReferenceTracker to pin references and prevent eviction.

func (*SharedMemoryStore) Release 

func (s *SharedMemoryStore) Release(id string)

Release decrements the reference count for a data chunk.

func (*SharedMemoryStore) Stats 

func (s *SharedMemoryStore) Stats() Stats

Stats returns statistics about the shared memory store.

func (*SharedMemoryStore) Store 

func (s *SharedMemoryStore) Store(id string, data []byte, contentType string, metadata map[string]string) (*loomv1.DataReference, error)

Store stores data in shared memory.

type Stats 

type Stats struct {
	CurrentSize  int64
	MaxSize      int64
	ItemCount    int
	Hits         int64
	Misses       int64
	Evictions    int64
	Compressions int64
}

Stats holds statistics about the shared memory store.

func GetGlobalSharedMemoryStats 

func GetGlobalSharedMemoryStats() *Stats

GetGlobalSharedMemoryStats returns statistics from the global shared memory store. Returns nil if the global store hasn't been initialized yet. Useful for monitoring and debugging tool result storage.

type TrackerStats 

type TrackerStats struct {
	// SessionCount is the number of sessions with pinned references
	SessionCount int

	// TotalRefs is the total number of references across all sessions
	TotalRefs int
}

TrackerStats holds reference tracker statistics.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.