storage

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 12, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Copyright 2026 Teradata

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const (
	// DefaultMaxMemoryBytes is 1GB
	DefaultMaxMemoryBytes = 1 * 1024 * 1024 * 1024
	// DefaultSharedMemoryThreshold is 16384 bytes — the one threshold value of
	// HLD §5.1, serving three roles: compile-time offload bound, persist-time
	// row bound, retrieval page bound. Per-agent override via
	// MemoryConfig.SharedMemoryThresholdBytes (cloud binds its
	// query_tool_result_threshold_bytes config through SetSharedMemoryThreshold).
	DefaultSharedMemoryThreshold = 16384
	// DefaultCompressionThreshold is 1MB
	DefaultCompressionThreshold = 1 * 1024 * 1024
	// DefaultTTLSeconds is 1 hour
	DefaultTTLSeconds = 3600
)
View Source 
const (
	// DefaultMaxDiskBytes is 10GB
	DefaultMaxDiskBytes = 10 * 1024 * 1024 * 1024
)
View Source 
const (
	// MaxPreviewChars limits progressive disclosure preview size
	// 20,000 chars ≈ 5K tokens (4 bytes/token estimate)
	// Large enough for full workflow examples (~20KB) with minimal headroom
	// Patterns are injected separately so don't need larger limits
	MaxPreviewChars = 20000
)

Variables

This section is empty.

Functions

func GenerateID 

func GenerateID() string

GenerateID generates a unique ID for a data reference.

func GetDefaultLoomDBPath 

func GetDefaultLoomDBPath() string

GetDefaultLoomDBPath returns the default database path in the loom data directory (respects LOOM_DATA_DIR).

func IsLargeData 

func IsLargeData(data []byte, threshold int64) bool

IsLargeData checks if data exceeds the threshold for shared memory.

func RefToString 

func RefToString(ref *loomv1.DataReference) string

RefToString converts a DataReference to a human-readable string.

func ResetGlobalSharedMemory 

func ResetGlobalSharedMemory()

ResetGlobalSharedMemory resets the global singleton (for testing only). This should NOT be used in production code.

func ShouldUseSharedMemory 

func ShouldUseSharedMemory(dataSize int64, threshold int64) bool

ShouldUseSharedMemory determines if data should go to shared memory.

func ValidateReference 

func ValidateReference(ref *loomv1.DataReference) error

ValidateReference validates a data reference.

Types

type Config 

type Config struct {
	MaxMemoryBytes       int64
	CompressionThreshold int64
	TTLSeconds           int64
	OverflowHandler      OverflowHandler
	Logger               *zap.Logger
}

Config configures the shared memory store.

type DataMetadata 

type DataMetadata struct {
	ID              string
	ContentType     string // "application/json", "text/csv", "text/plain"
	DataType        string // "json_array", "json_object", "csv", "text"
	SizeBytes       int64
	EstimatedTokens int64
	Schema          *SchemaInfo
	Preview         *PreviewData
	CreatedAt       time.Time
	ExpiresAt       time.Time
	Location        loomv1.StorageLocation
}

DataMetadata contains metadata about stored data for progressive disclosure. This enables agents to inspect data structure and size before retrieving full content.

type DiskMetadata 

type DiskMetadata struct {
	ID          string
	FilePath    string
	Size        int64
	StoredAt    time.Time
	AccessedAt  time.Time
	Checksum    string
	ContentType string
	Compressed  bool
}

DiskMetadata tracks metadata for disk-stored data.

type DiskOverflowConfig 

type DiskOverflowConfig struct {
	CachePath   string
	MaxDiskSize int64
	TTLSeconds  int64
}

DiskOverflowConfig configures the disk overflow manager.

type DiskOverflowManager 

type DiskOverflowManager struct {
	// contains filtered or unexported fields
}

DiskOverflowManager manages overflow data on disk.

func NewDiskOverflowManager 

func NewDiskOverflowManager(config *DiskOverflowConfig) (*DiskOverflowManager, error)

NewDiskOverflowManager creates a new disk overflow manager.

func (*DiskOverflowManager) Delete 

func (d *DiskOverflowManager) Delete(id string) error

Delete removes data from disk.

func (*DiskOverflowManager) Promote 

func (d *DiskOverflowManager) Promote(id string) ([]byte, error)

Promote retrieves data from disk and removes it (for promoting back to memory).

func (*DiskOverflowManager) Retrieve 

func (d *DiskOverflowManager) Retrieve(id string) ([]byte, error)

Retrieve retrieves data from disk.

func (*DiskOverflowManager) Stats 

func (d *DiskOverflowManager) Stats() DiskStats

Stats returns statistics about the disk overflow manager.

func (*DiskOverflowManager) Store 

func (d *DiskOverflowManager) Store(id string, data []byte, ref *loomv1.DataReference) error

Store stores data to disk.

type DiskStats 

type DiskStats struct {
	CurrentSize int64
	MaxSize     int64
	ItemCount   int
}

DiskStats holds statistics about the disk overflow manager.

type FieldInfo 

type FieldInfo struct {
	Name string
	Type string // "string", "number", "boolean", "object", "array", "null"
}

FieldInfo describes a field in JSON data.

type OverflowHandler 

type OverflowHandler interface {
	Store(id string, data []byte, metadata *loomv1.DataReference) error
	Retrieve(id string) ([]byte, error)
	Delete(id string) error
}

OverflowHandler handles data that doesn't fit in memory.

type PreviewData 

type PreviewData struct {
	First5 []any
	Last5  []any
}

PreviewData contains sample data for quick inspection.

type Purger added in v1.2.0 

type Purger interface {
	PurgeDeleted(ctx context.Context, graceInterval string) error
}

Purger is a local interface for the PurgeDeleted method. This avoids importing pkg/agent (which would cause an import cycle). Any type implementing agent.SoftDeleteStorage satisfies this interface.

type SchemaInfo 

type SchemaInfo struct {
	Type       string      // "array", "object", "table"
	ItemCount  int64       // Number of items in array or rows in table
	Fields     []FieldInfo // For JSON objects
	Columns    []string    // For CSV/tabular data
	SampleItem any         // Representative sample
}

SchemaInfo describes the structure of the data.

type SharedData 

type SharedData struct {
	ID          string
	Data        []byte
	Compressed  bool
	Size        int64
	Checksum    string
	ContentType string
	Metadata    map[string]string
	StoredAt    time.Time
	AccessedAt  time.Time
	RefCount    int32
	// contains filtered or unexported fields
}

SharedData represents a data chunk in shared memory.

type SharedMemoryStore 

type SharedMemoryStore struct {
	// contains filtered or unexported fields
}

SharedMemoryStore manages shared memory with LRU eviction.

func GetGlobalSharedMemory 

func GetGlobalSharedMemory(config *Config) *SharedMemoryStore

GetGlobalSharedMemory returns a singleton shared memory store with disk persistence. This ensures that tool result references work across agent instances and survive restarts.

The global store uses a two-tier architecture: 1. Hot data (frequently accessed) stays in memory (fast, up to MaxMemoryBytes) 2. Cold data (LRU evicted) overflows to disk (persistent, up to MaxDiskSize)

This design solves the reference-not-found problem where: - Multiple agent instances share the same reference store - References survive agent restarts (disk-backed) - References survive memory pressure (disk overflow)

Observability: The store automatically tracks hits, misses, evictions, and compressions. Use the Stats() method to retrieve metrics for monitoring.

func NewSharedMemoryStore 

func NewSharedMemoryStore(config *Config) *SharedMemoryStore

NewSharedMemoryStore creates a new shared memory store.

func (*SharedMemoryStore) Delete 

func (s *SharedMemoryStore) Delete(id string) error

Delete removes data from shared memory.

func (*SharedMemoryStore) Get 

func (s *SharedMemoryStore) Get(ref *loomv1.DataReference, callerSession string) ([]byte, error)

Get retrieves data from shared memory for the caller's partition. A handle owned by another session resolves to not-found (fail-closed): callerSession must equal the sessionID the record was stored under.

func (*SharedMemoryStore) GetMetadata 

func (s *SharedMemoryStore) GetMetadata(ref *loomv1.DataReference, callerSession string) (*DataMetadata, error)

GetMetadata returns metadata about stored data without loading full content. This enables progressive disclosure - agents can inspect data before retrieving. callerSession scopes the lookup to the caller's partition, same rule as Get.

func (*SharedMemoryStore) IncrementRefCount 

func (s *SharedMemoryStore) IncrementRefCount(id string)

IncrementRefCount increments the reference count for a data chunk, pinning it against eviction.

func (*SharedMemoryStore) Release 

func (s *SharedMemoryStore) Release(id string)

Release decrements the reference count for a data chunk.

func (*SharedMemoryStore) Stats 

func (s *SharedMemoryStore) Stats() Stats

Stats returns statistics about the shared memory store.

func (*SharedMemoryStore) Store 

func (s *SharedMemoryStore) Store(id string, data []byte, contentType string, metadata map[string]string, sessionID string) (*loomv1.DataReference, error)

Store stores data in shared memory owned by sessionID. The owning session is the partition key: only a Get carrying the same sessionID resolves the handle.

type SoftDeleteCleaner added in v1.2.0 

type SoftDeleteCleaner struct {
	// contains filtered or unexported fields
}

SoftDeleteCleaner manages the background goroutine that periodically purges soft-deleted records. Call Stop to cancel the goroutine and wait for it to exit.

func StartSoftDeleteCleanup added in v1.2.0 

func StartSoftDeleteCleanup(
	store Purger,
	gracePeriodSeconds int32,
	cleanupIntervalSeconds int32,
	logger *zap.Logger,
) *SoftDeleteCleaner

StartSoftDeleteCleanup starts a background goroutine that periodically purges soft-deleted records older than the grace period. It calls PurgeDeleted on the provided Purger at the configured interval.

Call Stop on the returned SoftDeleteCleaner during shutdown to cancel the goroutine and wait for it to exit cleanly.

func (*SoftDeleteCleaner) Stop added in v1.2.0 

func (c *SoftDeleteCleaner) Stop()

Stop cancels the cleanup goroutine and blocks until it has exited.

type Stats 

type Stats struct {
	CurrentSize  int64
	MaxSize      int64
	ItemCount    int
	Hits         int64
	Misses       int64
	Evictions    int64
	Compressions int64
}

Stats holds statistics about the shared memory store.

func GetGlobalSharedMemoryStats 

func GetGlobalSharedMemoryStats() *Stats

GetGlobalSharedMemoryStats returns statistics from the global shared memory store. Returns nil if the global store hasn't been initialized yet. Useful for monitoring and debugging tool result storage.

Source Files

View all Source files

Directories

Path Synopsis
Package backend defines the StorageBackend composite interface and factory.
 Package backend defines the StorageBackend composite interface and factory.
Package sqlite provides utility functions for SQLite database operations such as online backup and integrity verification.
 Package sqlite provides utility functions for SQLite database operations such as online backup and integrity verification.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.