storage

package
v1.4.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 22, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package storage provides a pluggable storage interface for the LUX indexer. Supported backends: PostgreSQL, MySQL, SQLite, MongoDB, ZapDB, Dgraph

Package storage provides unified storage with two layers:

  • KV Layer: Fast key-value storage using github.com/luxfi/database
  • Query Layer: SQL/Graph queries for indexed data

Default backend is SQLite. Use build tags for other backends: 

go build -tags postgres
go build -tags mysql
go build -tags mongo
go build -tags dgraph

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound      = fmt.Errorf("not found")
	ErrAlreadyExists = fmt.Errorf("already exists")
	ErrNotSupported  = fmt.Errorf("not supported by this backend")
	ErrClosed        = fmt.Errorf("store is closed")
)

Errors

Functions

This section is empty.

Types

type Backend 

type Backend string

Backend identifies the storage backend type 

const (
	BackendPostgres Backend = "postgres"
	BackendMySQL    Backend = "mysql"
	BackendSQLite   Backend = "sqlite"
	BackendMongo    Backend = "mongo"
	BackendBadger   Backend = "badger"
	BackendDgraph   Backend = "dgraph"
)

func ParseBackend 

func ParseBackend(s string) (Backend, error)

ParseBackend parses a backend string

type Block 

type Block struct {
	ID        string          `json:"id"`
	ParentID  string          `json:"parent_id"`
	Height    uint64          `json:"height"`
	Timestamp time.Time       `json:"timestamp"`
	Status    string          `json:"status"`
	TxCount   int             `json:"tx_count"`
	TxIDs     []string        `json:"tx_ids"`
	Data      json.RawMessage `json:"data"`
	Metadata  json.RawMessage `json:"metadata"`
	CreatedAt time.Time       `json:"created_at"`
}

Block represents a linear chain block for storage

type Column 

type Column struct {
	Name     string
	Type     ColumnType
	Nullable bool
	Default  string
	Primary  bool
}

Column defines a table column

type ColumnType 

type ColumnType string

ColumnType represents a column data type 

const (
	TypeText      ColumnType = "text"
	TypeInt       ColumnType = "int"
	TypeBigInt    ColumnType = "bigint"
	TypeFloat     ColumnType = "float"
	TypeBool      ColumnType = "bool"
	TypeTimestamp ColumnType = "timestamp"
	TypeJSON      ColumnType = "json"
	TypeBytes     ColumnType = "bytes"
)

type Config 

type Config struct {
	Backend  Backend
	URL      string            // Connection URL (postgres://, mysql://, mongo://, etc.)
	Database string            // Database name (for MongoDB)
	Options  map[string]string // Backend-specific options
	DataDir  string            // For file-based backends (ZapDB, SQLite)
}

Config for storage backend

type Edge 

type Edge struct {
	Source    string    `json:"source"`
	Target    string    `json:"target"`
	Type      string    `json:"type"`
	CreatedAt time.Time `json:"created_at"`
}

Edge represents a DAG edge for storage

type Index 

type Index struct {
	Name    string
	Table   string
	Columns []string
	Unique  bool
}

Index defines a database index

type KV 

type KV struct {
	Key   string
	Value []byte
}

KV represents a key-value pair

type Schema 

type Schema struct {
	Name    string
	Tables  []Table
	Indexes []Index
}

Schema defines the database schema

type Store 

type Store interface {
	// Backend returns the query backend type (sqlite, postgres, etc.)
	Backend() Backend

	// Lifecycle
	Init(ctx context.Context) error
	Close() error
	Ping(ctx context.Context) error

	// Schema management
	InitSchema(ctx context.Context, schema Schema) error
	MigrateSchema(ctx context.Context, schema Schema) error

	// Block/Vertex storage (for chain/dag indexers)
	StoreBlock(ctx context.Context, table string, block *Block) error
	GetBlock(ctx context.Context, table string, id string) (*Block, error)
	GetBlockByHeight(ctx context.Context, table string, height uint64) (*Block, error)
	GetRecentBlocks(ctx context.Context, table string, limit int) ([]*Block, error)

	StoreVertex(ctx context.Context, table string, vertex *Vertex) error
	GetVertex(ctx context.Context, table string, id string) (*Vertex, error)
	GetRecentVertices(ctx context.Context, table string, limit int) ([]*Vertex, error)

	// Edge storage (for DAG)
	StoreEdge(ctx context.Context, table string, edge *Edge) error
	GetEdges(ctx context.Context, table string, vertexID string) ([]*Edge, error)

	// Generic key-value storage
	Put(ctx context.Context, table string, key string, value []byte) error
	Get(ctx context.Context, table string, key string) ([]byte, error)
	Delete(ctx context.Context, table string, key string) error
	List(ctx context.Context, table string, prefix string, limit int) ([]KV, error)

	// Stats
	UpdateStats(ctx context.Context, table string, stats map[string]interface{}) error
	GetStats(ctx context.Context, table string) (map[string]interface{}, error)
	Count(ctx context.Context, table string, where string, args ...interface{}) (int64, error)

	// Queries
	Query(ctx context.Context, query string, args ...interface{}) ([]map[string]interface{}, error)
	Exec(ctx context.Context, query string, args ...interface{}) error

	// Transaction support (optional, may return ErrNotSupported)
	Begin(ctx context.Context) (Transaction, error)
}

Store is the main storage interface for indexed data

func New 

func New(cfg Config) (Store, error)

New creates a new unified storage backend based on config The unified storage uses a KV layer (ZapDB) + Query layer (SQLite/PostgreSQL)

type Table 

type Table struct {
	Name    string
	Columns []Column
}

Table defines a database table

type Transaction 

type Transaction interface {
	Store
	Commit() error
	Rollback() error
}

Transaction represents a storage transaction

type Unified added in v0.3.0 

type Unified struct {
	// contains filtered or unexported fields
}

Unified combines KV and Query layers into a single interface

func NewUnified added in v0.3.0 

func NewUnified(cfg UnifiedConfig) (*Unified, error)

NewUnified creates a new unified storage

func NewUnifiedWithDB added in v0.3.0 

func NewUnifiedWithDB(db database.Database, queryCfg query.Config) (*Unified, error)

NewUnifiedWithDB creates unified storage using an existing database This enables in-process mode where the indexer shares the node's database

func (*Unified) Backend added in v0.3.0 

func (u *Unified) Backend() Backend

Backend returns the query backend type

func (*Unified) Begin added in v0.3.0 

func (u *Unified) Begin(ctx context.Context) (Transaction, error)

Begin starts a transaction that satisfies the Store interface

func (*Unified) BeginStore added in v0.3.0 

func (u *Unified) BeginStore(ctx context.Context) (Transaction, error)

BeginStore begins a transaction that implements the Store interface Deprecated: Use Begin instead

func (*Unified) BeginTx added in v0.3.0 

func (u *Unified) BeginTx(ctx context.Context) (*UnifiedTx, error)

BeginTx starts a transaction returning the typed UnifiedTx

func (*Unified) Close added in v0.3.0 

func (u *Unified) Close() error

Close closes both layers

func (*Unified) Count added in v0.3.0 

func (u *Unified) Count(ctx context.Context, table string, where string, args ...interface{}) (int64, error)

Count counts records

func (*Unified) Database added in v0.3.0 

func (u *Unified) Database() database.Database

Database returns the underlying luxfi/database.Database

func (*Unified) Delete added in v0.3.0 

func (u *Unified) Delete(ctx context.Context, table string, key string) error

Delete removes a key from the KV layer

func (*Unified) Exec added in v0.3.0 

func (u *Unified) Exec(ctx context.Context, q string, args ...interface{}) error

func (*Unified) ExecSQL added in v0.3.0 

func (u *Unified) ExecSQL(ctx context.Context, q string, args ...interface{}) error

Exec executes a SQL command

func (*Unified) Get added in v0.3.0 

func (u *Unified) Get(ctx context.Context, table string, key string) ([]byte, error)

Get retrieves a raw key-value from the KV layer

func (*Unified) GetBlock added in v0.3.0 

func (u *Unified) GetBlock(ctx context.Context, table string, id string) (*Block, error)

GetBlock retrieves a block (tries KV first, falls back to Query)

func (*Unified) GetBlockByHeight added in v0.3.0 

func (u *Unified) GetBlockByHeight(ctx context.Context, table string, height uint64) (*Block, error)

GetBlockByHeight retrieves a block by height

func (*Unified) GetEdges added in v0.3.0 

func (u *Unified) GetEdges(ctx context.Context, table string, vertexID string) ([]*Edge, error)

GetEdges retrieves edges from a vertex

func (*Unified) GetRecentBlocks added in v0.3.0 

func (u *Unified) GetRecentBlocks(ctx context.Context, table string, limit int) ([]*Block, error)

GetRecentBlocks retrieves recent blocks (query layer only)

func (*Unified) GetRecentVertices added in v0.3.0 

func (u *Unified) GetRecentVertices(ctx context.Context, table string, limit int) ([]*Vertex, error)

GetRecentVertices retrieves recent vertices

func (*Unified) GetStats added in v0.3.0 

func (u *Unified) GetStats(ctx context.Context, table string) (map[string]interface{}, error)

GetStats retrieves stats from the KV layer

func (*Unified) GetVertex added in v0.3.0 

func (u *Unified) GetVertex(ctx context.Context, table string, id string) (*Vertex, error)

GetVertex retrieves a vertex

func (*Unified) Init added in v0.3.0 

func (u *Unified) Init(ctx context.Context) error

Init initializes both layers

func (*Unified) InitSchema added in v0.3.0 

func (u *Unified) InitSchema(ctx context.Context, schema Schema) error

func (*Unified) KV added in v0.3.0 

func (u *Unified) KV() *kv.Store

KV returns the KV layer for direct access

func (*Unified) List added in v0.3.0 

func (u *Unified) List(ctx context.Context, table string, prefix string, limit int) ([]KV, error)

List lists keys from the KV layer

func (*Unified) MigrateSchema added in v0.3.0 

func (u *Unified) MigrateSchema(ctx context.Context, schema Schema) error

func (*Unified) Ping added in v0.3.0 

func (u *Unified) Ping(ctx context.Context) error

Ping checks both layers

func (*Unified) Put added in v0.3.0 

func (u *Unified) Put(ctx context.Context, table string, key string, value []byte) error

Put stores a raw key-value in the KV layer

func (*Unified) Query added in v0.3.0 

func (u *Unified) Query(ctx context.Context, q string, args ...interface{}) ([]map[string]interface{}, error)

func (*Unified) QueryEngine added in v0.3.0 

func (u *Unified) QueryEngine() query.Engine

QueryEngine returns the Query layer for direct access

func (*Unified) QuerySQL added in v0.3.0 

func (u *Unified) QuerySQL(ctx context.Context, q string, args ...interface{}) ([]map[string]interface{}, error)

Query runs a SQL/Graph query

func (*Unified) StoreBlock added in v0.3.0 

func (u *Unified) StoreBlock(ctx context.Context, table string, block *Block) error

StoreBlock stores a block in both layers

func (*Unified) StoreEdge added in v0.3.0 

func (u *Unified) StoreEdge(ctx context.Context, table string, edge *Edge) error

StoreEdge stores an edge

func (*Unified) StoreVertex added in v0.3.0 

func (u *Unified) StoreVertex(ctx context.Context, table string, vertex *Vertex) error

StoreVertex stores a vertex in both layers

func (*Unified) UpdateStats added in v0.3.0 

func (u *Unified) UpdateStats(ctx context.Context, table string, stats map[string]interface{}) error

UpdateStats updates stats in the KV layer

type UnifiedConfig added in v0.3.0 

type UnifiedConfig struct {
	// KV layer config
	KV kv.Config

	// Query layer config
	Query query.Config

	// Enable dual-write to both layers
	DualWrite bool
}

UnifiedConfig configures the unified storage

func DefaultUnifiedConfig added in v0.3.0 

func DefaultUnifiedConfig(dataDir string) UnifiedConfig

DefaultUnifiedConfig returns the default configuration If DATABASE_URL environment variable is set, uses PostgreSQL backend

func InProcessConfig added in v0.3.0 

func InProcessConfig(nodeDB database.Database, dataDir string) UnifiedConfig

InProcessConfig returns config for in-process mode with a node database

type UnifiedTx added in v0.3.0 

type UnifiedTx struct {
	// contains filtered or unexported fields
}

UnifiedTx is a transaction over the unified store

func (*UnifiedTx) Commit added in v0.3.0 

func (t *UnifiedTx) Commit() error

func (*UnifiedTx) Rollback added in v0.3.0 

func (t *UnifiedTx) Rollback() error

func (*UnifiedTx) StoreBlock added in v0.3.0 

func (t *UnifiedTx) StoreBlock(ctx context.Context, table string, block *Block) error

func (*UnifiedTx) StoreVertex added in v0.3.0 

func (t *UnifiedTx) StoreVertex(ctx context.Context, table string, vertex *Vertex) error

type Vertex 

type Vertex struct {
	ID        string          `json:"id"`
	Type      string          `json:"type"`
	ParentIDs []string        `json:"parent_ids"`
	Height    uint64          `json:"height"`
	Epoch     uint32          `json:"epoch"`
	TxIDs     []string        `json:"tx_ids"`
	Timestamp time.Time       `json:"timestamp"`
	Status    string          `json:"status"`
	Data      json.RawMessage `json:"data"`
	Metadata  json.RawMessage `json:"metadata"`
	CreatedAt time.Time       `json:"created_at"`
}

Vertex represents a DAG vertex for storage

Source Files

View all Source files

Directories

Path Synopsis
Package kv provides a unified key-value storage layer using github.com/luxfi/database.
 Package kv provides a unified key-value storage layer using github.com/luxfi/database.
Package query provides a query layer for indexed data.
 Package query provides a query layer for indexed data.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.