json

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 31, 2025 License: Apache-2.0 Imports: 18 Imported by: 0

README

🧊 Icebox JSON Catalog for Apache Iceberg

A feature-complete, local-first JSON-based catalog for Apache Iceberg for sandbox environments, embedded use cases, and rapid prototyping.

⚡ Built into Icebox
🪶 Inspired by boring-catalog. Extended for full fidelity and concurrent workloads.

🚀 Highlights

  • Thread-safe & atomic: Full concurrency protection using RW mutexes and atomic file writes
  • No dependencies: Pure Go, no external services, and zero setup required
  • Iceberg compliant: Fully aligns with the Iceberg v2 metadata specification
  • Optimistic concurrency: Uses ETags to prevent race conditions
  • Metrics & observability: Tracks catalog ops, cache behavior, and detailed errors
  • Type-complete: Handles all Iceberg types — primitive, decimal, list, map, struct, and more
  • Graceful fallbacks: Load from .icebox/index for embedded catalog autodiscovery

✨ Features

🏗️ Core Architecture
  • Atomic JSON file writes with temporary files and os.Rename
  • Safe for concurrent access via read/write mutexes
  • Schema validation and integrity checks on load
  • Clean separation of catalog, namespace, and table structures
📊 Observability
  • Built-in metrics: table/namespace creation, cache hits/misses, operation errors
  • Structured logging with context-aware messages
  • Optional TTL-based cache layer with read-through logic
💪 Iceberg Support
  • Fully supports the Iceberg v2 table spec
  • Field ID tracking for schema evolution
  • Accurate partition specs, sort orders, and snapshot log scaffolding
  • UUID v4 table generation via google/uuid
🔁 Developer UX
  • Supports registration of existing tables with known metadata
  • Resilient startup via .icebox/index fallback
  • Compatible with the iceberg-go interface for transparent drop-in usage

Documentation

Index

Constants

View Source 
const (
	DefaultCatalogName = "icebox"
	// File permissions for catalog files
	CatalogFilePermissions = 0644
	// Maximum retry attempts for concurrent operations
	MaxRetryAttempts = 5
	// Retry delay base (exponential backoff)
	RetryDelayBase = 100 * time.Millisecond
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Catalog 

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog implements the iceberg-go catalog.Catalog interface using JSON file storage

func NewCatalog 

func NewCatalog(cfg *config.Config) (*Catalog, error)

NewCatalog creates a new JSON-based catalog with enterprise-grade features

func (*Catalog) CatalogType 

func (c *Catalog) CatalogType() catalog.Type

CatalogType returns the catalog type

func (*Catalog) CheckNamespaceExists 

func (c *Catalog) CheckNamespaceExists(ctx context.Context, namespace table.Identifier) (bool, error)

CheckNamespaceExists checks if a namespace exists

func (*Catalog) CheckTableExists 

func (c *Catalog) CheckTableExists(ctx context.Context, identifier table.Identifier) (bool, error)

CheckTableExists checks if a table exists in the catalog

func (*Catalog) Close 

func (c *Catalog) Close() error

Close closes the catalog gracefully

func (*Catalog) CommitTable 

func (c *Catalog) CommitTable(ctx context.Context, tbl *table.Table, reqs []table.Requirement, updates []table.Update) (table.Metadata, string, error)

CommitTable commits table changes to the catalog with enterprise-grade validation and proper metadata versioning

func (*Catalog) CreateNamespace 

func (c *Catalog) CreateNamespace(ctx context.Context, namespace table.Identifier, props iceberg.Properties) error

CreateNamespace creates a new namespace in the catalog

func (*Catalog) CreateTable 

func (c *Catalog) CreateTable(ctx context.Context, identifier table.Identifier, schema *iceberg.Schema, opts ...catalog.CreateTableOpt) (*table.Table, error)

CreateTable creates a new table in the catalog with enhanced options support

func (*Catalog) CreateView 

func (c *Catalog) CreateView(ctx context.Context, identifier table.Identifier, sql string, dialect string, schema *iceberg.Schema, properties map[string]string) (*View, error)

CreateView creates a new view in the catalog

func (*Catalog) DropNamespace 

func (c *Catalog) DropNamespace(ctx context.Context, namespace table.Identifier) error

DropNamespace removes a namespace from the catalog

func (*Catalog) DropTable 

func (c *Catalog) DropTable(ctx context.Context, identifier table.Identifier) error

DropTable drops a table from the catalog

func (*Catalog) DropView 

func (c *Catalog) DropView(ctx context.Context, identifier table.Identifier) error

DropView drops a view from the catalog

func (*Catalog) GetMetrics 

func (c *Catalog) GetMetrics() map[string]int64

GetMetrics returns the current catalog operation metrics

func (*Catalog) ListNamespaces 

func (c *Catalog) ListNamespaces(ctx context.Context, parent table.Identifier) ([]table.Identifier, error)

ListNamespaces lists all namespaces or child namespaces

func (*Catalog) ListTables 

func (c *Catalog) ListTables(ctx context.Context, namespace table.Identifier) iter.Seq2[table.Identifier, error]

ListTables lists all tables in a namespace

func (*Catalog) ListViews 

func (c *Catalog) ListViews(ctx context.Context, namespace table.Identifier) iter.Seq2[table.Identifier, error]

ListViews lists all views in a namespace

func (*Catalog) LoadNamespaceProperties 

func (c *Catalog) LoadNamespaceProperties(ctx context.Context, namespace table.Identifier) (iceberg.Properties, error)

LoadNamespaceProperties loads properties for a namespace

func (*Catalog) LoadTable 

func (c *Catalog) LoadTable(ctx context.Context, identifier table.Identifier, props iceberg.Properties) (*table.Table, error)

LoadTable loads a table from the catalog

func (*Catalog) LoadView 

func (c *Catalog) LoadView(ctx context.Context, identifier table.Identifier) (*View, error)

LoadView loads a view from the catalog

func (*Catalog) Name 

func (c *Catalog) Name() string

Name returns the catalog name

func (*Catalog) NamespaceExists 

func (c *Catalog) NamespaceExists(ctx context.Context, namespace table.Identifier) (bool, error)

NamespaceExists checks if a namespace exists in the catalog

func (*Catalog) RegisterTable 

func (c *Catalog) RegisterTable(ctx context.Context, identifier table.Identifier, metadataLocation string) (*table.Table, error)

RegisterTable registers an existing table with the catalog

func (*Catalog) RenameTable 

func (c *Catalog) RenameTable(ctx context.Context, from, to table.Identifier) (*table.Table, error)

RenameTable renames a table in the catalog

func (*Catalog) RenameView 

func (c *Catalog) RenameView(ctx context.Context, from, to table.Identifier) (*View, error)

RenameView renames a view

func (*Catalog) TableExists 

func (c *Catalog) TableExists(ctx context.Context, identifier table.Identifier) (bool, error)

TableExists checks if a table exists in the catalog

func (*Catalog) UpdateNamespaceProperties 

func (c *Catalog) UpdateNamespaceProperties(ctx context.Context, namespace table.Identifier, removals []string, updates iceberg.Properties) (catalog.PropertiesUpdateSummary, error)

UpdateNamespaceProperties updates properties for a namespace with comprehensive tracking

func (*Catalog) ViewExists 

func (c *Catalog) ViewExists(ctx context.Context, identifier table.Identifier) (bool, error)

ViewExists checks if a view exists in the catalog

type CatalogData 

type CatalogData struct {
	CatalogName string                    `json:"catalog_name"`
	Namespaces  map[string]NamespaceEntry `json:"namespaces"`
	Tables      map[string]TableEntry     `json:"tables"`
	Views       map[string]ViewEntry      `json:"views"`      // Added views support
	Version     int                       `json:"version"`    // Schema version for future migrations
	CreatedAt   time.Time                 `json:"created_at"` // When catalog was created
	UpdatedAt   time.Time                 `json:"updated_at"` // Last update timestamp
}

CatalogData represents the JSON structure stored in catalog.json

type CatalogMetrics 

type CatalogMetrics struct {
	TablesCreated     int64
	TablesDropped     int64
	ViewsCreated      int64
	ViewsDropped      int64
	NamespacesCreated int64
	NamespacesDropped int64
	OperationErrors   int64
	CacheHits         int64
	CacheMisses       int64
	// contains filtered or unexported fields
}

CatalogMetrics tracks operation metrics for monitoring

func (*CatalogMetrics) GetStats 

func (m *CatalogMetrics) GetStats() map[string]int64

func (*CatalogMetrics) IncrementCacheHits 

func (m *CatalogMetrics) IncrementCacheHits()

func (*CatalogMetrics) IncrementCacheMisses 

func (m *CatalogMetrics) IncrementCacheMisses()

func (*CatalogMetrics) IncrementNamespacesCreated 

func (m *CatalogMetrics) IncrementNamespacesCreated()

func (*CatalogMetrics) IncrementNamespacesDropped 

func (m *CatalogMetrics) IncrementNamespacesDropped()

func (*CatalogMetrics) IncrementOperationErrors 

func (m *CatalogMetrics) IncrementOperationErrors()

func (*CatalogMetrics) IncrementTablesCreated 

func (m *CatalogMetrics) IncrementTablesCreated()

func (*CatalogMetrics) IncrementTablesDropped 

func (m *CatalogMetrics) IncrementTablesDropped()

func (*CatalogMetrics) IncrementViewsCreated 

func (m *CatalogMetrics) IncrementViewsCreated()

func (*CatalogMetrics) IncrementViewsDropped 

func (m *CatalogMetrics) IncrementViewsDropped()

type ConcurrentModificationError 

type ConcurrentModificationError struct {
	// contains filtered or unexported fields
}

ConcurrentModificationError represents a concurrent modification error

func (*ConcurrentModificationError) Error 

func (e *ConcurrentModificationError) Error() string

type CreateTableOpt 

type CreateTableOpt func(*CreateTableOptions)

CreateTableOpt is a function that modifies CreateTableOptions

func WithLocation 

func WithLocation(location string) CreateTableOpt

WithLocation sets the table location

func WithPartitionSpec 

func WithPartitionSpec(spec *iceberg.PartitionSpec) CreateTableOpt

WithPartitionSpec sets the partition specification for the table

func WithProperties 

func WithProperties(properties iceberg.Properties) CreateTableOpt

WithProperties sets the table properties

type CreateTableOptions 

type CreateTableOptions struct {
	PartitionSpec *iceberg.PartitionSpec
	Properties    iceberg.Properties
	Location      string
}

CreateTableOptions represents options for creating a table

type IndexConfig 

type IndexConfig struct {
	CatalogName string                 `json:"catalog_name"`
	CatalogURI  string                 `json:"catalog_uri"`
	Properties  map[string]interface{} `json:"properties"`
}

IndexConfig represents the configuration stored in .icebox/index

type NamespaceEntry 

type NamespaceEntry struct {
	Properties iceberg.Properties `json:"properties"`
	CreatedAt  time.Time          `json:"created_at"`
	UpdatedAt  time.Time          `json:"updated_at"`
}

NamespaceEntry represents a namespace in the catalog

type TableEntry 

type TableEntry struct {
	Namespace                string    `json:"namespace"`
	Name                     string    `json:"name"`
	MetadataLocation         string    `json:"metadata_location"`
	PreviousMetadataLocation *string   `json:"previous_metadata_location,omitempty"`
	CreatedAt                time.Time `json:"created_at"`
	UpdatedAt                time.Time `json:"updated_at"`
}

TableEntry represents a table in the catalog

type ValidationError 

type ValidationError struct {
	Field   string
	Message string
}

ValidationError represents a validation error

func (*ValidationError) Error 

func (e *ValidationError) Error() string

type View 

type View struct {
	// contains filtered or unexported fields
}

View represents a view interface for compatibility

func (*View) Dialect 

func (v *View) Dialect() string

Dialect returns the SQL dialect of the current version

func (*View) Identifier 

func (v *View) Identifier() table.Identifier

Identifier returns the view identifier

func (*View) Metadata 

func (v *View) Metadata() *ViewMetadata

Metadata returns the view metadata

func (*View) SQL 

func (v *View) SQL() string

SQL returns the SQL representation of the current version

func (*View) Schema 

func (v *View) Schema() *ViewSchema

Schema returns the current schema of the view

type ViewEntry 

type ViewEntry struct {
	Namespace        string    `json:"namespace"`
	Name             string    `json:"name"`
	MetadataLocation string    `json:"metadata_location"`
	CreatedAt        time.Time `json:"created_at"`
	UpdatedAt        time.Time `json:"updated_at"`
}

ViewEntry represents a view in the catalog according to Iceberg V2 spec

type ViewMetadata 

type ViewMetadata struct {
	ViewUUID         string                `json:"view-uuid"`
	FormatVersion    int                   `json:"format-version"`
	Location         string                `json:"location"`
	Schemas          []ViewSchema          `json:"schemas"`
	CurrentVersionID int                   `json:"current-version-id"`
	Versions         []ViewVersion         `json:"versions"`
	VersionLog       []ViewVersionLogEntry `json:"version-log"`
	Properties       map[string]string     `json:"properties,omitempty"`
}

ViewMetadata represents the complete view metadata according to Iceberg V2 spec

type ViewRepresentation 

type ViewRepresentation struct {
	Type    string `json:"type"`
	SQL     string `json:"sql"`
	Dialect string `json:"dialect"`
}

ViewRepresentation represents a view representation (SQL, etc.)

type ViewSchema 

type ViewSchema struct {
	SchemaID int               `json:"schema-id"`
	Type     string            `json:"type"`
	Fields   []ViewSchemaField `json:"fields"`
}

ViewSchema represents a view schema

type ViewSchemaField 

type ViewSchemaField struct {
	ID       int    `json:"id"`
	Name     string `json:"name"`
	Required bool   `json:"required"`
	Type     string `json:"type"`
	Doc      string `json:"doc,omitempty"`
}

ViewSchemaField represents a field in a view schema

type ViewVersion 

type ViewVersion struct {
	VersionID        int                  `json:"version-id"`
	SchemaID         int                  `json:"schema-id"`
	TimestampMs      int64                `json:"timestamp-ms"`
	Summary          map[string]string    `json:"summary"`
	Representations  []ViewRepresentation `json:"representations"`
	DefaultCatalog   *string              `json:"default-catalog,omitempty"`
	DefaultNamespace []string             `json:"default-namespace"`
}

ViewVersion represents a version of a view

type ViewVersionLogEntry 

type ViewVersionLogEntry struct {
	TimestampMs int64 `json:"timestamp-ms"`
	VersionID   int   `json:"version-id"`
}

ViewVersionLogEntry represents an entry in the view version log

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.