Help us shape the future of Go! Take the 2026 Go Developer Survey

json

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 31, 2025 License: Apache-2.0 Imports: 18 Imported by: 0

README ΒΆ

🧊 Icebox JSON Catalog for Apache Iceberg

A feature-complete, local-first JSON-based catalog for Apache Iceberg for sandbox environments, embedded use cases, and rapid prototyping.

⚑ Built into Icebox
πŸͺΆ Inspired by boring-catalog. Extended for full fidelity and concurrent workloads.


πŸš€ Highlights

  • Thread-safe & atomic: Full concurrency protection using RW mutexes and atomic file writes
  • No dependencies: Pure Go, no external services, and zero setup required
  • Iceberg compliant: Fully aligns with the Iceberg v2 metadata specification
  • Optimistic concurrency: Uses ETags to prevent race conditions
  • Metrics & observability: Tracks catalog ops, cache behavior, and detailed errors
  • Type-complete: Handles all Iceberg types β€” primitive, decimal, list, map, struct, and more
  • Graceful fallbacks: Load from .icebox/index for embedded catalog autodiscovery

✨ Features

πŸ—οΈ Core Architecture
  • Atomic JSON file writes with temporary files and os.Rename
  • Safe for concurrent access via read/write mutexes
  • Schema validation and integrity checks on load
  • Clean separation of catalog, namespace, and table structures
πŸ“Š Observability
  • Built-in metrics: table/namespace creation, cache hits/misses, operation errors
  • Structured logging with context-aware messages
  • Optional TTL-based cache layer with read-through logic
πŸ’ͺ Iceberg Support
  • Fully supports the Iceberg v2 table spec
  • Field ID tracking for schema evolution
  • Accurate partition specs, sort orders, and snapshot log scaffolding
  • UUID v4 table generation via google/uuid
πŸ” Developer UX
  • Supports registration of existing tables with known metadata
  • Resilient startup via .icebox/index fallback
  • Compatible with the iceberg-go interface for transparent drop-in usage

Documentation ΒΆ

Index ΒΆ

Constants ΒΆ

View Source
const (
	DefaultCatalogName = "icebox"
	// File permissions for catalog files
	CatalogFilePermissions = 0644
	// Maximum retry attempts for concurrent operations
	MaxRetryAttempts = 5
	// Retry delay base (exponential backoff)
	RetryDelayBase = 100 * time.Millisecond
)

Variables ΒΆ

This section is empty.

Functions ΒΆ

This section is empty.

Types ΒΆ

type Catalog ΒΆ

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog implements the iceberg-go catalog.Catalog interface using JSON file storage

func NewCatalog ΒΆ

func NewCatalog(cfg *config.Config) (*Catalog, error)

NewCatalog creates a new JSON-based catalog with enterprise-grade features

func (*Catalog) CatalogType ΒΆ

func (c *Catalog) CatalogType() catalog.Type

CatalogType returns the catalog type

func (*Catalog) CheckNamespaceExists ΒΆ

func (c *Catalog) CheckNamespaceExists(ctx context.Context, namespace table.Identifier) (bool, error)

CheckNamespaceExists checks if a namespace exists

func (*Catalog) CheckTableExists ΒΆ

func (c *Catalog) CheckTableExists(ctx context.Context, identifier table.Identifier) (bool, error)

CheckTableExists checks if a table exists in the catalog

func (*Catalog) Close ΒΆ

func (c *Catalog) Close() error

Close closes the catalog gracefully

func (*Catalog) CommitTable ΒΆ

func (c *Catalog) CommitTable(ctx context.Context, tbl *table.Table, reqs []table.Requirement, updates []table.Update) (table.Metadata, string, error)

CommitTable commits table changes to the catalog with enterprise-grade validation and proper metadata versioning

func (*Catalog) CreateNamespace ΒΆ

func (c *Catalog) CreateNamespace(ctx context.Context, namespace table.Identifier, props iceberg.Properties) error

CreateNamespace creates a new namespace in the catalog

func (*Catalog) CreateTable ΒΆ

func (c *Catalog) CreateTable(ctx context.Context, identifier table.Identifier, schema *iceberg.Schema, opts ...catalog.CreateTableOpt) (*table.Table, error)

CreateTable creates a new table in the catalog with enhanced options support

func (*Catalog) CreateView ΒΆ

func (c *Catalog) CreateView(ctx context.Context, identifier table.Identifier, sql string, dialect string, schema *iceberg.Schema, properties map[string]string) (*View, error)

CreateView creates a new view in the catalog

func (*Catalog) DropNamespace ΒΆ

func (c *Catalog) DropNamespace(ctx context.Context, namespace table.Identifier) error

DropNamespace removes a namespace from the catalog

func (*Catalog) DropTable ΒΆ

func (c *Catalog) DropTable(ctx context.Context, identifier table.Identifier) error

DropTable drops a table from the catalog

func (*Catalog) DropView ΒΆ

func (c *Catalog) DropView(ctx context.Context, identifier table.Identifier) error

DropView drops a view from the catalog

func (*Catalog) GetMetrics ΒΆ

func (c *Catalog) GetMetrics() map[string]int64

GetMetrics returns the current catalog operation metrics

func (*Catalog) ListNamespaces ΒΆ

func (c *Catalog) ListNamespaces(ctx context.Context, parent table.Identifier) ([]table.Identifier, error)

ListNamespaces lists all namespaces or child namespaces

func (*Catalog) ListTables ΒΆ

func (c *Catalog) ListTables(ctx context.Context, namespace table.Identifier) iter.Seq2[table.Identifier, error]

ListTables lists all tables in a namespace

func (*Catalog) ListViews ΒΆ

func (c *Catalog) ListViews(ctx context.Context, namespace table.Identifier) iter.Seq2[table.Identifier, error]

ListViews lists all views in a namespace

func (*Catalog) LoadNamespaceProperties ΒΆ

func (c *Catalog) LoadNamespaceProperties(ctx context.Context, namespace table.Identifier) (iceberg.Properties, error)

LoadNamespaceProperties loads properties for a namespace

func (*Catalog) LoadTable ΒΆ

func (c *Catalog) LoadTable(ctx context.Context, identifier table.Identifier, props iceberg.Properties) (*table.Table, error)

LoadTable loads a table from the catalog

func (*Catalog) LoadView ΒΆ

func (c *Catalog) LoadView(ctx context.Context, identifier table.Identifier) (*View, error)

LoadView loads a view from the catalog

func (*Catalog) Name ΒΆ

func (c *Catalog) Name() string

Name returns the catalog name

func (*Catalog) NamespaceExists ΒΆ

func (c *Catalog) NamespaceExists(ctx context.Context, namespace table.Identifier) (bool, error)

NamespaceExists checks if a namespace exists in the catalog

func (*Catalog) RegisterTable ΒΆ

func (c *Catalog) RegisterTable(ctx context.Context, identifier table.Identifier, metadataLocation string) (*table.Table, error)

RegisterTable registers an existing table with the catalog

func (*Catalog) RenameTable ΒΆ

func (c *Catalog) RenameTable(ctx context.Context, from, to table.Identifier) (*table.Table, error)

RenameTable renames a table in the catalog

func (*Catalog) RenameView ΒΆ

func (c *Catalog) RenameView(ctx context.Context, from, to table.Identifier) (*View, error)

RenameView renames a view

func (*Catalog) TableExists ΒΆ

func (c *Catalog) TableExists(ctx context.Context, identifier table.Identifier) (bool, error)

TableExists checks if a table exists in the catalog

func (*Catalog) UpdateNamespaceProperties ΒΆ

func (c *Catalog) UpdateNamespaceProperties(ctx context.Context, namespace table.Identifier, removals []string, updates iceberg.Properties) (catalog.PropertiesUpdateSummary, error)

UpdateNamespaceProperties updates properties for a namespace with comprehensive tracking

func (*Catalog) ViewExists ΒΆ

func (c *Catalog) ViewExists(ctx context.Context, identifier table.Identifier) (bool, error)

ViewExists checks if a view exists in the catalog

type CatalogData ΒΆ

type CatalogData struct {
	CatalogName string                    `json:"catalog_name"`
	Namespaces  map[string]NamespaceEntry `json:"namespaces"`
	Tables      map[string]TableEntry     `json:"tables"`
	Views       map[string]ViewEntry      `json:"views"`      // Added views support
	Version     int                       `json:"version"`    // Schema version for future migrations
	CreatedAt   time.Time                 `json:"created_at"` // When catalog was created
	UpdatedAt   time.Time                 `json:"updated_at"` // Last update timestamp
}

CatalogData represents the JSON structure stored in catalog.json

type CatalogMetrics ΒΆ

type CatalogMetrics struct {
	TablesCreated     int64
	TablesDropped     int64
	ViewsCreated      int64
	ViewsDropped      int64
	NamespacesCreated int64
	NamespacesDropped int64
	OperationErrors   int64
	CacheHits         int64
	CacheMisses       int64
	// contains filtered or unexported fields
}

CatalogMetrics tracks operation metrics for monitoring

func (*CatalogMetrics) GetStats ΒΆ

func (m *CatalogMetrics) GetStats() map[string]int64

func (*CatalogMetrics) IncrementCacheHits ΒΆ

func (m *CatalogMetrics) IncrementCacheHits()

func (*CatalogMetrics) IncrementCacheMisses ΒΆ

func (m *CatalogMetrics) IncrementCacheMisses()

func (*CatalogMetrics) IncrementNamespacesCreated ΒΆ

func (m *CatalogMetrics) IncrementNamespacesCreated()

func (*CatalogMetrics) IncrementNamespacesDropped ΒΆ

func (m *CatalogMetrics) IncrementNamespacesDropped()

func (*CatalogMetrics) IncrementOperationErrors ΒΆ

func (m *CatalogMetrics) IncrementOperationErrors()

func (*CatalogMetrics) IncrementTablesCreated ΒΆ

func (m *CatalogMetrics) IncrementTablesCreated()

func (*CatalogMetrics) IncrementTablesDropped ΒΆ

func (m *CatalogMetrics) IncrementTablesDropped()

func (*CatalogMetrics) IncrementViewsCreated ΒΆ

func (m *CatalogMetrics) IncrementViewsCreated()

func (*CatalogMetrics) IncrementViewsDropped ΒΆ

func (m *CatalogMetrics) IncrementViewsDropped()

type ConcurrentModificationError ΒΆ

type ConcurrentModificationError struct {
	// contains filtered or unexported fields
}

ConcurrentModificationError represents a concurrent modification error

func (*ConcurrentModificationError) Error ΒΆ

type CreateTableOpt ΒΆ

type CreateTableOpt func(*CreateTableOptions)

CreateTableOpt is a function that modifies CreateTableOptions

func WithLocation ΒΆ

func WithLocation(location string) CreateTableOpt

WithLocation sets the table location

func WithPartitionSpec ΒΆ

func WithPartitionSpec(spec *iceberg.PartitionSpec) CreateTableOpt

WithPartitionSpec sets the partition specification for the table

func WithProperties ΒΆ

func WithProperties(properties iceberg.Properties) CreateTableOpt

WithProperties sets the table properties

type CreateTableOptions ΒΆ

type CreateTableOptions struct {
	PartitionSpec *iceberg.PartitionSpec
	Properties    iceberg.Properties
	Location      string
}

CreateTableOptions represents options for creating a table

type IndexConfig ΒΆ

type IndexConfig struct {
	CatalogName string                 `json:"catalog_name"`
	CatalogURI  string                 `json:"catalog_uri"`
	Properties  map[string]interface{} `json:"properties"`
}

IndexConfig represents the configuration stored in .icebox/index

type NamespaceEntry ΒΆ

type NamespaceEntry struct {
	Properties iceberg.Properties `json:"properties"`
	CreatedAt  time.Time          `json:"created_at"`
	UpdatedAt  time.Time          `json:"updated_at"`
}

NamespaceEntry represents a namespace in the catalog

type TableEntry ΒΆ

type TableEntry struct {
	Namespace                string    `json:"namespace"`
	Name                     string    `json:"name"`
	MetadataLocation         string    `json:"metadata_location"`
	PreviousMetadataLocation *string   `json:"previous_metadata_location,omitempty"`
	CreatedAt                time.Time `json:"created_at"`
	UpdatedAt                time.Time `json:"updated_at"`
}

TableEntry represents a table in the catalog

type ValidationError ΒΆ

type ValidationError struct {
	Field   string
	Message string
}

ValidationError represents a validation error

func (*ValidationError) Error ΒΆ

func (e *ValidationError) Error() string

type View ΒΆ

type View struct {
	// contains filtered or unexported fields
}

View represents a view interface for compatibility

func (*View) Dialect ΒΆ

func (v *View) Dialect() string

Dialect returns the SQL dialect of the current version

func (*View) Identifier ΒΆ

func (v *View) Identifier() table.Identifier

Identifier returns the view identifier

func (*View) Metadata ΒΆ

func (v *View) Metadata() *ViewMetadata

Metadata returns the view metadata

func (*View) SQL ΒΆ

func (v *View) SQL() string

SQL returns the SQL representation of the current version

func (*View) Schema ΒΆ

func (v *View) Schema() *ViewSchema

Schema returns the current schema of the view

type ViewEntry ΒΆ

type ViewEntry struct {
	Namespace        string    `json:"namespace"`
	Name             string    `json:"name"`
	MetadataLocation string    `json:"metadata_location"`
	CreatedAt        time.Time `json:"created_at"`
	UpdatedAt        time.Time `json:"updated_at"`
}

ViewEntry represents a view in the catalog according to Iceberg V2 spec

type ViewMetadata ΒΆ

type ViewMetadata struct {
	ViewUUID         string                `json:"view-uuid"`
	FormatVersion    int                   `json:"format-version"`
	Location         string                `json:"location"`
	Schemas          []ViewSchema          `json:"schemas"`
	CurrentVersionID int                   `json:"current-version-id"`
	Versions         []ViewVersion         `json:"versions"`
	VersionLog       []ViewVersionLogEntry `json:"version-log"`
	Properties       map[string]string     `json:"properties,omitempty"`
}

ViewMetadata represents the complete view metadata according to Iceberg V2 spec

type ViewRepresentation ΒΆ

type ViewRepresentation struct {
	Type    string `json:"type"`
	SQL     string `json:"sql"`
	Dialect string `json:"dialect"`
}

ViewRepresentation represents a view representation (SQL, etc.)

type ViewSchema ΒΆ

type ViewSchema struct {
	SchemaID int               `json:"schema-id"`
	Type     string            `json:"type"`
	Fields   []ViewSchemaField `json:"fields"`
}

ViewSchema represents a view schema

type ViewSchemaField ΒΆ

type ViewSchemaField struct {
	ID       int    `json:"id"`
	Name     string `json:"name"`
	Required bool   `json:"required"`
	Type     string `json:"type"`
	Doc      string `json:"doc,omitempty"`
}

ViewSchemaField represents a field in a view schema

type ViewVersion ΒΆ

type ViewVersion struct {
	VersionID        int                  `json:"version-id"`
	SchemaID         int                  `json:"schema-id"`
	TimestampMs      int64                `json:"timestamp-ms"`
	Summary          map[string]string    `json:"summary"`
	Representations  []ViewRepresentation `json:"representations"`
	DefaultCatalog   *string              `json:"default-catalog,omitempty"`
	DefaultNamespace []string             `json:"default-namespace"`
}

ViewVersion represents a version of a view

type ViewVersionLogEntry ΒΆ

type ViewVersionLogEntry struct {
	TimestampMs int64 `json:"timestamp-ms"`
	VersionID   int   `json:"version-id"`
}

ViewVersionLogEntry represents an entry in the view version log

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL