catalog

package
v0.1.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package catalog keeps what schema scans found, per database a datasource reads, so the console can search and plan over every scanned schema after a restart and on every replica. A scanned table is kept as a Table: the parts of an introspected entity that importing and planning read, in a stable JSON form independent of the runtime model.

Index

Constants

View Source
const (
	ItemPending  = "pending"
	ItemRunning  = "running"
	ItemDone     = "done"
	ItemFailed   = "failed"
	ItemCanceled = "canceled"
)

Item states: an item goes from pending to running, and from running to done, failed, back to pending (to try again after NotBefore) or, when the batch ends first, canceled.

Variables

View Source
var (
	// ErrTooManyBatches: the datasource has as many unfinished batches as
	// allowed.
	ErrTooManyBatches = errors.New("catalog: the datasource has too many scan batches running")
	// ErrNotOwner: the batch is finished or another run owns it.
	ErrNotOwner = errors.New("catalog: the scan batch is not this run's")
)

Batch errors.

Functions

func Catalog

func Catalog(schemas []Schema) introspect.Catalog

Catalog puts kept schemas together as an indexed catalog.

Types

type Batch

type Batch struct {
	ID         string
	Datasource string
	StartedBy  string
	// RequestKey names the request that started the batch: a request sent
	// again with the same key gets the same batch.
	RequestKey string
	// Owner is one run of the batch, never reused: a takeover makes a new
	// one.
	Owner      string
	StartedAt  time.Time
	FinishedAt time.Time
	Heartbeat  time.Time
	// Canceled asks the owner to stop.
	Canceled bool
	Items    []BatchItem
}

Batch is a scan of many schemas of a datasource, run by one process (the owner) at a time: the owner moves each schema through its states and beats, and another process takes over a batch whose owner stopped beating. Every write of the owner is conditioned on still owning the unfinished batch, in the same step as the write, so a run that lost the batch (paused, then resumed) writes nothing.

type BatchItem

type BatchItem struct {
	Schema string
	State  string
	Error  string
	// Attempts counts the scans of the schema started; NotBefore is when a
	// pending one that failed may be scanned again.
	Attempts  int
	NotBefore time.Time
}

BatchItem is one schema of a batch.

type BatchStore

type BatchStore interface {
	// CreateBatch creates b (its owner beating at b.Heartbeat) unless its
	// datasource has maxOpen unfinished batches (ErrTooManyBatches), checking
	// and creating in one step, so concurrent requests, of any process,
	// cannot pass the bound together. A batch started before with b's
	// RequestKey for the datasource is returned instead, with false.
	CreateBatch(ctx context.Context, b Batch, maxOpen int) (Batch, bool, error)
	Batch(ctx context.Context, id string) (Batch, bool, error)
	// Batches lists the batches of datasource, newest first.
	Batches(ctx context.Context, datasource string, limit int) ([]Batch, error)
	// UnfinishedBatches lists every batch not finished.
	UnfinishedBatches(ctx context.Context) ([]Batch, error)
	// ClaimBatch makes owner the owner of an unfinished batch whose heartbeat
	// is before staleBefore, or already owner's, beating at now; false when
	// another owner holds it.
	ClaimBatch(ctx context.Context, id, owner string, staleBefore, now time.Time) (bool, error)
	// MoveItem moves item (by its schema) to its state, error, attempts and
	// NotBefore when it is in one of from, the owner beating at now; false
	// when it is in none of them. ErrNotOwner, writing nothing, when the
	// batch is finished or not owner's.
	MoveItem(ctx context.Context, id, owner string, now time.Time, item BatchItem, from ...string) (bool, error)
	// FinishBatch ends a batch owner owns at at, canceling its pending and
	// running items in the same step; ErrNotOwner when it does not own it.
	FinishBatch(ctx context.Context, id, owner string, at time.Time) error
	// CancelBatch asks the owner of a batch to stop.
	CancelBatch(ctx context.Context, id string) error
}

BatchStore keeps scan batches.

type Column

type Column struct {
	Name          string `json:"name"`
	Type          string `json:"type"`
	Nullable      bool   `json:"nullable,omitempty"`
	Default       bool   `json:"default,omitempty"`
	Generated     bool   `json:"generated,omitempty"`
	AutoIncrement bool   `json:"autoIncrement,omitempty"`
	Description   string `json:"description,omitempty"`
}

Column is a column of a table.

type ForeignKey

type ForeignKey struct {
	Name       string   `json:"name"`
	Columns    []string `json:"columns"`
	RefSchema  string   `json:"refSchema,omitempty"`
	RefTable   string   `json:"refTable"`
	RefColumns []string `json:"refColumns"`
	OnDelete   string   `json:"onDelete,omitempty"`
	OnUpdate   string   `json:"onUpdate,omitempty"`
}

ForeignKey references another table.

type Index

type Index struct {
	Name    string   `json:"name"`
	Method  string   `json:"method"`
	Parts   []string `json:"parts,omitempty"`
	Unique  bool     `json:"unique,omitempty"`
	Primary bool     `json:"primary,omitempty"`
	Where   string   `json:"where,omitempty"`
}

Index is an index in the database's own terms.

type Key

type Key struct {
	Name       string   `json:"name"`
	Columns    []string `json:"columns"`
	Primary    bool     `json:"primary,omitempty"`
	Deferrable bool     `json:"deferrable,omitempty"`
	Reason     string   `json:"reason,omitempty"`
}

Key is a primary or unique key; Reason says why one cannot identify a row.

type List

type List struct {
	Schemas []string `json:"schemas"`
	// Current is where unqualified names resolve; empty when unknown.
	Current  string    `json:"current,omitempty"`
	ListedAt time.Time `json:"-"`
}

List is the schema list of a source.

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is a Store held in process memory, for a server without a configuration store and for tests.

func NewMemoryStore

func NewMemoryStore() *MemoryStore

NewMemoryStore returns an empty store.

func (*MemoryStore) Batch

func (m *MemoryStore) Batch(_ context.Context, id string) (Batch, bool, error)

func (*MemoryStore) Batches

func (m *MemoryStore) Batches(_ context.Context, datasource string, limit int) ([]Batch, error)

func (*MemoryStore) CancelBatch

func (m *MemoryStore) CancelBatch(_ context.Context, id string) error

func (*MemoryStore) CatalogList

func (m *MemoryStore) CatalogList(_ context.Context, source string) (List, bool, error)

func (*MemoryStore) CatalogSchemas

func (m *MemoryStore) CatalogSchemas(_ context.Context, source string, names []string) ([]Schema, error)

func (*MemoryStore) CatalogVersions

func (m *MemoryStore) CatalogVersions(_ context.Context, source string) ([]Version, error)

func (*MemoryStore) ClaimBatch

func (m *MemoryStore) ClaimBatch(_ context.Context, id, owner string, staleBefore, now time.Time) (bool, error)

func (*MemoryStore) CreateBatch

func (m *MemoryStore) CreateBatch(_ context.Context, b Batch, maxOpen int) (Batch, bool, error)

func (*MemoryStore) DeleteCatalogSchemas

func (m *MemoryStore) DeleteCatalogSchemas(_ context.Context, source string, names []string, before time.Time) error

func (*MemoryStore) FinishBatch

func (m *MemoryStore) FinishBatch(_ context.Context, id, owner string, at time.Time) error

func (*MemoryStore) MoveItem

func (m *MemoryStore) MoveItem(
	_ context.Context, id, owner string, now time.Time, item BatchItem, from ...string,
) (bool, error)

func (*MemoryStore) PruneCatalog

func (m *MemoryStore) PruneCatalog(_ context.Context, datasource, keep string, before time.Time) error

func (*MemoryStore) PutCatalogList

func (m *MemoryStore) PutCatalogList(_ context.Context, source, datasource string, l List) error

func (*MemoryStore) PutCatalogSchemas

func (m *MemoryStore) PutCatalogSchemas(_ context.Context, source, datasource string, schemas []Schema) error

func (*MemoryStore) UnfinishedBatches

func (m *MemoryStore) UnfinishedBatches(context.Context) ([]Batch, error)

type Schema

type Schema struct {
	Name      string    `json:"-"`
	ScannedAt time.Time `json:"-"`
	Default   string    `json:"default,omitempty"`
	FoldCase  bool      `json:"foldCase,omitempty"`
	Tables    []Table   `json:"tables"`
}

Schema is what one scan found in a schema; "" names the default schema of a datasource that cannot list schemas.

func FromCatalog

func FromCatalog(cat introspect.Catalog, schemas []string, at time.Time) []Schema

FromCatalog splits a scan into the schemas it covered: schemas lists the schemas scanned, none for the default schema of a datasource that cannot list them, which holds all of the scan.

type Source

type Source struct {
	Datasource string
	Driver     string
	DSN        string
}

Source is what a datasource reads: its driver and read connection. What is kept belongs to a source, so a datasource that keeps its name but reads another database does not show what the old one had.

func (Source) ID

func (s Source) ID() string

ID identifies the source in a store without revealing its DSN.

func (Source) Token

func (s Source) Token() string

Token names the database the source reads, without its DSN; the console compares it to tell results of another database apart.

type Store

type Store interface {
	// CatalogList returns the kept schema list of source; false when none.
	CatalogList(ctx context.Context, source string) (List, bool, error)
	// PutCatalogList replaces the schema list of source.
	PutCatalogList(ctx context.Context, source, datasource string, l List) error
	// CatalogVersions lists the kept schemas of source without their tables.
	CatalogVersions(ctx context.Context, source string) ([]Version, error)
	// CatalogSchemas returns the kept schemas of source among names.
	CatalogSchemas(ctx context.Context, source string, names []string) ([]Schema, error)
	// PutCatalogSchemas replaces the given schemas of source.
	PutCatalogSchemas(ctx context.Context, source, datasource string, schemas []Schema) error
	// PruneCatalog drops what was kept for datasource from sources other
	// than keep, listed or scanned before before.
	PruneCatalog(ctx context.Context, datasource, keep string, before time.Time) error
	// DeleteCatalogSchemas drops the schemas of source among names scanned
	// before before.
	DeleteCatalogSchemas(ctx context.Context, source string, names []string, before time.Time) error
	BatchStore
}

Store keeps schema lists and scanned schemas per source (by Source.ID). Implementations must be safe for concurrent use.

type Table

type Table struct {
	Schema      string       `json:"schema,omitempty"`
	Name        string       `json:"name"`
	View        bool         `json:"view,omitempty"`
	Derived     bool         `json:"derived,omitempty"`
	Description string       `json:"description,omitempty"`
	Columns     []Column     `json:"columns"`
	Keys        []Key        `json:"keys,omitempty"`
	Indexes     []Index      `json:"indexes,omitempty"`
	ForeignKeys []ForeignKey `json:"foreignKeys,omitempty"`
	SideEffects bool         `json:"sideEffects,omitempty"`
}

Table is one scanned table or view.

func FromEntity

func FromEntity(e entity.Entity) Table

FromEntity keeps what a scan found of a table.

func (Table) Entity

func (t Table) Entity() entity.Entity

Entity is the introspected entity t was kept from, as far as kept.

type Version

type Version struct {
	Name      string
	ScannedAt time.Time
	Tables    int
}

Version tells whether a kept schema changed since it was read.

Directories

Path Synopsis
Package catalogtest provides the conformance suite every catalog.Store implementation must pass, so SQL stores keep the MemoryStore semantics.
Package catalogtest provides the conformance suite every catalog.Store implementation must pass, so SQL stores keep the MemoryStore semantics.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL