Documentation
¶
Overview ¶
Package catalog keeps what schema scans found, per database a datasource reads, so the console can search and plan over every scanned schema after a restart and on every replica. A scanned table is kept as a Table: the parts of an introspected entity that importing and planning read, in a stable JSON form independent of the runtime model.
Index ¶
- Constants
- Variables
- func Catalog(schemas []Schema) introspect.Catalog
- type Batch
- type BatchItem
- type BatchStore
- type Column
- type ForeignKey
- type Index
- type Key
- type List
- type MemoryStore
- func (m *MemoryStore) Batch(_ context.Context, id string) (Batch, bool, error)
- func (m *MemoryStore) Batches(_ context.Context, datasource string, limit int) ([]Batch, error)
- func (m *MemoryStore) CancelBatch(_ context.Context, id string) error
- func (m *MemoryStore) CatalogList(_ context.Context, source string) (List, bool, error)
- func (m *MemoryStore) CatalogSchemas(_ context.Context, source string, names []string) ([]Schema, error)
- func (m *MemoryStore) CatalogVersions(_ context.Context, source string) ([]Version, error)
- func (m *MemoryStore) ClaimBatch(_ context.Context, id, owner string, staleBefore, now time.Time) (bool, error)
- func (m *MemoryStore) CreateBatch(_ context.Context, b Batch, maxOpen int) (Batch, bool, error)
- func (m *MemoryStore) DeleteCatalogSchemas(_ context.Context, source string, names []string, before time.Time) error
- func (m *MemoryStore) FinishBatch(_ context.Context, id, owner string, at time.Time) error
- func (m *MemoryStore) MoveItem(_ context.Context, id, owner string, now time.Time, item BatchItem, ...) (bool, error)
- func (m *MemoryStore) PruneCatalog(_ context.Context, datasource, keep string, before time.Time) error
- func (m *MemoryStore) PutCatalogList(_ context.Context, source, datasource string, l List) error
- func (m *MemoryStore) PutCatalogSchemas(_ context.Context, source, datasource string, schemas []Schema) error
- func (m *MemoryStore) UnfinishedBatches(context.Context) ([]Batch, error)
- type Schema
- type Source
- type Store
- type Table
- type Version
Constants ¶
const ( ItemPending = "pending" ItemRunning = "running" ItemDone = "done" ItemFailed = "failed" ItemCanceled = "canceled" )
Item states: an item goes from pending to running, and from running to done, failed, back to pending (to try again after NotBefore) or, when the batch ends first, canceled.
Variables ¶
var ( // ErrTooManyBatches: the datasource has as many unfinished batches as // allowed. ErrTooManyBatches = errors.New("catalog: the datasource has too many scan batches running") // ErrNotOwner: the batch is finished or another run owns it. ErrNotOwner = errors.New("catalog: the scan batch is not this run's") )
Batch errors.
Functions ¶
func Catalog ¶
func Catalog(schemas []Schema) introspect.Catalog
Catalog puts kept schemas together as an indexed catalog.
Types ¶
type Batch ¶
type Batch struct {
ID string
Datasource string
StartedBy string
// RequestKey names the request that started the batch: a request sent
// again with the same key gets the same batch.
RequestKey string
// Owner is one run of the batch, never reused: a takeover makes a new
// one.
Owner string
StartedAt time.Time
FinishedAt time.Time
Heartbeat time.Time
// Canceled asks the owner to stop.
Canceled bool
Items []BatchItem
}
Batch is a scan of many schemas of a datasource, run by one process (the owner) at a time: the owner moves each schema through its states and beats, and another process takes over a batch whose owner stopped beating. Every write of the owner is conditioned on still owning the unfinished batch, in the same step as the write, so a run that lost the batch (paused, then resumed) writes nothing.
type BatchItem ¶
type BatchItem struct {
Schema string
State string
Error string
// Attempts counts the scans of the schema started; NotBefore is when a
// pending one that failed may be scanned again.
Attempts int
NotBefore time.Time
}
BatchItem is one schema of a batch.
type BatchStore ¶
type BatchStore interface {
// CreateBatch creates b (its owner beating at b.Heartbeat) unless its
// datasource has maxOpen unfinished batches (ErrTooManyBatches), checking
// and creating in one step, so concurrent requests, of any process,
// cannot pass the bound together. A batch started before with b's
// RequestKey for the datasource is returned instead, with false.
CreateBatch(ctx context.Context, b Batch, maxOpen int) (Batch, bool, error)
Batch(ctx context.Context, id string) (Batch, bool, error)
// Batches lists the batches of datasource, newest first.
Batches(ctx context.Context, datasource string, limit int) ([]Batch, error)
// UnfinishedBatches lists every batch not finished.
UnfinishedBatches(ctx context.Context) ([]Batch, error)
// ClaimBatch makes owner the owner of an unfinished batch whose heartbeat
// is before staleBefore, or already owner's, beating at now; false when
// another owner holds it.
ClaimBatch(ctx context.Context, id, owner string, staleBefore, now time.Time) (bool, error)
// MoveItem moves item (by its schema) to its state, error, attempts and
// NotBefore when it is in one of from, the owner beating at now; false
// when it is in none of them. ErrNotOwner, writing nothing, when the
// batch is finished or not owner's.
MoveItem(ctx context.Context, id, owner string, now time.Time, item BatchItem, from ...string) (bool, error)
// FinishBatch ends a batch owner owns at at, canceling its pending and
// running items in the same step; ErrNotOwner when it does not own it.
FinishBatch(ctx context.Context, id, owner string, at time.Time) error
// CancelBatch asks the owner of a batch to stop.
CancelBatch(ctx context.Context, id string) error
}
BatchStore keeps scan batches.
type Column ¶
type Column struct {
Name string `json:"name"`
Type string `json:"type"`
Nullable bool `json:"nullable,omitempty"`
Default bool `json:"default,omitempty"`
Generated bool `json:"generated,omitempty"`
AutoIncrement bool `json:"autoIncrement,omitempty"`
Description string `json:"description,omitempty"`
}
Column is a column of a table.
type ForeignKey ¶
type ForeignKey struct {
Name string `json:"name"`
Columns []string `json:"columns"`
RefSchema string `json:"refSchema,omitempty"`
RefTable string `json:"refTable"`
RefColumns []string `json:"refColumns"`
OnDelete string `json:"onDelete,omitempty"`
OnUpdate string `json:"onUpdate,omitempty"`
}
ForeignKey references another table.
type Index ¶
type Index struct {
Name string `json:"name"`
Method string `json:"method"`
Parts []string `json:"parts,omitempty"`
Unique bool `json:"unique,omitempty"`
Primary bool `json:"primary,omitempty"`
Where string `json:"where,omitempty"`
}
Index is an index in the database's own terms.
type Key ¶
type Key struct {
Name string `json:"name"`
Columns []string `json:"columns"`
Primary bool `json:"primary,omitempty"`
Deferrable bool `json:"deferrable,omitempty"`
Reason string `json:"reason,omitempty"`
}
Key is a primary or unique key; Reason says why one cannot identify a row.
type List ¶
type List struct {
Schemas []string `json:"schemas"`
// Current is where unqualified names resolve; empty when unknown.
Current string `json:"current,omitempty"`
ListedAt time.Time `json:"-"`
}
List is the schema list of a source.
type MemoryStore ¶
type MemoryStore struct {
// contains filtered or unexported fields
}
MemoryStore is a Store held in process memory, for a server without a configuration store and for tests.
func (*MemoryStore) CancelBatch ¶
func (m *MemoryStore) CancelBatch(_ context.Context, id string) error
func (*MemoryStore) CatalogList ¶
func (*MemoryStore) CatalogSchemas ¶
func (*MemoryStore) CatalogVersions ¶
func (*MemoryStore) ClaimBatch ¶
func (*MemoryStore) CreateBatch ¶
func (*MemoryStore) DeleteCatalogSchemas ¶
func (*MemoryStore) FinishBatch ¶
func (*MemoryStore) PruneCatalog ¶
func (*MemoryStore) PutCatalogList ¶
func (*MemoryStore) PutCatalogSchemas ¶
func (*MemoryStore) UnfinishedBatches ¶
func (m *MemoryStore) UnfinishedBatches(context.Context) ([]Batch, error)
type Schema ¶
type Schema struct {
Name string `json:"-"`
ScannedAt time.Time `json:"-"`
Default string `json:"default,omitempty"`
FoldCase bool `json:"foldCase,omitempty"`
Tables []Table `json:"tables"`
}
Schema is what one scan found in a schema; "" names the default schema of a datasource that cannot list schemas.
func FromCatalog ¶
FromCatalog splits a scan into the schemas it covered: schemas lists the schemas scanned, none for the default schema of a datasource that cannot list them, which holds all of the scan.
type Source ¶
Source is what a datasource reads: its driver and read connection. What is kept belongs to a source, so a datasource that keeps its name but reads another database does not show what the old one had.
type Store ¶
type Store interface {
// CatalogList returns the kept schema list of source; false when none.
CatalogList(ctx context.Context, source string) (List, bool, error)
// PutCatalogList replaces the schema list of source.
PutCatalogList(ctx context.Context, source, datasource string, l List) error
// CatalogVersions lists the kept schemas of source without their tables.
CatalogVersions(ctx context.Context, source string) ([]Version, error)
// CatalogSchemas returns the kept schemas of source among names.
CatalogSchemas(ctx context.Context, source string, names []string) ([]Schema, error)
// PutCatalogSchemas replaces the given schemas of source.
PutCatalogSchemas(ctx context.Context, source, datasource string, schemas []Schema) error
// PruneCatalog drops what was kept for datasource from sources other
// than keep, listed or scanned before before.
PruneCatalog(ctx context.Context, datasource, keep string, before time.Time) error
// DeleteCatalogSchemas drops the schemas of source among names scanned
// before before.
DeleteCatalogSchemas(ctx context.Context, source string, names []string, before time.Time) error
BatchStore
}
Store keeps schema lists and scanned schemas per source (by Source.ID). Implementations must be safe for concurrent use.
type Table ¶
type Table struct {
Schema string `json:"schema,omitempty"`
Name string `json:"name"`
View bool `json:"view,omitempty"`
Derived bool `json:"derived,omitempty"`
Description string `json:"description,omitempty"`
Columns []Column `json:"columns"`
Keys []Key `json:"keys,omitempty"`
Indexes []Index `json:"indexes,omitempty"`
ForeignKeys []ForeignKey `json:"foreignKeys,omitempty"`
SideEffects bool `json:"sideEffects,omitempty"`
}
Table is one scanned table or view.
func FromEntity ¶
FromEntity keeps what a scan found of a table.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package catalogtest provides the conformance suite every catalog.Store implementation must pass, so SQL stores keep the MemoryStore semantics.
|
Package catalogtest provides the conformance suite every catalog.Store implementation must pass, so SQL stores keep the MemoryStore semantics. |