tableregister

package
v1.125.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 23, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package tableregister makes a file already in object storage readable as a table, without copying it and without giving anything a write tool.

A managed resource and a portal asset are both stored as one object under a per-object directory. Trino's Hive connector reads CSV from an external location, so "make this file queryable" is a CREATE TABLE naming that directory: no ingestion, no copy, and the table tracks whatever the object holds now. One registrar serves both kinds because a registration says the same thing about either.

What the registrar does NOT do is decide who may register. Every entry point -- the resources REST API, the asset REST API, the manage_table tool -- resolves its own caller and hands one in; the registrar applies the persona connection boundary to that caller and refuses on anything it cannot establish.

Index

Constants

View Source 
const (
	// KindResource is a managed resource: a file a person uploaded.
	KindResource = "resource"
	// KindAsset is a portal asset: a file the platform wrote, typically a
	// trino_export or a script's output.
	KindAsset = "asset"
)

Source kinds a registration can be built from. They are the two things a person can put a file into the platform through.

View Source 
const DefaultMaxBytes = 100 << 20

DefaultMaxBytes bounds the object the registrar reads to find a header row.

Neither S3 adapter has a range read, so learning the first line costs a full GetObject. The bound matches the managed-resource upload cap, which is the largest object either surface can have put there.

Variables

View Source 
var (
	// ErrNotFound is returned for a registration id that does not exist.
	ErrNotFound = errors.New("registration not found")

	// ErrNoScratchTarget means the connection has no scratch: block, so
	// nothing can be registered on it.
	ErrNoScratchTarget = errors.New("this connection has no scratch catalog and schema configured, so a table cannot be registered on it")

	// ErrConnectionDenied means the caller's persona is not granted the
	// connection. It is the same boundary a tool call meets.
	ErrConnectionDenied = errors.New("your persona is not granted this connection")

	// ErrNotCSV means the source object is not a CSV, which is the only format
	// a registration can be built from.
	ErrNotCSV = errors.New("only a CSV file can be registered as a table")

	// ErrEmptyHeader means the object had no header row to take columns from.
	ErrEmptyHeader = errors.New("the file has no header row, so the table has no column names")

	// ErrNoIdentity means the call carried no identity to register under.
	// Registration records who made it and decides replacement on that, so an
	// anonymous registration would be one nobody owns and anyone could take
	// over.
	ErrNoIdentity = errors.New("registering a table needs a signed-in identity")

	// ErrBadReference means the reference a caller passed is not one this
	// platform issues, or names something that is not a stored file. It is
	// separate from ErrNoSuchFile because the caller can see the difference
	// themselves: the string they sent is malformed, so telling them so
	// discloses nothing they did not already have.
	ErrBadReference = errors.New("not a reference to a stored file")

	// ErrNoSuchFile is the one answer to a reference that resolves to a record
	// this caller may not register: one that does not exist, one that was
	// deleted, and one that exists but belongs to somebody else are answered
	// identically, so the surface never confirms the existence of a record the
	// caller cannot act on. It is what `fetch` does with a reference outside
	// the caller's reach, held to here for the same reason.
	ErrNoSuchFile = errors.New("that reference names no stored file you can register")

	// ErrRefused marks a refusal the caller can act on -- a name already
	// taken, a sibling object in the way -- as opposed to a failure of the
	// platform. Every such refusal wraps it, so a surface can answer with a
	// status that says "your request was understood and declined" rather than
	// reporting a store outage as a conflict, or the reverse.
	ErrRefused = errors.New("registration refused")
)

Errors the registrar returns. Every surface renders these, so the wording a person sees comes from one place.

View Source 
var ErrNameTaken = errors.New("that table name was registered by someone else while this registration was being made")

ErrNameTaken is returned when the unique index on the table name rejects an insert. The registrar checks for a holder before it writes; this is the race between that check and this write, and it must not surface as a bare constraint violation.

View Source 
var ErrUnavailable = errors.New("table registration is not available on this deployment")

ErrUnavailable means the deployment has no registration mechanism wired.

Functions

func BuildDDL 

func BuildDDL(r Registration, replacing bool) []string

BuildDDL returns the statements that make a registration, in the order they must run.

CREATE SCHEMA comes first and is IF NOT EXISTS: the scratch schema is the target of every registration on a connection and the first one to arrive has to make it. DROP TABLE is issued only when replacing a registration the caller is entitled to replace -- an unconditional drop would let a name collision quietly take out somebody else's table, which is why the decision is made before this is called rather than here.

Every column is VARCHAR because Hive CSV admits nothing else; skipping the header line is what keeps the column names out of the rows.

func DirectoryOf 

func DirectoryOf(key string) string

DirectoryOf returns the directory portion of an object key, with its trailing slash. A key with no directory yields the empty string, which no registration can be built on.

func LocationURI 

func LocationURI(bucket, dir string) string

LocationURI renders the external location a directory in a bucket is addressed by.

func ParseReference 

func ParseReference(reference string) (kind, id string, err error)

ParseReference resolves the canonical reference an agent already holds -- the string `search` emits on a hit and `fetch` dereferences -- into the kind and id a registration is built over.

It is what makes one action serve every kind of stored file. The platform has exactly one vocabulary for naming a record across tools, so a registration keyed by that vocabulary needs no per-kind argument and no second tool; the kind travels inside the reference.

Only the two stored-file kinds resolve. Any other well-formed reference (a knowledge page, a dataset, a memory record) parses and is then refused by name, because naming what was passed tells the caller what to pass instead.

func PrefixedTableName 

func PrefixedTableName(persona, slug string) string

PrefixedTableName is the name a registration takes: the persona, then the slug.

The scratch schema is one shared workspace -- everyone granted the connection sees every table in it, and resource and asset permissions are not carried into Trino. The prefix is not a boundary and does not pretend to be one; it is what keeps two people who both registered "vendors" from colliding on the name, and what tells a reader of the schema whose working table they are looking at.

func QuoteIdentifier 

func QuoteIdentifier(name string) string

QuoteIdentifier renders a name as a Trino delimited identifier.

Every identifier the registrar puts in a statement goes through this, including ones it derived itself: a column name comes from the first line of a file somebody uploaded, and the DDL is assembled as text because Trino has no parameter binding for identifiers. Doubling the quote is what closes that.

func QuoteLiteral 

func QuoteLiteral(value string) string

QuoteLiteral renders a value as a Trino string literal, for the table properties -- the external location -- that are values rather than names.

func SampleJoinSQL 

func SampleJoinSQL(r Registration) string

SampleJoinSQL renders a statement showing how the registered table is used: a SELECT over it, with the CAST that joining it to a typed warehouse column requires. Every column is VARCHAR, so a reader who writes the obvious join gets a type error and no explanation of why; this is the explanation.

func SlugifyTableName 

func SlugifyTableName(raw string) string

SlugifyTableName turns a filename or a person's suggestion into a table name Trino accepts unquoted: lowercase, alphanumeric and underscore, never leading with a digit.

Types

type AuditLogger 

type AuditLogger interface {
	Log(ctx context.Context, event audit.Event) error
}

AuditLogger is the write half of audit.Logger. The registrar records events and never reads them, so it names only what it uses; audit.Logger satisfies it.

type Caller 

type Caller struct {
	UserID  string
	Email   string
	Persona string
	Roles   []string
	IsAdmin bool
}

Caller is who is asking. Persona drives the connection boundary and the table-name prefix; IsAdmin lifts the ownership check on replacing a registration.

type Column 

type Column struct {
	Name string `json:"name"`
	Type string `json:"type"`
}

Column is one column of a registered table.

Type is recorded even though Hive CSV admits exactly one: a reader of the record should not have to know the connector's rule to know what a query will get back, and a stored type is what a later format would vary.

func ReadHeaderColumns 

func ReadHeaderColumns(content []byte) ([]Column, error)

ReadHeaderColumns parses the first line of a CSV and returns the columns a table registered over it declares.

A blank column name is filled in positionally rather than refused: a trailing comma or an unnamed index column is ordinary in an exported CSV, and a table that refuses to exist over it helps nobody. A duplicate name is suffixed for the same reason -- the file is what it is, and the column still has to be addressable.

type ConnectionScope 

type ConnectionScope interface {
	AllowConnection(persona, connection string) bool
}

ConnectionScope is the persona connection boundary, the same predicate the authorizer applies to a tool call.

type Deps 

type Deps struct {
	Store Store
	Trino Executor
	// Objects reads a source's bytes, keyed by source kind. It is per kind
	// because the two kinds do not have to share an object store: a
	// deployment names the portal's S3 connection and the managed-resources
	// one separately, and reading a resource through the portal's client
	// would look in the wrong bucket on any deployment that split them.
	Objects  map[string]ObjectReader
	Scope    ConnectionScope
	Audit    AuditLogger
	NewID    func() (string, error)
	MaxBytes int64
}

Deps is what a Registrar needs.

type Executor 

type Executor interface {
	Exec(ctx context.Context, connection, sql string) error
	ScratchTarget(connection string) (trino.ScratchConfig, bool)
}

Executor runs a statement on a named Trino connection and reports the registration target that connection writes into. The Trino toolkit satisfies it.

type Lookup 

type Lookup struct {
	// contains filtered or unexported fields
}

Lookup adapts a Registrar to the discovery layer's TableLookup: it answers, for a page of search hits or one fetched record, which of them are readable as query-engine tables.

It lives here rather than in pkg/knowledge because staleness needs both halves -- the registration's recorded location and the record's current head key -- and only this side knows what a registration is.

func NewLookup 

func NewLookup(reg *Registrar) *Lookup

NewLookup adapts a Registrar for discovery. A nil or unwired Registrar yields a lookup that finds nothing, which is a deployment with no registration mechanism rather than an error every search has to carry.

func (*Lookup) TablesFor 

func (l *Lookup) TablesFor(
	ctx context.Context, subjects []knowledge.TableSubject,
) (map[string]*knowledge.HitTable, error)

TablesFor returns the table reference for every subject that has one.

A subject with several registrations -- the same file registered on two connections -- yields the first by the store's ordering, which is the most recent. A hit carries one table reference because it is a pointer to where the data can be queried, not an inventory of every place it was registered.

type ObjectEntry 

type ObjectEntry struct {
	Key  string
	Size int64
}

ObjectEntry names one object in a directory listing.

type ObjectReader 

type ObjectReader interface {
	GetObject(ctx context.Context, bucket, key string) (body []byte, contentType string, err error)
	ListDirectory(ctx context.Context, bucket, prefix string) (entries []ObjectEntry, truncated bool, err error)
}

ObjectReader reads the source object and lists what sits beside it.

type Record 

type Record struct {
	ID          string
	Name        string
	Bucket      string
	Key         string
	ContentType string
	OwnerID     string
}

Record is what a caller already holds about a stored file, in the terms both kinds share. It is the argument SourceFromResource and SourceFromAssetRecord take, so neither depends on the portal or resource types.

type Registrar 

type Registrar struct {
	// contains filtered or unexported fields
}

Registrar registers and unregisters tables over stored objects.

func New 

func New(deps Deps) *Registrar

New creates a Registrar. A nil Store or Trino executor makes every call report that registration is unavailable rather than panicking, which is the state of a deployment with no database or no Trino toolkit.

func (*Registrar) Available 

func (r *Registrar) Available() bool

Available reports whether registration is wired at all, so a surface can hide the action rather than offering one that always refuses.

func (*Registrar) BySource 

func (r *Registrar) BySource(ctx context.Context, kind, sourceID string) ([]Registration, error)

BySource returns every registration over one source.

func (*Registrar) ForSources 

func (r *Registrar) ForSources(ctx context.Context, kind string, ids []string) (map[string][]Registration, error)

ForSources returns the registrations of many sources at once.

func (*Registrar) Register 

func (r *Registrar) Register(ctx context.Context, caller Caller, src Source, req Request) (*Registration, error)

Register makes the source's directory readable as a table and records it.

The order is deliberate: everything that can refuse does so before any statement runs, so a refused registration leaves nothing behind in Trino. The record is written last, because a row naming a table that was never created is worse than a table with no row -- the first is a lie a search hit repeats, the second is an object in a scratch schema.

func (*Registrar) Unregister 

func (r *Registrar) Unregister(ctx context.Context, caller Caller, id, source string) error

Unregister drops a registered table and forgets it.

Dropping a Hive external table removes the metastore entry and leaves the objects, so unregistering never touches the file the person uploaded. The row goes even when the DROP fails: the alternative is a record of a table nobody can remove through the platform, and the DROP is IF EXISTS so a table already gone is not an error in the first place.

func (*Registrar) UnregisterAllForSource 

func (r *Registrar) UnregisterAllForSource(ctx context.Context, kind, sourceID string)

UnregisterAllForSource drops every table registered over a source. It is what a resource or asset delete calls: the file is going, and a table over where it used to be would return nothing and explain nothing.

It is best-effort by design. The delete that triggered it has its own reasons to succeed, and failing it because a scratch table could not be dropped would make an unrelated Trino outage look like a broken delete.

type Registration 

type Registration struct {
	ID           string    `json:"id"`
	SourceKind   string    `json:"source_kind"`
	SourceID     string    `json:"source_id"`
	Connection   string    `json:"connection"`
	Catalog      string    `json:"catalog"`
	Schema       string    `json:"schema"`
	Table        string    `json:"table"`
	Location     string    `json:"location"`
	Columns      []Column  `json:"columns"`
	RegisteredBy string    `json:"registered_by"`
	RegisteredAt time.Time `json:"registered_at"`
}

Registration records that a source object's directory is readable as a table on a connection.

func (Registration) IsStale 

func (r Registration) IsStale(bucket, currentHeadKey string) bool

IsStale reports whether the registration still points at the source's current content.

A resource revision and an asset version both write a new object under a new directory and move the head key to it. The table keeps serving the directory it was registered against, which is the revision that was current then -- correct SQL over the wrong bytes, and nothing about the table says so. This compares the recorded location against the directory of the head key the source carries now; re-registering targets the current head.

An overwrite in place is not staleness: replacing the object at the same key changes what the table returns on the next query, with no re-registration, which is what makes a repeating vendor drop a re-upload rather than a chore.

func (Registration) QualifiedName 

func (r Registration) QualifiedName() string

QualifiedName is the table as a query names it.

type Request 

type Request struct {
	Connection string
	// TableName is the caller's choice. Empty takes a slug of the source's
	// filename. Either way it is slugified and persona-prefixed.
	TableName string
	// Source names where the registration comes from: "portal" for a REST or
	// UI action, "mcp" for a tool call. It is recorded on the audit event.
	Source string
}

Request is one registration.

type Source 

type Source struct {
	Kind        string
	ID          string
	Name        string
	Bucket      string
	HeadKey     string
	ContentType string
	// OwnerID is who the source belongs to. It decides whether a caller may
	// replace an existing registration of the same name.
	OwnerID string
}

Source is the object a registration is built over: where it lives, what it is, and who it belongs to. Each surface builds one from its own record, so the registrar never learns what a resource or an asset is.

func SourceFromAssetRecord 

func SourceFromAssetRecord(rec Record) Source

SourceFromAssetRecord builds a portal-asset source.

The kind is what separates it from SourceFromResource, and it is not a detail: it selects which object store the file is read from and which rows a delete sweeps.

func SourceFromResource 

func SourceFromResource(rec Record) Source

SourceFromResource builds a managed-resource source: its head key is the current revision, and its directory holds only that file.

type Store 

type Store interface {
	Insert(ctx context.Context, r Registration) error
	Get(ctx context.Context, id string) (*Registration, error)
	// ByName returns the registration holding a name on a connection, or nil
	// when the name is free.
	ByName(ctx context.Context, connection, catalog, schema, table string) (*Registration, error)
	// BySource returns every registration of one resource or asset.
	BySource(ctx context.Context, kind, sourceID string) ([]Registration, error)
	// ForSources returns the registrations of many sources of one kind, keyed
	// by source id. It is the read a list view and a search result set use, so
	// a page of hits costs one query rather than one per hit.
	ForSources(ctx context.Context, kind string, sourceIDs []string) (map[string][]Registration, error)
	Delete(ctx context.Context, id string) error
}

Store persists registrations.

Insert is expected to fail when the name is already claimed; the registrar turns that into a refusal naming the holder rather than silently replacing a table someone else registered.

func NewPostgresStore 

func NewPostgresStore(db *sql.DB) Store

NewPostgresStore creates a registration store backed by PostgreSQL.

type Subject 

type Subject func(ctx context.Context, id string, caller Caller) (Source, bool)

Subject resolves the record an id names into what the registrar needs, and decides whether this caller may act on it. Returning ok=false means the caller may not act on the record at all, which every surface answers as a not-found so none of them reveals a record the caller cannot reach.

One resolver per kind serves both surfaces: the REST routes convert their authenticated portal user into a Caller and the tool reads one from the platform context, so the authorization rule for a kind is written once and cannot drift between the two doors.

type ToolAdapter 

type ToolAdapter struct {
	// contains filtered or unexported fields
}

ToolAdapter satisfies the asset toolkit's TableRegistrar over a Registrar, keyed by the canonical reference a caller already holds rather than by the id of one kind of record.

It exists because the tool's contract carries no caller: the acting identity belongs on the PlatformContext the middleware chain put in the request's context, and reading it here rather than taking it as an argument is what keeps a tool call from presenting an identity the rest of the platform would refuse.

func NewToolAdapter 

func NewToolAdapter(reg *Registrar, adminRoles []string, subjects map[string]Subject) *ToolAdapter

NewToolAdapter adapts a Registrar for the table tool. A nil or unwired Registrar, or one with no kind to resolve, yields nil, which the toolkit renders as "this deployment cannot register tables" rather than as a failure.

func (*ToolAdapter) DropAssetTables 

func (a *ToolAdapter) DropAssetTables(ctx context.Context, assetID string)

DropAssetTables removes every table registered over a deleted asset.

func (*ToolAdapter) Register 

func (a *ToolAdapter) Register(
	ctx context.Context, reference, connection, tableName string,
) (*portaltoolkit.TableRegistration, error)

Register registers a table over the current content of the file a reference names.

func (*ToolAdapter) Tables 

func (a *ToolAdapter) Tables(ctx context.Context, reference string) ([]portaltoolkit.TableRegistration, error)

Tables reports what is registered over the file a reference names.

func (*ToolAdapter) Unregister 

func (a *ToolAdapter) Unregister(ctx context.Context, registrationID string) error

Unregister drops a registered table.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.