datasource

package
v0.22.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 18, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package datasource reads a repo's data_sources.yml. The host owns this (it holds connection config / credentials and runs queries); only the name->dbtype mapping is ever passed to the sidecar for SQL-dialect selection, so secrets never cross the IPC boundary.

Index

Constants

View Source
const FileName = "data_sources.yml"

FileName is the data source manifest, inside the repo's config dir (repo.Repo.ConfigDir()).

Variables

View Source
var ErrUnknown = errors.New("no such data source")

ErrUnknown is Connection's answer for a name the repo has no data source by.

Functions

func ConnectionIdentity added in v0.21.0

func ConnectionIdentity(connection map[string]any) (string, error)

ConnectionIdentity is an identity for a data source whose access is all in its connection: the connection hashed, credentials included, so that another user, password, host or project is another identity. Only the hash is kept. The map's key order doesn't change it.

Types

type DataPermMode added in v0.11.1

type DataPermMode int

DataPermMode is how the SQL a connection carries reads the data: a fact about the statement, which the engine states and a host may check, never a decision. Every statement the engine runs is one of the two (internal/query). The zero value is no mode, which a host refuses.

const (
	// DataPermModeGoverned: SQL the engine compiled from a dataset, with the
	// invocation's DataPerms in it.
	DataPermModeGoverned DataPermMode = iota + 1
	// DataPermModeUngoverned: SQL as written, which nothing applies DataPerms
	// to; run only for an invocation whose DataPerms are Ungoverned.
	DataPermModeUngoverned
)

func (DataPermMode) String added in v0.11.1

func (m DataPermMode) String() string

String is the mode as logs name it.

type DataSource

type DataSource struct {
	Name       string         `json:"name"`
	DBType     string         `json:"dbtype"`
	Connection map[string]any `json:"-"`
	Settings   map[string]any `json:"-"`
	Unpooled   bool           `json:"-"`
	// Identity is who the warehouse sees when a query runs on it, as the query
	// cache's key holds it (docs/designs/query-cache.md#keys). Whoever loads
	// the data sources sets it, and changes it whenever access changes:
	// another warehouse user, role, project, or the viewer under OAuth
	// passthrough. It never holds a credential. Load sets ConnectionIdentity's.
	// A platform sets one stable across what doesn't change access, such as a
	// rotated secret. Empty, a query on it runs uncached.
	//
	// It is a field, not a method, because only whoever returns the connection
	// knows who it connects as: DataSources.Connection, in the same call, for
	// the caller in its ctx. A method could only hash the fields here, and a
	// separate call could disagree with the connection it describes.
	Identity string `json:"-"`
}

DataSource describes one data source. Name/DBType are what the sidecar needs to compile SQL (dialect). Connection holds the host-side connection config (passed to canal-query as dbconfig) and is never serialized to the sidecar. Settings are how canal-query connects and queries (timeouts, SSL, the type's own options), passed to it as dbsetting; none leaves each at canal's default. Unpooled has canal-query open a connection for each statement and close it after, rather than keep it in a pool: for a data source used once, such as a connection being checked, which a pool would outlive by hours.

type DataSources added in v0.9.0

type DataSources interface {
	// Dialects is the data sources the caller may compile against, by name and
	// type.
	Dialects(ctx context.Context) ([]Dialect, error)
	// Connection is the data source by name, connection included, for SQL that
	// reads the data as mode says. A host may refuse a mode it did not grant the
	// call on that data source: a second gate, behind its authorization and the
	// invocation's DataPerms, which must be right on their own. A name the repo
	// has no data source by is ErrUnknown; any other error is the host's, and
	// reaches the caller as it is.
	Connection(ctx context.Context, name string, mode DataPermMode) (DataSource, error)
}

DataSources is how the engine reaches a repo's data sources, asking for no more than a command needs: every data source's dialect, to compile (the engine cannot know which one a dataset uses until it is compiled); one connection, to run a query on the data source it targets. validate asks for neither.

A host serving a managed repo implements it over its own store: Dialects is metadata, and Connection is where one data source's connection is retrieved. A local repo's is its data_sources.yml (Load).

type Dialect added in v0.9.0

type Dialect struct {
	Name   string
	DBType string
}

Dialect is a data source as compiling needs it: its name and its type (the SQL dialect). No connection.

type Set added in v0.9.0

type Set map[string]DataSource

Set is data sources held in memory, by name: what data_sources.yml loads into, or what a host that already holds them all passes. It is a DataSources.

func Load

func Load(configDir string) (Set, error)

Load reads <configDir>/data_sources.yml into a Set.

func (Set) Connection added in v0.9.0

func (s Set) Connection(_ context.Context, name string, _ DataPermMode) (DataSource, error)

Connection is the data source named name in s, whatever the mode: a Set holds no grants.

func (Set) Dialects added in v0.9.0

func (s Set) Dialects(context.Context) ([]Dialect, error)

Dialects is every data source in s, sorted by name.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL