Documentation
¶
Overview ¶
Package query is the query use-case layer: it checks a (dataset, AQL) request, turns it into SQL and runs it, by loading the repo's data sources and driving the sidecars via their clients. What an invalid query is, and the code it fails with, are its own. It depends on the clients (not the process managers), so it works equally against host-spawned or external sidecars. The CLI command and the future HTTP/MCP handler are thin shells over these functions.
Index ¶
- Constants
- Variables
- func CheckUngoverned(perms dataperm.Set) error
- func Compile(ctx context.Context, node *anfranode.Client, src Source, dataset, aql string, ...) (anfranode.CompileToSQLResult, error)
- func CompileRequest(ctx context.Context, src Source, dataset, aql string) (anfranode.CompileToSQLRequest, error)
- func ExtractLimit(aql string) (string, int, error)
- func NewPageTooLarge(maxBytes int) error
- type DataSourceQuery
- func (q DataSourceQuery) DBType() string
- func (q DataSourceQuery) DataSource() string
- func (q DataSourceQuery) Identity() string
- func (q DataSourceQuery) SQL() string
- func (q DataSourceQuery) Stream(ctx context.Context, canal *canalquery.Client, bounds ResultBounds, ...) error
- func (q DataSourceQuery) WithLakeKey(key string) DataSourceQuery
- type Limits
- type PageLimit
- type QueryValidation
- type ResultBounds
- type ResultLimit
- type Run
- type Source
- type UngovernedSQL
Constants ¶
const DefaultMaxPageBytes = 16 << 20
DefaultMaxPageBytes is the most a page of a query's result may hold, in its rows' bytes, when the host sets no maximum (Limits.MaxPageBytes).
const DefaultMaxResultBytes = 256 << 20
DefaultMaxResultBytes is the most a query's result may hold, in bytes, when the host sets no maximum (Limits.MaxResultBytes).
const DefaultMaxResultRows = 1_000_000
DefaultMaxResultRows is the most rows a query's result may hold when the host sets no maximum (Limits.MaxResultRows).
const NoLimit = -1
NoLimit is the row limit meaning "no truncation" (canal's truncate_rows uses a negative value for this).
Variables ¶
var PageTooLarge = apperr.DefinePublicCodeWith[PageLimit](errcode.NS, "page_too_large", apperr.User,
"The page is larger than this server may return at once.")
PageTooLarge is a page of a query's result larger, in bytes, than the host answers at once: it fails rather than being cut short, so that page numbers stay exact. The maximum is in its details.
var QueryInvalid = apperr.DefinePublicCodeWith[QueryValidation](errcode.NS, "query_invalid", apperr.User, "The query is invalid.")
QueryInvalid is a query that cannot run or compile, with its diagnostics: the QueryValidation that says why. A command that judges the query answers the QueryValidation instead.
var ResultTooLarge = apperr.DefinePublicCodeWith[ResultLimit](errcode.NS, "result_too_large", apperr.User,
"The query's result is larger than this server allows.")
ResultTooLarge is a query whose result has more rows than the host allows: it fails rather than being cut short, since a chart drawn from part of a result says something the whole would not. The maximum is in its details.
var Unbounded = ResultBounds{Truncate: NoLimit}
Unbounded is every row, however many: for a result streamed to a file, which holds none of it.
Functions ¶
func CheckUngoverned ¶ added in v0.11.1
CheckUngoverned refuses SQL as written for a governed caller: nothing can govern it.
func Compile ¶
func Compile(ctx context.Context, node *anfranode.Client, src Source, dataset, aql string, run Run) (anfranode.CompileToSQLResult, error)
Compile compiles an AQL query against a dataset into SQL plus the data source it targets (dialect + execution routing), without executing. Shared by --generate and the run path so both fail identically on a bad query: with QueryInvalid and its diagnostics when the query does not compile. Any other refusal (a dataset that cannot be used, the run's shaping) is returned as it is, for the command to place on its arg (anfranode.Violation). The result's AQL is the query with run's Query Input applied.
func CompileRequest ¶
func CompileRequest(ctx context.Context, src Source, dataset, aql string) (anfranode.CompileToSQLRequest, error)
CompileRequest builds the sidecar compile request for a (dataset, aql) against src, with its data sources' dialects and no connection. Shared by SQL generation and AQL validation (both feed the same {repo, datasetFqn, aql, dataSources} to the sidecar).
func ExtractLimit ¶
ExtractLimit pulls an anfra-level `limit:` directive out of an AQL query and returns the query with the directive removed plus the row limit. The AQL engine does not support `limit`, so anfra strips it here and applies it at execution time via canal's truncate_rows. Returns NoLimit when no directive is present. TODO: remove this when AQL supports limit
func NewPageTooLarge ¶ added in v0.11.1
NewPageTooLarge is the error of a page past maxBytes.
Types ¶
type DataSourceQuery ¶ added in v0.21.0
type DataSourceQuery struct {
// contains filtered or unexported fields
}
DataSourceQuery is a query in its data source, ready to run there: its SQL, and the data source, connection included, as the engine's one call of DataSources.Connection gave it. What the query cache's key needs of the data source is read from it, so the key describes the connection the query runs on.
func CacheReadQuery ¶ added in v0.21.0
func CacheReadQuery(lakeKey string) DataSourceQuery
CacheReadQuery is a read of the query cache's entry at lakeKey, in canal-lake as its data source: through canal-query's canal_lake connector, which reads canal-lake as a data source, so a cached result is streamed and encoded as a warehouse's is, within the same bounds (docs/designs/query-cache.md#a-hit). Its rows are the entry's, as they were written.
func InDataSource ¶ added in v0.21.0
func InDataSource(ctx context.Context, sources datasource.DataSources, compiled anfranode.CompileToSQLResult) (DataSourceQuery, error)
InDataSource is compiled in the data source its dataset targets, asked of sources in the governed mode: the data source's name is the compiler's.
func (DataSourceQuery) DBType ¶ added in v0.21.0
func (q DataSourceQuery) DBType() string
DBType is the data source's type.
func (DataSourceQuery) DataSource ¶ added in v0.21.0
func (q DataSourceQuery) DataSource() string
DataSource is the name of the data source q runs on.
func (DataSourceQuery) Identity ¶ added in v0.21.0
func (q DataSourceQuery) Identity() string
Identity is the data source's identity: who the warehouse sees (datasource.DataSource.Identity).
func (DataSourceQuery) SQL ¶ added in v0.21.0
func (q DataSourceQuery) SQL() string
SQL is the SQL q runs.
func (DataSourceQuery) Stream ¶ added in v0.21.0
func (q DataSourceQuery) Stream(ctx context.Context, canal *canalquery.Client, bounds ResultBounds, h canalquery.Handler) error
Stream runs q, handing its rows within bounds to h as they arrive, as the package's Stream does.
func (DataSourceQuery) WithLakeKey ¶ added in v0.21.0
func (q DataSourceQuery) WithLakeKey(key string) DataSourceQuery
WithLakeKey is q also writing its result to canal-lake under key, as it streams it (canal-query's cache_key, canalquery.Statement.LakeKey).
type Limits ¶ added in v0.11.1
type Limits struct {
// MaxResultRows is the most rows a query's result may hold: a query that
// produces more fails with ResultTooLarge rather than being cut short. Zero:
// DefaultMaxResultRows.
MaxResultRows int
// MaxResultBytes is the most a query's result may hold, in bytes, as
// canal-query measures it while streaming it (each batch's Arrow data, about
// what its JSON takes; neither it nor the engine holds the result whole): a
// query that produces more fails with ResultTooLarge. It bounds how much one
// query moves, not memory. Zero: DefaultMaxResultBytes.
MaxResultBytes int64
// MaxPageBytes is the most a page answered whole may hold, in its rows'
// bytes: a page over it fails with PageTooLarge rather than being cut short,
// so that page numbers stay exact. Zero: DefaultMaxPageBytes.
MaxPageBytes int
}
Limits bound what a query may cost the host. The zero value is the defaults.
func (Limits) ResultBytes ¶ added in v0.11.1
ResultBytes is MaxResultBytes, or its default.
func (Limits) ResultRows ¶ added in v0.11.1
ResultRows is MaxResultRows, or its default.
type PageLimit ¶ added in v0.11.1
type PageLimit struct {
MaxBytes int `json:"max_bytes"`
}
PageLimit is the maximum a page went past.
type QueryValidation ¶
type QueryValidation struct {
// Valid: no diagnostic has "error" severity.
Valid bool `json:"valid"`
Diagnostics []anfranode.AQLDiagnostic `json:"diagnostics"`
}
QueryValidation is the outcome of validating a query: its verdict, and the type-check diagnostics for a single AQL query.
type ResultBounds ¶ added in v0.11.1
type ResultBounds struct {
Truncate int // NoLimit: every row the query produces
MaxRows int // zero: no maximum
MaxBytes int64 // zero: no maximum
}
ResultBounds bound how much of a query's result is fetched: up to Truncate rows, where the result is cut as the query asked (its limit: directive, or a page); and no more than MaxRows rows or MaxBytes bytes, past which the query fails with ResultTooLarge — the host's maximums (Limits), bounding what it holds.
func Bounded ¶ added in v0.11.1
func Bounded(truncate int, limits Limits) ResultBounds
Bounded are the bounds of the host's maximums (limits), the result truncated at truncate rows.
type ResultLimit ¶ added in v0.11.1
type ResultLimit struct {
MaxRows int `json:"max_rows,omitempty"`
MaxBytes int64 `json:"max_bytes,omitempty"`
}
ResultLimit is the maximum a result went past: its rows, or its bytes.
type Run ¶
type Run struct {
Input *anfranode.QueryTransforms
Pagination *anfranode.Pagination
Timezone string
WeekStartDay string
Seeds *anfranode.Seeds
}
Run is what shapes one run of a query beyond its AQL: its Query Input, applied to the AQL before it compiles; a page of rows (nil: every row); and the time zone relative dates and date truncation use ("": anfra-node's default); the day weeks start on ("": anfra-node's default, which differs by database); and its seeds (nil: generated by anfra-node).
type Source ¶ added in v0.9.0
type Source struct {
Repo repo.Repo
DataSources datasource.DataSources
}
Source is what a query compiles against and runs on: the repo's AML, and its data sources, whose dialects compiling asks for and one of whose connections a run asks for.
type UngovernedSQL ¶ added in v0.11.1
type UngovernedSQL struct{ DataSourceQuery }
UngovernedSQL is SQL as written, with the data source it runs on, connection included. Only InDataSourceUngoverned makes one.
func InDataSourceUngoverned ¶ added in v0.21.0
func InDataSourceUngoverned(ctx context.Context, perms dataperm.Set, sources datasource.DataSources, name, sql string) (UngovernedSQL, error)
InDataSourceUngoverned is sql, as written, in the data source named name: refused for a governed caller (CheckUngoverned), and asked of sources in the ungoverned mode. A name the repo has no data source by is datasource.ErrUnknown.