query

package
v0.24.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 20, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package query is the query commands: query runs a query, query.compile compiles one to SQL, and query.validate checks one. What they share, and only they, stays in here.

Index

Constants

View Source
const MaxPageSize = 10_000

MaxPageSize is the most rows a page holds: page_size's maximum, and the page a query without one is answered with when it is not streamed.

Variables

View Source
var Compile = command.Define(command.Def[QueryCompileRequestParams, CompiledQuery]{
	Name:     "query.compile",
	Short:    "Compile a query to SQL, without running it",
	ReadOnly: true,
	Timeout:  2 * time.Minute,
	Needs:    func(in QueryCompileRequestParams) command.Sidecars { return command.Sidecars{Node: in.Lang != "sql"} },
	Check:    func(in QueryCompileRequestParams) error { return in.queryParams().check() },
	Errors:   []apperr.AnyCode{query.QueryInvalid, errcode.DataPermsUnenforceable, errcode.RepoSettingsInvalid},
	Run: func(ctx context.Context, cc command.CommandContext, raw QueryCompileRequestParams) (CompiledQuery, error) {
		in, err := raw.queryParams().withRepoDefaults(cc)
		if err != nil {
			return CompiledQuery{}, err
		}
		if in.Lang == "sql" {

			if err := in.checkDataSource(ctx, cc); err != nil {
				return CompiledQuery{}, err
			}
			return CompiledQuery{SQL: in.Query}, nil
		}
		aql, _, err := in.aql()
		if err != nil {
			return CompiledQuery{}, err
		}
		if err := command.RequireSidecars(cc, command.Sidecars{Node: true}); err != nil {
			return CompiledQuery{}, err
		}
		src, err := sourceOf(cc)
		if err != nil {
			return CompiledQuery{}, err
		}
		compiled, err := compileAQL(ctx, cc, src, in.Dataset, aql, in.run())
		if err != nil {
			return CompiledQuery{}, err
		}
		return CompiledQuery{SQL: compiled.SQL, AQL: compiled.AQL, Input: in.Input.filled(compiled.Seeds), Columns: compiledColumns(compiled.Columns)}, nil
	},
})

Compile compiles a query to SQL, without running it.

View Source
var Export = command.Define(command.Def[QueryExportRequestParams, QueryExportResult]{
	Name:  "query.export",
	Short: "Export a query's whole result as a file to download",
	Long:  "Export a query's whole result as a file, and return a link to download it once the file is ready.",

	NonDestructive: true,
	Timeout:        30 * time.Minute,
	Needs: func(in QueryExportRequestParams) command.Sidecars {
		return command.Sidecars{Node: in.Lang != "sql", CanalQuery: true}
	},
	Check:  QueryExportRequestParams.check,
	Errors: []apperr.AnyCode{query.QueryInvalid, errcode.QueryFailed, errcode.DataPermsUnenforceable, errcode.ExportsUnavailable, errcode.RepoSettingsInvalid},
	Run:    runExport,
})

Export runs a query and answers its whole result as a file: a link to download it. See docs/designs/exports.md.

View Source
var Query = command.Define(command.Def[QueryRequestParams, QueryRun]{
	Name:  "query",
	Short: "Run a query, and return its rows and the SQL that produced them",
	Long: `Run a query, and return a page of its rows and the SQL that produced them.

A page is input.pagination.pageSize rows, or the first 10000 without it, and has_more says whether there may be more. Rather than paging through many rows, ask the query for what you need: an aggregate, a filter, or a limit: directive. To get every row of a large result, export it.

Over HTTP, a request that accepts application/x-ndjson gets the result as the query runs: its phase, its SQL, its columns, then its rows in batches. Without input.pagination, that's every row, up to the server's maximum.`,
	ReadOnly: true,
	Timeout:  10 * time.Minute,
	Needs: func(in QueryRequestParams) command.Sidecars {
		return command.Sidecars{Node: in.Lang != "sql", CanalQuery: true}
	},
	Normalize: QueryRequestParams.normalize,
	Check:     QueryRequestParams.check,
	Errors: []apperr.AnyCode{query.QueryInvalid, errcode.QueryFailed, errcode.DataPermsUnenforceable, query.ResultTooLarge, query.PageTooLarge,
		errcode.RepoSettingsInvalid,
		errcode.CacheMiss, errcode.CacheDisabled, errcode.CacheUnavailable},
	Events: queryEvents,
	Run:    newQueryRun,
})

Query runs a query: its rows, and the SQL that produced them — as one page, or streamed as it runs (QueryRun).

View Source
var Validate = command.Define(command.Def[QueryDefinition, query.QueryValidation]{
	Name:     "query.validate",
	Short:    "Validate a query without running it, and return its diagnostics",
	ReadOnly: true,
	Timeout:  2 * time.Minute,
	Needs:    func(QueryDefinition) command.Sidecars { return command.Sidecars{Node: true} },
	Check: func(in QueryDefinition) error {
		if in.Lang == "sql" {

			return command.InvalidArg("lang", "unsupported", "validating a SQL query is not supported yet")
		}
		return in.target()
	},
	Run: func(ctx context.Context, cc command.CommandContext, in QueryDefinition) (query.QueryValidation, error) {
		aql, _, err := in.aql()
		if err != nil {
			return query.QueryValidation{}, err
		}
		if err := command.RequireSidecars(cc, command.Sidecars{Node: true}); err != nil {
			return query.QueryValidation{}, err
		}
		src, err := sourceOf(cc)
		if err != nil {
			return query.QueryValidation{}, err
		}
		return checkAQL(ctx, cc, src, in.Dataset, aql)
	},
	Valid: func(r query.QueryValidation) bool { return r.Valid },
})

Validate checks a query: its diagnostics, without running it.

Functions

This section is empty.

Types

type CacheOptions added in v0.21.0

type CacheOptions struct {
	Duration   string `` /* 250-byte string literal not displayed */
	Refresh    bool   `json:"refresh,omitempty" doc:"true runs the query on the warehouse whatever is cached, and caches its result."`
	ReadOnly   bool   `` /* 140-byte string literal not displayed */
	CachedOnly bool   `json:"cached_only,omitempty" doc:"true answers from a cached result, or fails with cache_miss without running the query."`
	// MinCachedAt keeps a run's later pages and its export from being older
	// than its first page (docs/designs/query-cache.md#a-runs-later-pages).
	MinCachedAt string `` /* 352-byte string literal not displayed */
}

CacheOptions are how the query cache answers a request (docs/designs/query-cache.md#a-querys-input).

type CacheResult added in v0.21.0

type CacheResult struct {
	Key       string       `json:"key,omitempty" doc:"the query's key in the cache. Absent when the cache couldn't make one."`
	Hit       bool         `json:"hit" doc:"whether the rows came from the cache. false: they came from the warehouse."`
	CachedAt  time.Time    `json:"cached_at,omitzero" doc:"when the cached rows were read from the warehouse. Absent when the result wasn't cached."`
	ExpiresAt time.Time    `json:"expires_at,omitzero" doc:"when the cached result stops being used. Absent when the result wasn't cached."`
	Warning   CacheWarning `json:"warning,omitzero" doc:"why the query ran without the cache, though it asked for it"`
}

CacheResult is what the query cache did for a query (docs/designs/query-cache.md#a-querys-answer).

type CacheWarning added in v0.21.0

type CacheWarning struct {
	Code    string `` /* 165-byte string literal not displayed */
	Message string `json:"message"`
}

CacheWarning is why a query that asked for the cache ran without it. It doesn't fail the query.

type Column

type Column struct {
	Name        string   `json:"name" doc:"the column's key, as in fields"`
	FieldName   string   `json:"fieldName" doc:"the field it draws, or for an adhoc column, its name"`
	ModelID     string   `` /* 127-byte string literal not displayed */
	Label       string   `json:"label"`
	Adhoc       bool     `json:"adhoc" doc:"a query-local expression, not a field the dataset defines"`
	IsMeasure   bool     `json:"isMeasure"`
	Aggregation string   `json:"aggregation,omitempty" doc:"an aggregated field's aggregation, such as sum or count distinct"`
	Models      []string `json:"models,omitempty" doc:"the models a measure aggregates. Absent for a dimension."`
	Expression  string   `json:"expression,omitempty" doc:"a measure's AQL, as the query wrote it. Absent for a dimension."`

	SemanticType SemanticType         `` /* 482-byte string literal not displayed */
	ArrowType    canalquery.ArrowType `` /* 126-byte string literal not displayed */
	DBType       string               `` /* 197-byte string literal not displayed */
}

Column is one column of a query's answer: what it draws from the dataset, as anfra-node describes it, in amql's names (camelCase); and its types, which the core adds, in the API's (snake_case). See docs/designs/result-values.md.

type ColumnPage added in v0.21.0

type ColumnPage struct {
	Page     int `json:"page,omitempty" minimum:"1" doc:"the 1-based page of columns to return, 1 by default"`
	PageSize int `json:"pageSize" minimum:"1" maximum:"10000" doc:"how many of the grid's columns a page has"`
}

ColumnPage is one page of a pivot's columns.

type CompiledQuery

type CompiledQuery struct {
	SQL     string      `json:"sql"`
	AQL     string      `json:"aql,omitempty" doc:"the final AQL (with the Query Input applied). Absent for a SQL query."`
	Input   *QueryInput `` /* 198-byte string literal not displayed */
	Columns []Column    `` /* 135-byte string literal not displayed */
}

CompiledQuery is the `query compile` result: the SQL the query compiles to.

type ExportFormatOptions

type ExportFormatOptions struct {
	Header string `` /* 217-byte string literal not displayed */
	BOM    bool   `` /* 155-byte string literal not displayed */
}

ExportFormatOptions are a format's options. Each applies to the formats it names; given to another, it is refused (check).

type Pagination added in v0.21.0

type Pagination struct {
	Page     int         `json:"page,omitempty" minimum:"1" doc:"the 1-based page of rows to return, 1 by default"`
	PageSize int         `` /* 184-byte string literal not displayed */
	Columns  *ColumnPage `` /* 141-byte string literal not displayed */
}

Pagination is one page of a query's rows: of its result's, or of a pivot's grid, and of its columns too.

type PivotPage added in v0.21.0

type PivotPage struct {
	Rows           int  `json:"rows" doc:"how many rows the pivot's grid has, in all of its pages, the grand total not counted"`
	Columns        int  `json:"columns" doc:"how many columns the pivot's grid has, the total column not counted"`
	HasMoreColumns bool `json:"has_more_columns,omitempty" doc:"with input.pagination.columns: whether there are columns after this page's"`
}

PivotPage is a page of a pivot, as its answer says it: how big the pivot's grid is, so a caller can number its pages, and whether a page of columns has more after it.

type QueryCompileRequestParams added in v0.21.0

type QueryCompileRequestParams struct {
	QueryDefinition
	Input QueryInput `` /* 312-byte string literal not displayed */

	// These three moved into input, and are kept until they're removed. A
	// request that sets one both ways takes input's.
	Page     int    `` /* 134-byte string literal not displayed */
	PageSize int    `` /* 159-byte string literal not displayed */
	Timezone string `json:"timezone,omitempty" deprecated:"true" doc:"deprecated: use input.timezone. Ignored when input.timezone is set."`
}

QueryCompileRequestParams is what query.compile takes: a query and its input, as query takes them, without the cache, since compiling runs nothing.

func (QueryCompileRequestParams) TransformSchema added in v0.21.0

func (QueryCompileRequestParams) TransformSchema(_ huma.Registry, s *huma.Schema) *huma.Schema

type QueryCompiled added in v0.11.1

type QueryCompiled struct {
	Type  string      `json:"type" enum:"compiled"`
	SQL   string      `json:"sql"`
	AQL   string      `json:"aql,omitempty" doc:"the final AQL (with the Query Input applied). Absent for a SQL query."`
	Input *QueryInput `` /* 190-byte string literal not displayed */
}

QueryCompiled is the SQL the query runs, as soon as it is known.

type QueryDefinition added in v0.21.0

type QueryDefinition struct {
	Query      string `json:"query" cli:"positional,stdin" doc:"the query"`
	Lang       string `json:"lang,omitempty" short:"l" enum:"aql,sql" default:"aql" doc:"the language the query is written in"`
	Dataset    string `json:"dataset,omitempty" short:"d" doc:"the dataset an AQL query runs against"`
	DataSource string `json:"data_source,omitempty" short:"s" doc:"the data source a SQL query runs against"`
}

QueryDefinition is a query: its text, the language it is written in, and the target it runs against. Which target a language runs against is target's to say: AQL against a dataset, and SQL against a data source.

func (QueryDefinition) TransformSchema added in v0.21.0

func (QueryDefinition) TransformSchema(_ huma.Registry, s *huma.Schema) *huma.Schema

TransformSchema publishes the required query (command.ArgsSchema). The target is not a group in the schema: which one is required depends on lang, which a group cannot say, so target checks it.

type QueryEvent added in v0.11.1

type QueryEvent interface {
	// contains filtered or unexported methods
}

QueryEvent is one of query's own events, in a stream: compiled, then warehouse_query once the warehouse has the query, then header, then rows in batches, then summary. Which phase the query is in — compiling_query, then canal-query's connecting_to_dw, executing_query, fetching_query_result, and retrying_query — is the stream's status.

type QueryExportRequestParams added in v0.21.0

type QueryExportRequestParams struct {
	QueryDefinition
	Input         QueryInput          `` /* 276-byte string literal not displayed */
	Timezone      string              `json:"timezone,omitempty" deprecated:"true" doc:"deprecated: use input.timezone. Ignored when input.timezone is set."`
	Format        string              `json:"format,omitempty" enum:"csv" default:"csv" doc:"the file's format"`
	FormatOptions ExportFormatOptions `json:"format_options,omitempty" doc:"the format's options"`
	Filename      string              `` /* 181-byte string literal not displayed */
	Cache         CacheOptions        `` /* 354-byte string literal not displayed */
}

QueryExportRequestParams is what query.export takes: a query and its input, as query takes them (a page among it, which exports that page); and the file's format.

func (QueryExportRequestParams) TransformSchema added in v0.21.0

func (QueryExportRequestParams) TransformSchema(_ huma.Registry, s *huma.Schema) *huma.Schema

type QueryExportResult added in v0.21.0

type QueryExportResult struct {
	command.ExportLink
	Cache CacheResult `` /* 279-byte string literal not displayed */
}

QueryExportResult is query.export's answer: its file's link, and what the query cache did for its query, as a query's answer says it.

func (QueryExportResult) Warnings added in v0.21.0

func (r QueryExportResult) Warnings() []command.AnswerWarning

Warnings are the export's warnings: its query's cache's, when it ran without the cache it asked for.

type QueryHeader added in v0.11.1

type QueryHeader struct {
	Type    string   `json:"type" enum:"header"`
	Columns []Column `json:"columns" doc:"one per field, in order"`
	Fields  []string `json:"fields"`
}

QueryHeader is what each column of the result is, once the result's shape is known: before its rows.

type QueryInput

type QueryInput struct {
	anfranode.QueryTransforms
	Pagination *Pagination `` /* 156-byte string literal not displayed */
	Timezone   string      `` /* 186-byte string literal not displayed */
	// WeekStartDay's values are anfranode.WeekStartDays: a tag can't name
	// them, so a test holds the two the same.
	WeekStartDay string `` /* 285-byte string literal not displayed */
	Seeds        *Seeds `` /* 269-byte string literal not displayed */
}

QueryInput is a query's input: everything a caller chooses about running a query, other than its text. Kept in one object, in camelCase, so a caller runs the query again (another page, an export, a replay) by sending the input its answer returned (docs/designs/query-input.md).

func (QueryInput) IsZero added in v0.21.0

func (in QueryInput) IsZero() bool

IsZero reports an input that changes nothing about a query.

type QueryRecords added in v0.11.1

type QueryRecords struct {
	Type    string           `json:"type" enum:"rows"`
	Records []canalquery.Row `json:"records" doc:"one array per row, its values in fields' order, written as a page's records are"`
}

QueryRecords is a batch of the result's rows, as a page's records are.

type QueryRequestParams added in v0.21.0

type QueryRequestParams struct {
	QueryDefinition
	Input QueryInput `` /* 310-byte string literal not displayed */

	// These three moved into input, and are kept until they're removed. A
	// request that sets one both ways takes input's.
	Page     int    `` /* 134-byte string literal not displayed */
	PageSize int    `` /* 159-byte string literal not displayed */
	Timezone string `json:"timezone,omitempty" deprecated:"true" doc:"deprecated: use input.timezone. Ignored when input.timezone is set."`

	Cache CacheOptions `` /* 228-byte string literal not displayed */
	// contains filtered or unexported fields
}

QueryRequestParams is what query takes: a query, its input, and how the query cache answers it. query.compile takes the same but the cache (QueryCompileRequestParams), and query.validate the query alone (QueryDefinition), since none of the input applies to checking one.

func (QueryRequestParams) TransformSchema added in v0.21.0

func (QueryRequestParams) TransformSchema(_ huma.Registry, s *huma.Schema) *huma.Schema

type QueryResult

type QueryResult struct {
	SQL     string      `json:"sql" pretty:"-"`
	AQL     string      `json:"aql,omitempty" pretty:"-" doc:"the final AQL (with the Query Input applied). Absent for a SQL query."`
	Input   *QueryInput `` /* 200-byte string literal not displayed */
	Columns []Column    `json:"columns" pretty:"-" doc:"one per field of result, in order"`
	Result  QueryRows   `json:"result" pretty:",inline"`
	HasMore bool        `` /* 149-byte string literal not displayed */
	Pivot   *PivotPage  `` /* 145-byte string literal not displayed */
	Timings Timings     `` /* 341-byte string literal not displayed */
	Cache   CacheResult `` /* 182-byte string literal not displayed */
}

QueryResult is the `query` result answered whole: the SQL that ran, the AQL it was compiled from, what each column is, and one page of rows — page_size's, or the first MaxPageSize without one.

Its pretty view (docs/designs/cli-pretty.md) is its rows, labelled by each column's label, then has_more and how long each step took; the SQL, the AQL and what each column is are for --yaml.

func (QueryResult) Warnings added in v0.21.0

func (r QueryResult) Warnings() []command.AnswerWarning

Warnings are the query's warnings: its cache's, when it ran without the cache it asked for.

type QueryRows

type QueryRows struct {
	Fields  []string         `json:"fields" pretty:"-"`
	Records []canalquery.Row `` /* 396-byte string literal not displayed */
}

type QueryRun added in v0.11.1

type QueryRun = apikit.Run[QueryResult, QueryEvent]

QueryRun is how query answers: as a page (QueryResult), or as a stream of its events (QueryEvent).

type QuerySummary added in v0.11.1

type QuerySummary struct {
	Type     string     `json:"type" enum:"summary"`
	RowCount int        `json:"row_count"`
	HasMore  bool       `` /* 157-byte string literal not displayed */
	Pivot    *PivotPage `` /* 145-byte string literal not displayed */
	// BytesScanned is what the warehouse read to run the query, as it reports
	// it: BigQuery and Athena.
	BytesScanned *int64      `json:"bytes_scanned,omitempty" doc:"the bytes the warehouse read to run the query, where it reports them (BigQuery, Athena)"`
	Cache        CacheResult `` /* 171-byte string literal not displayed */
	Timings      Timings     `` /* 318-byte string literal not displayed */
}

QuerySummary ends the result: how many rows it held, and, for a page, whether there are more.

type QueryWarehouse added in v0.11.1

type QueryWarehouse struct {
	Type string `json:"type" enum:"warehouse_query"`
	ID   string `json:"id" doc:"the warehouse's own id of the query; for Postgres, the process id of the backend running it"`
}

QueryWarehouse is the query's id on the warehouse, as soon as it has one: to find it there — in its query history, its billing — or to cancel it there.

type Seeds added in v0.21.0

type Seeds struct {
	ReferenceTime string `` /* 157-byte string literal not displayed */
}

Seeds are a query's seeds: values generated for each request when unset.

type SemanticType

type SemanticType string

SemanticType is what a column's values mean: AML's types, plus bit.

const (
	SemanticText      SemanticType = "text"
	SemanticNumber    SemanticType = "number"
	SemanticDate      SemanticType = "date"
	SemanticDatetime  SemanticType = "datetime"
	SemanticTruefalse SemanticType = "truefalse"
	SemanticJSON      SemanticType = "json"
	SemanticBit       SemanticType = "bit"
	SemanticUnknown   SemanticType = "unknown"
)

func (SemanticType) Schema

Schema publishes the semantic types semanticTypes gives, and no other, so the enum can neither miss one the core answers nor list one it never does.

type TimedStep added in v0.11.1

type TimedStep = command.TimedStep

Timings and TimedStep are a run's timeline, command's (CommandContext's observers report them too), named here as the query's answer has always named them. A query's steps are documented on its answer's fields.

type Timings added in v0.11.1

type Timings = command.Timings

Timings and TimedStep are a run's timeline, command's (CommandContext's observers report them too), named here as the query's answer has always named them. A query's steps are documented on its answer's fields.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL