Documentation
¶
Overview ¶
Package query is the query commands: query runs a query, query.compile compiles one to SQL, and query.validate checks one. What they share, and only they, stays in here.
Index ¶
- Constants
- Variables
- type Column
- type CompiledQuery
- type ExportFormatOptions
- type ExportInput
- type QueryCompiled
- type QueryEvent
- type QueryHeader
- type QueryInput
- type QueryRecords
- type QueryResult
- type QueryRows
- type QueryRun
- type QueryRunInput
- type QuerySummary
- type QueryWarehouse
- type SemanticType
- type TimedStep
- type Timings
Constants ¶
const MaxPageSize = 10_000
MaxPageSize is the most rows a page holds: page_size's maximum, and the page a query without one is answered with when it is not streamed.
Variables ¶
var Compile = command.Define(command.Def[QueryRunInput, CompiledQuery]{ Name: "query.compile", Short: "Compile a query to SQL, without running it", ReadOnly: true, Timeout: 2 * time.Minute, Needs: func(in QueryRunInput) command.Sidecars { return command.Sidecars{Node: in.Lang != "sql"} }, Check: QueryRunInput.check, Errors: []apperr.AnyCode{query.QueryInvalid, errcode.DataPermsUnenforceable}, Run: func(ctx context.Context, cc command.CommandContext, in QueryRunInput) (CompiledQuery, error) { if in.Lang == "sql" { if err := in.checkDataSource(ctx, cc); err != nil { return CompiledQuery{}, err } return CompiledQuery{SQL: in.Query}, nil } aql, _, err := in.aql() if err != nil { return CompiledQuery{}, err } if err := command.RequireSidecars(cc, command.Sidecars{Node: true}); err != nil { return CompiledQuery{}, err } src, err := sourceOf(cc) if err != nil { return CompiledQuery{}, err } compiled, err := compileAQL(ctx, cc, src, in.Dataset, aql, in.run()) if err != nil { return CompiledQuery{}, err } return CompiledQuery{SQL: compiled.SQL, AQL: compiled.AQL, Columns: compiledColumns(compiled.Columns)}, nil }, })
Compile compiles a query to SQL, without running it.
var Export = command.Define(command.Def[ExportInput, command.ExportLink]{ Name: "query.export", Short: "Export a query's whole result as a file: a link to download it", Long: "Export a query's whole result as a file: a link to download it.\n\n" + "The query runs to completion before the answer, so a query that fails fails here. " + "The CLI downloads the file, to stdout or --output; --link prints the answer instead.", NonDestructive: true, Timeout: 30 * time.Minute, Needs: func(in ExportInput) command.Sidecars { return command.Sidecars{Node: in.Lang != "sql", CanalQuery: true} }, Check: ExportInput.check, Errors: []apperr.AnyCode{query.QueryInvalid, errcode.QueryFailed, errcode.DataPermsUnenforceable, errcode.ExportsUnavailable}, Run: runExport, })
Export runs a query and answers its whole result as a file: a link to download it. See docs/designs/exports.md.
var Query = command.Define(command.Def[QueryRunInput, QueryRun]{ Name: "query", Short: "Run a query: its rows, and the SQL that produced them", Long: `Answers one page of rows: page_size's, or without it the first 10000, with has_more saying whether there may be more. Ask the query for what you need (aggregate, filter, a limit: directive) rather than paging through rows; for every row of a large result, export it. Over HTTP, a request that accepts application/x-ndjson is answered as the query runs: its phase, its SQL, its columns, then its rows in batches — every row without page_size, up to the server's maximum.`, ReadOnly: true, Timeout: 10 * time.Minute, Needs: func(in QueryRunInput) command.Sidecars { return command.Sidecars{Node: in.Lang != "sql", CanalQuery: true} }, Check: QueryRunInput.check, Errors: []apperr.AnyCode{query.QueryInvalid, errcode.QueryFailed, errcode.DataPermsUnenforceable, query.ResultTooLarge, query.PageTooLarge}, Events: queryEvents, Run: newQueryRun, })
Query runs a query: its rows, and the SQL that produced them — as one page, or streamed as it runs (QueryRun).
var Validate = command.Define(command.Def[QueryInput, query.QueryValidation]{ Name: "query.validate", Short: "Validate a query: its diagnostics, without running it", ReadOnly: true, Timeout: 2 * time.Minute, Needs: func(QueryInput) command.Sidecars { return command.Sidecars{Node: true} }, Check: func(in QueryInput) error { if in.Lang == "sql" { return command.InvalidArg("lang", "unsupported", "validating a SQL query is not supported yet; you can still run it directly") } return in.target() }, Run: func(ctx context.Context, cc command.CommandContext, in QueryInput) (query.QueryValidation, error) { aql, _, err := in.aql() if err != nil { return query.QueryValidation{}, err } if err := command.RequireSidecars(cc, command.Sidecars{Node: true}); err != nil { return query.QueryValidation{}, err } src, err := sourceOf(cc) if err != nil { return query.QueryValidation{}, err } return checkAQL(ctx, cc, src, in.Dataset, aql) }, Valid: func(r query.QueryValidation) bool { return r.Valid }, })
Validate checks a query: its diagnostics, without running it.
Functions ¶
This section is empty.
Types ¶
type Column ¶
type Column struct {
Name string `json:"name" doc:"the column's key, as in fields"`
FieldName string `json:"fieldName" doc:"the field it draws; for an adhoc column, its name"`
ModelID string `` /* 126-byte string literal not displayed */
Label string `json:"label"`
Adhoc bool `json:"adhoc" doc:"a query-local expression, not a field the dataset defines"`
IsMeasure bool `json:"isMeasure"`
Aggregation string `json:"aggregation,omitempty" doc:"an aggregated field's aggregation, such as sum or count distinct"`
SemanticType SemanticType `` /* 389-byte string literal not displayed */
ArrowType canalquery.ArrowType `` /* 135-byte string literal not displayed */
DBType string `` /* 180-byte string literal not displayed */
}
Column is one column of a query's answer: what it draws from the dataset, as anfra-node describes it, in amql's names (camelCase); and its types, which the core adds, in the API's (snake_case). See docs/designs/result-values.md.
type CompiledQuery ¶
type CompiledQuery struct {
SQL string `json:"sql"`
AQL string `json:"aql,omitempty" doc:"the AQL compiled: the query with its Query Input applied; absent for a SQL query"`
Columns []Column `` /* 133-byte string literal not displayed */
}
CompiledQuery is the `query compile` result: the SQL the query compiles to.
type ExportFormatOptions ¶
type ExportFormatOptions struct {
Header string `` /* 203-byte string literal not displayed */
BOM bool `` /* 155-byte string literal not displayed */
}
ExportFormatOptions are a format's options. Each applies to the formats it names; given to another, it is refused (check).
type ExportInput ¶
type ExportInput struct {
QueryInput
Input anfranode.QueryTransforms `` /* 126-byte string literal not displayed */
Timezone string `json:"timezone,omitempty" doc:"the IANA time zone relative dates and date truncation use, such as Asia/Ho_Chi_Minh"`
Format string `json:"format,omitempty" enum:"csv" default:"csv" doc:"the file's format"`
FormatOptions ExportFormatOptions `json:"format_options,omitempty" doc:"the format's options"`
Filename string `` /* 181-byte string literal not displayed */
}
ExportInput is what query.export takes: a query and what shapes its run, as query takes them, but no page, since an export is the whole result; and the file's format.
func (ExportInput) TransformSchema ¶
type QueryCompiled ¶ added in v0.11.1
type QueryCompiled struct {
Type string `json:"type" enum:"compiled"`
SQL string `json:"sql"`
AQL string `json:"aql,omitempty" doc:"the AQL that runs: the query with its Query Input applied; absent for a SQL query"`
}
QueryCompiled is the SQL the query runs, as soon as it is known.
type QueryEvent ¶ added in v0.11.1
type QueryEvent interface {
// contains filtered or unexported methods
}
QueryEvent is one of query's own events, in a stream: compiled, then warehouse_query once the warehouse has the query, then header, then rows in batches, then summary. Which phase the query is in — compiling_query, then canal-query's connecting_to_dw, executing_query, fetching_query_result, and retrying_query — is the stream's status.
type QueryHeader ¶ added in v0.11.1
type QueryHeader struct {
Type string `json:"type" enum:"header"`
Columns []Column `json:"columns" doc:"one per field, in order"`
Fields []string `json:"fields"`
}
QueryHeader is what each column of the result is, once the result's shape is known: before its rows.
type QueryInput ¶
type QueryInput struct {
Query string `json:"query" cli:"positional,stdin" doc:"the query"`
Lang string `json:"lang,omitempty" short:"l" enum:"aql,sql" default:"aql" doc:"the language the query is written in"`
Dataset string `json:"dataset,omitempty" short:"d" doc:"the dataset an AQL query runs against"`
DataSource string `json:"data_source,omitempty" short:"s" doc:"the data source a SQL query runs against"`
}
QueryInput is a query: its text, the language it is written in, and the target it runs against. Which target a language runs against is target's to say: AQL against a dataset, and SQL against a data source.
func (QueryInput) TransformSchema ¶
TransformSchema publishes the required query (command.ArgsSchema). The target is not a group in the schema: which one is required depends on lang, which a group cannot say, so target checks it.
type QueryRecords ¶ added in v0.11.1
type QueryRecords struct {
Type string `json:"type" enum:"rows"`
Records []canalquery.Row `json:"records" doc:"one array per row, its values in fields' order, written as a page's records are"`
}
QueryRecords is a batch of the result's rows, as a page's records are.
type QueryResult ¶
type QueryResult struct {
SQL string `json:"sql"`
AQL string `json:"aql,omitempty" doc:"the AQL that ran: the query with its Query Input applied; absent for a SQL query"`
Columns []Column `json:"columns" doc:"one per field of result, in order"`
Result QueryRows `json:"result"`
HasMore bool `` /* 133-byte string literal not displayed */
Timings Timings `` /* 253-byte string literal not displayed */
}
QueryResult is the `query` result answered whole: the SQL that ran, the AQL it was compiled from, what each column is, and one page of rows — page_size's, or the first MaxPageSize without one.
type QueryRows ¶
type QueryRows struct {
Fields []string `json:"fields"`
Records []canalquery.Row `` /* 309-byte string literal not displayed */
}
type QueryRun ¶ added in v0.11.1
type QueryRun = apikit.Run[QueryResult, QueryEvent]
QueryRun is how query answers: as a page (QueryResult), or as a stream of its events (QueryEvent).
type QueryRunInput ¶
type QueryRunInput struct {
QueryInput
Input anfranode.QueryTransforms `` /* 126-byte string literal not displayed */
Page int `json:"page,omitempty" minimum:"1" doc:"the 1-based page of rows to answer; needs a page size"`
PageSize int `json:"page_size,omitempty" minimum:"1" maximum:"10000" doc:"rows per page; alone, the first page"`
Timezone string `json:"timezone,omitempty" doc:"the IANA time zone relative dates and date truncation use, such as Asia/Ho_Chi_Minh"`
}
QueryRunInput is what query and query.compile take: a query, and what shapes its run. query.validate takes the query alone (QueryInput), since none of it applies to checking one.
func (QueryRunInput) TransformSchema ¶
type QuerySummary ¶ added in v0.11.1
type QuerySummary struct {
Type string `json:"type" enum:"summary"`
RowCount int `json:"row_count"`
HasMore bool `` /* 139-byte string literal not displayed */
// BytesScanned is what the warehouse read to run the query, as it reports
// it: BigQuery and Athena.
BytesScanned *int64 `json:"bytes_scanned,omitempty" doc:"the bytes the warehouse read to run the query, where it reports them (BigQuery, Athena)"`
Timings Timings `` /* 253-byte string literal not displayed */
}
QuerySummary ends the result: how many rows it held, and, for a page, whether there are more.
type QueryWarehouse ¶ added in v0.11.1
type QueryWarehouse struct {
Type string `json:"type" enum:"warehouse_query"`
ID string `json:"id" doc:"the warehouse's own id of the query; for Postgres, the process id of the backend running it"`
}
QueryWarehouse is the query's id on the warehouse, as soon as it has one: to find it there — in its query history, its billing — or to cancel it there.
type SemanticType ¶
type SemanticType string
SemanticType is what a column's values mean: AML's types, plus bit.
const ( SemanticText SemanticType = "text" SemanticNumber SemanticType = "number" SemanticDate SemanticType = "date" SemanticDatetime SemanticType = "datetime" SemanticTruefalse SemanticType = "truefalse" SemanticJSON SemanticType = "json" SemanticBit SemanticType = "bit" SemanticUnknown SemanticType = "unknown" )