Documentation
¶
Index ¶
- type CancelStatementRequest
- type CancelStatementResponse
- type ChunkInfo
- type Client
- func (c *Client) CancelStatement(ctx context.Context, req CancelStatementRequest, opts ...call.Option) (*CancelStatementResponse, error)
- func (c *Client) ExecuteStatement(ctx context.Context, req ExecuteStatementRequest, opts ...call.Option) (*StatementResponse, error)
- func (c *Client) GetResultData(ctx context.Context, req GetResultDataRequest, opts ...call.Option) (*ResultData, error)
- func (c *Client) GetStatementResult(ctx context.Context, req GetStatementResultRequest, opts ...call.Option) (*StatementResponse, error)
- type ColumnInfo
- type ColumnTypeName
- type Disposition
- type ExecuteStatementRequest
- type ExternalLink
- type Format
- type GetResultDataRequest
- type GetStatementResultRequest
- type QueryTag
- type ResultData
- type ResultManifest
- type Schema
- type ServiceError
- type ServiceErrorCode
- type StatementParameter
- type StatementResponse
- type StatementStatus
- type StatementStatus_State
- type TimeoutAction
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CancelStatementRequest ¶
type CancelStatementRequest struct {
// The statement ID is returned upon successfully submitting a SQL statement,
// and is a required reference for all subsequent calls.
StatementId *string
}
type CancelStatementResponse ¶
type CancelStatementResponse struct {
}
type ChunkInfo ¶
type ChunkInfo struct {
// The position within the sequence of result set chunks.
ChunkIndex *int
// The starting row offset within the result set.
RowOffset *int64
// The number of rows within the result chunk.
RowCount *int64
// The number of bytes in the result chunk. This field is not available when
// using `INLINE` disposition.
ByteCount *int64
// When fetching, provides the `chunk_index` for the _next_ chunk. If absent,
// indicates there are no more chunks. The next chunk can be fetched with a
// :method:statementexecution/getstatementresultchunkn request.
NextChunkIndex *int
// When fetching, provides a link to fetch the _next_ chunk. If absent,
// indicates there are no more chunks. This link is an absolute `path` to be
// joined with your `$DATABRICKS_HOST`, and should be treated as an opaque link.
// This is an alternative to using `next_chunk_index`.
NextChunkInternalLink *string
}
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
func (*Client) CancelStatement ¶
func (c *Client) CancelStatement(ctx context.Context, req CancelStatementRequest, opts ...call.Option) (*CancelStatementResponse, error)
Requests that an executing statement be canceled. Callers must poll for status to see the terminal state. Cancel response is empty; receiving response indicates successful receipt.
func (*Client) ExecuteStatement ¶
func (c *Client) ExecuteStatement(ctx context.Context, req ExecuteStatementRequest, opts ...call.Option) (*StatementResponse, error)
Execute a SQL statement and optionally await its results for a specified time.
**Use case: small result sets with INLINE + JSON_ARRAY**
For flows that generate small and predictable result sets (<= 25 MiB), `INLINE` responses of `JSON_ARRAY` result data are typically the simplest way to execute and fetch result data.
**Use case: large result sets with EXTERNAL_LINKS**
Using `EXTERNAL_LINKS` to fetch result data allows you to fetch large result sets efficiently. The main differences from using `INLINE` disposition are that the result data is accessed with URLs, and that there are 3 supported formats: `JSON_ARRAY`, `ARROW_STREAM` and `CSV` compared to only `JSON_ARRAY` with `INLINE`.
** URLs**
External links point to data stored within your workspace's internal storage, in the form of a URL. The URLs are valid for only a short period, <= 15 minutes. Alongside each `external_link` is an expiration field indicating the time at which the URL is no longer valid. In `EXTERNAL_LINKS` mode, chunks can be resolved and fetched multiple times and in parallel.
----
### **Warning: Databricks strongly recommends that you protect the URLs that are returned by the `EXTERNAL_LINKS` disposition.**
When you use the `EXTERNAL_LINKS` disposition, a short-lived, URL is generated, which can be used to download the results directly from . As a short-lived is embedded in this URL, you should protect the URL.
Because URLs are already generated with embedded temporary s, you must not set an `Authorization` header in the download requests.
The `EXTERNAL_LINKS` disposition can be disabled upon request by creating a support case.
See also [Security best practices](/sql/admin/sql-execution-tutorial.html#security-best-practices).
----
StatementResponse contains `statement_id` and `status`; other fields might be absent or present depending on context. If the SQL warehouse fails to execute the provided statement, a 200 response is returned with `status.state` set to `FAILED` (in contrast to a failure when accepting the request, which results in a non-200 response). Details of the error can be found at `status.error` in case of execution failures.
func (*Client) GetResultData ¶
func (c *Client) GetResultData(ctx context.Context, req GetResultDataRequest, opts ...call.Option) (*ResultData, error)
After the statement execution has `SUCCEEDED`, this request can be used to fetch any chunk by index. Whereas the first chunk with `chunk_index=0` is typically fetched with :method:statementexecution/executeStatement or :method:statementexecution/getStatement, this request can be used to fetch subsequent chunks. The response structure is identical to the nested `result` element described in the :method:statementexecution/getStatement request, and similarly includes the `next_chunk_index` and `next_chunk_internal_link` fields for simple iteration through the result set. Depending on `disposition`, the response returns chunks of data either inline, or as links.
func (*Client) GetStatementResult ¶
func (c *Client) GetStatementResult(ctx context.Context, req GetStatementResultRequest, opts ...call.Option) (*StatementResponse, error)
This request can be used to poll for the statement's status. StatementResponse contains `statement_id` and `status`; other fields might be absent or present depending on context. When the `status.state` field is `SUCCEEDED` it will also return the result manifest and the first chunk of the result data. When the statement is in the terminal states `CANCELED`, `CLOSED` or `FAILED`, it returns HTTP 200 with the state set. After at least 12 hours in terminal state, the statement is removed from the warehouse and further calls will receive an HTTP 404 response.
**NOTE** This call currently might take up to 5 seconds to get the latest status and result.
type ColumnInfo ¶
type ColumnInfo struct {
// The name of the column.
Name *string
// The full SQL type specification.
TypeText *string
// The name of the base data type. This doesn't include details for complex
// types such as STRUCT, MAP or ARRAY.
TypeName ColumnTypeName
// The ordinal position of the column (starting at position 0).
Position *int
// Specifies the number of digits in a number. This applies to the DECIMAL type.
TypePrecision *int
// Specifies the number of digits to the right of the decimal point in a number.
// This applies to the DECIMAL type.
TypeScale *int
// The format of the interval type.
TypeIntervalType *string
}
type ColumnTypeName ¶
type ColumnTypeName string
The name of the base data type. This doesn't include details for complex types such as STRUCT, MAP or ARRAY.
const ( ColumnTypeName_Unspecified ColumnTypeName = "" ColumnTypeName_Boolean ColumnTypeName = "BOOLEAN" ColumnTypeName_Byte ColumnTypeName = "BYTE" ColumnTypeName_Short ColumnTypeName = "SHORT" ColumnTypeName_Int ColumnTypeName = "INT" ColumnTypeName_Long ColumnTypeName = "LONG" ColumnTypeName_Float ColumnTypeName = "FLOAT" ColumnTypeName_Double ColumnTypeName = "DOUBLE" ColumnTypeName_Date ColumnTypeName = "DATE" ColumnTypeName_Timestamp ColumnTypeName = "TIMESTAMP" ColumnTypeName_String ColumnTypeName = "STRING" ColumnTypeName_Binary ColumnTypeName = "BINARY" ColumnTypeName_Decimal ColumnTypeName = "DECIMAL" ColumnTypeName_Interval ColumnTypeName = "INTERVAL" ColumnTypeName_Array ColumnTypeName = "ARRAY" ColumnTypeName_Struct ColumnTypeName = "STRUCT" ColumnTypeName_Map ColumnTypeName = "MAP" ColumnTypeName_Char ColumnTypeName = "CHAR" ColumnTypeName_Null ColumnTypeName = "NULL" ColumnTypeName_UserDefinedType ColumnTypeName = "USER_DEFINED_TYPE" )
type Disposition ¶
type Disposition string
const ( Disposition_Unspecified Disposition = "" Disposition_Inline Disposition = "INLINE" Disposition_ExternalLinks Disposition = "EXTERNAL_LINKS" )
type ExecuteStatementRequest ¶
type ExecuteStatementRequest struct {
// The SQL statement to execute. The statement can optionally be parameterized,
// see `parameters`. The maximum query text size is 16 MiB.
Statement *string
// Warehouse upon which to execute a statement. See also [What are SQL
// warehouses?]
//
// [What are SQL warehouses?]: https://docs.databricks.com/sql/admin/warehouse-type.html
WarehouseId *string
// Sets default catalog for statement execution, similar to [`USE CATALOG`] in
// SQL.
//
// [`USE CATALOG`]: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-use-catalog.html
Catalog *string
// Sets default schema for statement execution, similar to [`USE SCHEMA`] in
// SQL.
//
// [`USE SCHEMA`]: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-use-schema.html
Schema *string
// Applies the given row limit to the statement's result set, but unlike the
// `LIMIT` clause in SQL, it also sets the `truncated` field in the response to
// indicate whether the result was trimmed due to the limit or not.
RowLimit *int64
// Applies the given byte limit to the statement's result size. Byte counts are
// based on internal data representations and might not match the final size in
// the requested `format`. If the result was truncated due to the byte limit,
// then `truncated` in the response is set to `true`. When using
// `EXTERNAL_LINKS` disposition, a default `byte_limit` of 100 GiB is applied if
// `byte_limit` is not explicitly set.
ByteLimit *int64
// Statement execution supports three result formats: `JSON_ARRAY` (default),
// `ARROW_STREAM`, and `CSV`.
//
// Important: The formats `ARROW_STREAM` and `CSV` are supported only with
// `EXTERNAL_LINKS` disposition. `JSON_ARRAY` is supported in `INLINE` and
// `EXTERNAL_LINKS` disposition.
//
// When specifying `format=JSON_ARRAY`, result data will be formatted as an
// array of arrays of values, where each value is either the *string
// representation* of a value, or `null`. For example, the output of `SELECT
// concat('id-', id) AS strCol, id AS intCol, null AS nullCol FROM range(3)`
// would look like this:
//
// “` [ [ "id-1", "1", null ], [ "id-2", "2", null ], [ "id-3", "3", null ], ]
// “`
//
// When specifying `format=JSON_ARRAY` and `disposition=EXTERNAL_LINKS`, each
// chunk in the result contains compact JSON with no indentation or extra
// whitespace.
//
// When specifying `format=ARROW_STREAM` and `disposition=EXTERNAL_LINKS`, each
// chunk in the result will be formatted as Apache Arrow Stream. See the [Apache
// Arrow streaming format].
//
// When specifying `format=CSV` and `disposition=EXTERNAL_LINKS`, each chunk in
// the result will be a CSV according to [RFC 4180] standard. All the columns
// values will have *string representation* similar to the `JSON_ARRAY` format,
// and `null` values will be encoded as “null”. Only the first chunk in the
// result would contain a header row with column names. For example, the output
// of `SELECT concat('id-', id) AS strCol, id AS intCol, null as nullCol FROM
// range(3)` would look like this:
//
// “` strCol,intCol,nullCol id-1,1,null id-2,2,null id-3,3,null “`
//
// [Apache Arrow streaming format]: https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format
// [RFC 4180]: https://www.rfc-editor.org/rfc/rfc4180
Format Format
// The fetch disposition provides two modes of fetching results: `INLINE` and
// `EXTERNAL_LINKS`.
//
// Statements executed with `INLINE` disposition will return result data inline,
// in `JSON_ARRAY` format, in a series of chunks. If a given statement produces
// a result set with a size larger than 25 MiB, that statement execution is
// aborted, and no result set will be available.
//
// **NOTE** Byte limits are computed based upon internal representations of the
// result set data, and might not match the sizes visible in JSON responses.
//
// Statements executed with `EXTERNAL_LINKS` disposition will return result data
// as external links: URLs that point to cloud storage internal to the
// workspace. Using `EXTERNAL_LINKS` disposition allows statements to generate
// arbitrarily sized result sets for fetching up to 100 GiB. The resulting links
// have two important properties:
//
// 1. They point to resources _external_ to the <Databricks> compute; therefore
// any associated authentication information (typically a personal access token,
// OAuth token, or similar) _must be removed_ when fetching from these links.
//
// 2. These are URLs with a specific expiration, indicated in the response. The
// behavior when attempting to use an expired link is cloud specific.
Disposition Disposition
// The time in seconds the call will wait for the statement's result set as
// `Ns`, where `N` can be set to 0 or to a value between 5 and 50.
//
// When set to `0s`, the statement will execute in asynchronous mode and the
// call will not wait for the execution to finish. In this case, the call
// returns directly with `PENDING` state and a statement ID which can be used
// for polling with :method:statementexecution/getStatement.
//
// When set between 5 and 50 seconds, the call will behave synchronously up to
// this timeout and wait for the statement execution to finish. If the execution
// finishes within this time, the call returns immediately with a manifest and
// result data (or a `FAILED` state in case of an execution error). If the
// statement takes longer to execute, `on_wait_timeout` determines what should
// happen after the timeout is reached.
WaitTimeout *string
// When `wait_timeout > 0s`, the call will block up to the specified time. If
// the statement execution doesn't finish within this time, `on_wait_timeout`
// determines whether the execution should continue or be canceled. When set to
// `CONTINUE`, the statement execution continues asynchronously and the call
// returns a statement ID which can be used for polling with
// :method:statementexecution/getStatement. When set to `CANCEL`, the statement
// execution is canceled and the call returns with a `CANCELED` state.
OnWaitTimeout TimeoutAction
// A list of parameters to pass into a SQL statement containing parameter
// markers. A parameter consists of a name, a value, and optionally a type. To
// represent a NULL value, the `value` field may be omitted or set to `null`
// explicitly. If the `type` field is omitted, the value is interpreted as a
// string.
//
// If the type is given, parameters will be checked for type correctness
// according to the given type. A value is correct if the provided string can be
// converted to the requested type using the `cast` function. The exact
// semantics are described in the section [`cast` function] of the SQL language
// reference.
//
// For example, the following statement contains two parameters, `my_name` and
// `my_date`:
//
// “` SELECT * FROM my_table WHERE name = :my_name AND date = :my_date “`
//
// The parameters can be passed in the request body as follows:
//
// ` { ..., "statement": "SELECT * FROM my_table WHERE name = :my_name AND date
// = :my_date", "parameters": [ { "name": "my_name", "value": "the name" }, {
// "name": "my_date", "value": "2020-01-01", "type": "DATE" } ] } `
//
// Currently, positional parameters denoted by a `?` marker are not supported by
// the Databricks SQL Statement Execution API.
//
// Also see the section [Parameter markers] of the SQL language reference.
//
// [Parameter markers]: https://docs.databricks.com/sql/language-manual/sql-ref-parameter-marker.html
// [`cast` function]: https://docs.databricks.com/sql/language-manual/functions/cast.html
Parameters []StatementParameter
// An array of query tags to annotate a SQL statement. A query tag consists of a
// non-empty key and, optionally, a value. To represent a NULL value, either
// omit the `value` field or manually set it to `null` or white space. Refer to
// the SQL language reference for the format specification of query tags.
// There's no significance to the order of tags. Only one value per key will be
// recorded. A sequence in excess of 20 query tags will be coerced to 20.
// Example:
//
// { ..., "query_tags": [ { "key": "team", "value": "eng" }, { "key": "some key
// only tag" } ] }
QueryTags []QueryTag
}
type ExternalLink ¶
type ExternalLink struct {
// A URL pointing to a chunk of result data, hosted by an external service, with
// a short expiration time (<= 15 minutes). As this URL contains a temporary
// credential, it should be considered sensitive and the client should not
// expose this URL in a log.
ExternalLink *string
// Indicates the date-time that the given external link will expire and becomes
// invalid, after which point a new `external_link` must be requested.
Expiration *string
// HTTP headers that must be included with a GET request to the `external_link`.
// Each header is provided as a key-value pair. Headers are typically used to
// pass a decryption key to the external service. The values of these headers
// should be considered sensitive and the client should not expose these values
// in a log.
HttpHeaders map[string]string
// The position within the sequence of result set chunks.
ChunkIndex *int
// The starting row offset within the result set.
RowOffset *int64
// The number of rows within the result chunk.
RowCount *int64
// The number of bytes in the result chunk. This field is not available when
// using `INLINE` disposition.
ByteCount *int64
// When fetching, provides the `chunk_index` for the _next_ chunk. If absent,
// indicates there are no more chunks. The next chunk can be fetched with a
// :method:statementexecution/getstatementresultchunkn request.
NextChunkIndex *int
// When fetching, provides a link to fetch the _next_ chunk. If absent,
// indicates there are no more chunks. This link is an absolute `path` to be
// joined with your `$DATABRICKS_HOST`, and should be treated as an opaque link.
// This is an alternative to using `next_chunk_index`.
NextChunkInternalLink *string
}
type GetResultDataRequest ¶
type GetStatementResultRequest ¶
type GetStatementResultRequest struct {
// The statement ID is returned upon successfully submitting a SQL statement,
// and is a required reference for all subsequent calls.
StatementId *string
}
type QueryTag ¶
* A query execution can be annotated with an optional key-value pair to allow users to attribute the executions by key and optional value to filter by. QueryTag is the user-facing representation..
type ResultData ¶
type ResultData struct {
ExternalLinks []ExternalLink
// The `JSON_ARRAY` format is an array of arrays of values, where each non-null
// value is formatted as a string. Null values are encoded as JSON `null`.
DataArray [][]json.RawMessage
// The position within the sequence of result set chunks.
ChunkIndex *int
// The starting row offset within the result set.
RowOffset *int64
// The number of rows within the result chunk.
RowCount *int64
// The number of bytes in the result chunk. This field is not available when
// using `INLINE` disposition.
ByteCount *int64
// When fetching, provides the `chunk_index` for the _next_ chunk. If absent,
// indicates there are no more chunks. The next chunk can be fetched with a
// :method:statementexecution/getstatementresultchunkn request.
NextChunkIndex *int
// When fetching, provides a link to fetch the _next_ chunk. If absent,
// indicates there are no more chunks. This link is an absolute `path` to be
// joined with your `$DATABRICKS_HOST`, and should be treated as an opaque link.
// This is an alternative to using `next_chunk_index`.
NextChunkInternalLink *string
}
Contains the result data of a single chunk when using `INLINE` disposition. When using `EXTERNAL_LINKS` disposition, the array `external_links` is used instead to provide URLs to the result data in cloud storage. Exactly one of these alternatives is used. (While the `external_links` array prepares the API to return multiple links in a single response. Currently only a single link is returned.).
type ResultManifest ¶
type ResultManifest struct {
Format Format
Schema *Schema
// The total number of chunks that the result set has been divided into.
TotalChunkCount *int
// Array of result set chunk metadata.
Chunks []ChunkInfo
// The total number of rows in the result set.
TotalRowCount *int64
// The total number of bytes in the result set. This field is not available when
// using `INLINE` disposition.
TotalByteCount *int64
// Indicates whether the result is truncated due to `row_limit` or `byte_limit`.
Truncated *bool
}
The result manifest provides schema and metadata for the result set..
type Schema ¶
type Schema struct {
ColumnCount *int
Columns []ColumnInfo
}
The schema is an ordered list of column descriptions..
type ServiceError ¶
type ServiceError struct {
ErrorCode ServiceErrorCode
// A brief summary of the error condition.
Message *string
}
type ServiceErrorCode ¶
type ServiceErrorCode string
const ( ServiceErrorCode_Unspecified ServiceErrorCode = "" ServiceErrorCode_Unknown ServiceErrorCode = "UNKNOWN" ServiceErrorCode_InternalError ServiceErrorCode = "INTERNAL_ERROR" ServiceErrorCode_IoError ServiceErrorCode = "IO_ERROR" ServiceErrorCode_BadRequest ServiceErrorCode = "BAD_REQUEST" ServiceErrorCode_ServiceUnderMaintenance ServiceErrorCode = "SERVICE_UNDER_MAINTENANCE" ServiceErrorCode_DeadlineExceeded ServiceErrorCode = "DEADLINE_EXCEEDED" ServiceErrorCode_Cancelled ServiceErrorCode = "CANCELLED" ServiceErrorCode_ResourceExhausted ServiceErrorCode = "RESOURCE_EXHAUSTED" ServiceErrorCode_Aborted ServiceErrorCode = "ABORTED" ServiceErrorCode_NotFound ServiceErrorCode = "NOT_FOUND" ServiceErrorCode_AlreadyExists ServiceErrorCode = "ALREADY_EXISTS" ServiceErrorCode_Unauthenticated ServiceErrorCode = "UNAUTHENTICATED" )
type StatementParameter ¶
type StatementParameter struct {
// The name of a parameter marker to be substituted in the statement.
Name *string
// The value to substitute, represented as a string. If omitted, the value is
// interpreted as NULL.
Value *string
// The data type, given as a string. For example: `INT`, `STRING`,
// `DECIMAL(10,2)`. If no type is given the type is assumed to be `STRING`.
// Complex types, such as `ARRAY`, `MAP`, and `STRUCT` are not supported. For
// valid types, refer to the section [Data types] of the SQL language reference.
//
// [Data types]: https://docs.databricks.com/sql/language-manual/functions/cast.html
Type *string
}
type StatementResponse ¶
type StatementResponse struct {
// The statement ID is returned upon successfully submitting a SQL statement,
// and is a required reference for all subsequent calls.
StatementId *string
Status *StatementStatus
Manifest *ResultManifest
Result *ResultData
}
type StatementStatus ¶
type StatementStatus struct {
// Statement execution state: - `PENDING`: waiting for warehouse - `RUNNING`:
// running - `SUCCEEDED`: execution was successful, result data available for
// fetch - `FAILED`: execution failed; reason for failure described in
// accompanying error message - `CANCELED`: user canceled; can come from
// explicit cancel call, or timeout with `on_wait_timeout=CANCEL` - `CLOSED`:
// execution successful, and statement closed; result no longer available for
// fetch
State StatementStatus_State
Error *ServiceError
// SQLSTATE error code returned when the statement execution fails. Only
// populated when the statement status is `FAILED`.
SqlState *string
}
The status response includes execution state and if relevant, error information..
type StatementStatus_State ¶
type StatementStatus_State string
const ( StatementStatus_State_Unspecified StatementStatus_State = "" StatementStatus_State_Pending StatementStatus_State = "PENDING" StatementStatus_State_Running StatementStatus_State = "RUNNING" StatementStatus_State_Succeeded StatementStatus_State = "SUCCEEDED" StatementStatus_State_Failed StatementStatus_State = "FAILED" StatementStatus_State_Canceled StatementStatus_State = "CANCELED" StatementStatus_State_Closed StatementStatus_State = "CLOSED" )
type TimeoutAction ¶
type TimeoutAction string
When `wait_timeout > 0s`, the call will block up to the specified time. If the statement execution doesn't finish within this time, `on_wait_timeout` determines whether the execution should continue or be canceled. When set to `CONTINUE`, the statement execution continues asynchronously and the call returns a statement ID which can be used for polling with :method:statementexecution/getStatement. When set to `CANCEL`, the statement execution is canceled and the call returns with a `CANCELED` state.
const ( TimeoutAction_Unspecified TimeoutAction = "" TimeoutAction_Continue TimeoutAction = "CONTINUE" TimeoutAction_Cancel TimeoutAction = "CANCEL" )