Documentation
¶
Index ¶
- Constants
- Variables
- func CollectBigQueryStats(ctx context.Context, job *bigquery.Job)
- func IsAuthError(err error) bool
- func IsPermissionError(err error) bool
- func NewHTTPClient(ctx context.Context, conf *BigQueryConf, scopes ...string) (*http.Client, error)
- func ValidateServiceAccountKey(key []byte) error
- type BigQueryConf
- type BigQueryExecutor
- func (e *BigQueryExecutor) Close() error
- func (e *BigQueryExecutor) Exec(ctx context.Context, sql string) error
- func (e *BigQueryExecutor) GetBigQueryClient() *bigquery.Client
- func (e *BigQueryExecutor) NewQuery(ctx context.Context, sql string, args ...any) *bigquery.Query
- func (e *BigQueryExecutor) QueryRowsIterator(ctx context.Context, sql string, args ...interface{}) (*bigquery.RowIterator, error)
- type Executor
- type Querier
- func (q *Querier[T]) Close() error
- func (q *Querier[T]) Exec(ctx context.Context, sql string) error
- func (q *Querier[T]) QueryAndProcessMany(ctx context.Context, sql string, ...) error
- func (q *Querier[T]) QueryMany(ctx context.Context, sql string, opts ...exec.QueryManyOpt[T]) ([]*T, error)
- func (q *Querier[T]) QueryMaps(ctx context.Context, sql string) ([]exec.QueryMapResult, error)
Constants ¶
const GoogleTokenURI = "https://oauth2.googleapis.com/token"
GoogleTokenURI is the token endpoint every Google service account key names.
Variables ¶
var ErrNoCredential = errors.New("bigquery: no credential configured, set a service account key or an access token")
ErrNoCredential is returned for a connection that names no credential and does not allow the process's default credentials.
Functions ¶
func CollectBigQueryStats ¶ added in v0.9.0
CollectBigQueryStats populates the DriverStats with BigQuery job statistics if a stats callback is registered in the context.
func IsAuthError ¶ added in v0.23.0
IsAuthError reports whether err is Google rejecting the credentials: the token endpoint refusing a service account key or a refresh token (invalid_grant and the like, any 4xx), or the API refusing the token it was sent (401).
A 403 from the API is not one: the identity was accepted and lacks a grant, which IsPermissionError reports.
func IsPermissionError ¶ added in v0.15.3
IsPermissionError reports whether err indicates the DWH credentials lack the privileges required for the attempted operation.
func NewHTTPClient ¶ added in v0.21.0
NewHTTPClient returns an HTTP client authenticated the way conf says (see BigQueryConf) for the given OAuth scopes, with conf's per-request timeout. The BigQuery client and the scrapper's Resource Manager, IAM and Data Catalog calls all go through it, so they act as the same identity.
func ValidateServiceAccountKey ¶ added in v0.24.7
ValidateServiceAccountKey accepts only a Google service account key that gets its tokens from Google. Every other credential type makes the process read a file or call a URL the credential names (external_account reads its subject token from one), and a key with another token_uri posts to that address. A credential that someone other than the host's owner supplies must pass this.
Types ¶
type BigQueryConf ¶
type BigQueryConf struct {
// ProjectId is the GCP project containing BigQuery datasets.
ProjectId string
// Region restricts queries to a specific BQ region (e.g. "EU", "us-central1").
Region string
// CredentialsJson is a GCP service account key in JSON format.
CredentialsJson string
// CredentialsFile is a path to a GCP service account key file.
CredentialsFile string
// AccessToken is an OAuth access token for user-level authentication.
// When set, it takes precedence over CredentialsJson and CredentialsFile.
AccessToken string
// AllowDefaultCredentials opts into Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS,
// gcloud's login, the GKE or GCE metadata server) when no credential is set. Only a process running
// in the customer's own environment sets it, such as the synq-dwh agent. On the hosted side it
// stays false, so a connection with no credential is refused instead of running as our own
// service account.
AllowDefaultCredentials bool
// AllowAnyCredentialType accepts every credential type Google's client libraries load in
// CredentialsJson and CredentialsFile (external_account, impersonated_service_account,
// authorized_user). Some of them read a file or call a URL that the credential names. Only a
// process whose configuration comes from the host's owner sets it, such as the synq-dwh agent.
// When false, the credential must pass ValidateServiceAccountKey.
AllowAnyCredentialType bool
// HTTPTimeout caps each BigQuery REST round-trip (table/dataset metadata,
// table listing, query-result pages). Guards against a stalled connection
// hanging a scrape until the caller's job deadline. Zero uses
// defaultHTTPTimeout.
HTTPTimeout time.Duration
}
BigQueryConf configures the BigQuery executor connection. Authentication precedence: AccessToken > CredentialsJson > CredentialsFile > Application Default Credentials, the last only when AllowDefaultCredentials is set.
type BigQueryExecutor ¶
type BigQueryExecutor struct {
// contains filtered or unexported fields
}
func NewBigqueryExecutor ¶
func NewBigqueryExecutor(ctx context.Context, conf *BigQueryConf) (*BigQueryExecutor, error)
func NewBigqueryExecutorFromClient ¶ added in v0.21.0
func NewBigqueryExecutorFromClient(client *bigquery.Client, conf *BigQueryConf) *BigQueryExecutor
NewBigqueryExecutorFromClient wraps a client the caller has already built, without the connection check NewBigqueryExecutor makes.
func (*BigQueryExecutor) Close ¶
func (e *BigQueryExecutor) Close() error
func (*BigQueryExecutor) Exec ¶
func (e *BigQueryExecutor) Exec(ctx context.Context, sql string) error
func (*BigQueryExecutor) GetBigQueryClient ¶
func (e *BigQueryExecutor) GetBigQueryClient() *bigquery.Client
func (*BigQueryExecutor) NewQuery ¶ added in v0.21.0
NewQuery builds a query job for sql that carries the query context of ctx: the SQL comment, which INFORMATION_SCHEMA.JOBS keeps in the query text, and the job labels. args become positional parameters. Every job we start is built here, so query-history readers can tell our own traffic apart.
func (*BigQueryExecutor) QueryRowsIterator ¶
func (e *BigQueryExecutor) QueryRowsIterator( ctx context.Context, sql string, args ...interface{}, ) (*bigquery.RowIterator, error)
type Querier ¶
type Querier[T any] struct { // contains filtered or unexported fields }
func NewQuerier ¶
func NewQuerier[T any](conn *BigQueryExecutor) *Querier[T]