Documentation
¶
Index ¶
- Constants
- Variables
- func CollectBigQueryStats(ctx context.Context, job *bigquery.Job)
- func IsAuthError(err error) bool
- func IsPermissionError(err error) bool
- func NewHTTPClient(ctx context.Context, conf *BigQueryConf, scopes ...string) (*http.Client, error)
- func ValidateServiceAccountKey(key []byte) error
- type BigQueryConf
- type BigQueryExecutor
- func (e *BigQueryExecutor) Close() error
- func (e *BigQueryExecutor) Exec(ctx context.Context, sql string) error
- func (e *BigQueryExecutor) GetBigQueryClient() *bigquery.Client
- func (e *BigQueryExecutor) NewQuery(ctx context.Context, sql string, args ...any) *bigquery.Query
- func (e *BigQueryExecutor) QueryRowsIterator(ctx context.Context, sql string, args ...interface{}) (*bigquery.RowIterator, error)
- type Executor
- type ImpersonationError
- type Querier
- func (q *Querier[T]) Close() error
- func (q *Querier[T]) Exec(ctx context.Context, sql string) error
- func (q *Querier[T]) QueryAndProcessMany(ctx context.Context, sql string, ...) error
- func (q *Querier[T]) QueryMany(ctx context.Context, sql string, opts ...exec.QueryManyOpt[T]) ([]*T, error)
- func (q *Querier[T]) QueryMaps(ctx context.Context, sql string) ([]exec.QueryMapResult, error)
Constants ¶
const GoogleTokenURI = "https://oauth2.googleapis.com/token"
GoogleTokenURI is the token endpoint every Google service account key names.
Variables ¶
var ErrNoCredential = errors.New("bigquery: no credential configured, set a service account key or an access token")
ErrNoCredential is returned for a connection that names no credential and does not allow the process's default credentials.
Functions ¶
func CollectBigQueryStats ¶ added in v0.9.0
CollectBigQueryStats populates the DriverStats with BigQuery job statistics if a stats callback is registered in the context.
func IsAuthError ¶ added in v0.23.0
IsAuthError reports whether err is Google rejecting the credentials: the token endpoint refusing a service account key or a refresh token (invalid_grant and the like, any 4xx), or the API refusing the token it was sent (401).
A denied impersonation hop (see ImpersonationError.Denied) is one too: Google refused to mint a token for the identity the connection names.
A 403 from the API is not one: the identity was accepted and lacks a grant, which IsPermissionError reports.
func IsPermissionError ¶ added in v0.15.3
IsPermissionError reports whether err indicates the DWH credentials lack the privileges required for the attempted operation. A refused impersonation hop is not one, even though Google answers it with a 403: no identity was established, which IsAuthError reports.
func NewHTTPClient ¶ added in v0.21.0
NewHTTPClient returns an HTTP client authenticated the way conf says (see BigQueryConf) for the given OAuth scopes, with conf's per-request timeout. The BigQuery client and the scrapper's Resource Manager, IAM and Data Catalog calls all go through it, so they act as the same identity.
func ValidateServiceAccountKey ¶ added in v0.24.7
ValidateServiceAccountKey accepts only a Google service account key that gets its tokens from Google. Every other credential type makes the process read a file or call a URL the credential names (external_account reads its subject token from one), and a key with another token_uri posts to that address. A credential that someone other than the host's owner supplies must pass this.
Types ¶
type BigQueryConf ¶
type BigQueryConf struct {
// ProjectId is the GCP project containing BigQuery datasets.
ProjectId string
// Region restricts queries to a specific BQ region (e.g. "EU", "us-central1").
Region string
// CredentialsJson is a GCP service account key in JSON format.
CredentialsJson string
// CredentialsFile is a path to a GCP service account key file.
CredentialsFile string
// AccessToken is an OAuth access token for user-level authentication.
// When set, it takes precedence over CredentialsJson and CredentialsFile.
AccessToken string
// ImpersonateServiceAccounts is a chain of service account emails. The process's own default
// credentials get a token for the first one, that one for the next, and BigQuery sees the last.
// Each one must grant roles/iam.serviceAccountTokenCreator to the identity before it. Google
// mints the tokens, so no key leaves anyone's project, and the chain needs no
// AllowDefaultCredentials: whoever builds it decides which service accounts it names. It can not
// be combined with CredentialsJson or CredentialsFile.
ImpersonateServiceAccounts []string
// AllowDefaultCredentials opts into Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS,
// gcloud's login, the GKE or GCE metadata server) when no credential is set. Only a process running
// in the customer's own environment sets it, such as the synq-dwh agent. On the hosted side it
// stays false, so a connection with no credential is refused instead of running as our own
// service account.
AllowDefaultCredentials bool
// AllowAnyCredentialType accepts every credential type Google's client libraries load in
// CredentialsJson and CredentialsFile (external_account, impersonated_service_account,
// authorized_user). Some of them read a file or call a URL that the credential names. Only a
// process whose configuration comes from the host's owner sets it, such as the synq-dwh agent.
// When false, the credential must pass ValidateServiceAccountKey.
AllowAnyCredentialType bool
// HTTPTimeout caps each BigQuery REST round-trip (table/dataset metadata,
// table listing, query-result pages). Guards against a stalled connection
// hanging a scrape until the caller's job deadline. Zero uses
// defaultHTTPTimeout.
HTTPTimeout time.Duration
}
BigQueryConf configures the BigQuery executor connection. Authentication precedence: AccessToken > ImpersonateServiceAccounts > CredentialsJson > CredentialsFile > Application Default Credentials, the last only when AllowDefaultCredentials is set.
type BigQueryExecutor ¶
type BigQueryExecutor struct {
// contains filtered or unexported fields
}
func NewBigqueryExecutor ¶
func NewBigqueryExecutor(ctx context.Context, conf *BigQueryConf) (*BigQueryExecutor, error)
func NewBigqueryExecutorFromClient ¶ added in v0.21.0
func NewBigqueryExecutorFromClient(client *bigquery.Client, conf *BigQueryConf) *BigQueryExecutor
NewBigqueryExecutorFromClient wraps a client the caller has already built, without the connection check NewBigqueryExecutor makes.
func (*BigQueryExecutor) Close ¶
func (e *BigQueryExecutor) Close() error
func (*BigQueryExecutor) Exec ¶
func (e *BigQueryExecutor) Exec(ctx context.Context, sql string) error
func (*BigQueryExecutor) GetBigQueryClient ¶
func (e *BigQueryExecutor) GetBigQueryClient() *bigquery.Client
func (*BigQueryExecutor) NewQuery ¶ added in v0.21.0
NewQuery builds a query job for sql that carries the query context of ctx: the SQL comment, which INFORMATION_SCHEMA.JOBS keeps in the query text, and the job labels. args become positional parameters. Every job we start is built here, so query-history readers can tell our own traffic apart.
func (*BigQueryExecutor) QueryRowsIterator ¶
func (e *BigQueryExecutor) QueryRowsIterator( ctx context.Context, sql string, args ...interface{}, ) (*bigquery.RowIterator, error)
type ImpersonationError ¶
ImpersonationError is a refused or failed token request for ServiceAccount, one hop of BigQueryConf.ImpersonateServiceAccounts. Caller is the identity that asked for it: the process's own default credentials for the first hop ("" then), the previous service account after that.
func AsImpersonationError ¶
func AsImpersonationError(err error) (*ImpersonationError, bool)
AsImpersonationError returns the impersonation hop err failed on, if it failed on one.
func (*ImpersonationError) Denied ¶
func (e *ImpersonationError) Denied() bool
Denied reports whether Google refused the hop: the caller lacks roles/iam.serviceAccountTokenCreator on the service account (403), or the service account does not exist or is disabled (404, 400).
func (*ImpersonationError) Error ¶
func (e *ImpersonationError) Error() string
func (*ImpersonationError) Unwrap ¶
func (e *ImpersonationError) Unwrap() error
type Querier ¶
type Querier[T any] struct { // contains filtered or unexported fields }
func NewQuerier ¶
func NewQuerier[T any](conn *BigQueryExecutor) *Querier[T]