Help us shape the future of Go! Take the 2026 Go Developer Survey

bigquery

package
v0.25.0-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

View Source
const GoogleTokenURI = "https://oauth2.googleapis.com/token"

GoogleTokenURI is the token endpoint every Google service account key names.

Variables

View Source
var ErrNoCredential = errors.New("bigquery: no credential configured, set a service account key or an access token")

ErrNoCredential is returned for a connection that names no credential and does not allow the process's default credentials.

Functions

func CollectBigQueryStats added in v0.9.0

func CollectBigQueryStats(ctx context.Context, job *bigquery.Job)

CollectBigQueryStats populates the DriverStats with BigQuery job statistics if a stats callback is registered in the context.

func IsAuthError added in v0.23.0

func IsAuthError(err error) bool

IsAuthError reports whether err is Google rejecting the credentials: the token endpoint refusing a service account key or a refresh token (invalid_grant and the like, any 4xx), or the API refusing the token it was sent (401).

A denied impersonation hop (see ImpersonationError.Denied) is one too: Google refused to mint a token for the identity the connection names.

A 403 from the API is not one: the identity was accepted and lacks a grant, which IsPermissionError reports.

func IsPermissionError added in v0.15.3

func IsPermissionError(err error) bool

IsPermissionError reports whether err indicates the DWH credentials lack the privileges required for the attempted operation. A refused impersonation hop is not one, even though Google answers it with a 403: no identity was established, which IsAuthError reports.

func NewHTTPClient added in v0.21.0

func NewHTTPClient(ctx context.Context, conf *BigQueryConf, scopes ...string) (*http.Client, error)

NewHTTPClient returns an HTTP client authenticated the way conf says (see BigQueryConf) for the given OAuth scopes, with conf's per-request timeout. The BigQuery client and the scrapper's Resource Manager, IAM and Data Catalog calls all go through it, so they act as the same identity.

func ValidateServiceAccountKey added in v0.24.7

func ValidateServiceAccountKey(key []byte) error

ValidateServiceAccountKey accepts only a Google service account key that gets its tokens from Google. Every other credential type makes the process read a file or call a URL the credential names (external_account reads its subject token from one), and a key with another token_uri posts to that address. A credential that someone other than the host's owner supplies must pass this.

Types

type BigQueryConf

type BigQueryConf struct {
	// ProjectId is the GCP project containing BigQuery datasets.
	ProjectId string
	// Region restricts queries to a specific BQ region (e.g. "EU", "us-central1").
	Region string
	// CredentialsJson is a GCP service account key in JSON format.
	CredentialsJson string
	// CredentialsFile is a path to a GCP service account key file.
	CredentialsFile string
	// AccessToken is an OAuth access token for user-level authentication.
	// When set, it takes precedence over CredentialsJson and CredentialsFile.
	AccessToken string
	// ImpersonateServiceAccounts is a chain of service account emails. The process's own default
	// credentials get a token for the first one, that one for the next, and BigQuery sees the last.
	// Each one must grant roles/iam.serviceAccountTokenCreator to the identity before it. Google
	// mints the tokens, so no key leaves anyone's project, and the chain needs no
	// AllowDefaultCredentials: whoever builds it decides which service accounts it names. It can not
	// be combined with CredentialsJson or CredentialsFile.
	ImpersonateServiceAccounts []string
	// AllowDefaultCredentials opts into Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS,
	// gcloud's login, the GKE or GCE metadata server) when no credential is set. Only a process running
	// in the customer's own environment sets it, such as the synq-dwh agent. On the hosted side it
	// stays false, so a connection with no credential is refused instead of running as our own
	// service account.
	AllowDefaultCredentials bool
	// AllowAnyCredentialType accepts every credential type Google's client libraries load in
	// CredentialsJson and CredentialsFile (external_account, impersonated_service_account,
	// authorized_user). Some of them read a file or call a URL that the credential names. Only a
	// process whose configuration comes from the host's owner sets it, such as the synq-dwh agent.
	// When false, the credential must pass ValidateServiceAccountKey.
	AllowAnyCredentialType bool
	// HTTPTimeout caps each BigQuery REST round-trip (table/dataset metadata,
	// table listing, query-result pages). Guards against a stalled connection
	// hanging a scrape until the caller's job deadline. Zero uses
	// defaultHTTPTimeout.
	HTTPTimeout time.Duration
}

BigQueryConf configures the BigQuery executor connection. Authentication precedence: AccessToken > ImpersonateServiceAccounts > CredentialsJson > CredentialsFile > Application Default Credentials, the last only when AllowDefaultCredentials is set.

type BigQueryExecutor

type BigQueryExecutor struct {
	// contains filtered or unexported fields
}

func NewBigqueryExecutor

func NewBigqueryExecutor(ctx context.Context, conf *BigQueryConf) (*BigQueryExecutor, error)

func NewBigqueryExecutorFromClient added in v0.21.0

func NewBigqueryExecutorFromClient(client *bigquery.Client, conf *BigQueryConf) *BigQueryExecutor

NewBigqueryExecutorFromClient wraps a client the caller has already built, without the connection check NewBigqueryExecutor makes.

func (*BigQueryExecutor) Close

func (e *BigQueryExecutor) Close() error

func (*BigQueryExecutor) Exec

func (e *BigQueryExecutor) Exec(ctx context.Context, sql string) error

func (*BigQueryExecutor) GetBigQueryClient

func (e *BigQueryExecutor) GetBigQueryClient() *bigquery.Client

func (*BigQueryExecutor) NewQuery added in v0.21.0

func (e *BigQueryExecutor) NewQuery(ctx context.Context, sql string, args ...any) *bigquery.Query

NewQuery builds a query job for sql that carries the query context of ctx: the SQL comment, which INFORMATION_SCHEMA.JOBS keeps in the query text, and the job labels. args become positional parameters. Every job we start is built here, so query-history readers can tell our own traffic apart.

func (*BigQueryExecutor) QueryRowsIterator

func (e *BigQueryExecutor) QueryRowsIterator(
	ctx context.Context,
	sql string,
	args ...interface{},
) (*bigquery.RowIterator, error)

type Executor

type Executor interface {
	QueryRowsIterator(ctx context.Context, q string, args ...interface{}) (*bigquery.RowIterator, error)
}

type ImpersonationError

type ImpersonationError struct {
	ServiceAccount string
	Caller         string
	Err            error
}

ImpersonationError is a refused or failed token request for ServiceAccount, one hop of BigQueryConf.ImpersonateServiceAccounts. Caller is the identity that asked for it: the process's own default credentials for the first hop ("" then), the previous service account after that.

func AsImpersonationError

func AsImpersonationError(err error) (*ImpersonationError, bool)

AsImpersonationError returns the impersonation hop err failed on, if it failed on one.

func (*ImpersonationError) Denied

func (e *ImpersonationError) Denied() bool

Denied reports whether Google refused the hop: the caller lacks roles/iam.serviceAccountTokenCreator on the service account (403), or the service account does not exist or is disabled (404, 400).

func (*ImpersonationError) Error

func (e *ImpersonationError) Error() string

func (*ImpersonationError) Unwrap

func (e *ImpersonationError) Unwrap() error

type Querier

type Querier[T any] struct {
	// contains filtered or unexported fields
}

func NewQuerier

func NewQuerier[T any](conn *BigQueryExecutor) *Querier[T]

func (*Querier[T]) Close

func (q *Querier[T]) Close() error

func (*Querier[T]) Exec

func (q *Querier[T]) Exec(ctx context.Context, sql string) error

func (*Querier[T]) QueryAndProcessMany

func (q *Querier[T]) QueryAndProcessMany(
	ctx context.Context,
	sql string,
	handler func(ctx context.Context, batch []*T) error,
	opts ...exec.QueryManyOpt[T],
) error

func (*Querier[T]) QueryMany

func (q *Querier[T]) QueryMany(
	ctx context.Context,
	sql string,
	opts ...exec.QueryManyOpt[T],
) ([]*T, error)

func (*Querier[T]) QueryMaps

func (q *Querier[T]) QueryMaps(ctx context.Context, sql string) ([]exec.QueryMapResult, error)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL