Documentation
¶
Index ¶
- Constants
- func ExecuteWithDefaultRetry(ctx context.Context, fn func() error) error
- func ExecuteWithRetry(ctx context.Context, opts RetryOptions, fn func() error) error
- func GetErrorCode(err error) int
- func GetErrorReason(err error) string
- func IsBigQueryRateLimitError(err error) bool
- func IsBigQueryRetryableError(err error) bool
- func NewConnector(projectID string, opts ...option.ClientOption) driver.Connector
- func NewConnectorWithRetry(projectID string, retryConfig *RetryConfig, opts ...option.ClientOption) driver.Connector
- type RetryConfig
- type RetryOptions
Constants ¶
const ( // ReasonRateLimitExceeded is returned for standard rate limit errors (HTTP 403/429) ReasonRateLimitExceeded = "rateLimitExceeded" // ReasonQuotaExceeded is returned when quota is exceeded ReasonQuotaExceeded = "quotaExceeded" // ReasonInvalidQuery is returned for query errors AND some rate limit errors (HTTP 400) // BigQuery API quirk: table metadata rate limits return invalidQuery, not rateLimitExceeded ReasonInvalidQuery = "invalidQuery" // ReasonBackendError is returned for transient backend errors ReasonBackendError = "backendError" // ReasonJobRateLimitExceeded is returned when job rate limits are exceeded ReasonJobRateLimitExceeded = "jobRateLimitExceeded" // ReasonInternalError is returned for internal BigQuery errors ReasonInternalError = "internalError" )
BigQuery error reasons (from google-cloud-go/bigquery/bigquery.go) These are the machine-readable reason codes returned in googleapi.ErrorItem.Reason
const ( DefaultInitialBackoff = 1 * time.Second DefaultMaxBackoff = 32 * time.Second DefaultMultiplier = 2.0 DefaultMaxRetries = 0 // 0 = unlimited, matching google-cloud-go's default behavior )
Default retry parameters (aligned with google-cloud-go/bigquery defaults)
Variables ¶
This section is empty.
Functions ¶
func ExecuteWithDefaultRetry ¶
ExecuteWithDefaultRetry is a convenience function that uses default retry options.
func ExecuteWithRetry ¶
func ExecuteWithRetry(ctx context.Context, opts RetryOptions, fn func() error) error
ExecuteWithRetry executes the given function with retry logic for BigQuery rate limit errors. This is necessary because google-cloud-go does NOT automatically retry "invalidQuery" errors, which includes table metadata rate limits.
The function implements exponential backoff with optional jitter: - Backoff sequence: 1s → 2s → 4s → 8s → 16s → 32s → 32s → ... - Jitter: randomizes actual delay to 50-100% of calculated backoff
Retry stops when: - Operation succeeds (returns nil) - Non-retryable error occurs (returns that error) - MaxRetries exceeded (returns wrapped error) - MaxDuration exceeded (returns wrapped error) - Context cancelled/deadline exceeded (returns context error)
func GetErrorCode ¶
GetErrorCode extracts the HTTP status code from a googleapi.Error. Returns 0 if the error is not a googleapi.Error.
func GetErrorReason ¶
GetErrorReason extracts the primary reason from a googleapi.Error. Returns empty string if the error is not a googleapi.Error or has no reasons.
func IsBigQueryRateLimitError ¶
IsBigQueryRateLimitError determines if an error is a BigQuery rate limit error that should be retried. This uses a type-safe hybrid approach:
- Type-safe: Uses errors.As() to extract googleapi.Error
- Type-safe: Checks HTTP Code and Reason fields (structured data)
- Pattern match: Only checks Message when Code+Reason indicate potential rate limit
BigQuery API quirk: Table metadata rate limits (e.g., "too many table update operations") return HTTP 400 with reason "invalidQuery" instead of HTTP 429 with "rateLimitExceeded". The google-cloud-go library does NOT automatically retry these errors.
func IsBigQueryRetryableError ¶
IsBigQueryRetryableError determines if an error is retryable (broader than just rate limits). This includes: - Rate limit errors (via IsBigQueryRateLimitError) - Backend errors (HTTP 5xx) - Transient errors with specific reasons
func NewConnector ¶
func NewConnector(projectID string, opts ...option.ClientOption) driver.Connector
func NewConnectorWithRetry ¶
func NewConnectorWithRetry(projectID string, retryConfig *RetryConfig, opts ...option.ClientOption) driver.Connector
Types ¶
type RetryConfig ¶
RetryConfig contains retry configuration for BigQuery API calls
type RetryOptions ¶
type RetryOptions struct {
// InitialBackoff is the initial delay before the first retry.
// Default: 1 second
InitialBackoff time.Duration
// MaxBackoff is the maximum delay between retries.
// Default: 32 seconds
MaxBackoff time.Duration
// Multiplier is the factor by which backoff increases after each retry.
// Default: 2.0
Multiplier float64
// MaxRetries is the maximum number of retry attempts.
// Default: 10 (set to 0 for unlimited, though not recommended)
MaxRetries int
// MaxDuration limits the total time spent retrying.
// If set, retries will stop when this duration is exceeded.
// Default: 0 (no limit, relies on MaxRetries and context)
MaxDuration time.Duration
// Jitter adds randomness to backoff to prevent thundering herd.
// When true, actual backoff is between 0.5x and 1.0x of calculated backoff.
// Default: true
Jitter bool
// RetryableFunc determines if an error should be retried.
// Default: IsBigQueryRateLimitError (only retry rate limit errors)
// Use IsBigQueryRetryableError for broader retry coverage
RetryableFunc func(error) bool
}
RetryOptions configures the retry behavior for BigQuery operations.
func DefaultRetryOptions ¶
func DefaultRetryOptions() RetryOptions
DefaultRetryOptions returns retry options with sensible defaults.
func RetryOptionsFromConfig ¶
func RetryOptionsFromConfig(config *RetryConfig) RetryOptions
RetryOptionsFromConfig creates RetryOptions from RetryConfig.
Source Files