Documentation
¶
Overview ¶
Package spark provides Bruin's ADBC Spark SQL platform and shared utilities for Spark-based execution environments such as EMR Serverless and Dataproc Serverless.
Index ¶
- Variables
- func ADBCDriverName() string
- func EnsureADBCDriverInstalled(ctx context.Context) error
- func Exclude(path string) bool
- func NewColumnCheckOperator(manager config.ConnectionGetter) *ansisql.ColumnCheckOperator
- func NewMaterializer(fullRefresh bool) *pipeline.Materializer
- func PackageContext(zw *zip.Writer, context fs.FS) error
- type BasicOperator
- type Client
- func (c *Client) CreateSchemaIfNotExist(ctx context.Context, asset *pipeline.Asset, pipelineName string) error
- func (c *Client) GetDatabaseSummary(ctx context.Context) (*ansisql.DBDatabase, error)
- func (c *Client) Ping(ctx context.Context) error
- func (c *Client) RunQueriesWithoutResult(ctx context.Context, queries []*query.Query) error
- func (c *Client) RunQueryWithoutResult(ctx context.Context, queryObj *query.Query) error
- func (c *Client) Select(ctx context.Context, queryObj *query.Query) ([][]interface{}, error)
- func (c *Client) SelectWithSchema(ctx context.Context, queryObj *query.Query) (*query.QueryResult, error)
- func (c *Client) TableExists(ctx context.Context, tableName string) (bool, error)
- func (c *Client) TablesExist(ctx context.Context, tableNames []string) (map[string]bool, error)
- type ClientResolver
- type Config
- type Renderer
- type SeedOperator
- type TableSensor
Constants ¶
This section is empty.
Variables ¶
var BruinExcludes = []string{
"README.md",
".bruin.yml",
"pipeline.yml",
"pipeline.yaml",
}
BruinExcludes contains files that should be excluded from Spark context packages.
var DirExcludes = []*regexp.Regexp{ regexp.MustCompile(`(^|[/\\])\.venv([/\\]|$)`), regexp.MustCompile(`(^|[/\\])venv([/\\]|$)`), regexp.MustCompile(`^logs([/\\]|$)`), regexp.MustCompile(`^\.git([/\\]|$)`), }
DirExcludes contains regex patterns for directories that should be excluded from Spark context packages.
Functions ¶
func ADBCDriverName ¶ added in v0.11.697
func ADBCDriverName() string
func EnsureADBCDriverInstalled ¶ added in v0.11.697
func NewColumnCheckOperator ¶ added in v0.11.697
func NewColumnCheckOperator(manager config.ConnectionGetter) *ansisql.ColumnCheckOperator
NewColumnCheckOperator uses Spark SQL-specific accepted-values and regex checks shared with Sail.
func NewMaterializer ¶ added in v0.11.697
func NewMaterializer(fullRefresh bool) *pipeline.Materializer
NewMaterializer uses the Spark SQL materialization implementation shared with Sail and replaces the strategies that need Spark table-provider features, such as Iceberg partition specs, sort orders, and MERGE INTO.
func PackageContext ¶
PackageContext creates a zip archive from the given filesystem, suitable for Spark execution. It's a modified version of zip.AddFS() with:
- Exclusion of Bruin configuration files and virtual environments
- Automatic creation of __init__.py files in directories for Python package support
Spark requires directories to contain __init__.py to be treated as packages.
Types ¶
type BasicOperator ¶ added in v0.11.697
type BasicOperator struct {
// contains filtered or unexported fields
}
func NewBasicOperator ¶ added in v0.11.697
func NewBasicOperator(conn config.ConnectionGetter, extractor query.QueryExtractor, fullRefresh bool, _ pipeline.DeclareHoister, parser *sqlparser.SQLParser) *BasicOperator
func (BasicOperator) Run ¶ added in v0.11.697
func (o BasicOperator) Run(ctx context.Context, ti scheduler.TaskInstance) error
type Client ¶ added in v0.11.697
type Client struct {
// contains filtered or unexported fields
}
func (*Client) CreateSchemaIfNotExist ¶ added in v0.11.697
func (*Client) GetDatabaseSummary ¶ added in v0.11.697
func (*Client) RunQueriesWithoutResult ¶ added in v0.11.697
func (*Client) RunQueryWithoutResult ¶ added in v0.11.697
func (*Client) SelectWithSchema ¶ added in v0.11.697
func (*Client) TableExists ¶ added in v0.11.697
type ClientResolver ¶ added in v0.11.708
type ClientResolver interface {
ResolveSparkClient(ctx context.Context, name string) (*Client, error)
}
ClientResolver allows execution backends such as Microsoft Fabric to derive a Spark client from another Bruin connection type.
type Config ¶ added in v0.11.697
type Config struct {
URI string
Catalog string
IngestLocation string
IngestStagingArea string
Options map[string]string
}
Config contains the options passed to the ADBC Spark driver. URI uses the driver's spark:// URI format; Options exposes driver-specific settings that are not modeled explicitly by Bruin.
func (Config) GetDatabase ¶ added in v0.11.697
func (Config) IngestOptions ¶ added in v0.11.697
IngestOptions returns statement-level options for ADBC bulk ingestion. The Spark driver does not accept these options when opening a database.
type Renderer ¶ added in v0.11.697
type Renderer struct {
// contains filtered or unexported fields
}
Renderer applies Spark's multi-statement execution semantics to an already-rendered script before materializing its main query.
func NewRenderer ¶ added in v0.11.697
func (*Renderer) WrapsHooks ¶ added in v0.11.697
WrapsHooks tells command renderers that Spark has already placed hooks relative to session statements using the same ordering as runtime execution.
type SeedOperator ¶ added in v0.11.697
type SeedOperator struct {
// contains filtered or unexported fields
}
func NewSeedOperator ¶ added in v0.11.697
func NewSeedOperator(connection config.ConnectionGetter, renderer jinja.RendererInterface) *SeedOperator
func (SeedOperator) Run ¶ added in v0.11.697
func (o SeedOperator) Run(ctx context.Context, ti scheduler.TaskInstance) error
type TableSensor ¶ added in v0.11.697
type TableSensor struct {
// contains filtered or unexported fields
}
func NewTableSensor ¶ added in v0.11.697
func NewTableSensor(connection config.ConnectionGetter, sensorMode string) *TableSensor
func (*TableSensor) Run ¶ added in v0.11.697
func (s *TableSensor) Run(ctx context.Context, ti scheduler.TaskInstance) error
Source Files