spark

package

v0.11.430 Latest Latest Go to latest Published: Jan 27, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/bruin-data/bruin

Links

Documentation ¶

Overview ¶

Package spark provides shared utilities for Spark-based execution environments such as EMR Serverless and Dataproc Serverless.

Index ¶

Variables
func Exclude(path string) bool
func PackageContext(zw *zip.Writer, context fs.FS) error

Constants ¶

This section is empty.

Variables ¶

View Source

var BruinExcludes = []string{
	"README.md",
	".bruin.yml",
	"pipeline.yml",
	"pipeline.yaml",
}

BruinExcludes contains files that should be excluded from Spark context packages.

View Source

var DirExcludes = []*regexp.Regexp{
	regexp.MustCompile(`(^|[/\\])\.venv([/\\]|$)`),
	regexp.MustCompile(`(^|[/\\])venv([/\\]|$)`),
	regexp.MustCompile(`^logs([/\\]|$)`),
	regexp.MustCompile(`^\.git([/\\]|$)`),
}

DirExcludes contains regex patterns for directories that should be excluded from Spark context packages.

Functions ¶

func Exclude ¶

func Exclude(path string) bool

Exclude returns true if the given path should be excluded from packaging.

func PackageContext ¶

func PackageContext(zw *zip.Writer, context fs.FS) error

PackageContext creates a zip archive from the given filesystem, suitable for Spark execution. It's a modified version of zip.AddFS() with:

Exclusion of Bruin configuration files and virtual environments
Automatic creation of __init__.py files in directories for Python package support

Spark requires directories to contain __init__.py to be treated as packages.

Types ¶

This section is empty.

Source Files ¶

View all Source files

package.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL