bundlev1

package
v1.0.0-beta2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 11, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package bundlev1 defines the interfaces and types need to be an Airflow Dag Bundle

The main entry point should call bundlev1server/Serve.

Package bundlev1 defines the types and interfaces needed to implement v1 of the Bundle Plugin

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Bundle

type Bundle = worker.Bundle

Bundle is the execution-time view of a registry: it looks up a task by dag_id and task_id. Registry embeds it so the object built during RegisterDags can also serve tasks when they run.

type BundleInfo

type BundleInfo = api.BundleInfo

BundleInfo identifies a bundle by name and version. It is returned by BundleProvider.GetBundleVersion and tells Airflow which bundle to run a task with.

type BundleProvider

type BundleProvider interface {
	// GetBundleVersion returns upfront information about the bundle name and version without needing to load
	// the full dag and task information, which could be memory intensive.
	GetBundleVersion() BundleInfo

	// RegisterDags declares every dag and task in this bundle on the supplied
	// Registry, for example:
	//
	//	func (m *myBundle) RegisterDags(dagbag bundlev1.Registry) error {
	//		dag := dagbag.AddDag("simple_dag")
	//		dag.AddTask(extract)
	//		dag.AddTask(transform)
	//		return nil
	//	}
	//
	// Register all dags and tasks here. It is called once per process per
	// bundle and cached internally, so you do not have to cache it yourself.
	RegisterDags(Registry) error
}

BundleProvider is the single interface a bundle author implements. Construct one in main and pass it to bundlev1server.Serve; the runtime calls GetBundleVersion to identify the bundle and RegisterDags to load its tasks.

type Dag

type Dag interface {
	// AddTask registers fn as a task, deriving the task_id from fn's own
	// name (so it must match the @task.stub name in the Python dag).
	//
	// fn is an ordinary Go function whose parameters are injected by type
	// and may appear in any order. Recognised parameters are:
	//   - context.Context: cancelled when the task is asked to stop
	//   - *slog.Logger: writes to the task's Airflow log
	//   - sdk.Client (or a narrower sdk.VariableClient / sdk.ConnectionClient):
	//     access to Variables, Connections, and XCom
	//
	// fn must return either error or (result, error): a non-nil error fails
	// the task, and a non-nil first result is pushed as the task's
	// return-value XCom. Passing a non-function, or a function whose return
	// signature does not match, panics at registration time.
	AddTask(fn any)

	// AddTaskWithName is like AddTask but sets task_id explicitly instead of
	// deriving it from the function name. Use it when the Go function name
	// cannot match the Python @task.stub id, for example for an anonymous
	// function or a differing name.
	AddTaskWithName(taskId string, fn any)
}

Dag is the handle returned by Registry.AddDag. Use it to attach the Go functions that implement the dag's tasks.

type DagInfo

type DagInfo struct {
	DagID string
	Tasks []TaskInfo
}

DagInfo describes a registered dag together with its tasks in registration order.

type EnumerableBundle

type EnumerableBundle interface {
	OrderedDags() []DagInfo
}

EnumerableBundle exposes the dag/task identity recorded by RegisterDags. The default registry implements it; airflow-go-pack relies on it to read a bundle's dag/task ids without executing any task.

type ExecuteTaskWorkload

type ExecuteTaskWorkload struct {
	Token string `json:"token"`

	TI         api.TaskInstance `json:"ti"`
	BundleInfo api.BundleInfo   `json:"bundle_info"`
	LogPath    *string          `json:"log_path,omitempty"`
}

ExecuteTaskWorkload is the runtime payload describing one task to run: its TaskInstance, the bundle to load, and an optional log path. The worker delivers it to the bundle; authors do not build it themselves.

type GetMetadataResponse

type GetMetadataResponse struct {
	Bundle BundleInfo
}

GetMetadataResponse is the runtime reply that carries a bundle's BundleInfo back over the go-plugin transport. It is plumbing between the worker and the bundle, not something authors construct.

type Registry

type Registry interface {
	Bundle
	// AddDag registers a dag by its dag_id (matching the Python stub dag)
	// and returns a Dag handle for attaching tasks. Registering the same
	// dag_id twice panics.
	AddDag(dagId string) Dag
}

Registry is the recorder passed to BundleProvider.RegisterDags. Use it to declare the dags this bundle can run; it also extends Bundle so the same object serves task lookups at execution time.

func New

func New() Registry

New returns an empty Registry on which dags and tasks can be registered. The runtime creates one and hands it to BundleProvider.RegisterDags, so bundle authors rarely call this directly; it is handy for unit-testing a RegisterDags implementation.

type Task

type Task = worker.Task

Task is one registered task: something the runtime can Execute. Bundle authors do not implement this directly; Dag.AddTask wraps a plain Go function into a Task for you.

func NewTaskFunction

func NewTaskFunction(fn any) (Task, error)

NewTaskFunction wraps a plain Go function as a Task, validating its signature (injectable parameters, and a return of error or (result, error)). Bundle authors normally use Dag.AddTask, which calls this for them; use it directly only when building a Task outside the registry.

type TaskInfo

type TaskInfo struct {
	ID string
}

TaskInfo describes a registered task by its user-visible id.

type TaskInstance

type TaskInstance = api.TaskInstance

TaskInstance identifies the running task by dag_id, run_id, task_id, and map_index. It is an alias for the Execution-API type and is what XComClient.PushXCom takes.

Directories

Path Synopsis
Package bundlev1client implements the worker-side Client to speak to Dag bundles compiled with the github.com/apache/airflow/go-sdk/bundle/bundlev1/bundlev1server package.
Package bundlev1client implements the worker-side Client to speak to Dag bundles compiled with the github.com/apache/airflow/go-sdk/bundle/bundlev1/bundlev1server package.
Package bundlev1server implements a server implementation to run bundlev1 as gRPC servers, making it accessible to the Airflow Go Workers.
Package bundlev1server implements a server implementation to run bundlev1 as gRPC servers, making it accessible to the Airflow Go Workers.
impl
Package impl contains internal GPRC implementation types.
Package impl contains internal GPRC implementation types.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL