Documentation
¶
Overview ¶
Package bundlev1 defines the interfaces and types need to be an Airflow Dag Bundle
The main entry point should call bundlev1server/Serve.
Package bundlev1 defines the types and interfaces needed to implement v1 of the Bundle Plugin
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Bundle ¶
Bundle is the execution-time view of a registry: it looks up a task by dag_id and task_id. Registry embeds it so the object built during RegisterDags can also serve tasks when they run.
type BundleInfo ¶
type BundleInfo = api.BundleInfo
BundleInfo identifies a bundle by name and version. It is returned by BundleProvider.GetBundleVersion and tells Airflow which bundle to run a task with.
type BundleProvider ¶
type BundleProvider interface {
// GetBundleVersion returns upfront information about the bundle name and version without needing to load
// the full dag and task information, which could be memory intensive.
GetBundleVersion() BundleInfo
// RegisterDags declares every dag and task in this bundle on the supplied
// Registry, for example:
//
// func (m *myBundle) RegisterDags(dagbag bundlev1.Registry) error {
// dag := dagbag.AddDag("simple_dag")
// dag.AddTask(extract)
// dag.AddTask(transform)
// return nil
// }
//
// Register all dags and tasks here. It is called once per process per
// bundle and cached internally, so you do not have to cache it yourself.
RegisterDags(Registry) error
}
BundleProvider is the single interface a bundle author implements. Construct one in main and pass it to bundlev1server.Serve; the runtime calls GetBundleVersion to identify the bundle and RegisterDags to load its tasks.
type Dag ¶
type Dag interface {
// AddTask registers fn as a task, deriving the task_id from fn's own
// name (so it must match the @task.stub name in the Python dag).
//
// fn is an ordinary Go function whose parameters are injected by type
// and may appear in any order. Recognised parameters are:
// - context.Context: cancelled when the task is asked to stop
// - *slog.Logger: writes to the task's Airflow log
// - sdk.Client (or a narrower sdk.VariableClient / sdk.ConnectionClient):
// access to Variables, Connections, and XCom
//
// fn must return either error or (result, error): a non-nil error fails
// the task, and a non-nil first result is pushed as the task's
// return-value XCom. Passing a non-function, or a function whose return
// signature does not match, panics at registration time.
AddTask(fn any)
// AddTaskWithName is like AddTask but sets task_id explicitly instead of
// deriving it from the function name. Use it when the Go function name
// cannot match the Python @task.stub id, for example for an anonymous
// function or a differing name.
AddTaskWithName(taskId string, fn any)
}
Dag is the handle returned by Registry.AddDag. Use it to attach the Go functions that implement the dag's tasks.
type EnumerableBundle ¶
type EnumerableBundle interface {
OrderedDags() []DagInfo
}
EnumerableBundle exposes the dag/task identity recorded by RegisterDags. The default registry implements it; airflow-go-pack relies on it to read a bundle's dag/task ids without executing any task.
type ExecuteTaskWorkload ¶
type ExecuteTaskWorkload struct {
Token string `json:"token"`
TI api.TaskInstance `json:"ti"`
BundleInfo api.BundleInfo `json:"bundle_info"`
LogPath *string `json:"log_path,omitempty"`
}
ExecuteTaskWorkload is the runtime payload describing one task to run: its TaskInstance, the bundle to load, and an optional log path. The worker delivers it to the bundle; authors do not build it themselves.
type GetMetadataResponse ¶
type GetMetadataResponse struct {
Bundle BundleInfo
}
GetMetadataResponse is the runtime reply that carries a bundle's BundleInfo back over the go-plugin transport. It is plumbing between the worker and the bundle, not something authors construct.
type Registry ¶
type Registry interface {
Bundle
// AddDag registers a dag by its dag_id (matching the Python stub dag)
// and returns a Dag handle for attaching tasks. Registering the same
// dag_id twice panics.
AddDag(dagId string) Dag
}
Registry is the recorder passed to BundleProvider.RegisterDags. Use it to declare the dags this bundle can run; it also extends Bundle so the same object serves task lookups at execution time.
type Task ¶
Task is one registered task: something the runtime can Execute. Bundle authors do not implement this directly; Dag.AddTask wraps a plain Go function into a Task for you.
func NewTaskFunction ¶
NewTaskFunction wraps a plain Go function as a Task, validating its signature (injectable parameters, and a return of error or (result, error)). Bundle authors normally use Dag.AddTask, which calls this for them; use it directly only when building a Task outside the registry.
type TaskInfo ¶
type TaskInfo struct {
ID string
}
TaskInfo describes a registered task by its user-visible id.
type TaskInstance ¶
type TaskInstance = api.TaskInstance
TaskInstance identifies the running task by dag_id, run_id, task_id, and map_index. It is an alias for the Execution-API type and is what XComClient.PushXCom takes.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bundlev1client implements the worker-side Client to speak to Dag bundles compiled with the github.com/apache/airflow/go-sdk/bundle/bundlev1/bundlev1server package.
|
Package bundlev1client implements the worker-side Client to speak to Dag bundles compiled with the github.com/apache/airflow/go-sdk/bundle/bundlev1/bundlev1server package. |
|
Package bundlev1server implements a server implementation to run bundlev1 as gRPC servers, making it accessible to the Airflow Go Workers.
|
Package bundlev1server implements a server implementation to run bundlev1 as gRPC servers, making it accessible to the Airflow Go Workers. |
|
impl
Package impl contains internal GPRC implementation types.
|
Package impl contains internal GPRC implementation types. |