optimus

package

v0.13.0 Latest Latest Go to latest Published: Apr 19, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/raystack/meteor

Links

Open Source Insights

README ¶

Raystack Optimus

Extract job metadata from Optimus, including lineage and ownership.

Usage

source:
  name: optimus
  scope: my-optimus
  config:
    host: optimus.com:80
    max_size_in_mb: 45

Configuration

Key	Type	Required	Default	Description
`host`	`string`	Yes		Optimus gRPC host address.
`max_size_in_mb`	`int`	No	`45`	Maximum message size in MB for the gRPC client.

Entities

Type: job
URN format: urn:optimus:{scope}:job:{project}.{namespace}.{job_name}

Properties

Property	Type	Description
`properties.version`	`int32`	Job specification version.
`properties.project`	`string`	Optimus project name.
`properties.project_id`	`string`	Optimus project name (alias).
`properties.namespace`	`string`	Optimus namespace name.
`properties.owner`	`string`	Job owner email.
`properties.interval`	`string`	Job schedule interval.
`properties.depends_on_past`	`bool`	Whether the job depends on past runs.
`properties.task_name`	`string`	Task plugin name.
`properties.window_size`	`string`	Window size.
`properties.window_offset`	`string`	Window offset.
`properties.window_truncate_to`	`string`	Window truncation setting.
`properties.start_date`	`string`	Job start date.
`properties.end_date`	`string`	Job end date.
`properties.sql`	`string`	SQL query from `query.sql` resource (if present).
`properties.task`	`map[string]any`	Task details (`name`, `description`, `image`).

Edges

Edge Type	Source URN	Target URN	Description
`derived_from`	Upstream resource URN	Job URN	Upstream dependency (BigQuery or MaxCompute table).
`generates`	Job URN	Downstream resource URN	Downstream destination (BigQuery or MaxCompute table).
`owned_by`	Job URN	`urn:user:{owner_email}`	Job owner.

MaxCompute support

When a job references MaxCompute resources (prefix maxcompute://), the extractor parses fully-qualified names in the format maxcompute://project.schema.table and generates URNs accordingly.

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation ¶

Index ¶

type Config
type Extractor
- func New(l log.Logger, c client.Client) *Extractor
- func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error
- func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Config ¶

type Config struct {
	Host        string `json:"host" yaml:"host" mapstructure:"host" validate:"required"`
	MaxSizeInMB int    `json:"max_size_in_mb" yaml:"max_size_in_mb" mapstructure:"max_size_in_mb"`
}

Config holds the set of configuration for the bigquery extractor

type Extractor ¶

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the communication with the bigquery service

func New ¶

func New(l log.Logger, c client.Client) *Extractor

func (*Extractor) Extract ¶

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the table is valid and extracts the table schema

func (*Extractor) Init ¶

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Init initializes the extractor

Source Files ¶

View all Source files

optimus.go

Directories ¶

Path	Synopsis
client

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL