optimus

package

v0.12.0 Latest Latest Go to latest Published: Mar 28, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/raystack/meteor

Links

Open Source Insights

README ¶

optimus

Usage

source:
  type: optimus
  config:
    host: optimus.com:80

Inputs

Key	Value	Example	Description
`host`	`string`	`optimus.com:80`	Optimus' GRPC host	required
`max_size_in_mb`	`int`	`45`	Max megabytes for GRPC client to receive message. Default to 45.

Outputs

Field	Sample Value
`resource.urn`	`optimus::https://optimus-host.com/project.namespace.job`
`resource.name`	`job-name`
`resource.service`	`optimus`
`resource.description`	`Sample job description`
`ownership.owners[0].urn`	`john_doe@example.com`
`ownership.owners[0].name`	`john_doe@example.com`
`lineage.upstreams[].urn`	`bigquery::project/dataset/table`
`lineage.upstreams[].type`	`table`
`lineage.upstreams[].service`	`bigquery`
`lineage.downstreams[0].urn`	`bigquery::project/dataset/table`
`lineage.downstreams[0].type`	`table`
`lineage.downstreams[0].service`	`bigquery`
`properties.attributes`	`{}`

MaxCompute Support

The Optimus extractor supports MaxCompute as a dependency source. When a job references MaxCompute resources (with the maxcompute:// prefix), the extractor will:

Parse MaxCompute fully-qualified names in the format maxcompute://project.schema.table
Generate URNs using maxcompute://{project}.{schema}.{table}
Include MaxCompute resources as upstream or downstream lineage entries with service maxcompute

You can also specify a project_id in the extractor config to scope lineage extraction to a specific project.

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation ¶

Index ¶

type Config
type Extractor
- func New(l log.Logger, c client.Client) *Extractor
- func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error
- func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Config ¶

type Config struct {
	Host        string `json:"host" yaml:"host" mapstructure:"host" validate:"required"`
	MaxSizeInMB int    `json:"max_size_in_mb" yaml:"max_size_in_mb" mapstructure:"max_size_in_mb"`
}

Config holds the set of configuration for the bigquery extractor

type Extractor ¶

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the communication with the bigquery service

func New ¶

func New(l log.Logger, c client.Client) *Extractor

func (*Extractor) Extract ¶

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the table is valid and extracts the table schema

func (*Extractor) Init ¶

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Init initializes the extractor

Source Files ¶

View all Source files

optimus.go

Directories ¶

Path	Synopsis
client

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL