optimus

package

v0.9.0 Latest Latest Go to latest Published: Jul 26, 2023 License: Apache-2.0 Imports: 20 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/raystack/meteor

Links

Open Source Insights

README ¶

optimus

Usage

source:
  type: optimus
  config:
    host: optimus.com:80

Inputs

Key	Value	Example	Description
`host`	`string`	`optimus.com:80`	Optimus' GRPC host	required
`max_size_in_mb`	`int`	`45`	Max megabytes for GRPC client to receive message. Default to 45.

Outputs

Field	Sample Value
`resource.urn`	`optimus::https://optimus-host.com/project.namespace.job`
`resource.name`	`job-name`
`resource.service`	`optimus`
`resource.description`	`Sample job description`
`ownership.owners[0].urn`	`john_doe@example.com`
`ownership.owners[0].name`	`john_doe@example.com`
`lineage.upstreams[].urn`	`bigquery::project/dataset/table`
`lineage.upstreams[].type`	`table`
`lineage.upstreams[].service`	`bigquery`
`lineage.downstreams[0].urn`	`bigquery::project/dataset/table`
`lineage.downstreams[0].type`	`table`
`lineage.downstreams[0].service`	`bigquery`
`properties.attributes`	`{}`

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation ¶

Index ¶

Constants
type Client
type Config
type Extractor
- func New(logger log.Logger, client Client) *Extractor
- func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error
- func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Constants ¶

View Source

const (
	GRPCMaxClientSendSizeMB      = 45
	GRPCMaxClientRecvSizeMB      = 45
	GRPCMaxRetry            uint = 3
)

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Client ¶

type Client interface {
	pb.NamespaceServiceClient
	pb.ProjectServiceClient
	pb.JobSpecificationServiceClient
	pb.JobRunServiceClient
	Connect(ctx context.Context, host string, maxSizeInMB int) error
	Close() error
}

type Config ¶

type Config struct {
	Host        string `json:"host" yaml:"host" mapstructure:"host" validate:"required"`
	MaxSizeInMB int    `json:"max_size_in_mb" yaml:"max_size_in_mb" mapstructure:"max_size_in_mb"`
}

Config holds the set of configuration for the bigquery extractor

type Extractor ¶

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the communication with the bigquery service

func New ¶

func New(logger log.Logger, client Client) *Extractor

func (*Extractor) Extract ¶

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the table is valid and extracts the table schema

func (*Extractor) Init ¶

func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Init initializes the extractor

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL