caramlstore

package

v0.9.2 Latest Latest Go to latest Published: Sep 24, 2023 License: Apache-2.0 Imports: 22 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/raystack/meteor

Links

Open Source Insights

README ¶

caramlstore

Extractor for Machine Learning(ML) Features from caraml-store.

The extractor uses the CoreService.proto service exposed by caraml-store to extract feature tables.

Usage

source:
  name: caramlstore
  scope: caramlstore-stg
  config:
    url: staging.caraml-store.com:80
    max_size_in_mb: 10
    request_timeout: 30s

Inputs

Key	Value	Example	Description	Required?
`url`	`string`	`caraml-store.com:80`	caraml-store's host URL	✅
`max_size_in_mb`	`int`	`10`	Max MB for gRPC client to receive message. Default is 45.	✘
`request_timeout`	`string`	`10s`	Timeout for gRPC requests to caraml-store. Default is 10s.	✘

Outputs

The feature tables are mapped to an Asset with model specific metadata stored using FeatureTable. Please refer the proto definitions for more information.

Field	Value	Sample Value
`resource.urn`	`urn:caramlstore:{scope}:feature_table:{feature_table.project}.{feature_table.name}`	`urn:caramlstore:caramlstore-stg:feature_table:my_project.merchant_uuid_t2_discovery`
`resource.name`	`{feature_table.name}`	`merchant_uuid_t2_discovery`
`resource.service`	`caramlstore`	`caramlstore`
`resource.type`	`feature_table`	`feature_table`
`namespace`	`{feature_table.project}`	`my_project`
`entities`	`[]Entity`
`features`	`[]Feature`
`create_time`	`{feature_table.created_timestamp}`	`2022-08-08T03:17:54Z`
`update_time`	`{feature_table.updated_timestamp}`	`2022-08-08T03:57:54Z`

Entity

Field	Value	Sample Value
`name`	`{entity.name}`	`service_type`
`labels`	`map[string]string`	`{"description":"merchant uuid","value_type":"STRING"}`

Feature

Field	Value	Sample Value
`name`	`{feature.name}`	`avg_gmv_merchant_1d`
`data_type`	`{features.value_type}`	`INT64`

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation ¶

Index ¶

type Client
type Config
type Extractor
- func New(logger log.Logger, client Client) *Extractor

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Client ¶

type Client interface {
	Connect(ctx context.Context, host string, maxSizeInMB int, timeout time.Duration) error
	Projects(ctx context.Context) ([]string, error)
	Entities(ctx context.Context, project string) (map[string]*core.Entity, error)
	FeatureTables(ctx context.Context, project string) ([]*core.FeatureTable, error)
	Close() error
}

type Config ¶

type Config struct {
	URL            string        `json:"url" yaml:"url" mapstructure:"url" validate:"required"`
	MaxSizeInMB    int           `json:"max_size_in_mb" yaml:"max_size_in_mb" mapstructure:"max_size_in_mb"`
	RequestTimeout time.Duration `json:"request_timeout" yaml:"request_timeout" mapstructure:"request_timeout" validate:"min=1ms" default:"10s"`
}

Config holds the set of configuration for the CaraML Store extractor

type Extractor ¶

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the communication with the CaraML Store service

func New ¶

func New(logger log.Logger, client Client) *Extractor

func (*Extractor) Close ¶ added in v0.9.1

func (e *Extractor) Close() error

func (*Extractor) Extract ¶

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the table is valid and extracts the table schema

func (*Extractor) Init ¶

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Init initializes the extractor

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
internal
core
mocks

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL