core

package
v0.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 16, 2019 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Index

Constants

View Source
const CfgRoot = "pipeline."

CfgRoot configuration root path

Variables

This section is empty.

Functions

func CfgAuthOpts

func CfgAuthOpts(cfg *viper.Viper) *shttp.AuthenticationOpts

CfgAuthOpts creates the auth options form configuration

func NewSubscriber

func NewSubscriber(pipeline *Pipeline, cfg *viper.Viper) (*websocket.StructSpeaker, error)

NewSubscriber returns a new flow subscriber writing to object store

func SubscriberRun

func SubscriberRun(s *websocket.StructSpeaker)

SubscriberRun runs the subscriber under main

Types

type Classifier

type Classifier interface {
	GetFlowTag(fl *flow.Flow) Tag
}

Classifier exposes the interface for tag based classification

func NewClassify

func NewClassify(cfg *viper.Viper) (Classifier, error)

NewClassify returns a new classify, based on the given cluster net masks

type Compressor

type Compressor interface {
	Compress(b []byte) (bytes.Buffer, error)
}

Compressor exposes the interface for compressesing encoded flows

func NewCompressGzip

func NewCompressGzip() (Compressor, error)

NewCompressGzip create an encode object

type Encoder

type Encoder interface {
	Encode(in []interface{}) ([]byte, error)
}

Encoder exposes the interface for encoding flows

func NewEncodeJSON

func NewEncodeJSON() (Encoder, error)

NewEncodeJSON create an encode object

type Filterer

type Filterer interface {
	IsExcluded(tag Tag) bool
}

Filterer exposes the interface for tag based filtering

func NewFilter

func NewFilter(tags ...string) (Filterer, error)

NewFilter returns a new filter based on a list of excluded tags

func NewFilterFromConfig

func NewFilterFromConfig(cfg *viper.Viper) (Filterer, error)

NewFilterFromConfig returns a new filter based on config

type Pipeline

type Pipeline struct {
	Transformer Transformer
	Classifier  Classifier
	Filterer    Filterer
	Encoder     Encoder
	Compressor  Compressor
	Storer      Storer
}

Pipeline manager

func NewPipeline

func NewPipeline(transformer Transformer, classifier Classifier, filterer Filterer, encoder Encoder, compressor Compressor, storer Storer) *Pipeline

NewPipeline defines the pipeline elements

func (*Pipeline) OnStructMessage

func (p *Pipeline) OnStructMessage(c ws.Speaker, msg *ws.StructMessage)

OnStructMessage is triggered when WS server sends us a message.

type StoreS3

type StoreS3 struct {
	// contains filtered or unexported fields
}

StoreS3 allows writing flows to an object storage service

func NewStoreS3

func NewStoreS3(client objectStoreClient, bucket, objectPrefix string, maxFlowArraySize, maxFlowsPerObject, maxSecondsPerObject, maxSecondsPerStream int) (*StoreS3, error)

NewStoreS3 creates a store

func NewStoreS3FromConfig

func NewStoreS3FromConfig(cfg *viper.Viper) (*StoreS3, error)

NewStoreS3FromConfig returns a new storage interface for storing flows to object store

func (*StoreS3) DeleteObject

func (s *StoreS3) DeleteObject(objectKey *string) error

DeleteObject deletes an object

func (*StoreS3) ListObjects

func (s *StoreS3) ListObjects() ([]*string, error)

ListObjects lists all stored objects

func (*StoreS3) ReadObjectFlows

func (s *StoreS3) ReadObjectFlows(objectKey *string, objectFlows interface{}) error

ReadObjectFlows reads flows from object

func (*StoreS3) SetPipeline

func (s *StoreS3) SetPipeline(pipeline *Pipeline)

SetPipeline setup

func (*StoreS3) StoreFlows

func (s *StoreS3) StoreFlows(flows []*flow.Flow) error

StoreFlows store flows in memory, before being written to the object store

type Storer

type Storer interface {
	StoreFlows(flows []*flow.Flow) error
	SetPipeline(p *Pipeline)
}

Storer interface of a store object

func NewStoreFromConfig

func NewStoreFromConfig(cfg *viper.Viper) (Storer, error)

NewStoreFromConfig creates store from config

func NewStoreLog

func NewStoreLog() (Storer, error)

NewStoreLog returns a new storage interface for storing flows to object store

type Tag

type Tag string

Tag represents the flow classification

type Transformer

type Transformer interface {
	// Transform transforms a flow before being stored
	Transform(f *flow.Flow) interface{}
}

Transformer allows generic transformations of a flow

Source Files

  • classify.go
  • compress.go
  • encode.go
  • filter.go
  • pipeline.go
  • s3_client.go
  • store.go
  • store_log.go
  • store_s3.go
  • subscriber.go
  • transform.go

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL