flow

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 7, 2022 License: Apache-2.0 Imports: 11 Imported by: 0

README

Flow Distributed Workflow System

Flow Server

The Flow server manages job submissions. It provides a GRPC API that is used by clients to submit and manage workflows.

Flow CLI

The Flow CLI manages workflows from the command line. You can queue, view, and delete workflows.

NAME:
   fctl workflows queue - queue workflow

USAGE:
   fctl workflows queue [command options] [arguments...]

OPTIONS:
   --name value, -n value               workflow instance name
   --type value, -t value               workflow type
   --input-file value, -f value         file path to use as workflow input
   --input-workflow-id value, -w value  output from another workflow to use as input (can be multiple)
   --priority value                     workflow priority (low, normal, urgent) (default: "normal")
   --parameter value, -p value          specify workflow parameters (KEY=VAL)
   --help, -h                           show help (default: false)

Redis

Redis is used for database storage for workflow metadata, user accounts, and queueing.

Minio

Minio is used for workflow input and output storage. Minio is lightweight and works very well with Flow but any compatible S3 system should work.

Flow Processors

Flow processors provide the processing logic for queued workflows. Processors use GRPC to access the Flow server which streams workflow events to the corresponding processor type.

Documentation

Overview

Copyright 2022 Evan Hazlett

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const (
	// QueueSubjectJobPriorityNormal is normal priority for workflows
	QueueSubjectJobPriorityNormal = "normal"
	// QueueSubjectJobPriorityUrgent is urgent priority for workflows (processed first)
	QueueSubjectJobPriorityUrgent = "urgent"
	// QueueSubjectJobPriorityLow is low priority for workflows (processed last)
	QueueSubjectJobPriorityLow = "low"

	// S3WorkflowPath is the path in the s3 bucket for workflow content
	S3WorkflowPath = "workflows"

	// CtxTokenKey is the key stored in the context for the token
	CtxTokenKey = "token"
	// CtxAPITokenKey is the user api token key stored in the context
	CtxAPITokenKey = "api-token"
	// CtxServiceTokenKey is the service key stored in the context
	CtxServiceTokenKey = "service-token"
	// CtxTokenKey is the key stored in the context for the username
	CtxUsernameKey = "username"
	// CtxTokenKey is the key stored in the context if the user is an admin
	CtxAdminKey = "isAdmin"
	// CtxNamespaceKey is the key stored in the context for the namespace
	CtxNamespaceKey = "namespace"
	// CtxDefaultNamespace is the default key used when unauthenticated and no auth
	CtxDefaultNamespace = "default"
)

Variables

This section is empty.

Functions

func GenerateHash 

func GenerateHash(v ...string) string

GenerateHash generates a sha256 hash of the string

func GenerateToken 

func GenerateToken(v ...string) string

GenerateToken returns a sha256 of the specified data

func GetContentType 

func GetContentType(f *os.File) (string, error)

GetContentType returns the content type of the specified file

func GetMinioClient 

func GetMinioClient(cfg *Config) (*minio.Client, error)

GetMinioClient returns a MinIO client using the specified flow.Config

func GetNamespaceFromContext 

func GetNamespaceFromContext(ctx context.Context) (string, error)

GetNamespaceFromContext returns the namespace from the specified context

func GetTokenFromContext 

func GetTokenFromContext(ctx context.Context) (string, error)

GetTokenFromContext returns the first auth token from the context

Types

type AuthenticatorConfig 

type AuthenticatorConfig struct {
	// Name is the name of the authenticator
	Name string
	// Options are passed to the authenticator
	Options map[string]string
}

type Config 

type Config struct {
	// GRPCAddress is the address for the grpc server
	GRPCAddress string
	// TLSCertificate is the certificate used for grpc communication
	TLSServerCertificate string
	// TLSKey is the key used for grpc communication
	TLSServerKey string
	// TLSClientCertificate is the client certificate used for client communication
	TLSClientCertificate string
	// TLSClientKey is the client key used for client communication
	TLSClientKey string
	// TLSInsecureSkipVerify disables certificate verification
	TLSInsecureSkipVerify bool
	// S3Endpoint is the endpoint for the S3 compatible service
	S3Endpoint string
	// S3AccessID is the S3 access id
	S3AccessID string
	// S3AccessKey is the S3 key
	S3AccessKey string
	// S3Bucket is the S3 bucket
	S3Bucket string
	// S3UseSSL enables SSL for the S3 service
	S3UseSSL bool
	// DatabaseAddress is the address of the database
	DatabaseAddress string
	// QueueAddress is the address of the queue
	QueueAddress string
	// ProfilerAddress enables the performance profiler on the specified address
	ProfilerAddress string
	// TraceEndpoint is the endpoint of the telemetry tracer
	TraceEndpoint string
	// Environment is the environment the app is running in
	Environment string
	// InitialAdminPassword is the password used when creating the initial admin account. If empty, a random one is generated.
	InitialAdminPassword string
	// Authenticator is the auth configuration
	Authenticator *AuthenticatorConfig
}

Config is the configuration used for the server

func DefaultConfig 

func DefaultConfig() *Config

func LoadConfig 

func LoadConfig(configPath string) (*Config, error)

LoadConfig returns a Flow config from the specified file path

Source Files

View all Source files

Directories

Path Synopsis
api
services/accounts/v1
services/info/v1
services/workflows/v1
cmd
fctl command
flow command
flow-workflow-example command
pkg
auth
auth/providers/none
auth/providers/token
middleware
middleware/admin
profiler
queue
tracing
workflows
accounts
info
workflows

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.