sources

package
v0.3.29 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 14, 2026 License: MIT Imports: 16 Imported by: 3

Documentation

Index

Constants

View Source 
const (
	Postgres  types.DataSourceType = "postgres"
	DuckDB    types.DataSourceType = "duckdb"
	MySQL     types.DataSourceType = "mysql"
	MSSQL     types.DataSourceType = "mssql"
	Http      types.DataSourceType = "http"
	Runtime   types.DataSourceType = "runtime"
	Extension types.DataSourceType = "extension"
	Embedding types.DataSourceType = "embedding"

	Airport  types.DataSourceType = "airport"
	DuckLake types.DataSourceType = "ducklake"
	Iceberg  types.DataSourceType = "iceberg"
	HugrApp  types.DataSourceType = "hugr-app"

	LLMOpenAI    types.DataSourceType = "llm-openai"
	LLMAnthropic types.DataSourceType = "llm-anthropic"
	LLMGemini    types.DataSourceType = "llm-gemini"
	Redis        types.DataSourceType = "redis"
)

Variables

View Source 
var (
	ErrDataSourceNotFound                  = errors.New("data source not found")
	ErrDataSourceExists                    = errors.New("data source already exists")
	ErrDataSourceAttached                  = errors.New("data source is attached")
	ErrDataSourceNotAttached               = errors.New("data source is not attached")
	ErrUnknownDataSourceType               = errors.New("unknown data source type")
	ErrDataSourceAttachedWithDifferentType = errors.New("data source already attached with different type exists")
	ErrEmptyQuery                          = errors.New("empty query")
	ErrQueryParsingFailed                  = errors.New("query parsing failed")
	ErrInvalidDataSourcePath               = errors.New("invalid data source path")
)

Functions

func ApplyEnvVars added in v0.1.9 

func ApplyEnvVars(dsn string) (string, error)

func CheckDBExists 

func CheckDBExists(ctx context.Context, db *db.Pool, name string, dsType types.DataSourceType) error

func EnsurePgDatabase added in v0.3.3 

func EnsurePgDatabase(ctx context.Context, host, port, user, password, dbName string) error

EnsurePgDatabase connects to the PostgreSQL "postgres" maintenance database and creates the target database if it doesn't exist. This is idempotent — it does nothing if the database already exists.

NOTE: The connection to the maintenance database uses sslmode=disable because it targets the same PostgreSQL instance as the main DuckLake connection. In production, the DuckLake PostgreSQL backend should be co-located (same network) with the Hugr query engine, so TLS is not required for the maintenance connection. If your PostgreSQL instance requires TLS, do not use ensure_db — create the database manually instead.

func EscapeSQLString added in v0.3.3 

func EscapeSQLString(s string) string

EscapeSQLString escapes single quotes in SQL string literals by doubling them.

Types

type DataSourceResolver added in v0.3.16 

type DataSourceResolver interface {
	Resolve(name string) (Source, error)
	ResolveAll() []Source
}

DataSourceResolver resolves registered data sources by name.

type EmbeddingResult added in v0.3.16 

type EmbeddingResult = types.EmbeddingResult

Type aliases for convenience — re-export from types sub-module.

type EmbeddingSource added in v0.1.28 

type EmbeddingSource interface {
	ModelSource
	CreateEmbedding(ctx context.Context, input string) (*types.EmbeddingResult, error)
	CreateEmbeddings(ctx context.Context, input []string) (*types.EmbeddingsResult, error)
}

type EmbeddingsResult added in v0.3.16 

type EmbeddingsResult = types.EmbeddingsResult

Type aliases for convenience — re-export from types sub-module.

type ExtensionSource added in v0.1.12 

type ExtensionSource interface {
	IsExtension() bool
}

The data source is a catalog extension.

type HeartbeatConfig added in v0.3.11 

type HeartbeatConfig struct {
	Interval   time.Duration
	Timeout    time.Duration
	MaxRetries int
}

HeartbeatConfig holds heartbeat monitoring settings.

type Heartbeater added in v0.3.11 

type Heartbeater interface {
	StartHeartbeat(
		config HeartbeatConfig,
		onSuspend func(ctx context.Context, name string) error,
		onRecover func(ctx context.Context, name string) error,
	)
	StopHeartbeat()
}

Heartbeater is implemented by sources that need periodic health monitoring. The service calls StartHeartbeat after successful Attach and StopHeartbeat before Detach.

type LLMMessage added in v0.3.16 

type LLMMessage = types.LLMMessage

Type aliases for convenience — re-export from types sub-module.

type LLMOptions added in v0.3.16 

type LLMOptions = types.LLMOptions

Type aliases for convenience — re-export from types sub-module.

type LLMResult added in v0.3.16 

type LLMResult = types.LLMResult

Type aliases for convenience — re-export from types sub-module.

type LLMSource added in v0.3.16 

type LLMSource interface {
	ModelSource
	CreateCompletion(ctx context.Context, prompt string, opts types.LLMOptions) (*types.LLMResult, error)
	CreateChatCompletion(ctx context.Context, messages []types.LLMMessage, opts types.LLMOptions) (*types.LLMResult, error)
}

LLMSource provides text generation capabilities.

type LLMStreamEvent added in v0.3.20 

type LLMStreamEvent = types.LLMStreamEvent

Type alias for LLMStreamEvent convenience.

type LLMStreamingSource added in v0.3.20 

type LLMStreamingSource interface {
	LLMSource
	CreateChatCompletionStream(ctx context.Context, messages []types.LLMMessage, opts types.LLMOptions,
		onEvent func(event *types.LLMStreamEvent) error) error
}

LLMStreamingSource extends LLMSource with streaming completion support.

type LLMTool added in v0.3.16 

type LLMTool = types.LLMTool

Type aliases for convenience — re-export from types sub-module.

type LLMToolCall added in v0.3.16 

type LLMToolCall = types.LLMToolCall

Type aliases for convenience — re-export from types sub-module.

type ModelInfo added in v0.3.16 

type ModelInfo = types.ModelInfo

Type aliases for convenience — re-export from types sub-module.

type ModelSource added in v0.3.16 

type ModelSource interface {
	ModelInfo() types.ModelInfo
}

ModelSource identifies a data source as an AI model.

type ParsedDSN 

type ParsedDSN struct {
	Proto    string
	Host     string
	Port     string
	User     string
	Password string
	DBName   string

	Params map[string]string
}

func ParseDSN 

func ParseDSN(dsn string) (ParsedDSN, error)

func (ParsedDSN) String added in v0.1.9 

func (p ParsedDSN) String() string

type Provisioner added in v0.3.11 

type Provisioner interface {
	Provision(ctx context.Context, querier types.Querier) error
}

Provisioner is implemented by sources that need to provision external resources (databases, schemas) after attachment. Called by the data source service after Attach() succeeds. Querier provides access to hugr's GraphQL API for registering/loading data sources and querying system configuration.

type PubSubStoreSource added in v0.3.20 

type PubSubStoreSource interface {
	StoreSource
	Subscribe(ctx context.Context, channel string) (*SubscriptionResult, error)
	PSubscribe(ctx context.Context, pattern string) (*SubscriptionResult, error)
	Publish(ctx context.Context, channel string, message string) error
	ConfigGet(ctx context.Context, key string) (string, error)
	ConfigSet(ctx context.Context, key string, value string) error
}

PubSubStoreSource extends StoreSource with Pub/Sub capabilities.

type Querier added in v0.3.11 

type Querier = types.Querier

Querier provides GraphQL query access for provisioning operations.

type RuntimeSource 

type RuntimeSource interface {
	Name() string
	Engine() engines.Engine
	IsReadonly() bool
	AsModule() bool
	Attach(ctx context.Context, db *db.Pool) error
	Catalog(ctx context.Context) (cs.Catalog, error)
}

RuntimeSource is a data source that is attached on start and provides a catalog source.

type RuntimeSourceDataSourceUser added in v0.3.16 

type RuntimeSourceDataSourceUser interface {
	DataSourceServiceSetup(resolver DataSourceResolver)
}

RuntimeSourceDataSourceUser is implemented by runtime sources that need access to registered data sources. Follows RuntimeSourceQuerier pattern.

type RuntimeSourceQuerier 

type RuntimeSourceQuerier interface {
	QueryEngineSetup(querier types.Querier)
}

type SelfDescriber 

type SelfDescriber interface {
	CatalogSource(ctx context.Context, db *db.Pool) (cs.Catalog, error)
}

type Source 

type Source interface {
	Name() string
	Definition() types.DataSource
	Engine() engines.Engine
	IsAttached() bool
	ReadOnly() bool
	Attach(ctx context.Context, db *db.Pool) error
	Detach(ctx context.Context, db *db.Pool) error
}

type SourceDataSourceUser added in v0.3.16 

type SourceDataSourceUser interface {
	SetDataSourceResolver(resolver DataSourceResolver)
}

SourceDataSourceUser is implemented by regular (non-runtime) data sources that need access to other data sources (e.g., LLM sources needing Redis for rate limiting).

type StoreSource added in v0.3.16 

type StoreSource interface {
	Get(ctx context.Context, key string) (string, bool, error)
	Set(ctx context.Context, key string, value string, ttlSeconds int) error
	Del(ctx context.Context, key string) error
	Incr(ctx context.Context, key string) (int64, error)
	IncrBy(ctx context.Context, key string, value int64) (int64, error)
	Expire(ctx context.Context, key string, ttlSeconds int) error
	Keys(ctx context.Context, pattern string) ([]string, error)
}

StoreSource provides key-value store operations. Named StoreSource (not RedisSource) to allow future non-Redis implementations.

type SubscriptionResult added in v0.3.20 

type SubscriptionResult struct {
	Reader array.RecordReader // Yields records incrementally; schema included.
	Cancel func()             // Stops the subscription.
}

SubscriptionResult is returned by SubscriptionSource.Subscribe.

type SubscriptionSource added in v0.3.20 

type SubscriptionSource interface {
	Subscribe(ctx context.Context, field *ast.Field, vars map[string]any) (*SubscriptionResult, error)
}

SubscriptionSource is implemented by data sources that handle native subscriptions.

Source Files

View all Source files

Directories

Path Synopsis
Package llm implements LLM data source types for AI model providers.
 Package llm implements LLM data source types for AI model providers.
Package ratelimit provides request and token rate limiting for LLM sources.
 Package ratelimit provides request and token rate limiting for LLM sources.
auth
core-db
data-sources
db
ducklake
gis
meta-info
models
storage
store

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.