runtime

package
v0.78.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 22, 2025 License: Apache-2.0 Imports: 52 Imported by: 0

README

runtime

The runtime a data infrastructure proxy and orchestrator – our data plane. It connects to data infrastructure and is responsible for dashboard queries, parsing code files, reconciling infra state, implementing connectors, enforcing (row-based) access policies, scheduling tasks, triggering reports, and much more.

It's designed as a modular component that can be embedded in local applications (as it is into Rill Developer) or deployed stand-alone in a cloud environment.

Code structure

The base directory contains a Runtime type that represents the lifecycle of the runtime. It ties together the sub-directories:

  • client contains a Go client library for connecting to a runtime server.
  • drivers contains interfaces and drivers for external data infrastructure that the runtime interfaces with (like DuckDB and Druid).
  • metricsview contains the metrics layer that converts metrics definitions and queries to raw SQL queries.
  • parser contains logic for parsing Rill projects.
  • pkg contains utility libraries.
  • queries contains the underlying implementation of the analytical APIs used for profiling and dashboards (note: gradually being replaced by resolvers/)
  • reconcilers contains logic that for each project resource reconciles the desired state expressed in code with the actual state observed in external data systems.
  • resolvers contains implementations of a unified interface for resolving data queries, which is used in the metrics APIs, alerts, reports, and more.
  • server contains a server that implements the runtime's APIs.
  • storage contains logic that wraps a project's persistent file storage (on disk and in an object store).
  • testruntime contains helper functions for initializing a test runtime with test data and test connectors.

Development

Developing the local application

Run rill devtool local. You need to stop and restart it using ctrl+C when you make code changes.

Developing for cloud

In one terminal, start a full cloud development environment except the runtime:

rill devtool start cloud --except runtime

In a separate terminal, start a runtime server:

go run ./cli runtime start

Optionally, deploy a seed project:

rill devtool seed cloud
Running tests

You can run all tests using:

go test ./runtime/...

Configuration

The runtime server is configured using environment variables parsed in cli/cmd/runtime/start.go.

Adding a new endpoint

We define our APIs using gRPC and use gRPC-Gateway to map the RPCs to a RESTful API. See proto/README.md for details.

To add a new endpoint:

  1. Describe the endpoint in proto/rill/runtime/v1/api.proto
  2. Re-generate gRPC and OpenAPI interfaces by running make proto.generate
  3. Copy the new handler signature from the RuntimeServiceServer interface in proto/gen/rill/runtime/v1/api_grpc_pb.go
  4. Paste the handler signature and implement it in a relevant file in runtime/server/

Adding a new analytical query endpoint

  1. Add a new endpoint for the query by following the steps in the section above ("Adding a new endpoint")
  2. Implement the query in runtime/queries by following the instructions in runtime/queries/README.md

Using a DuckDB nightly build

The following steps apply for macOS, but a similar approach should work for Linux.

  1. Download the latest DuckDB nightly from Github from the "Artifacts" section on the newest workflow run here)
  2. Unzip the downloaded archive and copy the libduckdb.dylib file in the libduckdb-osx-universal folder to /usr/local/lib
  • You must use the command-line to copy the file. If you touch it using the Finder, macOS will quarantine it. To remove a quarantine, run: xattr -d com.apple.quarantine libduckdb.dylib.
  1. DuckDB usually does not support older file formats, so delete the stage.db and stage.db.wal files in your dev-project
  2. Add the flag -tags=duckdb_use_lib when running go run or go build to use the nightly build of DuckDB
  • If testing the local frontend, you need to temporarily set it in the dev-runtime script in package.json
  • For details, see Linking DuckDB

Note: DuckDB often makes breaking changes to its APIs, so you may encounter other errors when using a dev version of DuckDB.

Documentation

Index

Constants

View Source 
const (
	ResourceKindProjectParser  string = "rill.runtime.v1.ProjectParser"
	ResourceKindSource         string = "rill.runtime.v1.Source"
	ResourceKindModel          string = "rill.runtime.v1.Model"
	ResourceKindMetricsView    string = "rill.runtime.v1.MetricsView"
	ResourceKindExplore        string = "rill.runtime.v1.Explore"
	ResourceKindMigration      string = "rill.runtime.v1.Migration"
	ResourceKindReport         string = "rill.runtime.v1.Report"
	ResourceKindAlert          string = "rill.runtime.v1.Alert"
	ResourceKindRefreshTrigger string = "rill.runtime.v1.RefreshTrigger"
	ResourceKindTheme          string = "rill.runtime.v1.Theme"
	ResourceKindComponent      string = "rill.runtime.v1.Component"
	ResourceKindCanvas         string = "rill.runtime.v1.Canvas"
	ResourceKindAPI            string = "rill.runtime.v1.API"
	ResourceKindConnector      string = "rill.runtime.v1.Connector"
)

Built-in resource kinds

Variables

View Source 
var AllPermissions = []Permission{
	ManageInstances,
	ReadInstance,
	EditInstance,
	EditTrigger,
	ReadRepo,
	EditRepo,
	ReadObjects,
	ReadOLAP,
	ReadMetrics,
	ReadProfiling,
	ReadAPI,
	ReadResolvers,
	UseAI,
}

AllPermissions is a list of all valid Permission values.

View Source 
var BuiltinAPIs = map[string]*runtimev1.API{}

BuiltinAPIs is a map of built-in APIs (i.e. predefined APIs that are not created dynamically from a project's YAML files.)

View Source 
var ErrAINotConfigured = fmt.Errorf("an AI service is not configured for this instance")
View Source 
var ErrAdminNotConfigured = fmt.Errorf("an admin service is not configured for this instance")
View Source 
var ErrForbidden = errors.New("action not allowed")

ErrForbidden is returned when an action is not allowed.

View Source 
var ErrMetricsViewCachingDisabled = errors.New("metrics_cache_key: caching is disabled")
View Source 
var GlobalProjectParserName = &runtimev1.ResourceName{Kind: ResourceKindProjectParser, Name: "parser"}

GlobalProjectParserName is the name of the instance-global project parser resource that is created for each new instance.

View Source 
var ReconcilerInitializers = make(map[string]ReconcilerInitializer)

ReconcilerInitializers is a registry of reconciler initializers for different resource kinds. There can be only one reconciler per resource kind.

View Source 
var ResolvedSecurityClosed = &ResolvedSecurity{
	access:      nil,
	fieldAccess: nil,
	rowFilter:   "",
	queryFilter: nil,
}

ResolvedSecurityClosed is a ResolvedSecurity that denies access.

View Source 
var ResolvedSecurityOpen = &ResolvedSecurity{
	access:      &truth,
	fieldAccess: nil,
	rowFilter:   "",
	queryFilter: nil,
}

ResolvedSecurityOpen is a ResolvedSecurity that allows access with no restrictions.

View Source 
var ResolverInitializers = make(map[string]ResolverInitializer)

ResolverInitializers tracks resolver initializers by name.

Functions

func NewDependencyError added in v0.56.0 

func NewDependencyError(err error) error

func PrettifyReconcileStatus added in v0.49.0 

func PrettifyReconcileStatus(s runtimev1.ReconcileStatus) string

PrettifyReconcileStatus returns the reconcile status in a user-friendly format suitable for printing.

func PrettifyResourceKind added in v0.49.0 

func PrettifyResourceKind(k string) string

PrettifyResourceKind returns the resource kind in a user-friendly format suitable for printing.

func RegisterBuiltinAPI added in v0.42.0 

func RegisterBuiltinAPI(opts *BuiltinAPIOptions)

RegisterBuiltinAPI adds a built-in API with the given options.

func RegisterReconcilerInitializer added in v0.32.0 

func RegisterReconcilerInitializer(resourceKind string, initializer ReconcilerInitializer)

RegisterReconciler registers a reconciler initializer for a specific resource kind

func RegisterResolverInitializer added in v0.42.0 

func RegisterResolverInitializer(name string, initializer ResolverInitializer)

RegisterResolverInitializer registers a resolver initializer by name.

func ResolveFeatureFlags added in v0.78.0 

func ResolveFeatureFlags(inst *drivers.Instance, userAttributes map[string]any, camelCase bool) (map[string]bool, error)

ResolveFeatureFlags resolves feature flags for the given instance and the provided user attributes. Set the camelCase flag to true when the flags will be returned to the UI since it currently expects the keys in that format. NOTE: Currently, feature flags are mainly honored in the UI. They are only enforced on the backend in a few places.

func ResourceKindFromParser added in v0.60.7 

func ResourceKindFromParser(kind parser.ResourceKind) string

ResourceKindFromParser converts a parser resource kind to a runtime resource kind.

func ResourceKindFromShorthand added in v0.58.0 

func ResourceKindFromShorthand(kind string) string

ResourceKindFromPretty converts a user-friendly resource kind to a runtime resource kind. If the kind doesn't match a known shorthand, it is returned as-is.

func ResourceKindToParser added in v0.60.7 

func ResourceKindToParser(kind string) parser.ResourceKind

ResourceKindToParser converts a runtime resource kind to a parser resource kind.

func ResourceNameFromParser added in v0.60.7 

func ResourceNameFromParser(name parser.ResourceName) *runtimev1.ResourceName

ResourceNameFromParser converts a parser resource name to a runtime resource name.

func ResourceNameToParser added in v0.60.7 

func ResourceNameToParser(name *runtimev1.ResourceName) parser.ResourceName

ResourceNameToParser converts a runtime resource name to a parser resource name.

func SelfAllowRuleAccess added in v0.78.0 

func SelfAllowRuleAccess(res *runtimev1.Resource) *runtimev1.SecurityRule_Access

Types

type BuiltinAPIOptions added in v0.64.1 

type BuiltinAPIOptions struct {
	Name                  string
	Resolver              string
	ResolverProperties    map[string]any
	OpenAPISummary        string
	OpenAPIRequestSchema  string
	OpenAPIResponseSchema string
}

BuiltinAPIOptions hold options for registering built-in APIs.

type ConnectorConfig added in v0.43.0 

type ConnectorConfig struct {
	Driver  string
	Preset  map[string]any
	Project map[string]any
	Env     map[string]string
	// Provision will cause it to request the admin service to provision the connector.
	Provision bool
	// ProvisionArgs provide provisioning args for when ProvisionName is set.
	ProvisionArgs map[string]any
}

ConnectorConfig holds and resolves connector configuration. We support three levels of configuration: 1. Preset: provided when creating the instance (or set by the system, such as allow_host_access). Cannot be overridden. 2. Project: defined in the rill.yaml file. Can be overridden by the env. 3. Env: defined in the instance's variables (in the format "connector.name.var").

func (*ConnectorConfig) Resolve added in v0.43.0 

func (c *ConnectorConfig) Resolve() map[string]any

Resolve returns the final resolved connector configuration. It guarantees that all keys in the result are lowercase.

type Controller added in v0.32.0 

type Controller struct {
	Runtime    *Runtime
	InstanceID string
	Logger     *zap.Logger
	Activity   *activity.Client
	// contains filtered or unexported fields
}

Controller manages the catalog for a single instance and runs reconcilers to migrate the catalog (and related resources in external databases) into the desired state. For information about how the controller schedules reconcilers, see `runtime/reconcilers/README.md`.

func NewController added in v0.32.0 

func NewController(ctx context.Context, rt *Runtime, instanceID string, logger *zap.Logger, ac *activity.Client) (*Controller, error)

NewController creates a new Controller

func (*Controller) AcquireConn added in v0.32.0 

func (c *Controller) AcquireConn(ctx context.Context, connector string) (drivers.Handle, func(), error)

AcquireOLAP gets a handle for a connector in the controller's instance.

func (*Controller) AcquireOLAP added in v0.32.0 

func (c *Controller) AcquireOLAP(ctx context.Context, connector string) (drivers.OLAPStore, func(), error)

AcquireOLAP gets an OLAP handle for a connector in the controller's instance.

func (*Controller) Cancel added in v0.33.0 

func (c *Controller) Cancel(ctx context.Context, name *runtimev1.ResourceName) error

Cancel cancels the current invocation of a resource's reconciler (if it's running). It does not re-enqueue the resource for reconciliation.

func (*Controller) Create added in v0.32.0 

func (c *Controller) Create(ctx context.Context, name *runtimev1.ResourceName, refs []*runtimev1.ResourceName, owner *runtimev1.ResourceName, paths []string, hidden bool, r *runtimev1.Resource) error

Create creates a resource and enqueues it for reconciliation. If a resource with the same name is currently being deleted, the deletion will be cancelled.

func (*Controller) Delete added in v0.32.0 

func (c *Controller) Delete(ctx context.Context, name *runtimev1.ResourceName) error

Delete soft-deletes a resource and enqueues it for reconciliation (with DeletedOn != nil). Once the deleting reconciliation has been completed, the resource will be hard deleted. If Delete is called from the resource's own reconciler, the resource will be hard deleted immediately (and the calling reconcile's ctx will be canceled immediately).

func (*Controller) Flush added in v0.32.0 

func (c *Controller) Flush(ctx context.Context) error

Flush forces a flush of the controller's catalog changes to persistent storage.

func (*Controller) Get added in v0.32.0 

func (c *Controller) Get(ctx context.Context, name *runtimev1.ResourceName, clone bool) (*runtimev1.Resource, error)

Get returns a resource by name. Soft-deleted resources (i.e. resources where DeletedOn != nil) are not returned.

func (*Controller) List added in v0.32.0 

func (c *Controller) List(ctx context.Context, kind, path string, clone bool) ([]*runtimev1.Resource, error)

List returns a list of resources of the specified kind. If kind is empty, all resources are returned. Soft-deleted resources (i.e. resources where DeletedOn != nil) are not returned.

func (*Controller) Lock added in v0.32.0 

func (c *Controller) Lock(ctx context.Context)

Lock locks the controller's catalog and delays scheduling of new reconciliations until the lock is released. It can only be called from within a reconciler invocation. While the lock is held, resources can only be edited by a caller using the ctx passed to Lock.

func (*Controller) Reconcile added in v0.33.0 

func (c *Controller) Reconcile(ctx context.Context, name *runtimev1.ResourceName) error

Reconcile enqueues a resource for reconciliation. If the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) Run added in v0.32.0 

func (c *Controller) Run(ctx context.Context) error

Run starts and runs the controller's event loop. It returns when ctx is cancelled or an unrecoverable error occurs. Before returning, it closes the controller, so it must only be called once. The event loop schedules/invokes resource reconciliation and periodically flushes catalog changes to persistent storage. The implementation centers around these internal functions: enqueue, processQueue (uses markPending, trySchedule, invoke), and processCompletedInvocation. See their docstrings for further details.

func (*Controller) Subscribe added in v0.33.1 

func (c *Controller) Subscribe(ctx context.Context, fn SubscribeCallback) error

Subscribe registers a callback that will receive resource update events. The same callback function will not be invoked concurrently. The callback function is invoked under a lock and must not call the controller.

func (*Controller) Unlock added in v0.32.0 

func (c *Controller) Unlock(ctx context.Context)

Unlock releases the lock acquired by Lock.

func (*Controller) UpdateError added in v0.32.0 

func (c *Controller) UpdateError(ctx context.Context, name *runtimev1.ResourceName, reconcileErr error) error

UpdateError updates a resource's error. Unlike UpdateMeta and UpdateSpec, it does not cancel or enqueue reconciliation for the resource.

func (*Controller) UpdateMeta added in v0.32.0 

func (c *Controller) UpdateMeta(ctx context.Context, name *runtimev1.ResourceName, refs []*runtimev1.ResourceName, owner *runtimev1.ResourceName, paths []string) error

UpdateMeta updates a resource's meta fields and enqueues it for reconciliation. If called from outside the resource's reconciler and the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) UpdateName added in v0.33.0 

func (c *Controller) UpdateName(ctx context.Context, name, newName, owner *runtimev1.ResourceName, paths []string) error

UpdateName renames a resource and updates annotations, and enqueues it for reconciliation. If called from outside the resource's reconciler and the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) UpdateSpec added in v0.32.0 

func (c *Controller) UpdateSpec(ctx context.Context, name *runtimev1.ResourceName, r *runtimev1.Resource) error

UpdateSpec updates a resource's spec and enqueues it for reconciliation. If called from outside the resource's reconciler and the resource is currently reconciling, the current reconciler will be cancelled first.

func (*Controller) UpdateState added in v0.32.0 

func (c *Controller) UpdateState(ctx context.Context, name *runtimev1.ResourceName, r *runtimev1.Resource) error

UpdateState updates a resource's state. It can only be called from within the resource's reconciler. NOTE: Calls to UpdateState succeed even if ctx is cancelled. This enables cancelled reconcilers to update state before finishing.

func (*Controller) WaitUntilIdle added in v0.35.0 

func (c *Controller) WaitUntilIdle(ctx context.Context, ignoreHidden bool) error

WaitUntilIdle returns when the controller is idle (i.e. no reconcilers are pending or running).

type ExportOptions added in v0.29.1 

type ExportOptions struct {
	// Format of the export.
	Format runtimev1.ExportFormat
	// Priority of the export. Higher priority exports are executed first.
	Priority int
	// PreWriteHook is called with information about the file before writes begin to the io.Writer.
	PreWriteHook func(filename string) error
	// IncludeHeader indicates whether to include a header with metadata in the export.
	// Not supported for all file formats (currently only CSV and Excel).
	IncludeHeader bool
	// OriginDashboard is the dashboard that the export originated from (optional).
	// It is only used if IncludeHeader is true.
	OriginDashboard *runtimev1.ResourceName
	// OriginURL is the UI URL that the export originated from (optional).
	// It is only used if IncludeHeader is true.
	OriginURL string
}

type Health added in v0.47.0 

type Health struct {
	HangingConn     error
	Registry        error
	InstancesHealth map[string]*InstanceHealth
}

Health reports a combined health status for the runtime and all instances.

type InstanceHealth added in v0.47.0 

type InstanceHealth struct {
	// Controller error. The controller is considered healthy if this is empty.
	Controller string `json:"controller"`
	// ControllerVersion is the version of the controller that cached this health information.
	// It is used for health cache checks.
	ControllerVersion int64 `json:"controller_version"`
	// OLAP error. May be cached for OLAPs that scale to zero. The OLAP is considered healthy if this is empty.
	OLAP string `json:"olap"`
	// Repo error. The repo is considered healthy if this is empty.
	Repo string `json:"repo"`
	// MetricsViews contains health checks for metrics views.
	MetricsViews map[string]InstanceHealthMetricsViewError `json:"metrics_views"`
	// ParseErrCount is the number of parse errors in the project parser.
	ParseErrCount int `json:"parse_error_count"`
	// ReconcileErrCount is the number of resources with reconcile errors.
	ReconcileErrCount int `json:"reconcile_error_count"`
}

InstanceHealth contains health information for a single instance. The information about OLAP and metrics views is cached in the catalog. We want to avoid hitting the underlying OLAP engine when OLAP engine can scale to zero when no queries are generated within TTL. We do not want to keep it running just to check health. In such cases, we use the cached health information.

func (*InstanceHealth) Proto added in v0.78.0 

func (h *InstanceHealth) Proto() *runtimev1.InstanceHealth

Proto converts InstanceHealth to the proto representation.

type InstanceHealthMetricsViewError added in v0.51.0 

type InstanceHealthMetricsViewError struct {
	Err     string `json:"err"`
	Version int64  `json:"version"`
}

InstanceHealthMetricsViewError contains health information for a single metrics view.

type Options added in v0.16.0 

type Options struct {
	MetastoreConnector           string
	SystemConnectors             []*runtimev1.Connector
	ConnectionCacheSize          int
	QueryCacheSizeBytes          int64
	SecurityEngineCacheSize      int
	ControllerLogBufferCapacity  int
	ControllerLogBufferSizeBytes int64
	AllowHostAccess              bool
	Version                      version.Version
}

type Permission added in v0.78.0 

type Permission int

Permission represents runtime access permissions. 

const (
	// System-level permissions
	ManageInstances Permission = 0x01

	// Instance-level permissions
	ReadInstance  Permission = 0x11
	EditInstance  Permission = 0x12
	EditTrigger   Permission = 0x20
	ReadRepo      Permission = 0x13
	EditRepo      Permission = 0x14
	ReadObjects   Permission = 0x15
	ReadOLAP      Permission = 0x16
	ReadMetrics   Permission = 0x17
	ReadProfiling Permission = 0x18
	ReadAPI       Permission = 0x19
	ReadResolvers Permission = 0x1A
	UseAI         Permission = 0x1B
)

type Query added in v0.16.0 

type Query interface {
	// Key should return a cache key that uniquely identifies the query
	Key() string
	// Deps should return the resource names that the query targets.
	// It's used to invalidate cached queries when the underlying data changes.
	// If a dependency doesn't exist, it is ignored. (So if the underlying resource kind is unknown, it can return all possible dependency names.)
	Deps() []*runtimev1.ResourceName
	// MarshalResult should return the query result and estimated cost in bytes for caching
	MarshalResult() *QueryResult
	// UnmarshalResult should populate a query with a cached result
	UnmarshalResult(v any) error
	// Resolve should execute the query against the instance's infra.
	// Error can be nil along with a nil result in general, i.e. when a model contains no rows aggregation results can be nil.
	Resolve(ctx context.Context, rt *Runtime, instanceID string, priority int) error
	// Export resolves the query and serializes the result to the writer.
	Export(ctx context.Context, rt *Runtime, instanceID string, w io.Writer, opts *ExportOptions) error
}

type QueryResult added in v0.26.0 

type QueryResult struct {
	Value any
	Bytes int64
}

type ReconcileResult added in v0.32.0 

type ReconcileResult struct {
	Err       error
	Retrigger time.Time
}

ReconcileResult propagates results from a reconciler invocation

type Reconciler added in v0.32.0 

type Reconciler interface {
	Close(ctx context.Context) error
	AssignSpec(from, to *runtimev1.Resource) error
	AssignState(from, to *runtimev1.Resource) error
	ResetState(r *runtimev1.Resource) error
	Reconcile(ctx context.Context, n *runtimev1.ResourceName) ReconcileResult
	// ResolveTransitiveAccess resolves transitive access rules for the resource of this type.
	// For example, for a report resource, this determines all the resources needed to access the report like the underlying metrics view, explore, etc. and adds the corresponding security rules to the list of rules to be applied.
	// And use the underlying query to determine the fields that are accessible in the report and where clause that needs to be applied.
	ResolveTransitiveAccess(ctx context.Context, claims *SecurityClaims, res *runtimev1.Resource) ([]*runtimev1.SecurityRule, error)
}

Reconciler implements reconciliation logic for all resources of a specific kind. Reconcilers are managed and invoked by a Controller.

type ReconcilerInitializer added in v0.32.0 

type ReconcilerInitializer func(context.Context, *Controller) (Reconciler, error)

ReconcilerInitializer is a function that initializes a new reconciler for a specific controller

type ResolveOptions added in v0.42.0 

type ResolveOptions struct {
	InstanceID         string
	Resolver           string
	ResolverProperties map[string]any
	Args               map[string]any
	Claims             *SecurityClaims
}

ResolveOptions are the options passed to the runtime's Resolve method.

type ResolvedSecurity added in v0.47.0 

type ResolvedSecurity struct {
	// contains filtered or unexported fields
}

ResolvedSecurity represents the resolved security rules for a given claims against a specific resource.

func (*ResolvedSecurity) CanAccess added in v0.47.0 

func (r *ResolvedSecurity) CanAccess() bool

CanAccess returns whether the resource can be accessed.

func (*ResolvedSecurity) CanAccessAllFields added in v0.47.0 

func (r *ResolvedSecurity) CanAccessAllFields() bool

CanAccessAllFields returns whether all fields in the resource are allowed.

func (*ResolvedSecurity) CanAccessField added in v0.47.0 

func (r *ResolvedSecurity) CanAccessField(field string) bool

CanAccessField evaluates whether a specific field in the resource is allowed.

func (*ResolvedSecurity) QueryFilter added in v0.47.0 

func (r *ResolvedSecurity) QueryFilter() *runtimev1.Expression

QueryFilter returns a query expression to apply when querying the resource.

func (*ResolvedSecurity) RowFilter added in v0.47.0 

func (r *ResolvedSecurity) RowFilter() string

RowFilter returns a raw SQL expression to apply to the WHERE clause when querying the resource.

type Resolver added in v0.42.0 

type Resolver interface {
	// Close is called when done with the resolver.
	// Note that the Resolve method may not have been called when Close is called (in case of cache hits or validation failures).
	Close() error
	// CacheKey returns a key that can be used for caching. It can be a large string since the value will be hashed.
	// The key should include all the properties and args that affect the output.
	// It does not need to include the instance ID or resolver name, as those are added separately to the cache key.
	//
	// If the resolver result is not cacheable, ok is set to false.
	CacheKey(ctx context.Context) (key []byte, ok bool, err error)
	// Refs access by the resolver. The output may be approximate, i.e. some of the refs may not exist.
	// The output should avoid duplicates and be stable between invocations.
	Refs() []*runtimev1.ResourceName
	// Validate the properties and args without running any expensive operations.
	Validate(ctx context.Context) error
	// ResolveInteractive resolves data for interactive use (e.g. API requests or alerts).
	ResolveInteractive(ctx context.Context) (ResolverResult, error)
	// ResolveExport resolve data for export (e.g. downloads or reports).
	ResolveExport(ctx context.Context, w io.Writer, opts *ResolverExportOptions) error
	// InferRequiredSecurityRules attempts to infer the security rules that are required to be able to execute the resolver for the currently configured properties.
	InferRequiredSecurityRules() ([]*runtimev1.SecurityRule, error)
}

Resolver represents logic, such as a SQL query, that produces output data. Resolvers are used to evaluate API requests, alerts, reports, etc.

A resolver has two levels of configuration: static properties and dynamic arguments. For example, a SQL resolver has a static property for the SQL query and dynamic arguments for the query parameters. The static properties are usually declared in advance, such as in the YAML for a custom API, whereas the dynamic arguments are provided just prior to execution, such as in an API request.

type ResolverExportOptions added in v0.42.0 

type ResolverExportOptions struct {
	// Format is the format to export the result in.
	Format runtimev1.ExportFormat
	// PreWriteHook is a function that is called after the export has been prepared, but before the first bytes are output to the io.Writer.
	PreWriteHook func(filename string) error
}

ResolverExportOptions are the options passed to a resolver's ResolveExport method.

type ResolverInitializer added in v0.42.0 

type ResolverInitializer func(ctx context.Context, opts *ResolverOptions) (Resolver, error)

ResolverInitializer is a function that initializes a resolver.

type ResolverOptions added in v0.42.0 

type ResolverOptions struct {
	Runtime    *Runtime
	InstanceID string
	Properties map[string]any
	Args       map[string]any
	Claims     *SecurityClaims
	ForExport  bool
}

ResolverOptions are the options passed to a resolver initializer.

type ResolverResult added in v0.42.0 

type ResolverResult interface {
	// Close should be called to release resources
	Close() error
	// Meta can contain arbitrary metadata about the result.
	// For example, the metrics resolver will return information about the result fields, like display names and formatting rules.
	Meta() map[string]any
	// Schema is the schema for the Data
	Schema() *runtimev1.StructType
	// Next returns the next row of data. It returns io.EOF when there are no more rows.
	Next() (map[string]any, error)
	// MarshalJSON is a convenience method to serialize the result to JSON.
	MarshalJSON() ([]byte, error)
}

ResolverResult is the result of a resolver's execution.

func NewDriverResolverResult added in v0.48.0 

func NewDriverResolverResult(result *drivers.Result, meta map[string]any) ResolverResult

NewDriverResolverResult creates a ResolverResult from a drivers.Result.

func NewMapsResolverResult added in v0.48.0 

func NewMapsResolverResult(result []map[string]any, schema *runtimev1.StructType) ResolverResult

NewMapsResolverResult creates a ResolverResult from a slice of maps.

type Runtime 

type Runtime struct {
	Email *email.Client

	Logger *zap.Logger
	// contains filtered or unexported fields
}

func New 

func New(ctx context.Context, opts *Options, logger *zap.Logger, st *storage.Client, ac *activity.Client, emailClient *email.Client) (*Runtime, error)

func (*Runtime) AI added in v0.41.0 

func (r *Runtime) AI(ctx context.Context, instanceID string) (drivers.AIService, func(), error)

func (*Runtime) APIForName added in v0.42.0 

func (r *Runtime) APIForName(ctx context.Context, instanceID, name string) (*runtimev1.API, error)

APIForName returns the API with the given name for the given instance. It gives precedence to built-in APIs over project-specific dynamically created APIs.

func (*Runtime) AcquireHandle added in v0.32.0 

func (r *Runtime) AcquireHandle(ctx context.Context, instanceID, connector string) (drivers.Handle, func(), error)

AcquireHandle returns instance specific handle

func (*Runtime) AcquireSystemHandle added in v0.32.0 

func (r *Runtime) AcquireSystemHandle(ctx context.Context, connector string) (drivers.Handle, func(), error)

func (*Runtime) Admin added in v0.37.0 

func (r *Runtime) Admin(ctx context.Context, instanceID string) (drivers.AdminService, func(), error)

func (*Runtime) AllowHostAccess added in v0.27.0 

func (r *Runtime) AllowHostAccess() bool

func (*Runtime) ApplySecurityPolicy added in v0.78.0 

func (r *Runtime) ApplySecurityPolicy(ctx context.Context, instID string, claims *SecurityClaims, res *runtimev1.Resource) (*runtimev1.Resource, bool, error)

ApplySecurityPolicy applies relevant security policies to the resource. The input resource will not be modified in-place (so no need to set clone=true when obtaining it from the catalog).

func (*Runtime) Catalog added in v0.16.0 

func (r *Runtime) Catalog(ctx context.Context, instanceID string) (drivers.CatalogStore, func(), error)

func (*Runtime) Close added in v0.18.0 

func (r *Runtime) Close() error

func (*Runtime) ConnectorConfig added in v0.43.0 

func (r *Runtime) ConnectorConfig(ctx context.Context, instanceID, name string) (*ConnectorConfig, error)

func (*Runtime) Controller added in v0.33.1 

func (r *Runtime) Controller(ctx context.Context, instanceID string) (*Controller, error)

Controller returns the controller for the given instance. If the controller is currently initializing, the call will wait until the controller is ready. If the controller has closed with a fatal error, that error will be returned here until it's restarted.

func (*Runtime) CreateInstance 

func (r *Runtime) CreateInstance(ctx context.Context, inst *drivers.Instance) error

CreateInstance creates a new instance and starts a controller for it.

func (*Runtime) DataDir added in v0.47.0 

func (r *Runtime) DataDir(instanceID string, elem ...string) (string, error)

DataDir returns the path to a persistent data directory for the given instance. The directory is created if it doesn't exist. Storage usage in the returned directory will be reported in the instance's heartbeat events.

func (*Runtime) DefaultInstanceID added in v0.64.1 

func (r *Runtime) DefaultInstanceID() (string, bool)

DefaultInstanceID returns the instance ID for the default instance. It returns false on runtimes with none or multiple instances.

func (*Runtime) DeleteFile added in v0.16.0 

func (r *Runtime) DeleteFile(ctx context.Context, instanceID, path string, force bool) error

func (*Runtime) DeleteInstance added in v0.16.0 

func (r *Runtime) DeleteInstance(ctx context.Context, instanceID string) error

DeleteInstance deletes an instance and stops its controller.

func (*Runtime) EditInstance added in v0.23.0 

func (r *Runtime) EditInstance(ctx context.Context, inst *drivers.Instance, restartController bool) error

EditInstance edits an existing instance. If restartController is true, the instance's controller will be re-opened and all cached connections for the instance will be evicted. Until the controller and connections have been closed and re-opened, calls related to the instance may return transient errors.

func (*Runtime) FeatureFlags added in v0.78.0 

func (r *Runtime) FeatureFlags(ctx context.Context, instanceID string, claims *SecurityClaims) (map[string]bool, error)

FeatureFlags finds and resolves the feature flags for the given instance ID and claims. It's designed for use in the backend. Use runtime.ResolveFeatureFlags for resolving flags that will be exposed to the UI.

func (*Runtime) GetFile added in v0.16.0 

func (r *Runtime) GetFile(ctx context.Context, instanceID, path string) (string, time.Time, error)

func (*Runtime) GetInstanceAttributes added in v0.32.0 

func (r *Runtime) GetInstanceAttributes(ctx context.Context, instanceID string) []attribute.KeyValue

GetInstanceAttributes fetches an instance and converts its annotations to attributes nil is returned if an error occurred or instance was not found

func (*Runtime) Health added in v0.47.0 

func (r *Runtime) Health(ctx context.Context, fullStatus bool) (*Health, error)

func (*Runtime) Instance added in v0.35.0 

func (r *Runtime) Instance(ctx context.Context, instanceID string) (*drivers.Instance, error)

Instance looks up an instance by ID. Instances are cached in-memory, so this is a cheap operation.

func (*Runtime) InstanceConfig added in v0.43.0 

func (r *Runtime) InstanceConfig(ctx context.Context, instanceID string) (drivers.InstanceConfig, error)

InstanceConfig returns the instance's dynamic configuration.

func (*Runtime) InstanceHealth added in v0.47.0 

func (r *Runtime) InstanceHealth(ctx context.Context, instanceID string) (*InstanceHealth, error)

func (*Runtime) InstanceLogger added in v0.40.0 

func (r *Runtime) InstanceLogger(ctx context.Context, instanceID string) (*zap.Logger, error)

InstanceLogger returns a logger scoped for the given instance. Logs emitted to the logger will also be available in the instance's log buffer.

func (*Runtime) InstanceLogs added in v0.40.0 

func (r *Runtime) InstanceLogs(ctx context.Context, instanceID string) (*logbuffer.Buffer, error)

InstanceLogs returns an in-memory buffer of recent logs related to the given instance.

func (*Runtime) Instances added in v0.35.0 

func (r *Runtime) Instances(ctx context.Context) ([]*drivers.Instance, error)

Instances returns all instances managed by the runtime.

func (*Runtime) ListFiles added in v0.16.0 

func (r *Runtime) ListFiles(ctx context.Context, instanceID, glob string) ([]drivers.DirEntry, error)

func (*Runtime) MkdirAll added in v0.78.0 

func (r *Runtime) MkdirAll(ctx context.Context, instanceID, path string) error

func (*Runtime) OLAP added in v0.16.0 

func (r *Runtime) OLAP(ctx context.Context, instanceID, connector string) (drivers.OLAPStore, func(), error)

OLAP returns a handle for an OLAP data store. The connector argument is optional. If not provided, the instance's default OLAP connector is used.

func (*Runtime) PutFile added in v0.16.0 

func (r *Runtime) PutFile(ctx context.Context, instanceID, path string, blob io.Reader, create, createOnly bool) error

func (*Runtime) Query added in v0.16.0 

func (r *Runtime) Query(ctx context.Context, instanceID string, query Query, priority int) error

func (*Runtime) RenameFile added in v0.16.0 

func (r *Runtime) RenameFile(ctx context.Context, instanceID, fromPath, toPath string) error

func (*Runtime) Repo added in v0.16.0 

func (r *Runtime) Repo(ctx context.Context, instanceID string) (drivers.RepoStore, func(), error)

func (*Runtime) Resolve added in v0.42.0 

func (r *Runtime) Resolve(ctx context.Context, opts *ResolveOptions) (res ResolverResult, resErr error)

Resolve resolves a query using the given options. The caller must call Close on the result when done consuming it.

func (*Runtime) ResolveSecurity added in v0.47.0 

func (r *Runtime) ResolveSecurity(ctx context.Context, instanceID string, claims *SecurityClaims, res *runtimev1.Resource) (*ResolvedSecurity, error)

func (*Runtime) TempDir added in v0.47.0 

func (r *Runtime) TempDir(instanceID string, elem ...string) (string, error)

TempDir returns the path to a temporary directory for the given instance. The directory is created if it doesn't exist. The TempDir is a fixed location. The caller is responsible for cleaning up after use. The TempDir may be cleared after restarts. Storage usage in the returned directory will be reported in the instance's heartbeat events.

func (*Runtime) UpdateInstanceConnector added in v0.46.0 

func (r *Runtime) UpdateInstanceConnector(ctx context.Context, instanceID, name string, connector *runtimev1.ConnectorSpec) error

UpdateInstanceConnector upserts or removes a connector from an instance If connector is nil, the connector is removed; otherwise, it is upserted

func (*Runtime) UpdateInstanceWithRillYAML added in v0.45.0 

func (r *Runtime) UpdateInstanceWithRillYAML(ctx context.Context, instanceID string, p *parser.Parser, restartController bool) error

func (*Runtime) Version added in v0.64.1 

func (r *Runtime) Version() version.Version

func (*Runtime) WaitUntilIdle added in v0.35.0 

func (r *Runtime) WaitUntilIdle(ctx context.Context, instanceID string, ignoreHidden bool) error

WaitUntilIdle waits until the instance's controller is idle (not reconciling any resources).

type SecurityClaims added in v0.47.0 

type SecurityClaims struct {
	// UserID is the ID of the end user (or service account).
	UserID string
	// UserAttributes about the current user (or service account). Usually exposed through templating as {{ .user }}.
	UserAttributes map[string]any
	// Permissions is a list of assigned permissions.
	Permissions []Permission
	// AdditionalRules are optional security rules to apply *in addition* to the built-in rules and the rules defined on the requested resource.
	// These are currently leveraged by the admin service to enforce restrictions for magic auth tokens.
	AdditionalRules []*runtimev1.SecurityRule
	// SkipChecks enables completely skipping all security checks. Used in local development.
	SkipChecks bool
}

SecurityClaims represents contextual information for the enforcement of security rules. Note that it does not consider instance IDs, which must be handled/checked by the code that creates the SecurityClaims.

func (*SecurityClaims) Admin added in v0.47.0 

func (c *SecurityClaims) Admin() bool

Admin is a convenience function for extracting an "admin" bool from the user attributes.

func (*SecurityClaims) Can added in v0.78.0 

func (c *SecurityClaims) Can(p Permission) bool

Can returns true if the claims have the specified permission.

func (*SecurityClaims) MarshalJSON added in v0.47.0 

func (c *SecurityClaims) MarshalJSON() ([]byte, error)

MarshalJSON serializes the SecurityClaims to JSON. It serializes the AdditionalRules using protojson.

func (*SecurityClaims) UnmarshalJSON added in v0.47.0 

func (c *SecurityClaims) UnmarshalJSON(data []byte) error

UnmarshalJSON deserializes the SecurityClaims from JSON. It deserializes the AdditionalRules using protojson.

type SubscribeCallback added in v0.33.1 

type SubscribeCallback func(e runtimev1.ResourceEvent, n *runtimev1.ResourceName, r *runtimev1.Resource)

SubscribeCallback is the callback type passed to Subscribe.

Source Files

View all Source files

Directories

Path Synopsis
admin
athena
azure
bigquery
clickhouse
clickhouse/testclickhouse
druid
druid/druidsqldriver
duckdb
duckdb/extensions
file
gcs
https
mock
mock/ai
mock/object_store
mysql
openai
pinot
pinot/sqldriver
postgres
redshift
s3
salesforce
slack
snowflake
sqlite
executor
metricssql
pkg
activity
ai
archive
arrayutil
blob
bufferutil
conncache
container
ctxsync
dag
debugserver
duckdbsql
duration
Package duration copied as it is from github.com/senseyeio/duration
 Package duration copied as it is from github.com/senseyeio/duration
email
env
examples
expressionpb
fieldselectorpb
fileutil
filewatcher
formatter
gcputil
globutil
graceful
httputil
jsonschemautil
jsonval
logbuffer
logutil
mapstructureutil
middleware
observability
openapiutil
pagination
pathutil
pbutil
priorityqueue
ratelimit
rduckdb
retrier
rilltime
schedule
securetoken
singleflight
Package singleflight provides a duplicate function call suppression mechanism.
 Package singleflight provides a duplicate function call suppression mechanism.
sqlconvert
sqlparse
sqlstring
timeutil
typepb
auth
testmode

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.