plugin

package module
v0.19.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 21, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package plugin is the OpenRun Starlark plugin provider SDK.

An OpenRun plugin provider is a standalone executable that serves one or more Starlark plugin modules to the OpenRun server over gRPC (hashicorp/go-plugin). Apps load provider-served modules with the .ex suffix — load("notes.ex", "notes") — and use them exactly like builtin .in plugins: the same permission approvals, the same plugin_response error handling, and the same request-end resource cleanup apply. The SDK has no Starlark dependency; plugin functions are plain Go methods over plain Go values, and the server converts Starlark values at the process boundary with full fidelity (exact big ints, int/float distinction, dict order, tuples, sets, bytes, typed records).

Writing a provider

A provider's main function calls Serve with a ServeConfig declaring its modules. Each ModuleDef names a builder and the module's functions:

func main() {
	plugin.Serve(&plugin.ServeConfig{
		ProviderVersion: version,
		Modules: map[string]plugin.ModuleDef{
			"notes": { // served to apps as "notes.ex"
				Builder: NewNotesModule,
				Functions: []plugin.FuncDef{
					{Name: "add", Type: plugin.WRITE, Method: "Add"},
					{Name: "list", Type: plugin.READ, Method: "List"},
				},
			},
		},
	})
}

The builder returns a Module; one instance is created per (module, account) pair in each provider process. Module.InitModule receives a ModuleInit with the app identity, the per-account settings from openrun.toml (secrets already expanded by the server), and the raw schema.star bytes for schema-aware plugins.

Every declared function must be an exported method with the Func signature; Serve validates this at startup:

func (m *NotesModule) Add(ctx context.Context, call *plugin.Call) (any, error) {
	var text string
	if err := plugin.UnpackArgs("add", call, "text", &text); err != nil {
		return nil, err
	}
	...
	return id, nil
}

UnpackArgs binds positional and keyword arguments with starlark.UnpackArgs semantics. The returned value becomes response.value in the app; a returned error becomes response.error (use ErrorWithCode for an explicit error_code) and feeds OpenRun's automatic error handling.

Sessions, cursors, and resource lifetimes

All calls made during one app request share Call.Session. Session state (Session.Set, Session.Get) and deferred cleanup (Session.Defer, Session.ClearDefer) scope cross-call resources such as transactions to the request; remaining defers run when the request ends.

A function may return a *Cursor to stream results lazily: the app receives an iterable, batches are fetched on demand, and a cursor the app never consumes is closed at request end and reported as a leaked resource, failing the request under the cursor's LeakKey.

The one rule stateful providers must follow: resources that outlive a single call — a sql.Tx, the sql.Rows behind a Cursor — must be created on Session.Context, not the per-call context. The per-call gRPC context is cancelled when the call returns, which would roll back the transaction or close the rows before the next call arrives.

How the server runs providers

At registration time the server launches the provider briefly and calls Describe; the returned module manifests (functions, read/write flags, constants) are what app loading and permission auditing use, so no provider process runs until an app actually calls the plugin. At runtime the server keeps one provider process per app (mutual-TLS gRPC over a unix socket, binary checksum verified at launch), initializes it once via InitApp/InitModule, and stops it when the app is closed or reloaded. All permission checks and secret expansion happen server-side before a call is dispatched.

Application errors travel inside responses; a gRPC transport error means the provider process failed, its sessions are lost, and calls are never retried automatically.

The out-of-process build of the OpenRun store plugin (internal/app/store/storeprovider in the OpenRun repository) is the reference provider implementation, covering settings, schema access, transactions, and cursors.

Index

Constants

View Source
const MaxMessageSize = 64 * 1024 * 1024

MaxMessageSize is the gRPC send/receive message limit on both sides of the provider connection. Larger results must use a Cursor.

View Source
const MaxValueDepth = 100

MaxValueDepth bounds StarValue nesting. Starlark collections are mutable and can be made cyclic; the encoder fails cleanly instead of recursing forever.

View Source
const PluginName = "starplugin"

PluginName is the go-plugin dispense name for the plugin provider plugin.

View Source
const ProtocolVersion = 1

ProtocolVersion is the go-plugin protocol version for the v1 Starlark plugin provider protocol. Incompatible protocol changes bump this and are served side by side through VersionedPlugins during a transition.

Variables

View Source
var (
	ErrAppAlreadyInited = errors.New("app already initialized in this provider process")
	ErrAppNotInited     = errors.New("app not initialized")
	ErrUnknownModule    = errors.New("unknown module")
	ErrModuleNotInited  = errors.New("module not initialized")
	ErrUnknownFunction  = errors.New("unknown function")
)

Typed errors the gRPC shim maps to protocol-level status errors; every other error is an application-level failure carried in response fields.

View Source
var Handshake = goplugin.HandshakeConfig{
	ProtocolVersion:  ProtocolVersion,
	MagicCookieKey:   "OPENRUN_PLUGIN_PROVIDER",
	MagicCookieValue: "8e41c2d7-openrun-starlark-plugin-provider",
}

Handshake is the go-plugin handshake shared by the server and providers. The magic cookie is a sanity check that the launched executable is a Starlark plugin provider (not, for example, a binding provider); it is not a security measure.

Functions

func DecodeValue

func DecodeValue(v *pb.StarValue) (any, error)

DecodeValue converts a wire value into its Go representation: nil, bool, int64, *big.Int, float64, string, []byte, []any, Tuple, Set, map[string]any (string-keyed dicts) or *Dict, *Struct, and time.Time.

func DecodeValueMap

func DecodeValueMap(m map[string]*pb.StarValue) (map[string]any, error)

DecodeValueMap decodes a settings-style map.

func EmbeddedProviders

func EmbeddedProviders() map[string]*ServeConfig

EmbeddedProviders returns the registered embedded providers by name.

func EncodeValue

func EncodeValue(v any) (*pb.StarValue, error)

EncodeValue converts a Go value into its wire representation. It accepts the value shapes plugin functions return: nil, bool, all int/uint widths, *big.Int, float32/64, string, []byte, time.Time, Tuple, Set, *Dict/Dict, *Struct/Struct, []any (and common typed slices), and map[string]any (and common typed maps). Cursors are not encoded here; the provider serve loop registers them and encodes a cursor handle.

func EncodeValueMap

func EncodeValueMap(m map[string]any) (map[string]*pb.StarValue, error)

EncodeValueMap encodes a settings-style map.

func ErrorWithCode

func ErrorWithCode(code int64, err error) error

ErrorWithCode returns an error that carries a plugin error code, surfaced to Starlark as response.error_code.

func RegisterEmbedded

func RegisterEmbedded(name string, config *ServeConfig)

RegisterEmbedded registers a provider config to be served in-process by an OpenRun binary this package is compiled into. Call it from an init function; the config is validated and conflicts (same provider name, or a module already served by another embedded provider) panic, so a bad custom build fails at startup rather than on a user request.

func Serve

func Serve(config *ServeConfig)

Serve runs the provider plugin. It is called from a provider executable's main function and blocks until the server side closes the plugin. The OPENRUN_PROVIDER_LOG_LEVEL environment variable (set by the server from its own log level) controls provider log verbosity.

func UnpackArgs

func UnpackArgs(fnName string, call *Call, pairs ...any) error

UnpackArgs binds a call's positional and keyword arguments to Go variables, mirroring starlark.UnpackArgs: pairs alternate a parameter name and a pointer. A name ending in "?" marks an optional parameter. Supported pointer types: *string, *bool, *int64, *[]string, *[]any, *map[string]any, **Struct, and *any.

Types

type AppInfo

type AppInfo struct {
	AppId     string
	AppPath   string
	IsDev     bool
	AppSchema []byte // raw schema.star contents, nil if the app has none
}

AppInfo identifies the app a Host serves.

type Call

type Call struct {
	Function string
	Args     []any
	Kwargs   []Kwarg
	Thread   ThreadState
	Session  *Session

	// Host is non-nil only when the module runs in-process. See HostServices.
	Host HostServices
}

Call carries one plugin function invocation.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is the typed gRPC client for the provider protocol, dispensed by go-plugin. Server code uses Provider, which wraps process lifecycle around it.

type Cursor

type Cursor struct {
	// TypeName names the iterable's Starlark type ("<TypeName> iterator").
	TypeName string

	// Stream marks a cursor that the app returns from its handler as a
	// streaming HTTP response (response.is_stream), instead of iterating it
	// in Starlark. A stream cursor is detached from its session when the
	// call returns — it is consumed after the request's plugin cleanup runs —
	// and no leak entry is registered for it. Supported for in-process
	// modules only; an external provider returning a stream cursor fails
	// the call.
	Stream bool

	// LeakKey names the strict cleanup entry the server registers for this
	// cursor, e.g. "rows_cursor_<table>_0x...". If the app never consumes or
	// closes the cursor, the request fails citing this key.
	LeakKey string

	// Next returns up to max items and whether iteration is complete. The
	// provider closes the cursor itself when it reports done.
	Next func(ctx context.Context, max int) ([]any, bool, error)

	// Close releases the cursor before exhaustion.
	Close func(ctx context.Context) error
}

Cursor is a lazy result iterator that stays in the provider process, scoped to the session of the Call that returned it. The server wraps it in a Starlark iterable; an unconsumed cursor fails the request through the strict deferred-cleanup mechanism, under the name LeakKey.

func PushCursor

func PushCursor(typeName, leakKey string, stream bool, seq func(yield func(any, error) bool)) *Cursor

PushCursor adapts a push-style stream function (repeatedly calling yield with values, like a range-over-func iterator) into a pull-based Cursor. The stream function runs in its own goroutine, started lazily on the first Next; it stops when it returns, yields an error, or the cursor is closed. Next returns the first available item promptly (it does not wait to fill a batch), so live streams flush without delay.

type CursorInfo

type CursorInfo struct {
	CursorId string
	TypeName string
	LeakKey  string
	Stream   bool
}

CursorInfo is the handle for a cursor returned by a call, to be drained with CursorNext / released with CursorClose in the same session (or, for a stream cursor, detached with DetachCursor and consumed directly).

type Dict

type Dict struct {
	Entries []DictEntry
}

Dict is an order-preserving Starlark dict. Incoming dicts whose keys are all strings arrive as map[string]any instead; Dict is used for dicts with non-string keys, and may be returned by a function that needs to control the entry order of a returned dict.

type DictEntry

type DictEntry struct {
	Key   any
	Value any
}

DictEntry is one entry of an order-preserving Dict.

type Download

type Download struct {
	Name     string
	Producer func(w io.Writer) error
}

Download is a file download whose content is produced at response-write time: the app passes it to ace.response(content, download=name) and the producer writes the bytes directly to the HTTP response, after the request's plugin cleanup has run. Supported for in-process modules only; an external provider returning a Download fails the call.

type Func

type Func = func(ctx context.Context, call *Call) (any, error)

Func is the signature every plugin function method must have.

type FuncDef

type FuncDef struct {
	Name   string
	Type   FunctionType
	Method string
}

FuncDef declares one plugin function: its Starlark name, read/write classification, and the Go method (on the Module implementation) that handles it. The method must have the Func signature.

type FuncRef

type FuncRef struct {
	Function string
	Args     []any
}

FuncRef is a value the server materializes as a zero-argument callable in the app: calling it invokes the named plugin function with Args, in the same module, account, and session as the call that returned it. Use it as a Struct field for members whose result is computed lazily against live plugin state, e.g. the http plugin's response.body() reading a response held open in the session. Function names starting with "_" are internal: they are declared in the module's Functions list (so the method binding is validated) but are not exposed as module attributes, and are callable only through a FuncRef.

type FunctionType

type FunctionType int

FunctionType classifies a plugin function as a read or a write operation. Writes are blocked for stage/preview apps without write access.

const (
	READ FunctionType = iota
	WRITE
)

type Host

type Host struct {
	// contains filtered or unexported fields
}

Host runs a provider's modules for one app, on plain Go values, with no transport involved. It owns the module instances (one per (module, account) pair), the sessions with their deferred cleanups, and the cursors returned by calls. Both plugin transports are thin layers over it: the gRPC providerServer (external providers) decodes wire values at its edges, and the OpenRun server embeds one Host per (app, provider) for internal plugins, converting Starlark values with the direct bridge. Keeping the semantics here — lazy module init, session-scoped state, strict-leak reporting, cursor batching, module Close on shutdown — is what guarantees a plugin behaves identically internal and external.

func NewHost

func NewHost(config *ServeConfig, logger *Logger) (*Host, error)

NewHost validates the config and returns a Host. The logger may be nil.

func (*Host) Call

func (h *Host) Call(ctx context.Context, call *HostCall) (*HostResult, error)

Call invokes one plugin function. An error returned by the function is returned as-is (possibly a *ProviderError carrying an error code), with the result's StrictKeys still valid; ErrModuleNotInited / ErrUnknownFunction signal caller bugs.

func (*Host) Close

func (h *Host) Close(ctx context.Context)

Close calls Close on every initialized module instance, logging failures. Used at provider process shutdown and at app close/reload for in-process hosts.

func (*Host) Config

func (h *Host) Config() *ServeConfig

Config returns the provider config the Host serves.

func (*Host) CursorClose

func (h *Host) CursorClose(ctx context.Context, sessionId, cursorId string) error

CursorClose releases a cursor before exhaustion. Closing an unknown cursor or session is not an error.

func (*Host) CursorNext

func (h *Host) CursorNext(ctx context.Context, sessionId, cursorId string, max int) ([]any, bool, error)

CursorNext returns up to max items from a cursor and whether iteration is complete. When done, the cursor and its session cleanup are forgotten (the cursor released its own resources).

func (*Host) DetachCursor

func (h *Host) DetachCursor(sessionId, cursorId string) (*Cursor, error)

DetachCursor removes a cursor from its session and returns it for direct consumption. Used for stream cursors, which are consumed after the request's session has ended; the caller owns closing the cursor.

func (*Host) EndSession

func (h *Host) EndSession(ctx context.Context, sessionId string) error

EndSession runs the session's remaining deferred cleanups and forgets it. On a retired host, the last session ending closes the host's modules.

func (*Host) InitApp

func (h *Host) InitApp(info AppInfo) error

InitApp establishes the app identity. Called once, before any InitModule.

func (*Host) InitModule

func (h *Host) InitModule(ctx context.Context, module, account string, settings map[string]any) error

InitModule initializes one (module, account) instance, if not already initialized. Settings are the per-account plugin settings with secrets expanded. An error returned by the module's InitModule is returned as-is. Concurrent first calls for the same (module, account) are serialized: one runs the initialization, the others wait for its outcome, so exactly one instance is ever built (a failed init is forgotten, so a later call retries fresh).

func (*Host) Retire

func (h *Host) Retire()

Retire marks a host that is being replaced (app reload): in-flight requests finish on the old module instances, and the modules are closed when the last active session ends (immediately when none are active). New requests go to the replacement host; a straggler call racing the retire still works — InitModule lazily rebuilds the instance and the session-end close runs again.

func (*Host) StartSession

func (h *Host) StartSession(sessionId string)

StartSession registers a request session before its first call. Callers register the session at host-acquisition time so a concurrent Retire (app reload) counts the request as active and keeps the host open until the matching EndSession, instead of closing the modules mid-request.

type HostCall

type HostCall struct {
	Module    string
	Account   string
	Function  string
	Args      []any
	Kwargs    []Kwarg
	Thread    ThreadState
	SessionId string

	// Host provides host-process services to in-process modules; nil for
	// external provider processes.
	Host HostServices
}

HostCall is one plugin function invocation at the Host level.

type HostResult

type HostResult struct {
	Value      any
	Cursor     *CursorInfo // set instead of Value when the function returned a *Cursor
	StrictKeys []string
}

HostResult is the outcome of one call. StrictKeys is always populated (even alongside an error) so the caller can mirror strict deferred-cleanup entries after every call.

type HostServices

type HostServices interface {
	// Value returns a host-scoped value by key, nil if not set. Keys are
	// host-defined (e.g. the container plugin's request container handler).
	Value(key string) any
}

HostServices exposes host-process services to a module running in-process (compiled into the OpenRun binary). It is nil when the module runs in an external provider process, so a module using it is host-bound: it works internal-only and must fail gracefully when Host is nil.

type Kwarg

type Kwarg struct {
	Name  string
	Value any
}

Kwarg is one keyword argument, in call-site order.

type LaunchConfig

type LaunchConfig struct {
	// ExecPath is the provider executable.
	ExecPath string
	// Logger receives go-plugin lifecycle logs and the provider's forwarded
	// log lines. Nil uses go-plugin's default (stderr).
	Logger hclog.Logger
	// LogLevel is passed to the provider via OPENRUN_PROVIDER_LOG_LEVEL.
	LogLevel string
	// SecureConfig, when set, verifies the executable's checksum before
	// launch.
	SecureConfig *goplugin.SecureConfig
}

LaunchConfig configures launching a provider process.

type Logger

type Logger struct {
	*zerolog.Logger
}

Logger is the logger passed to provider implementations. It mirrors the OpenRun server's logger type (a zerolog wrapper) so binding code written against the server's internal interface ports without changes.

func NewLogger

func NewLogger(level string) *Logger

NewLogger returns a plain stderr logger at the given level ("WARN", "INFO", "DEBUG", "TRACE"; anything else means INFO). Intended for provider tests; provider processes get their logger from Serve.

type Module

type Module interface {
	// InitModule initializes the instance before its first call.
	InitModule(ctx context.Context, init ModuleInit) error

	// Close releases the instance's resources. Called at process shutdown.
	Close(ctx context.Context) error
}

Module is a plugin module instance, scoped to one (module, account) pair within one app's provider process.

type ModuleDef

type ModuleDef struct {
	// Builder returns a new module instance. One instance is created per
	// (module, account) pair in each provider process (one process per app).
	Builder func() Module

	// Functions declares the module's plugin functions.
	Functions []FuncDef

	// Constants are exported as module constants, e.g. store.MAX_LIMIT.
	Constants map[string]any
}

ModuleDef declares one Starlark module served by a provider. A module named "store" is loaded by apps as load("store.ex", "store").

type ModuleInit

type ModuleInit struct {
	AppId   string
	AppPath string
	Account string // "" for the default account
	IsDev   bool

	// AppSchema is the raw contents of the app's schema.star, nil if the app
	// has none.
	AppSchema []byte

	// Settings are the per-account plugin settings from the server config
	// (e.g. [plugin."store.ex#myaccount"]), with secrets already expanded.
	Settings map[string]any

	Logger *Logger
}

ModuleInit carries the app and account context for a module instance.

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider is a running plugin provider process, launched by the OpenRun server. Application-level failures are returned as *ProviderError; any other error is a transport failure (the process died or broke protocol) and the server treats the process and its sessions as dead.

func LaunchProvider

func LaunchProvider(config LaunchConfig) (*Provider, error)

LaunchProvider starts the provider executable and completes the go-plugin handshake. The returned Provider must be closed with Kill.

func (*Provider) Call

func (p *Provider) Call(ctx context.Context, req *pb.CallRequest) (*pb.CallResponse, error)

Call invokes a plugin function. An application-level failure is returned as *ProviderError together with the (possibly nil) response; a transport failure is any other error.

func (*Provider) CheckHealth

func (p *Provider) CheckHealth(ctx context.Context) error

CheckHealth reports whether the provider process is responsive.

func (*Provider) CursorClose

func (p *Provider) CursorClose(ctx context.Context, req *pb.CursorCloseRequest) error

CursorClose releases a cursor before exhaustion. Closing an unknown cursor or session is not an error.

func (*Provider) CursorNext

func (p *Provider) CursorNext(ctx context.Context, req *pb.CursorNextRequest) (*pb.CursorNextResponse, error)

CursorNext fetches the next batch of items from a cursor returned by a previous Call in the same session.

func (*Provider) Describe

func (p *Provider) Describe(ctx context.Context) (*pb.DescribeResponse, error)

Describe reports the provider's version and module manifests. Valid before InitApp; used at registration time to capture the manifests.

func (*Provider) EndSession

func (p *Provider) EndSession(ctx context.Context, req *pb.EndSessionRequest) error

EndSession releases all provider-side state for a session: remaining deferred cleanups run and the session is forgotten. Called by the server when the request handler finishes.

func (*Provider) Exited

func (p *Provider) Exited() bool

Exited reports whether the provider process has exited.

func (*Provider) InitApp

func (p *Provider) InitApp(ctx context.Context, req *pb.InitAppRequest) error

InitApp establishes the app identity for this provider process. Called once per process, immediately after launch.

func (*Provider) InitModule

func (p *Provider) InitModule(ctx context.Context, req *pb.InitModuleRequest) error

InitModule initializes one (module, account) instance in the provider process, called lazily before the instance's first Call.

func (*Provider) Kill

func (p *Provider) Kill()

Kill terminates the provider process.

type ProviderError

type ProviderError struct {
	Message string
	Code    int64 // plugin error code, 1 unless the function set one
}

ProviderError is an application-level error reported by a provider: the plugin function ran and returned a failure. Transport-level failures (provider crashed, protocol error) are returned as ordinary gRPC errors instead; the server treats those as fatal for the provider process.

func (*ProviderError) Error

func (e *ProviderError) Error() string

type ServeConfig

type ServeConfig struct {
	// Modules maps each module name served by this provider to its
	// definition. A module "store" is loaded by apps as "store.in" (or
	// "store.ex" to require the external build).
	Modules map[string]ModuleDef

	// ProviderVersion is the provider's release version, reported via
	// Describe.
	ProviderVersion string
}

ServeConfig configures a Starlark plugin provider process.

type Session

type Session struct {
	// contains filtered or unexported fields
}

Session groups the plugin calls of one app request. Cross-call state (transactions, handles) and deferred cleanup are scoped to it; the server ends the session when the request handler finishes.

func NewSession

func NewSession(id string) *Session

NewSession creates a standalone session, for unit tests of plugin modules. In a provider the host creates and ends sessions itself.

func (*Session) ClearDefer

func (s *Session) ClearDefer(key string)

ClearDefer removes a previously registered cleanup, typically after the resource was released explicitly (e.g. a transaction was committed).

func (*Session) Context

func (s *Session) Context() context.Context

Context returns a context that stays alive until the session ends. Use it (not the per-call context) for resources that outlive one call, such as the query backing a returned Cursor: the per-call gRPC context is cancelled when the call returns, which would invalidate the resource before the next CursorNext arrives.

func (*Session) Defer

func (s *Session) Defer(key string, strict bool, fn func(ctx context.Context) error)

Defer registers a cleanup function run when the session ends (unless cleared first). Cleanups run in reverse registration order. Strict entries are mirrored to the server after every call: a strict entry still registered when the request ends fails the request as a leaked resource, matching the strict cleanup contract of builtin plugins. Use strict for resources the app is required to consume or release explicitly, and ClearDefer when it does.

func (*Session) End

func (s *Session) End(ctx context.Context) error

End runs the session's remaining deferred cleanups, for unit tests of plugin modules. In a provider the host ends sessions itself.

func (*Session) Get

func (s *Session) Get(key string) any

Get returns a session-scoped value, or nil if not set.

func (*Session) Id

func (s *Session) Id() string

Id returns the session id assigned by the server.

func (*Session) Set

func (s *Session) Set(key string, value any)

Set stores a session-scoped value.

type Set

type Set []any

Set is a Starlark set argument or return value.

type Struct

type Struct struct {
	TypeName string
	Fields   map[string]any
}

Struct is a schema-typed record. The server materializes returned Structs as typed Starlark values with attribute access (row.field), matching what builtin plugins return; typed values passed as arguments arrive as Structs.

type ThreadState

type ThreadState struct {
	RequestId   string
	UserId      string
	UserSubject string
	UserEmail   string
	Groups      []string
	AppUrl      string
}

ThreadState snapshots the request-scoped state a plugin function may read.

type Thunk

type Thunk struct {
	Name  string
	Value any
	Error string
}

Thunk is a value the server materializes as a zero-argument callable in the app: calling it returns Value, or fails with Error if set. Use it as a Struct field to return a record with callable members over pre-computed data. The value must be transportable (EncodeValue rules); Name names the callable in error messages.

type Tuple

type Tuple []any

Tuple is a Starlark tuple argument or return value.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL