scriptrun

package
v1.124.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 22, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package scriptrun is the managed-script execution engine: an embedded Starlark interpreter, the curated host stdlib scripts are allowed to call, and the static validator the authoring loop answers with.

Starlark (go.starlark.net) is the engine because determinism is a property of the LANGUAGE rather than of a blocklist the platform has to maintain: the language has no ambient clock, randomness, filesystem, or network, iteration order is specified, and unbounded loops and recursion are off by default. A script can only affect the world through bindings this package predeclares, and every one of them is one ordinary platform tool call: platform.call names the tool, and the three named helpers are that call with a constant and some behavior worth a name. What a script may reach is what its author's persona authorizes, at every call, at run time; what a READER can enumerate is what Validate reads out of the source.

The determinism contract this engine supports, stated exactly:

same script version + same parameters + same underlying data => same output

It is not "identical forever". The warehouse changes between runs, and that is the point of re-running. What the platform eliminates is every source of variation it controls: no clock or RNG is predeclared, the fire time arrives as a pinned parameter rather than a clock read, and tool results reaching a script carry no semantic enrichment (which varies with catalog state).

Resource limits, honestly: starlark-go bounds CPU with an execution-step limit and wall-clock through thread cancellation, and this package adds a hard byte cap on every host result plus bounded log capture. The ROW cap and its push-down into the query belong to platform.query, which is the reason that helper exists: a script that calls the query tool through platform.call is handed the tool's own result, its truncation flag included, and reads it itself. Neither starlark-go nor any comparable embedded interpreter offers a hard MEMORY cap, so a pathological script can still grow the process heap. That residual risk is recorded in docs/scripts/security.md rather than papered over.

Index

Constants

View Source
const (
	CapabilityQuery  = "platform.query"
	CapabilityExport = "platform.export"
	// CapabilityPublishData replaces the data region of a portal document this
	// script already publishes, and touches nothing else in it. It is separate
	// from CapabilityExport, whose document arm composes whole documents,
	// because the two describe different behavior a reader should be able to
	// tell apart.
	CapabilityPublishData = "platform.publish_data"
	// CapabilityCall invokes any platform tool by name. It resolves to the same
	// Caller the three helpers resolve to, so a tool the run's persona may not
	// call is refused by the middleware in the middleware's own words, exactly
	// as it refuses an agent.
	CapabilityCall = "platform.call"
)

Member names of the platform module.

Three of them are named helpers over one tool call each, kept because they carry behavior worth a name: the typed parameter binding on query, the format and destination resolution on export, the structural splice on publish_data. CapabilityCall is the same mechanism with the tool name left to the author, and it is what makes the surface open.

The set is not access control. What a script may reach is what its author's persona authorizes, decided by the persona filter at every call, at run time (#1419).

View Source
const (
	// DraftMaxSteps caps interpreter execution steps for a draft run.
	DraftMaxSteps = 2_000_000

	// DraftTimeout caps wall-clock time for a draft run, including the time
	// spent inside host calls.
	DraftTimeout = 60 * time.Second

	// DraftMaxRows caps the rows one platform.query may return to a draft.
	DraftMaxRows = 5_000

	// DraftMaxResultBytes caps the serialized size of one platform.query result.
	DraftMaxResultBytes = 8 << 20

	// MaxLogBytes caps captured print output. Anything a script needs to emit
	// that is larger than a log is an output asset, not a log line.
	MaxLogBytes = 64 << 10
)

Draft-execution limits. A draft runs interactively, under its author's own identity, while they iterate — so it is bounded more tightly than a platform run will be: the author is waiting for the answer, and a runaway draft should fail fast with a legible message instead of occupying a serving replica.

View Source
const (
	// RunMaxSteps caps interpreter execution steps for a platform run.
	RunMaxSteps = 20_000_000

	// RunTimeout caps wall-clock time for one platform run, including the
	// time spent inside host calls. It matches the ceiling trino_export already
	// applies to a synchronous export.
	RunTimeout = 10 * time.Minute

	// RunMaxRows caps the rows one platform.query may return.
	RunMaxRows = 20_000

	// RunMaxResultBytes caps the serialized size of one query result.
	RunMaxResultBytes = 32 << 20
)

Platform-run limits. A worker-executed run is looser than a draft on every axis, because nobody is waiting at a prompt for it — but it is still bounded on all of them, because the one resource no embedded interpreter of this class can cap is memory, and every limit here is part of what keeps a runaway script from taking the process with it.

View Source
const (
	SeverityError   = "error"
	SeverityWarning = "warning"
)

Finding severities. An error means the script cannot run as written; a warning means it will run but somebody should look.

View Source
const MaxOutputBytes = 100 << 20

MaxOutputBytes caps one serialized output. It matches the ceiling the portal export path applies, so a script cannot write an asset a human could not have exported by hand.

View Source
const PublishFormat = "json"

PublishFormat is the one format a data-region payload has. The region is a JSON data island by contract, so unlike an export there is no format axis for the script to choose on. Exported because the writer records the same fact on the run row, and the two spellings must be one.

View Source
const TextResultKey = "text"

TextResultKey is the single field a tool result arrives under when the tool returned no structured object: the text it produced, verbatim (SessionCaller. CallTool). It is one key rather than a shape per tool so the rule an author has to remember is one sentence, and it sits with the other result-shape constants because that is what it is — the shape a host call hands back. Exported because the dialect contract states it.

Variables

View Source
var (
	ErrStepLimit = errors.New("script exceeded its execution-step limit")
	ErrTimeout   = errors.New("script exceeded its time limit")
)

ErrStepLimit marks a run stopped by the execution-step limit, and ErrTimeout a run stopped by the wall-clock limit. Both are script-side failures: the same script on the same inputs will hit them again, so a caller must never retry them.

Capabilities is the full member set of the platform module, in the order help and validate report it.

It is a report ordering and the spelling check behind validate's "no such member" refusal. It is not a boundary: platform.call reaches every tool the run's persona authorizes, and what a script reaches is read from the source by Validate, which reports the tool names it names.

View Source
var PredeclaredNames = []string{"platform", "json", "date", "run", sumBuiltinName}

PredeclaredNames are the globals the platform adds on top of the Starlark universe, in the order the dialect contract introduces them.

It is the one definition of that set. predeclared() builds the bindings and isPredeclaredName answers for them while validating, and a name present in one and absent from the other is the defect that let the contract advertise a built-in the environment did not have (#1414): validation would resolve a name the run cannot bind, or refuse one it can.

Functions

func FormatDataPayload added in v1.123.0

func FormatDataPayload(name string, data any) ([]byte, error)

FormatDataPayload serializes one publish_data payload as the JSON the data region will hold, checked against the same output ceiling every export is.

It is the single serializer for the payload — a draft measures exactly the bytes a platform run splices — and it keeps encoding/json's default escaping, which writes <, > and & as \u escapes, so no string in the payload can ever terminate the <script> element it lands inside. Go serializes map keys in sorted order, so the bytes are deterministic for a given payload. The indentation is for the reader of a version diff: a refreshed dashboard's history should read field by field, not as one replaced line.

func PublishRowCount added in v1.123.0

func PublishRowCount(data any) int

PublishRowCount reports the honest row count of a payload: the length of a list, and zero for a dict, whose size is not a row count.

func ResolveDestination added in v1.123.1

func ResolveDestination(name string, declared []script.Destination) (script.Destination, error)

ResolveDestination turns a destination name into the address it stands for, against the set a deployment declares. The portal is built in; every other name comes from the scripts.destinations configuration.

It is exported because the run is no longer the only caller: validate reports a script that names a destination this deployment does not declare (#1415), and the two must refuse in the same words for the same reason. A script whose export was accepted by validate and then refused at run time had already executed its queries by the time it learned.

Types

type Caller

type Caller interface {
	// CallTool invokes the named tool and returns its structured content. A
	// tool that reports an error returns a non-nil error carrying the tool's
	// message; the script sees it as a Starlark error and the run fails.
	CallTool(ctx context.Context, name string, args map[string]any) (map[string]any, error)
}

Caller issues one platform tool call on behalf of a running script and returns the tool's structured result.

Every host binding goes through this one seam, and the production implementation drives the fully assembled MCP server over an in-memory session — so persona and connection authorization, rate limiting, and audit all apply to a script's calls exactly as they apply to an agent's, with no second implementation to keep in step. Binding host functions straight onto narrow Go interfaces would be faster per call and would mean re-implementing authorization for user-authored code, which is the drift this platform's single-funnel design exists to prevent.

func Connect

func Connect(ctx context.Context, server *mcp.Server, label string) (Caller, func(), error)

Connect opens an in-memory MCP session against server and returns the Caller that drives it, plus the teardown for both ends.

It returns the Caller interface rather than the concrete SessionCaller because that is the whole of what a run does with it: both call sites hand the result straight to Options.Caller. Naming the interface here is what makes the relationship between the session plumbing and the engine legible — implementing an interface is invisible in Go's reference graph, and a SessionCaller nothing visibly connects to the engine reads as a second package sharing an import path.

The identity the session authenticates as comes from ctx, which the caller has already decorated: a draft run carries its author's own identity, a platform run carries the script principal and the version author's captured roles. This function deliberately establishes no identity of its own — there is one place a script's authority is decided, and it is not here. label names the client in the handshake so the two run kinds are distinguishable in logs.

type ExportRecord

type ExportRecord struct {
	Name string `json:"name"`
	// Destination is the name the script wrote, so a run that sends one result
	// to two places reads as two records rather than as a repeat.
	Destination string `json:"destination"`
	Format      string `json:"format"`
	RowCount    int    `json:"row_count"`
	// Document marks an output written verbatim from a string body, whose
	// RowCount is therefore not a fact about it: without the marker a surface
	// rendering "N rows as html" would describe a dashboard as an empty table.
	Document bool `json:"document,omitempty"`
	// Refresh marks a platform.publish_data call: the run replaced the data
	// region of an existing asset rather than writing a whole output, and
	// Bytes is the payload spliced in, not the document around it.
	Refresh bool `json:"refresh,omitempty"`
	// Bytes is the serialized length of the output in its declared format. A
	// preview serializes to measure rather than estimating, so the number is
	// the same one a real run would report for the same rows.
	Bytes int `json:"bytes"`
	// Preview is true when nothing was persisted.
	Preview      bool   `json:"preview"`
	AssetID      string `json:"asset_id,omitempty"`
	AssetVersion int    `json:"asset_version,omitempty"`
	Bucket       string `json:"bucket,omitempty"`
	Key          string `json:"key,omitempty"`
}

ExportRecord is what one platform.export call did, in call order on the run's Result. A record with Preview set measured the output and wrote nothing, which is what a draft run does; otherwise it names the asset version written or the object delivered.

type ExportRequest

type ExportRequest struct {
	// Name identifies the output across runs. It is the stable half of the
	// output's identity: the same name from the same script maps to one asset
	// whose versions are its run history.
	Name string
	// Format is one of the formats exportFormats admits.
	Format string
	// Columns is the column order the script wrote, read from the rows before
	// they became order-free Go maps. A tabular format writes its columns in
	// this order; see columnOrder.
	Columns []string
	// Rows is the list of row dicts to write. Nil when the script passed a
	// string body instead, which Body then carries.
	Rows []any
	// Body is the document arm: a string the script composed, persisted
	// verbatim. Valid for the document formats (markdown, text, html, jsx) and
	// nil for a tabular output — exactly one of Body and Rows is the content.
	Body *string
	// Destination is the destination the output goes to, already resolved from
	// the name the script wrote to the address configuration declares for it.
	Destination script.Destination
	// Key is the object key the script asked for beneath the destination's
	// configured prefix, empty when it named none and never set for the portal.
	Key string
}

ExportRequest is one output as the script produced it.

type ExportResult

type ExportResult struct {
	AssetID      string
	AssetVersion int
	Bucket       string
	Key          string
	// Bytes is the serialized size actually written.
	Bytes int
}

ExportResult is where one output landed. A portal output reports the asset version it created; a delivered one reports the object it wrote.

type Exporter

type Exporter interface {
	// Export writes one output and reports where it landed. An error fails the
	// run: a report whose output did not persist did not happen.
	Export(ctx context.Context, req ExportRequest) (*ExportResult, error)

	// PublishData replaces the data region of the named output asset with the
	// request's payload and reports the version that created. An error fails
	// the run: a dashboard that did not refresh did not refresh.
	PublishData(ctx context.Context, req PublishRequest) (*ExportResult, error)
}

Exporter persists one script output. The engine holds it behind this interface so nothing here knows what an asset, a bucket, or a portal is: the interpreter's job ends at "these rows, under this name, in this format".

type Finding

type Finding struct {
	Severity string `json:"severity" example:"error"`
	Line     int    `json:"line,omitempty" example:"12"`
	Message  string `json:"message"`
	// Hint is the corrective action. It carries most of the value of this
	// validator: an author who writes Python at a Starlark interpreter needs to
	// be told what to write instead, not merely that the parser disagreed.
	Hint string `json:"hint,omitempty"`
}

Finding is one thing the validator noticed, addressed to the author.

func CheckDestinations added in v1.123.1

func CheckDestinations(report Report, declared []script.Destination) []Finding

CheckDestinations reports each destination a source names literally that the deployment does not declare.

Validate itself is deployment-independent — it parses source and reads what the code reaches — so this is a separate pass over its report, applied by the surfaces that know the configured set. Splitting it that way keeps a save working when configuration changes underneath a stored script, while the surface whose job is answering "would this run" answers it (#1415).

It reads report.Destinations, which holds only the destinations named as string literals in the source. A call that computes its destination is invisible there and is reported by report.DynamicDestinations instead: its address is not readable from the source, so there is nothing to check.

The refusal is ResolveDestination's, so validate and the run say the same thing about the same script.

type Options

type Options struct {
	// Source is the Starlark source to execute, and Name labels it in tracebacks.
	Source string
	Name   string

	// RunID, FireTime and Params populate the frozen run dict — the script's
	// only source of time and of caller input.
	RunID    string
	FireTime time.Time
	Params   map[string]any

	// Caller issues the script's platform tool calls. A nil Caller leaves the
	// platform module predeclared but every call on it fails, which is what a
	// syntax-only execution wants.
	Caller Caller

	// Destinations is the deployment's configured bucket destinations, the set
	// a platform.export destination name resolves against at run time. The
	// portal is built in and never listed here. A draft run and a platform run
	// carry the same set, so the script an author finishes is the script that
	// runs.
	Destinations []script.Destination

	// Exporter persists what platform.export produces. nil previews instead,
	// which is what a draft run does.
	Exporter Exporter

	// Limits. Zero means the draft default.
	MaxSteps       uint64
	Timeout        time.Duration
	MaxRows        int
	MaxResultBytes int
	MaxLogBytes    int
}

Options configures one script execution.

func RunLimits added in v1.123.0

func RunLimits() Options

RunLimits returns the limit set a platform run executes under, so a caller configures them by naming the policy rather than by copying four numbers it would then have to keep in step.

type OutputIdentity added in v1.123.0

type OutputIdentity struct {
	ContentType string
	Extension   string
}

OutputIdentity is how one output is stored: the media type it carries and the file extension its object keys take. It is a value rather than the formatter that produced the bytes, because the bytes are already serialized when FormatOutput returns — nothing downstream may re-serialize, and a document has no serializer to hand back.

func FormatOutput added in v1.122.0

func FormatOutput(req ExportRequest) ([]byte, OutputIdentity, error)

FormatOutput serializes one export request in its declared format, checks it against the output ceiling, and returns the identity the bytes are stored under.

It is the single serializer for a script's output, and the ceiling is applied here rather than by the writer so that the two runs of a script agree: a platform run persists exactly these bytes, and a draft run measures them, so an output too large to write is refused while the author is still iterating rather than at the first scheduled fire.

type PublishRequest added in v1.123.0

type PublishRequest struct {
	// Name identifies the target asset through the same identity rule an
	// export's name resolves by: one (script, name) pair is one asset. The
	// asset must already exist — this call refreshes a region of it and can
	// create nothing.
	Name string
	// Data is the payload the asset's data region will hold, already converted
	// to plain Go values. FormatDataPayload is its one serializer, shared by
	// the draft preview and the platform run's write.
	Data any
}

PublishRequest is one data-region refresh as the script produced it (platform.publish_data, #1389).

type Report

type Report struct {
	OK       bool      `json:"ok"`
	Findings []Finding `json:"findings"`
	// Capabilities is the set of host bindings the source references, and
	// Connections the connection names it names literally, across every call.
	Capabilities []string `json:"capabilities"`
	Connections  []string `json:"connections"`
	// Destinations is where this script's OUTPUTS go: the destination names
	// platform.export writes to, plus the portal for an export that names none
	// and for every platform.publish_data. It is a statement about the output
	// surface, not about every byte the script can move — a script that writes
	// through a tool, say platform.call("s3_put_object", ...), produces no
	// output in this sense and is read in Tools instead.
	Destinations []string `json:"destinations"`
	// Tools is the tool names the source passes to platform.call literally,
	// sorted. It is where a reader learns the reach of the open half of the
	// surface: the persona filter decides what a run MAY call, and this says
	// what this source DOES call (#1419).
	Tools []string `json:"tools"`
	// RefreshTargets is the output names platform.publish_data refreshes, read
	// literally from the calls, so a reader sees WHICH asset's data region a
	// script rewrites.
	RefreshTargets []string `json:"refresh_targets"`
	// DynamicConnections is true when a platform.query call computes its
	// connection instead of naming one literally, DynamicDestinations when
	// a platform.export call computes its destination, and DynamicRefreshTargets
	// when a platform.publish_data call computes the name it refreshes, so the
	// list in question is known to be incomplete. Reporting the gap is the
	// point: a reader shown a list that silently omitted a computed name would
	// be reading a false statement.
	DynamicConnections    bool `json:"dynamic_connections"`
	DynamicDestinations   bool `json:"dynamic_destinations"`
	DynamicRefreshTargets bool `json:"dynamic_refresh_targets"`
	// DynamicTools is true when a platform.call computes the tool it invokes,
	// so the tool list is known to be incomplete. A call that computes its
	// ARGUMENT SET leaves the tool list intact and sets DynamicConnections
	// instead, because the connection is the only claim this report makes
	// about what is inside those arguments.
	DynamicTools bool `json:"dynamic_tools"`
}

Report is the result of validating one script's source: whether it can run, what it would reach if it did, and everything the author or a reviewer should know first.

func Validate

func Validate(source string) Report

Validate parses and resolves a script without executing it, and reports what it would reach. It is the fast half of the authoring loop: an author gets interpreter-accurate errors, the Python-isms their instincts produce get a specific correction, and what the script reaches is extracted for a reader — all without a query running or a row moving.

func WithDestinationCheck added in v1.123.1

func WithDestinationCheck(report Report, declared []script.Destination) Report

WithDestinationCheck returns report with CheckDestinations' findings folded in and OK recomputed, which is the whole of what a validating surface does with them. It exists so the tool arm and the portal editor cannot fold them in differently.

type Result

type Result struct {
	// Log is the captured print output, truncated at the log cap.
	Log string `json:"log"`
	// LogTruncated is true when output was dropped at the cap.
	LogTruncated bool `json:"log_truncated"`
	// Steps is the interpreter step count the run consumed, and Duration its
	// wall-clock time. Both are the raw material for sizing a platform run's
	// limits against a draft that already works.
	Steps    uint64        `json:"steps"`
	Duration time.Duration `json:"-"`
	// Queries counts the platform.query calls the run issued.
	Queries int `json:"queries"`
	// Exports lists what every platform.export call did, in call order.
	Exports []ExportRecord `json:"exports"`
}

Result reports one completed execution.

func Run

func Run(ctx context.Context, opts Options) (*Result, error)

Run executes a script and returns its result. The error is non-nil when the script itself failed — a Starlark error, a refused host call, or a limit — and the returned Result still carries whatever log and metrics the run produced before failing, because that log is exactly what the author needs.

Failures here are deterministic by construction: the same source on the same inputs fails the same way. Callers must not retry them.

type SessionCaller

type SessionCaller struct {
	// contains filtered or unexported fields
}

SessionCaller issues a script's platform calls over one in-memory MCP session against the fully assembled server. It is the production Caller: every host binding a script invokes becomes an ordinary tool call, crossing the same authentication, authorization, gate, rate-limit, and audit middleware an agent's call crosses, with no second implementation to keep in step.

func (*SessionCaller) CallTool

func (c *SessionCaller) CallTool(ctx context.Context, name string, args map[string]any) (map[string]any, error)

CallTool invokes one tool and returns its structured content.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL