ferret

package module
v2.0.0-alpha.35 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 23, 2026 License: Apache-2.0 Imports: 24 Imported by: 10

README

Ferret

Go Report Status Build Status Mastodon Follow Telegram Group Ferret release Apache-2.0 License

Try it! Docs CLI Test runner Web worker

Explore Ferret v2

Ferret v2 is currently in alpha. You can try the new syntax in the playground and read more about the design behind the new runtime and language capabilities:

Notice: This branch contains the upcoming Ferret v2. For the stable v1 release, please visit Ferret v1.

What is it?

Ferret is a declarative runtime for structured data extraction and automation.

It lets you query web pages, browser state, documents, APIs, and host-provided data sources with a dedicated query language, then return the result as structured data.

Instead of writing page-specific glue code for browser control, DOM traversal, waiting, extraction, and transformation, Ferret lets you describe the data you want and run that workflow from the CLI, a worker, or an embedded Go application.

Features
  • Declarative query language for structured data workflows
  • Support for static pages, dynamic pages, and browser-driven extraction
  • CLI tooling, including formatting and debugging support
  • Embeddable Go runtime for integrating Ferret into applications
  • Extensible module, function, and runtime capability system
  • Structured results for testing, analytics, AI/ML, and automation pipelines
  • Portable execution model with a focused VM

Getting started

go get github.com/MontFerret/ferret/v2@latest

There are currently two ways to start with Ferret v2:

  • Native v2 API - recommended for new projects
  • compat module - recommended as a first migration step for existing v1 integrations
New projects

Use the native v2 API built around the following flow:

Engine -> compile query -> create session -> run

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/MontFerret/ferret/v2/pkg/engine"
)

func main() {
	ctx := context.Background()

	eng, err := engine.New()
	if err != nil {
		log.Fatal(err)
	}
	defer eng.Close()

	plan, err := eng.Compile(`RETURN 1 + 1`)
	if err != nil {
		log.Fatal(err)
	}

	session, err := plan.NewSession()
	if err != nil {
		log.Fatal(err)
	}
	defer session.Close()

	result, err := session.Run(ctx)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(result.Content)
}
Migration from v1

Ferret v2 introduces a new architecture and public API, so embedding it directly is different from v1.

To make migration easier, v2 includes a compat module that provides a v1-style API. Its goal is to make upgrades incremental instead of forcing a full rewrite up front.

For many projects, the easiest migration path will be:

  • switch imports from v1 to the compat package
  • get the project compiling again
  • migrate incrementally to the native v2 API over time

A small helper script for rewriting import paths is planned to simplify this process further.

The compatibility layer is intended as a migration aid, not the long-term preferred API. New projects should use the native v2 packages directly.

Alpha status

Ferret v2 is currently in active development.

Alpha releases are intended for early adopters, experimentation, and feedback. Some APIs and language features may still change before the stable v2 release.

Documentation

Index

Constants

View Source 
const (
	DebugReasonEntry        = debugger.ReasonEntry
	DebugReasonBreakpoint   = debugger.ReasonBreakpoint
	DebugReasonStep         = debugger.ReasonStep
	DebugReasonPause        = debugger.ReasonPause
	DebugReasonRuntimeError = debugger.ReasonRuntimeError
	DebugReasonCompleted    = debugger.ReasonCompleted
	DebugReasonTerminated   = debugger.ReasonTerminated

	DebugBreakpointBindNextExecutableInFile     = debugger.BreakpointBindNextExecutableInFile
	DebugBreakpointBindExact                    = debugger.BreakpointBindExact
	DebugBreakpointBindNextExecutableInFunction = debugger.BreakpointBindNextExecutableInFunction
)
View Source 
const (
	// ProgramFormatJSON specifies the JSON format identifier for program serialization or artifact representation.
	ProgramFormatJSON = artifact.FormatJSON

	// ProgramFormatMsgPack specifies the MsgPack format identifier for program serialization or artifact representation.
	ProgramFormatMsgPack = artifact.FormatMsgPack
)

Variables

View Source 
var (
	// WithProgramFormat specifies an artifact format by applying the provided FormatID to the options configuration.
	WithProgramFormat = artifact.WithFormat

	// MarshalProgram serializes the given bytecode program into a byte slice using the provided artifact options.
	// Returns an error if the program is nil or fails during the marshaling process.
	MarshalProgram = artifact.Marshal

	// UnmarshalProgram decodes a byte slice into a *bytecode.Program object or returns an error if unmarshaling fails.
	UnmarshalProgram = artifact.Unmarshal
)
View Source 
var (
	// ParseLogLevel parses a log level string and returns the corresponding LogLevel and an error if the string is invalid.
	ParseLogLevel = logging.ParseLogLevel
	// MustParseLogLevel parses a log level string and panics if the string is not a valid log level.
	MustParseLogLevel = logging.MustParseLogLevel
	// FormatError formats a diagnostic error into a human-readable string.
	FormatError = diagnostics.Format
)

Functions

This section is empty.

Types

type DebugBreakpoint 

type DebugBreakpoint = debugger.Breakpoint

type DebugBreakpointBindingMode 

type DebugBreakpointBindingMode = debugger.BreakpointBindingMode

type DebugBreakpointID 

type DebugBreakpointID = debugger.BreakpointID

type DebugBreakpointOptions 

type DebugBreakpointOptions = debugger.BreakpointOptions

type DebugEvent 

type DebugEvent = debugger.Event

type DebugFormatOptions 

type DebugFormatOptions = debugger.FormatOptions

type DebugFrame 

type DebugFrame = debugger.Frame

type DebugLocation 

type DebugLocation = debugger.Location

type DebugReason 

type DebugReason = debugger.Reason

type DebugSession 

type DebugSession = debugger.Session

type DebugSourceLocation 

type DebugSourceLocation = debugger.SourceLocation

type DebugStateError 

type DebugStateError = debugger.StateError

type DebugValue 

type DebugValue = debugger.Value

type DebugValueReference 

type DebugValueReference = debugger.ValueReference

type DebugVariable 

type DebugVariable = debugger.Variable

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine compiles queries into reusable plans and runs them against the configured host.

func New 

func New(setters ...Option) (*Engine, error)

New constructs an Engine from the provided options, registers all modules, builds the host, and runs engine init hooks. It returns an error if any option, module registration, or init hook fails.

func (*Engine) Close 

func (e *Engine) Close() error

Close runs the engine close hooks and releases engine-scoped resources.

func (*Engine) Compile 

func (e *Engine) Compile(ctx context.Context, src *source.Source) (*Plan, error)

Compile compiles source into a reusable execution plan.

func (*Engine) CompileDebug 

func (e *Engine) CompileDebug(ctx context.Context, src *source.Source) (*Plan, error)

CompileDebug compiles source into a reusable plan with source-level debugger metadata. Debug compilation uses effective O0 optimization.

func (*Engine) Load 

func (e *Engine) Load(data []byte) (*Plan, error)

Load decodes a serialized program artifact and wraps it in a reusable plan.

func (*Engine) Run 

func (e *Engine) Run(ctx context.Context, src *source.Source, opts ...SessionOption) (*Output, error)

Run compiles source, executes it in a fresh session, and returns encoded output and an error. Similar to Session.Run, it may return a non-nil *Output together with a non-nil error (for example, if execution produced output but a deferred cleanup step failed).

type Option 

type Option func(env *options) error

Option configures an Engine during construction.

func WithAfterCompileHook 

func WithAfterCompileHook(hook module.AfterCompileHook) Option

WithAfterCompileHook returns an Option that registers a hook to execute after each compilation attempt. The hook receives the compilation error (if any). It returns an error if hook is nil.

func WithAfterRunHook 

func WithAfterRunHook(hook module.AfterRunHook) Option

WithAfterRunHook returns an Option that registers a hook to execute after each session run attempt. The hook receives the run error (if any). It returns an error if hook is nil.

func WithBeforeCompileHook 

func WithBeforeCompileHook(hook module.BeforeCompileHook) Option

WithBeforeCompileHook returns an Option that registers a hook to execute before each compilation attempt. It returns an error if hook is nil.

func WithBeforeRunHook 

func WithBeforeRunHook(hook module.BeforeRunHook) Option

WithBeforeRunHook returns an Option that registers a hook to execute before each session run. The hook can replace the context used by subsequent hooks and VM execution. It returns an error if hook is nil.

func WithCompilerOptions 

func WithCompilerOptions(opts ...compiler.Option) Option

WithCompilerOptions creates an Option that appends the provided compiler options to the options if not empty.

func WithEncodingCodec 

func WithEncodingCodec(contentType string, codec encoding.Codec) Option

WithEncodingCodec registers or overrides a codec for the given content type.

func WithEncodingRegistry 

func WithEncodingRegistry(registry *encoding.Registry) Option

WithEncodingRegistry sets a custom encoding registry for query execution.

func WithEngineCloseHook 

func WithEngineCloseHook(hook module.EngineCloseHook) Option

WithEngineCloseHook returns an Option that registers a hook to execute when the engine is closed. It returns an error if hook is nil.

func WithEngineInitHook 

func WithEngineInitHook(hook module.EngineInitHook) Option

WithEngineInitHook returns an Option that registers a hook to execute during engine initialization. It returns an error if hook is nil.

func WithFSReadOnly 

func WithFSReadOnly() Option

WithFSReadOnly sets the engine's file system to read-only mode.

func WithFSRoot 

func WithFSRoot(root string) Option

WithFSRoot sets the root directory for the engine's file system.

func WithFunctions 

func WithFunctions(funcs *runtime.Functions) Option

WithFunctions merges the provided *runtime.Functions into the engine's function library.

func WithFunctionsRegistrar 

func WithFunctionsRegistrar(setter func(ns runtime.Namespace)) Option

WithFunctionsRegistrar creates an Option that invokes the provided registrar with the engine's runtime.Namespace if the registrar is not nil.

func WithLog 

func WithLog(writer io.Writer) Option

WithLog sets the writer for logging output. The writer can be any io.Writer, such as os.Stdout or a file.

func WithLogFields 

func WithLogFields(fields map[string]any) Option

WithLogFields sets the fields to be included in log entries. These fields can provide additional context for debugging and monitoring purposes.

func WithLogLevel 

func WithLogLevel(lvl logging.LogLevel) Option

WithLogLevel sets the logging level for the engine. The logging level determines the severity of log messages that will be recorded.

func WithMaxActiveSessions 

func WithMaxActiveSessions(n int) Option

WithMaxActiveSessions sets an engine-wide limit on concurrently active sessions.

This limit applies to Session objects created from any plan compiled by the engine. When the limit is reached, Plan.NewSession blocks until another session is closed or the provided context is canceled.

Use this when you want to put a global cap on query execution concurrency and the host-side resources that come with it, such as CPU, memory, network traffic, or downstream service pressure.

This is different from WithMaxIdleVMsPerPlan and WithMaxVMsPerPlan: WithMaxActiveSessions controls how many sessions may be running or checked out at once across the engine, while the VM options control how each individual plan manages its VM pool.

A value of 0 disables the limit.

func WithMaxIdleVMsPerPlan 

func WithMaxIdleVMsPerPlan(n int) Option

WithMaxIdleVMsPerPlan sets how many closed-session VMs each plan keeps warm for reuse after they become idle.

This is a retention setting, not a concurrency limit. It only controls how many unused VMs remain cached in a plan's pool after sessions are closed. When the idle cache is full, additional returned VMs are closed instead of retained.

Use this when the same compiled plan is executed repeatedly and you want to trade some steady-state memory for faster session creation by reusing already initialized VMs.

This is different from WithMaxVMsPerPlan: WithMaxIdleVMsPerPlan controls how many unused VMs stay cached, while WithMaxVMsPerPlan controls the maximum total number of VMs the plan may own at all, including both idle and currently borrowed VMs.

A value of 0 disables idle retention for the plan.

func WithMaxVMsPerPlan 

func WithMaxVMsPerPlan(n int) Option

WithMaxVMsPerPlan sets a hard per-plan limit on the total number of VMs the plan's pool may own at one time.

The total includes both idle VMs kept in the pool and VMs currently borrowed by active sessions created from that plan. When the limit is reached and no idle VM is available to reuse, session creation fails with vm.ErrPoolExhausted.

Use this when you need a strict upper bound on the memory or resource footprint of a single hot plan, even if that plan is under heavy concurrent load.

This is different from WithMaxActiveSessions: WithMaxVMsPerPlan limits VM ownership for one plan, while WithMaxActiveSessions limits active session concurrency across the entire engine.

This is also different from WithMaxIdleVMsPerPlan: WithMaxVMsPerPlan is a hard cap, while WithMaxIdleVMsPerPlan only decides how many unused VMs are retained after demand drops.

A value of 0 means the plan may create as many VMs as needed, subject only to other limits such as WithMaxActiveSessions.

func WithModules 

func WithModules(mods ...module.Module) Option

WithModules creates an Option that appends the provided modules to the options if not empty.

func WithNamespace 

func WithNamespace(ns runtime.Namespace) Option

WithNamespace merges the functions from the provided runtime.Namespace into the engine's function library.

func WithNetwork 

func WithNetwork(network ferretnet.Network) Option

WithNetwork sets the engine network service used by derived executions.

func WithParam 

func WithParam(name string, value any) Option

WithParam returns an Option that sets a parameter with the specified name and value in the options configuration. The name cannot be empty, and the value cannot be nil. It ensures the parameter value is correctly parsed and stored.

func WithParams 

func WithParams(params map[string]any) Option

WithParams applies custom parameters to the options by merging them with existing ones, initializing if necessary. If a parameter already exists, it will be overwritten. All host values will be converted to a runtime.Value.

func WithPlanCloseHook 

func WithPlanCloseHook(hook module.PlanCloseHook) Option

WithPlanCloseHook returns an Option that registers a hook to execute when a plan is closed. It returns an error if hook is nil.

func WithProgramLoader 

func WithProgramLoader(loader *artifact.Loader) Option

WithProgramLoader sets a custom artifact loader for Engine.Load.

func WithRuntimeParam 

func WithRuntimeParam(name string, value runtime.Value) Option

WithRuntimeParam returns an Option that sets a runtime parameter with the specified name and value in the options configuration. The name cannot be empty, and the value cannot be nil.

func WithRuntimeParams 

func WithRuntimeParams(params runtime.Params) Option

WithRuntimeParams configures runtime parameters by merging the provided params with existing ones in options. If a parameter already exists, it will be overwritten.

func WithSessionCloseHook 

func WithSessionCloseHook(hook module.SessionCloseHook) Option

WithSessionCloseHook returns an Option that registers a hook to execute when a session is closed. It returns an error if hook is nil.

func WithStdlib 

func WithStdlib(set stdlib.Set) Option

WithStdlib configures which standard library groups are registered by default.

func WithoutStdlib 

func WithoutStdlib() Option

WithoutStdlib disables the standard library, so no built-in functions are registered by default.

type Output 

type Output = encoding.Output

Output is the encoded result returned from session or engine execution.

type Plan 

type Plan struct {
	// contains filtered or unexported fields
}

Plan wraps a compiled program together with the host state needed to execute it.

func (*Plan) Close 

func (p *Plan) Close() error

Close runs plan cleanup hooks and closes the plan's VM pool.

func (*Plan) Marshal 

func (p *Plan) Marshal(opts ...artifact.Option) ([]byte, error)

Marshal serializes the plan's compiled program into a byte slice using the provided artifact options.

func (*Plan) NewDebugSession 

func (p *Plan) NewDebugSession(ctx context.Context, setters ...SessionOption) (*DebugSession, error)

NewDebugSession creates a retained-state source-level debugging session.

func (*Plan) NewSession 

func (p *Plan) NewSession(ctx context.Context, setters ...SessionOption) (*Session, error)

NewSession creates a session for executing the plan with optional per-run settings.

func (*Plan) Params 

func (p *Plan) Params() []string

Params returns the list of parameter names declared in the query.

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session represents the execution of a compiled Ferret program. It holds the state of the execution, including the virtual machine, environment, and encoding registry. A Session is created from a Plan and can be run to obtain results.

Session is not safe for concurrent use by multiple goroutines, except that Close is idempotent and safe to call multiple times, including concurrently. It is typically used for a single logical execution. When a Session is created directly via Plan.NewSession, it may be reused for multiple sequential runs as long as the environment and encoding registry are not modified between runs. Helper APIs such as Engine.Run may take ownership of the Session and close it after a single execution, in which case the caller must not attempt to reuse it.

func (*Session) Close 

func (s *Session) Close() error

Close releases the session's borrowed VM and runs close hooks. It is idempotent and safe to call multiple times, including concurrently.

func (*Session) Run 

func (s *Session) Run(c context.Context) (*Output, error)

Run executes the session with the provided context and returns encoded output.

type SessionOption 

type SessionOption func(*sessionOptions) error

SessionOption configures a Session created from a Plan.

func WithDebugFormat 

func WithDebugFormat(options DebugFormatOptions) SessionOption

WithDebugFormat configures bounded debugger value formatting.

func WithEnvironmentOptions 

func WithEnvironmentOptions(opts ...vm.EnvironmentOption) SessionOption

WithEnvironmentOptions appends VM environment options to the created session.

func WithOutputContentType 

func WithOutputContentType(contentType string) SessionOption

WithOutputContentType selects the output codec content type for session results.

func WithSessionLog 

func WithSessionLog(writer io.Writer) SessionOption

WithSessionLog sets the writer for logging output. The writer can be any io.Writer, such as os.Stdout or a file.

func WithSessionLogFields 

func WithSessionLogFields(fields map[string]any) SessionOption

WithSessionLogFields sets the fields to be included in log entries for the session. These fields can provide additional context for debugging and monitoring purposes.

func WithSessionLogLevel 

func WithSessionLogLevel(lvl logging.LogLevel) SessionOption

WithSessionLogLevel sets the logging level for the session. The logging level determines the severity of log messages that will be recorded.

func WithSessionParam 

func WithSessionParam(name string, value any) SessionOption

WithSessionParam adds or overrides a single session parameter.

func WithSessionParams 

func WithSessionParams(params map[string]any) SessionOption

WithSessionParams merges the provided parameter map into the session environment, overriding existing keys while preserving any other previously defined parameters.

func WithSessionRuntimeParam 

func WithSessionRuntimeParam(name string, value runtime.Value) SessionOption

WithSessionRuntimeParam adds or overrides a single session parameter using a pre-converted runtime.Value.

func WithSessionRuntimeParams 

func WithSessionRuntimeParams(params runtime.Params) SessionOption

WithSessionRuntimeParams merges the provided runtime.Params into the session environment, overriding existing keys while preserving any other previously defined parameters.

Source Files

View all Source files

Directories

Path Synopsis
Package compat provides a Ferret v1-compatible public API surface, built on top of the Ferret v2 engine.
 Package compat provides a Ferret v1-compatible public API surface, built on top of the Ferret v2 engine.
compiler
Package compiler provides a v1-compatible Compiler for the Ferret compatibility layer.
 Package compiler provides a v1-compatible Compiler for the Ferret compatibility layer.
runtime
Package runtime provides v1-compatible runtime types for the Ferret compatibility layer.
 Package runtime provides v1-compatible runtime types for the Ferret compatibility layer.
runtime/core
Package core provides v1-compatible runtime core types for the Ferret compatibility layer.
 Package core provides v1-compatible runtime core types for the Ferret compatibility layer.
runtime/values
Package values provides v1-compatible concrete value types and helpers for the Ferret compatibility layer.
 Package values provides v1-compatible concrete value types and helpers for the Ferret compatibility layer.
runtime/values/types
Package types provides v1-compatible type constants for the Ferret compatibility layer.
 Package types provides v1-compatible type constants for the Ferret compatibility layer.
examples
embedded command
extensible command
pkg
asm
bytecode
bytecode/artifact
bytecode/format
bytecode/format/json
bytecode/format/msgpack
bytecode/internal/persist
compiler
compiler/internal
compiler/internal/core
compiler/internal/optimization
debugger
Package debugger implements source-level debugger policy and orchestration over a retained VM execution.
 Package debugger implements source-level debugger policy and orchestration over a retained VM execution.
diagnostics
encoding
encoding/json
encoding/msgpack
formatter
formatter/internal
fs
internal/debugpoint
internal/valueset
logging
module
Package module defines the extension contracts used to customize engine bootstrap and lifecycle behavior.
 Package module defines the extension contracts used to customize engine bootstrap and lifecycle behavior.
net
net/http
Package http provides Ferret's sandboxed HTTP client and its reusable access policy.
 Package http provides Ferret's sandboxed HTTP client and its reusable access policy.
parser
parser/diagnostics
parser/fql
parser/tools command
runtime
sdk
Package sdk is the supported authoring layer for Ferret modules and host values.
 Package sdk is the supported authoring layer for Ferret modules and host values.
sdk/sdktest
Package sdktest provides a small black-box harness for module authors.
 Package sdktest provides a small black-box harness for module authors.
source
stdlib
stdlib/arrays
stdlib/collections
stdlib/datetime
stdlib/io
stdlib/io/fs
stdlib/io/net
stdlib/io/net/http
stdlib/math
stdlib/objects
stdlib/path
stdlib/strings
stdlib/testing
stdlib/testing/base
stdlib/types
stdlib/utils
vm
vm/internal/data
vm/internal/diagnostics
vm/internal/frame
vm/internal/mem
Package mem provides VM storage helpers and the narrow ownership layer used to clean up direct register-held closers.
 Package mem provides VM storage helpers and the narrow ownership layer used to clean up direct register-held closers.
vm/test
spec
spec/assert
spec/compile
spec/compile/inspect
spec/exec
spec/format
spec/mock
spec/optimize

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.