Documentation
¶
Overview ¶
Package engine is the top-level inference entry point for running ONNX models.
Index ¶
- Variables
- func LoadAndRun(modelPath string, inputs map[string]*tensor.Tensor) (map[string]*tensor.Tensor, error)
- func SidecarPathFor(modelPath string) string
- func WriteQ8Sidecar(path string, entries []Q8SidecarEntry) error
- type Engine
- func (e *Engine) Close() error
- func (e *Engine) InputNames() []string
- func (e *Engine) MaxConvParams() (maxInChannels, maxOutChannels, maxKernelH, maxKernelW int)
- func (e *Engine) Run(inputs map[string]*tensor.Tensor) (map[string]*tensor.Tensor, error)
- func (e *Engine) ValidationReport() *validate.ValidationReport
- type Option
- type Q8SidecarEntry
Constants ¶
This section is empty.
Variables ¶
var ( // ErrMissingInput indicates that a required graph input was not // provided in the inputs map passed to Run. ErrMissingInput = errors.New("engine: missing input") // ErrExecutionFailed indicates that a node failed during inference, // wrapping operator lookup, input resolution, or execution errors // with node name and op type context. ErrExecutionFailed = errors.New("engine: execution failed") // ErrValidationFailed indicates that pre-execution model validation // detected unsupported operators or shape mismatches. ErrValidationFailed = errors.New("engine: validation failed") )
Sentinel errors returned by the inference engine.
var ErrQ8SidecarInvalid = errors.New("engine: invalid Q8 sidecar")
ErrQ8SidecarInvalid wraps malformed-sidecar errors. The engine treats every parse failure as fatal (rather than silently falling back to fp32) because a present-but-broken sidecar usually signals a stale quantize_q8 build that would otherwise produce silent inference regressions on disk.
Functions ¶
func LoadAndRun ¶
func LoadAndRun(modelPath string, inputs map[string]*tensor.Tensor) (map[string]*tensor.Tensor, error)
LoadAndRun is a convenience function that loads a model and runs inference in a single call. It uses CPU-only execution (no options).
func SidecarPathFor ¶
SidecarPathFor returns the conventional sidecar path for an .onnx model: same directory and stem, .q8.bin extension. Other extensions pass through unchanged so callers can pass an arbitrary path and get a sibling .q8.bin back.
func WriteQ8Sidecar ¶
func WriteQ8Sidecar(path string, entries []Q8SidecarEntry) error
WriteQ8Sidecar writes the entries to path in the sidecar binary format documented above. The output is byte-for-byte deterministic given the same input, which makes round-tripping cheap to test.
Types ¶
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine holds the immutable parts of a compiled inference plan: the graph, the (shared, thread-safe) tensor pool, the provider chain, and the optional validation report.
All per-Run state lives in runState (created on the stack inside Run()), so multiple goroutines can call Run() concurrently on the same Engine without data races. This unlocks tile-level parallelism in the orchestrator.
cachedOps is parallel to graph.ExecutionOrder. When the first matching provider for a node is the CPU provider, we pre-fetch the Operator from the ops registry at construction time so Run() can dispatch directly without re-acquiring the registry mutex per node. nil entries fall back to the provider-loop path (preserves correctness for tests that bypass New() and any future non-CPU provider).
func New ¶
New loads an ONNX model from modelPath and builds an execution plan. It returns an Engine ready for inference or an error if loading or graph construction fails. Optional functional options configure GPU acceleration or custom provider chains. With no options the engine uses CPU-only execution, preserving backward compatibility.
func NewFromBytes ¶
NewFromBytes constructs an Engine directly from in-memory ONNX bytes. No temporary file is written — the bytes are parsed via onnx.Load from a bytes.Reader. Q8_0 sidecar quantisation is not available in this path (there is no file path to derive the sidecar name from).
func (*Engine) Close ¶
Close releases resources held by all providers in the engine's provider chain. It iterates each provider and calls Close, collecting any errors encountered.
func (*Engine) InputNames ¶
InputNames returns the names of the graph's input tensors in declaration order.
func (*Engine) MaxConvParams ¶
MaxConvParams scans the model graph for Conv nodes and returns the maximum input channels, output channels, kernel height, and kernel width across all Conv layers. Conv weight tensors have shape [outChannels, inChannels, kH, kW]. If no Conv nodes are found, all return values are 0.
func (*Engine) Run ¶
Run executes inference on the compiled graph using the provided input tensors. It returns the graph's output tensors keyed by name. Intermediate tensors are allocated from the engine's pool and released back to the pool once their last consumer has executed.
Run is reentrant: each call creates a fresh runState on the stack, so multiple goroutines can call Run on the same Engine concurrently (this is what tile-level parallelism in the pipeline orchestrator relies on). Only the underlying tensor.Pool is shared, and it is mutex-protected internally.
func (*Engine) ValidationReport ¶
func (e *Engine) ValidationReport() *validate.ValidationReport
ValidationReport returns the validation report produced by the WithValidation option, or nil if validation was not requested.
type Option ¶
Option configures an Engine during construction. Options are applied after the model is loaded and the default CPU-only provider chain is created.
func WithProviders ¶
func WithProviders(providers ...provider.ExecutionProvider) Option
WithGPU returns an Option that creates a GPUProvider backed by the WithProviders returns an Option that replaces the default provider list with the explicitly provided providers. This gives callers full control over provider ordering for advanced use cases.
func WithValidation ¶
WithValidation returns an Option that runs model validation after the graph is built. If validation fails (unsupported ops or shape errors), engine construction returns an error wrapping ErrValidationFailed. If validation passes, the report is stored and accessible via ValidationReport().
type Q8SidecarEntry ¶
type Q8SidecarEntry struct {
// Name is the ONNX initializer name this entry replaces. It MUST
// match an initializer in the .onnx exactly (case-sensitive).
Name string
// Shape is the logical (unquantized) tensor shape, e.g.
// [outC, inC, kH, kW] for a Conv weight.
Shape []int
// BlocksPerRow is ceil(inC*kH*kW / 32). Stored in the sidecar so
// loaders don't have to recompute it from the shape (the math is
// the same, but storing it future-proofs the format against
// non-standard block sizes).
BlocksPerRow int
// Blocks is the packed Q8_0 byte stream for this entry. Length is
// outC * BlocksPerRow * 36 bytes.
Blocks []byte
}
Q8SidecarEntry is one quantised Conv weight inside the sidecar. Tools (tools/quantize_q8) construct entries; the loader consumes them. Exported so the tool package, which lives under tools/quantize_q8 and only depends on engine for the format constants, can build the payload without duplicating the byte layout in two places.
func ReadQ8Sidecar ¶
func ReadQ8Sidecar(path string) ([]Q8SidecarEntry, error)
ReadQ8Sidecar parses a sidecar file written by WriteQ8Sidecar. It allocates a fresh Q8SidecarEntry slice; the byte slices in each entry are sub-slices of a single backing buffer so the caller pays O(1) heap allocations regardless of entry count.
Source Files