inference

package

v1.18.2 Latest Latest Go to latest Published: Oct 13, 2026 License: MIT Imports: 12 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/gsoultan/hermod

Links

Open Source Insights

Documentation ¶

Overview ¶

Package inference calls a model server and turns its answer back into rows.

Hermod does not run models in its own process: it is built without cgo, and a model that runs out of memory must not take a CDC pipeline down with it. A model is served by something else — Hermod's own hermod-ml worker, KServe, Triton, Seldon MLServer or an MLflow scoring server — and this package is the one client every caller (the Predict node, the REST and gRPC serving endpoints) goes through.

Two wire protocols cover those servers:

BackendOIP, the Open Inference Protocol (KServe "V2"), spoken by KServe, Triton, MLServer and hermod-ml: POST {url}/v2/models/{model}[/versions/{v}]/infer.
BackendMLflow, the MLflow scoring server: POST {url}/invocations.

Index ¶

Constants
type Backend
type Client
- func NewClient(hc *http.Client) *Client
- func (c *Client) Predict(ctx context.Context, t Target, rows []Row) ([]Row, error)
type Row
type Target
- func (t Target) Validate() error

Constants ¶

View Source

const DefaultTimeout = 30 * time.Second

DefaultTimeout bounds a call whose Target names none.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Backend ¶

type Backend string

Backend names the wire protocol a model server speaks.

const (
	// BackendOIP is the Open Inference Protocol (KServe V2).
	BackendOIP Backend = "oip"
	// BackendMLflow is the MLflow scoring server protocol.
	BackendMLflow Backend = "mlflow"
)

type Client ¶

type Client struct {
	// contains filtered or unexported fields
}

Client calls model servers.

func NewClient ¶

func NewClient(hc *http.Client) *Client

NewClient returns a client using hc, or Hermod's data client when hc is nil. The data client allows private addresses on purpose: a model server is usually inside the same network, like a database.

func (*Client) Predict ¶

func (c *Client) Predict(ctx context.Context, t Target, rows []Row) ([]Row, error)

Predict sends rows to the model and returns one prediction per row, in order.

type Row ¶

type Row = map[string]any

Row is one record sent to a model, or one prediction read back.

type Target ¶

type Target struct {
	Backend Backend
	// URL is the server's base URL, without the protocol's path.
	URL string
	// Model and Version name the model on an OIP server. Version may be empty
	// for the server's default. MLflow serves one model per URL and ignores both.
	Model   string
	Version string
	// Token, when set, is sent as "Authorization: Bearer <Token>".
	Token string
	// InputName switches OIP to matrix mode: one FP32 tensor of this name,
	// shaped [rows, len(Features)], with the values in Features order. This is
	// what an ONNX model exported from scikit-learn, PyTorch or TensorFlow
	// usually takes. Empty means column mode: one tensor per field.
	InputName string
	Features  []string
	// Timeout bounds the whole call. Zero means DefaultTimeout.
	Timeout time.Duration
}

Target says where a model is served and how to call it.

func (Target) Validate ¶

func (t Target) Validate() error

Validate reports what is wrong with a target before anything is sent to it.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL