inference

inference

github.com/looprig/inference is Looprig's provider-neutral inference seam. It defines the Client / Request / Response contract over Core's content transcript types. It also has one wire codec per model API format, each translating between the neutral transcript and that format's published contract, and a generic HTTP transport that combines a codec, a router and an authorizer into a Client.

The module is provider-policy-free. It holds no provider default endpoints, no model catalogue and no credential discovery. Those live in llm.

Status

Released. Implemented API formats (model.APIFormat):

Also implemented:

• Request and stream encoding and decoding, with SSE, NDJSON and AWS Event Stream framing.
• Structured output (Request.Output, DecodeOutput) and tool choice.
• Normalized token usage and deterministic context counting.
• Bounded, classified retry through retry.
• The loopback gateway, which lets a client speaking one dialect reach any injected inference.Client target. See docs/inference-gateway.md.

Each codec's encoded request bodies are checked in tests against the format's official request schema. The schemas are in codec/conformance and are regenerated with make conformance-schemas.

Install

go get github.com/looprig/inference@latest

Packages

Usage

Build a request and compose an HTTP client for an OpenAI-compatible endpoint:

m := model.CustomModel("local", model.APIFormatOpenAI, "http://127.0.0.1:8080/v1", "demo-model")

client := transport.New(
	transport.Endpoint{BaseURL: m.BaseURL, Provider: m.Provider, APIFormat: m.APIFormat},
	route.StaticChat("/chat/completions"),
	openaiapi.Codec{},
	auth.None(), // or a credentials/httpauth Authorizer
)

resp, err := client.Invoke(ctx, inference.Request{
	Model:  m,
	System: "Answer briefly.",
	Messages: content.AgenticMessages{&content.UserMessage{Message: content.Message{
		Role:   content.RoleUser,
		Blocks: []content.Block{&content.TextBlock{Text: "Say hello."}},
	}}},
})

To use a hosted provider with its endpoint, auth policy and model catalogue already configured, take the provider package from llm. Runnable programs are in examples/: invoke, stream, retry and gateway.

Running without a provider: inferencetest

inferencetest.New returns a scripted inference.Client that needs no API key or network. Each call, through Invoke or Stream, consumes the next step of its script, and inferencetest.Model() is a descriptor that validates and declares tool and structured-output support:

client := inferencetest.New(
	inferencetest.Text("Let me check.").ToolCall("read_file", `{"path":"go.mod"}`),
	inferencetest.Text("It is module example.com/demo."),
)
def, err := loop.Define(
	loop.WithName("assistant"),
	loop.WithInference(client, inferencetest.Model()),
	// tools, access gate, ...
)
// After the turn: client.Requests() is what the model saw; client.Err()
// reports an exhausted script or a failed Step.Expect check.

A step can carry reasoning (Thinking), text, tool calls with deterministic ids (call_<call>_<n>), a finish reason and usage. It can also fail before output (Fail), fail mid-stream (StreamError), stream at a visible pace (ChunkDelay), or check its request (Expect). Echo().Repeat() answers anything with the user's last message, which suits a keyless demo. Stream emits the chunk shapes a real codec produces, and folding it yields the message Invoke returns. Like a real client, each call validates the request first.

Calls with no execution ceiling

transport.Client bounds Invoke by a whole-request timeout (5 minutes by default, WithInvokeTimeout) and Stream by a 60-second wait for response headers. For a call that may legitimately run longer, such as a slow local model or a long reasoning turn, mark its context:

ctx = transport.WithoutExecutionTimeout(ctx)
resp, err := client.Invoke(ctx, req) // or client.Stream(ctx, req)

A marked call has neither ceiling. Everything else is unchanged:

• TLS roots and a WithRoundTripper transport;
• dial and TLS-handshake limits;
• redirect refusal and authorization;
• response-size limits.

Unmarked calls are unaffected, and nothing is sent on the wire. Cancellation becomes the caller's job: the context's own cancel or deadline is the only end, and a half-open connection is detected only by TCP keepalive. llm providers built on transport.Client honour the marker as they are. llm's gemini and bedrock providers use their own HTTP client and do not.

Consumer obligation: Request.SessionID

Request.SessionID is an optional, stable identifier for a conversation. Providers that document a per-conversation header can use it. "" means absent, and no neutral codec encodes it.

ValidateRequestFeatures, which every codec runs, checks the id for every provider and refuses one that could not be sent unchanged as an HTTP header value. The reasons are control bytes, surrounding whitespace, or more than MaxSessionIDBytes (256) bytes. The error is *InvalidSessionIDError.

A refusal fails the whole request, not just the header. If you wire a session id, pre-check it with inference.ValidateRequestFeatures(inference.Request{SessionID: id}), then omit or map a refused id rather than failing every turn.

Where it sits

Tier 2 in the Looprig workspace. Direct Looprig dependencies: core, credentials, secrets. Consumers include llm, harness, eval, classifiers, mcp, tui and pluto.

Development

The Go baseline is 1.26.8. The module does not vendor. Verify it standalone:

GOWORK=off go test ./...
make check                # gofmt check, vet, staticcheck, gosec, govulncheck, race tests, build
make conformance-schemas  # re-derive codec/conformance/schema from upstream specs (network)

Other targets: fmt, fmt-check, vet, test, lint, vuln, secure, check-staticcheck, check-gosec, check-vuln, build. make fuzz only prints how to run a fuzz target:

go test -fuzz=FuzzXxx ./path/to/pkg -fuzztime=30s

The lint and security tools are pinned by tool directives in go.mod. The tests need no network, credentials or build tags.

License

Apache License 2.0. See LICENSE.