batch

package
v0.31.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 31, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package batch implements the cross-provider LLM Batch coordinator.

One plugin, two providers, three event surfaces:

  • Subscribe: "llm.batch.submit" (events.BatchSubmit)
  • Emit: "llm.batch.status" (events.BatchStatus)
  • Emit: "llm.batch.results" (events.BatchResults)

The coordinator dispatches each submit to the configured provider's batch endpoint (Anthropic Messages Batches or OpenAI Batch API), persists state to disk so pollers survive restart, and polls in the background until the batch completes — at which point per-request results are emitted via the bus.

v1 scope notes (intentionally explicit so future readers know what's not here):

  • Direct-API auth only. Bedrock / Vertex / Azure modes are not plumbed through; the per-provider auth code stays in providers/* and isn't reused here. A second-caller refactor is the right time to extract.
  • Text-only LLMRequest -> wire-format adapters. Multimodal, extended thinking, prompt caching, predicted outputs and citations are NOT carried into batched requests. The synchronous llm.request path remains the supported way to use those features.
  • OpenAI reasoning IS carried, on the Responses surface only: a batch line's `url` follows the role's effective `api:` and, on `api: responses`, the resolved `reasoning` object rides the body. The resolution is shared with nexus.llm.openai through engine.ResolveModelConfig; the serialization is still a second, minimal implementation, because the provider's builders are unexported methods on another plugin's state. See openai_surface.go for the full statement of what is shared, what is duplicated, and why.
  • Cross-provider batches (split N requests across providers) are out of scope — one BatchSubmit = one provider.
  • Cancellation is not exposed; pollers run until completion or process exit.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func New

func New() engine.Plugin

New constructs a fresh Plugin instance.

Types

type Plugin

type Plugin struct {
	// contains filtered or unexported fields
}

Plugin is the batch coordinator. It owns:

  • the bus subscription on "llm.batch.submit"
  • the on-disk state directory
  • the in-memory map of active batches and their pollers
  • the per-provider auth credentials (api keys only in v1)

func (*Plugin) Capabilities

func (p *Plugin) Capabilities() []engine.Capability

func (*Plugin) ConfigSchema

func (p *Plugin) ConfigSchema() []byte

ConfigSchema implements engine.ConfigSchemaProvider.

func (*Plugin) Dependencies

func (p *Plugin) Dependencies() []string

func (*Plugin) Emissions

func (p *Plugin) Emissions() []string

func (*Plugin) ID

func (p *Plugin) ID() string

func (*Plugin) Init

func (p *Plugin) Init(ctx engine.PluginContext) error

func (*Plugin) Name

func (p *Plugin) Name() string

func (*Plugin) Ready

func (p *Plugin) Ready() error

func (*Plugin) Requires

func (p *Plugin) Requires() []engine.Requirement

func (*Plugin) Shutdown

func (p *Plugin) Shutdown(ctx context.Context) error

Shutdown unsubscribes and stops every poller. State files are intentionally left on disk so the next process boot resumes the same batches.

func (*Plugin) Subscriptions

func (p *Plugin) Subscriptions() []engine.EventSubscription

func (*Plugin) Version

func (p *Plugin) Version() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL