Documentation
¶
Overview ¶
Package batch implements the cross-provider LLM Batch coordinator.
One plugin, two providers, three event surfaces:
- Subscribe: "llm.batch.submit" (events.BatchSubmit)
- Emit: "llm.batch.status" (events.BatchStatus)
- Emit: "llm.batch.results" (events.BatchResults)
The coordinator dispatches each submit to the configured provider's batch endpoint (Anthropic Messages Batches or OpenAI Batch API), persists state to disk so pollers survive restart, and polls in the background until the batch completes — at which point per-request results are emitted via the bus.
v1 scope notes (intentionally explicit so future readers know what's not here):
- Direct-API auth only. Bedrock / Vertex / Azure modes are not plumbed through; the per-provider auth code stays in providers/* and isn't reused here. A second-caller refactor is the right time to extract.
- Text-only LLMRequest -> wire-format adapters. Multimodal, extended thinking, prompt caching, predicted outputs and citations are NOT carried into batched requests. The synchronous llm.request path remains the supported way to use those features.
- OpenAI reasoning IS carried, on the Responses surface only: a batch line's `url` follows the role's effective `api:` and, on `api: responses`, the resolved `reasoning` object rides the body. The resolution is shared with nexus.llm.openai through engine.ResolveModelConfig; the serialization is still a second, minimal implementation, because the provider's builders are unexported methods on another plugin's state. See openai_surface.go for the full statement of what is shared, what is duplicated, and why.
- Cross-provider batches (split N requests across providers) are out of scope — one BatchSubmit = one provider.
- Cancellation is not exposed; pollers run until completion or process exit.
Index ¶
- func New() engine.Plugin
- type Plugin
- func (p *Plugin) Capabilities() []engine.Capability
- func (p *Plugin) ConfigSchema() []byte
- func (p *Plugin) Dependencies() []string
- func (p *Plugin) Emissions() []string
- func (p *Plugin) ID() string
- func (p *Plugin) Init(ctx engine.PluginContext) error
- func (p *Plugin) Name() string
- func (p *Plugin) Ready() error
- func (p *Plugin) Requires() []engine.Requirement
- func (p *Plugin) Shutdown(ctx context.Context) error
- func (p *Plugin) Subscriptions() []engine.EventSubscription
- func (p *Plugin) Version() string
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Plugin ¶
type Plugin struct {
// contains filtered or unexported fields
}
Plugin is the batch coordinator. It owns:
- the bus subscription on "llm.batch.submit"
- the on-disk state directory
- the in-memory map of active batches and their pollers
- the per-provider auth credentials (api keys only in v1)
func (*Plugin) Capabilities ¶
func (p *Plugin) Capabilities() []engine.Capability
func (*Plugin) ConfigSchema ¶
ConfigSchema implements engine.ConfigSchemaProvider.
func (*Plugin) Dependencies ¶
func (*Plugin) Requires ¶
func (p *Plugin) Requires() []engine.Requirement
func (*Plugin) Shutdown ¶
Shutdown unsubscribes and stops every poller. State files are intentionally left on disk so the next process boot resumes the same batches.
func (*Plugin) Subscriptions ¶
func (p *Plugin) Subscriptions() []engine.EventSubscription