classify

type AudioChunk ¶

type AudioChunk struct {
	// PCM is the raw PCM audio bytes for this chunk. Format
	// declared by the parent AudioStreamOptions.
	PCM []byte
	// Final marks the last chunk; backends may flush state.
	Final bool
}

AudioChunk is a single packet in a streaming audio classification request. Backends that stream consume these from a channel until it closes.

type AudioClassifier ¶

type AudioClassifier interface {
	ClassifyAudio(ctx context.Context, audio []byte, opts AudioOptions) ([]LabelScore, error)
}

AudioClassifier classifies a single audio clip into one or more labeled scores. Required interface for every backend that supports audio (HF base, Hume, Deepgram, ONNX).

Implementations must be safe for concurrent use; the registry hands the same instance to multiple eval handlers.

type AudioOptions ¶

type AudioOptions struct {
	// Model is the backend-specific model identifier (e.g.
	// "superb/wav2vec2-base-superb-er" on HuggingFace).
	Model string

	// Language is an optional ISO-639-1 hint (e.g. "en"). Only used
	// by backends that route by language.
	Language string

	// SampleRate is the source audio sample rate in Hz. Backends
	// that can resample internally use this to drive it. The
	// shipped HuggingFace backend does NOT resample — handlers
	// are expected to deliver audio at the rate the target model
	// wants (typically 16 kHz mono for SER models). The field is
	// kept on the options struct so future backends (ONNX, Hume,
	// etc.) that do resample have a place to read the source rate
	// from without growing a per-backend option type.
	SampleRate int

	// MIMEType is the audio container / codec hint (e.g. "audio/wav",
	// "audio/L16"). Backends that need explicit format selection use
	// this; others ignore.
	MIMEType string
}

AudioOptions carries per-call knobs for audio classification. Backends ignore fields they don't support; consumers set what matters for their target model.

type AudioStreamOptions ¶

type AudioStreamOptions struct {
	AudioOptions
	// ChunkInterval is the source's emit cadence; backends use it
	// to size analysis windows and stamp output timestamps.
	ChunkInterval time.Duration
}

AudioStreamOptions carries the framing contract for a streaming audio classification request.

type Backend ¶ added in v1.4.16

type Backend = any

Backend is whatever a factory returns: a value that implements one or more of the task interfaces (AudioClassifier, TextClassifier, …). RegisterBackend type-asserts it against each. It is intentionally an alias for any — there is no method common to all task interfaces.

func CreateFromSpec(spec ProviderSpec) (Backend, error)

type EmbedOptions ¶

type EmbedOptions struct {
	// Model is the backend-specific model identifier (e.g.
	// "voyage-3", "text-embedding-3-large").
	Model string

	// Dimensions optionally requests truncation to a specific
	// dimensionality. Backend ignores if not supported.
	Dimensions int

	// InputType disambiguates query vs document embedding for
	// asymmetric models (e.g. Voyage's `query` / `document`).
	// Empty = backend default.
	InputType string
}

EmbedOptions carries per-call knobs for text embedding.

type Embedder ¶

type Embedder interface {
	Embed(ctx context.Context, inputs []string, opts EmbedOptions) ([][]float32, error)
}

Embedder turns text inputs into dense vectors. Co-located with the classifiers because it shares the same shape (task interface + multiple backends + registry lookup) and because moving embeddings here breaks the historical dependency between runtime/evals/handlers/cosine_similarity.go and concrete provider packages.

The current cosine_similarity handler reads pre-computed embeddings from EvalContext.Metadata — it doesn't call an Embedder directly. Migrating to this interface is queued (see phase 2 of the inference abstraction proposal); the interface lives here from day one so the migration doesn't introduce a new package later.

type Factory ¶ added in v1.4.16

type Factory = base.Factory[Backend]

Factory builds a Backend from a spec. Per-backend packages register one via init() (see runtime/classify/backends/hf/register.go) so this package never imports them.

type ImageClassifier ¶

type ImageClassifier interface {
	ClassifyImage(ctx context.Context, img []byte, opts ImageOptions) ([]LabelScore, error)
}

ImageClassifier classifies a single still image. Visual content moderation (NSFW, violence, brand safety), scene tagging, object recognition all go through this. No streaming sibling — a still image is a single inference.

type ImageOptions ¶

type ImageOptions struct {
	// Model is the backend-specific model identifier (e.g.
	// "Falconsai/nsfw_image_detection" on HuggingFace).
	Model string

	// MIMEType is the image format hint (e.g. "image/png", "image/jpeg").
	MIMEType string
}

ImageOptions carries per-call knobs for image classification.

type LabelScore ¶

type LabelScore struct {
	Label string
	Score float64
}

LabelScore pairs a classifier label with a confidence score in [0, 1]. Backends return slices of these sorted by descending score where the model supports ranking; the caller treats the order as hint only and looks up by Label when checking expected values.

type LabelScoreEvent ¶

type LabelScoreEvent struct {
	Timestamp time.Duration
	Window    time.Duration
	Scores    []LabelScore
}

LabelScoreEvent is one window's worth of classification output from a streaming backend. Timestamp is relative to the start of the input stream.

type ProviderSpec ¶ added in v1.4.16

type ProviderSpec = base.CapabilitySpec

ProviderSpec is the runtime form of an inference-provider declaration. Aliased to base.CapabilitySpec so the field shape is shared with the TTS, STT, embedding, and image factories (id/type/model/base_url/ credential/additional_config).

type Registry ¶

type Registry struct {
	// contains filtered or unexported fields
}

Registry holds named classifier and embedder instances keyed by an id supplied at config time (e.g. "hf", "hf-azure", "onnx-local"). Eval handlers look up by id; the id-to-backend mapping is the only thing the handler config needs to know about.

Backends register themselves into a Registry at engine startup; the registry then travels with context.Context to handlers via WithRegistry / FromContext.

func BuildRegistry(specs []ProviderSpec, defaults RegistryDefaults) (*Registry, error)

func FromContext(ctx context.Context) *Registry

func NewRegistry() *Registry

func (r *Registry) AudioClassifier(id string) (AudioClassifier, error)

func (r *Registry) Embedder(id string) (Embedder, error)

func (r *Registry) ImageClassifier(id string) (ImageClassifier, error)

func (r *Registry) RegisterAudio(id string, c AudioClassifier)

func (r *Registry) RegisterEmbedder(id string, e Embedder)

func (r *Registry) RegisterImage(id string, c ImageClassifier)

func (r *Registry) RegisterText(id string, c TextClassifier)

func (r *Registry) RegisterVideo(id string, c VideoClassifier)

func (r *Registry) SetDefaultAudio(id string) error

func (r *Registry) SetDefaultEmbedder(id string) error

func (r *Registry) SetDefaultImage(id string) error

func (r *Registry) SetDefaultText(id string) error

func (r *Registry) SetDefaultVideo(id string) error

func (r *Registry) TextClassifier(id string) (TextClassifier, error)

func (r *Registry) VideoClassifier(id string) (VideoClassifier, error)

type RegistryDefaults ¶ added in v1.4.16

type RegistryDefaults struct {
	AudioClassifier string
	TextClassifier  string
	ImageClassifier string
	VideoClassifier string
	Embedder        string
}

RegistryDefaults pins which provider id serves each task when a handler doesn't name one. Mirrors config.InferenceDefaults but lives in runtime so both Arena and the SDK map onto it.

type StreamingAudioClassifier ¶

type StreamingAudioClassifier interface {
	AudioClassifier
	ClassifyAudioStream(
		ctx context.Context,
		chunks <-chan AudioChunk,
		opts AudioStreamOptions,
	) (<-chan LabelScoreEvent, error)
}

StreamingAudioClassifier is the optional capability for live classification — Hume, Deepgram, and locally-windowed ONNX implement it; HF Inference API base does not. Eval handlers that only need end-of-turn classification depend on AudioClassifier; hooks and guardrails that react mid-call type-assert to this.

type StreamingVideoClassifier ¶

type StreamingVideoClassifier interface {
	VideoClassifier
	ClassifyVideoStream(
		ctx context.Context,

		chunks <-chan struct{},
		opts VideoOptions,
	) (<-chan LabelScoreEvent, error)
}

StreamingVideoClassifier is the optional capability for live video classification (mostly a future concern — agentic-UI replay, security camera streams). No MVP backend implements it; the interface stays in the package so the synchronous one doesn't break when a streaming consumer arrives.

type TextClassifier ¶

type TextClassifier interface {
	ClassifyText(ctx context.Context, text string, opts TextOptions) ([]LabelScore, error)
}

TextClassifier classifies a single text string. Toxicity, sentiment, emotion-from-text, intent, language detection — anything that takes text in and returns labeled scores.

Text inputs aren't temporally extended, so there's no streaming sibling. Batch text classification (many strings at once) goes through repeated calls — backends that support batching internally can pool requests.

type TextOptions ¶

type TextOptions struct {
	// Model is the backend-specific model identifier (e.g.
	// "unitary/toxic-bert" on HuggingFace).
	Model string

	// Language is an optional ISO-639-1 hint.
	Language string

	// MultiLabel asks the backend to return scores for every label
	// rather than a single best one. Useful for toxicity panels
	// (toxic, severe_toxic, obscene, …) where multiple may apply.
	MultiLabel bool
}

TextOptions carries per-call knobs for text classification.

type VideoClassifier ¶

type VideoClassifier interface {
	ClassifyVideo(ctx context.Context, video []byte, opts VideoOptions) ([]LabelScore, error)
}

VideoClassifier classifies a video clip. Two physical shapes:

• Whole-clip backends (e.g. HuggingFace Video Classification API) accept the full clip and return labels for it.
• Decomposing backends extract the audio track and frame-sample stills, route those through AudioClassifier + ImageClassifier, and aggregate the per-modality scores. The interface is the same; backends choose the implementation.

VideoOptions.FrameSampleRate and .ExtractAudio steer decomposing backends; whole-clip backends ignore them.

type VideoOptions ¶

type VideoOptions struct {
	// Model is the backend-specific model identifier.
	Model string

	// MIMEType is the video container hint (e.g. "video/mp4").
	MIMEType string

	// FrameSampleRate is the per-second frame extraction rate used
	// by decomposing backends (audio + sampled image classifier).
	// Whole-clip backends ignore this. 0 = backend default.
	FrameSampleRate float64

	// ExtractAudio toggles whether decomposing backends route the
	// audio track through their AudioClassifier as well as their
	// ImageClassifier. Default false.
	ExtractAudio bool
}

VideoOptions carries per-call knobs for video classification.