dispatcher

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 27, 2026 License: MIT Imports: 20 Imported by: 0

README

Dispatcher Package

Pluggable event dispatcher service that consumes events from the event bus and forwards them to external observability platforms.

Purpose & Responsibilities

  • Event Consumption: Subscribes to the event bus and processes incoming events
  • Event Transformation: Converts internal events to formats required by external services
  • Batching: Groups events for efficient network transmission
  • Retry Logic: Handles transient failures with exponential backoff
  • Plugin Architecture: Supports multiple backend services via plugins

Architecture

flowchart LR
    subgraph Input
        EB[EventBus]
    end
    
    subgraph Dispatcher
        T[Transformer]
        B[Batcher]
        R[Retry Logic]
    end
    
    subgraph Plugins
        F[File Plugin]
        L[Lunary Plugin]
        H[Helicone Plugin]
    end
    
    EB --> T
    T --> B
    B --> R
    R --> F
    R --> L
    R --> H

Available Backend Plugins

Plugin Description Use Case
File Writes events to JSONL file Local storage, debugging
Lunary Sends to Lunary.ai LLM observability
Helicone Sends to Helicone LLM analytics
Plugin Configuration
Plugin Key Description Required Default
File endpoint File path for JSONL output Yes -
Lunary api-key Lunary API key Yes -
Lunary endpoint API endpoint URL No https://api.lunary.ai/v1/runs/ingest
Helicone api-key Helicone API key Yes -
Helicone endpoint API endpoint URL No https://api.worker.helicone.ai/custom/v1/log

Helicone-Specific Features: Automatic provider detection, token usage injection, request ID propagation, non-JSON response handling (base64).

Service Configuration

Parameter Description Default
BufferSize Event bus buffer size 1000
BatchSize Events per batch 100
FlushInterval Max time between flushes 5s
RetryAttempts Retry count on failure 3
RetryBackoff Initial backoff duration 1s
Plugin Backend plugin (required) -
PluginName Plugin name for offset tracking -
Verbose Include debug info in payloads false

CLI Usage

# File output
llm-proxy dispatcher --service file --endpoint events.jsonl

# Lunary integration
llm-proxy dispatcher --service lunary --api-key $LUNARY_API_KEY

# Helicone integration
llm-proxy dispatcher --service helicone --api-key $HELICONE_API_KEY
CLI Options
Flag Description Default
--service Backend service (file, lunary, helicone) file
--endpoint API endpoint or file path Service-specific
--api-key API key for external services -
--buffer Event bus buffer size 1000
--batch-size Batch size for sending events 100
--detach Run in background (daemon mode) false

Environment Variables

Variable Description Default
LLM_PROXY_API_KEY API key for selected service -
LLM_PROXY_ENDPOINT Default endpoint URL -
LLM_PROXY_EVENT_BUS Event bus backend (in-memory or redis-streams) redis-streams

Batching and Retry Flow

flowchart TD
    E[Events Arrive] --> A{Batch Full?}
    A -->|Yes| S[Send Batch]
    A -->|No| T{Flush Timer?}
    T -->|Yes| S
    T -->|No| E
    
    S --> R{Success?}
    R -->|Yes| C[Clear Batch]
    R -->|No| B{Retries Left?}
    B -->|Yes| W[Backoff Wait]
    W --> S
    B -->|No| D[Drop & Log]
    
    C --> E
    D --> E

Retry Behavior:

  • Exponential backoff: attempt * RetryBackoff
  • Permanent errors (HTTP 4xx) are not retried
  • After all retries exhausted, batch is dropped and logged

Event Transformation

The transformer converts eventbus.Event to EventPayload for external services.

EventPayload fields: Type, Event, RunID, Timestamp, Input, Output, TokensUsage, Metadata, Tags, LogID

Key Functions: NewDefaultEventTransformer(verbose), Transform(evt), implement EventTransformer interface for custom logic.

Integration Patterns

flowchart TB
    subgraph "Single Process"
        P1[Proxy] --> EB1[In-Memory EventBus]
        EB1 --> D1[Dispatcher]
    end
    
    subgraph "Distributed"
        P2[Proxy 1] --> R[Redis EventBus]
        P3[Proxy 2] --> R
        R --> D2[Standalone Dispatcher]
    end

Testing Guidance

  • Unit Tests: Create a mock implementing BackendPlugin interface to capture sent events
  • Transformer Tests: Use NewDefaultEventTransformer with test events
  • Existing Tests: See service_test.go, transformer_test.go, plugins/plugins_test.go

Files

File Description
service.go Main dispatcher service implementation
plugin.go BackendPlugin interface and EventPayload struct
transformer.go Event transformation logic
errors.go Error types including PermanentBackendError
plugins/registry.go Plugin factory registry
plugins/file.go File backend plugin
plugins/lunary.go Lunary.ai backend plugin
plugins/helicone.go Helicone backend plugin

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BackendPlugin

type BackendPlugin interface {
	// Init initializes the plugin with configuration
	Init(cfg map[string]string) error

	// SendEvents sends a batch of events to the backend
	SendEvents(ctx context.Context, events []EventPayload) error

	// Close cleans up any resources used by the plugin
	Close() error
}

BackendPlugin defines the interface for dispatcher backends

type Config

type Config struct {
	BufferSize       int
	BatchSize        int
	FlushInterval    time.Duration
	RetryAttempts    int
	RetryBackoff     time.Duration
	Plugin           BackendPlugin
	EventTransformer EventTransformer
	PluginName       string
	Verbose          bool // If true, include response_headers and extra debug info
}

Config holds configuration for the dispatcher service

type DefaultEventTransformer

type DefaultEventTransformer struct {
	Verbose bool
}

DefaultEventTransformer provides a basic transformation from eventbus.Event to EventPayload Verbose: if true, includes response_headers in metadata Use NewDefaultEventTransformer(verbose) to construct

func NewDefaultEventTransformer

func NewDefaultEventTransformer(verbose bool) *DefaultEventTransformer

NewDefaultEventTransformer creates a transformer with the given verbosity

func (*DefaultEventTransformer) Transform

Transform converts an eventbus.Event to an EventPayload

type EventPayload

type EventPayload struct {
	Type         string          `json:"type"`
	Event        string          `json:"event"`
	RunID        string          `json:"runId"`
	ParentRunID  *string         `json:"parentRunId,omitempty"`
	Name         *string         `json:"name,omitempty"`
	Timestamp    time.Time       `json:"timestamp"`
	Input        json.RawMessage `json:"input,omitempty"`
	InputBase64  string          `json:"input_base64,omitempty"` // base64 if not valid JSON
	Output       json.RawMessage `json:"output,omitempty"`
	OutputBase64 string          `json:"output_base64,omitempty"` // base64 if not valid JSON

	// Additional fields for enhanced observability
	UserID      *string        `json:"userId,omitempty"`
	TokensUsage *TokensUsage   `json:"tokensUsage,omitempty"`
	UserProps   map[string]any `json:"userProps,omitempty"`
	Extra       map[string]any `json:"extra,omitempty"`
	Metadata    map[string]any `json:"metadata,omitempty"`
	Tags        []string       `json:"tags,omitempty"`
	LogID       int64          `json:"log_id"` // Unique, monotonic event log ID
}

EventPayload represents the extended event format for external services

type EventTransformer

type EventTransformer interface {
	Transform(evt eventbus.Event) (*EventPayload, error)
}

EventTransformer transforms eventbus.Event to EventPayload

type HealthStatus

type HealthStatus struct {
	// Healthy indicates whether the dispatcher is currently healthy.
	Healthy bool `json:"healthy"`
	// Status is a human-readable string describing the current health status.
	Status string `json:"status"`
	// EventsProcessed is the total number of events processed by the dispatcher.
	EventsProcessed int64 `json:"events_processed"`
	// EventsDropped is the total number of events that were dropped and not processed.
	EventsDropped int64 `json:"events_dropped"`
	// EventsSent is the total number of events successfully sent to the backend.
	EventsSent int64 `json:"events_sent"`
	// ProcessingRate is the average number of events processed per second.
	ProcessingRate float64 `json:"processing_rate"`
	// LagCount is the number of pending messages in the event bus that have not yet been processed.
	// For Redis Streams, this is the pending entries count (XPENDING).
	LagCount int64 `json:"lag_count"`
	// StreamLength is the total number of messages currently in the Redis Stream.
	// For Redis Streams, this is the stream length (XLEN).
	StreamLength int64 `json:"stream_length"`
	// LastProcessedAt is the timestamp of the last successfully processed event.
	LastProcessedAt time.Time `json:"last_processed_at"`
	// Message provides additional information about the health status, if any.
	Message string `json:"message,omitempty"`
}

HealthStatus represents the health status of the dispatcher.

type PermanentBackendError

type PermanentBackendError struct {
	Msg string
}

PermanentBackendError is a custom error type for permanent backend errors (e.g., Helicone 500s that should not be retried)

func (*PermanentBackendError) Error

func (e *PermanentBackendError) Error() string

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service represents the event dispatcher service

func NewService

func NewService(cfg Config, logger *zap.Logger) (*Service, error)

NewService creates a new dispatcher service

func NewServiceWithBus

func NewServiceWithBus(cfg Config, logger *zap.Logger, bus eventbus.EventBus) (*Service, error)

NewServiceWithBus creates a new dispatcher service with a provided event bus.

func (*Service) DetailedStats

func (s *Service) DetailedStats() map[string]interface{}

DetailedStats returns detailed service statistics including lag and rate

func (*Service) EventBus

func (s *Service) EventBus() eventbus.EventBus

EventBus returns the event bus for this service (for connecting with other components)

func (*Service) Health

func (s *Service) Health(ctx context.Context) HealthStatus

Health returns the health status of the dispatcher

func (*Service) Run

func (s *Service) Run(ctx context.Context, detach bool) error

Run starts the dispatcher service and blocks until stopped

func (*Service) Stats

func (s *Service) Stats() (processed, dropped, sent int64)

Stats returns service statistics

func (*Service) Stop

func (s *Service) Stop() error

Stop gracefully stops the dispatcher service

type TokensUsage

type TokensUsage struct {
	Completion int `json:"completion"`
	Prompt     int `json:"prompt"`
}

TokensUsage represents token usage statistics

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL