modelrouter

package
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 23, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

README

ModelRouter — Worker 内置 LLM 模型代理

概述

backend/internal/modelrouter 是 Worker 进程内的 LLM 模型代理层，负责注册 HTTP 端点、解析模型上游配置、转发请求、编排 SSE 流和记录调试日志。

LLM 协议转换内核已经拆分到公共包：

github.com/insmtx/Leros/backend/pkg/llmprotocol

llmprotocol 负责 OpenAI Chat Completions、OpenAI Responses、Anthropic Messages、Gemini 与统一 IR 之间的请求/响应/流事件转换。

职责边界

包 职责
backend/internal/modelrouter Worker 内部 HTTP 模型代理、上游配置存储、HTTP 转发、SSE 编排、debug log
backend/pkg/llmprotocol 协议枚举、IR、协议 adapter、能力归一化、StreamAggregator、协议 golden tests

当前文件

文件 说明
config.go 内部上游模型配置 UpstreamConfig
handler.go Gin 路由注册、请求处理、上游 HTTP 调用、SSE 流转换编排
debug.go 请求级 JSON Lines 调试日志
handler_test.go 模型代理 HTTP 行为测试

外部接口

  • DefaultStore() — 获取进程级模型配置存储单例。
  • RegisterRoutes(r gin.IRouter) — 注册 Worker 模型代理端点。
  • UpstreamConfig — Worker 内部上游模型配置，由 runtime lifecycle 写入。

调试

export LEROS_MODELROUTER_DEBUG=true
cat logs/modelrouter/<uuid>.jsonl

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ProxyBaseURL added in v0.3.0 

func ProxyBaseURL(workerAddr string) string

ProxyBaseURL returns the built-in worker model proxy BaseURL. workerAddr is the worker's listen address (e.g., ":8081" or "127.0.0.1:8081"). Requests sent to this address are transparently routed to the upstream provider according to the config registered in the worker-scoped ModelStore.

Returns empty string when workerAddr is empty.

func RegisterRoutes 

func RegisterRoutes(r gin.IRouter, store *ModelStore)

RegisterRoutes registers model routing endpoints on the given Gin router. Each endpoint supports all entry protocols and transparently converts between protocols when upstream targets a different protocol.

func SplitProxyModel added in v0.3.0 

func SplitProxyModel(model string) (realModelName, runID string)

SplitProxyModel separates the proxy model name into the real model name and run ID. "gpt-4o:run_abc" → "gpt-4o", "run_abc". Returns original model and empty runID if no colon.

Types

type BusinessKeys added in v0.3.0 

type BusinessKeys struct {
	ProjectID   uint
	SessionID   uint
	MessageID   uint
	AssistantID uint
	Uin         uint
}

BusinessKeys holds business primary key IDs for a single run.

type DebugLogger 

type DebugLogger struct {
	// contains filtered or unexported fields
}

DebugLogger 记录一次完整的模型路由请求生命周期。 未启用时所有方法均为空操作，无性能开销。

func NewDebugLogger 

func NewDebugLogger(enabled bool) *DebugLogger

NewDebugLogger 创建调试日志器。enabled 为 false 时返回空操作实例。

func (*DebugLogger) Close 

func (dl *DebugLogger) Close()

Close 关闭日志文件并输出完成信息。空操作实例安全调用。

func (*DebugLogger) LogEntryResponse 

func (dl *DebugLogger) LogEntryResponse(body []byte)

LogEntryResponse 记录转换后返回给客户端的最终响应体（非流式）。

func (*DebugLogger) LogEntryStreamChunk 

func (dl *DebugLogger) LogEntryStreamChunk(data []byte)

LogEntryStreamChunk 记录转换后的入口协议 SSE chunk。

func (*DebugLogger) LogError 

func (dl *DebugLogger) LogError(stage string, err error)

LogError 记录特定阶段的错误信息。

func (*DebugLogger) LogIRDecoded 

func (dl *DebugLogger) LogIRDecoded(ir *llmprotocol.IRRequest)

LogIRDecoded 记录从入口协议解码后的 IR（第 3 步）。

func (*DebugLogger) LogIRNormalized 

func (dl *DebugLogger) LogIRNormalized(ir *llmprotocol.IRRequest)

LogIRNormalized 记录经过能力裁剪后的归一化 IR（第 4 步）。

func (*DebugLogger) LogOriginalRequest 

func (dl *DebugLogger) LogOriginalRequest(body []byte)

LogOriginalRequest 记录原始请求体（第 1 步：客户端发送的原始 JSON）。

func (*DebugLogger) LogRequestMeta 

func (dl *DebugLogger) LogRequestMeta(entryProtocol, upstreamProtocol llmprotocol.Protocol, model string, stream bool)

LogRequestMeta 记录请求元信息（入口协议、上游协议、模型名、是否流式）。

func (*DebugLogger) LogStreamChunkSeparator 

func (dl *DebugLogger) LogStreamChunkSeparator()

LogStreamChunkSeparator 写入空行，在 chunk 组之间形成视觉分隔。

func (*DebugLogger) LogUpstreamErrorResponse 

func (dl *DebugLogger) LogUpstreamErrorResponse(body []byte)

LogUpstreamErrorResponse 记录上游返回的错误响应体（非流式）。

func (*DebugLogger) LogUpstreamRequest 

func (dl *DebugLogger) LogUpstreamRequest(body []byte)

LogUpstreamRequest 记录发送给上游的请求体（第 5 步：转换后的上游协议 JSON）。

func (*DebugLogger) LogUpstreamResponse 

func (dl *DebugLogger) LogUpstreamResponse(body []byte)

LogUpstreamResponse 记录上游返回的原始响应体（非流式）。

func (*DebugLogger) LogUpstreamStreamChunk 

func (dl *DebugLogger) LogUpstreamStreamChunk(data []byte)

LogUpstreamStreamChunk 记录上游原始 SSE chunk。

type InvokeOption added in v0.3.0 

type InvokeOption func(*invokeOptions)

InvokeOption 配置 Call() 的行为选项。

func WithModelCode added in v0.3.0 

func WithModelCode(code string) InvokeOption

WithModelCode 按 llm_models.code 指定要使用的模型。 不调用此选项时 Call() 使用组织默认模型。

func WithModelID added in v0.3.0 

func WithModelID(id uint) InvokeOption

WithModelID 按 llm_models.id 指定要使用的模型。

type Invoker added in v0.3.0 

type Invoker interface {
	Call(ctx context.Context, orgID uint, req *llm.CallRequest, opts ...InvokeOption) (*llm.CallResult, error)
}

Invoker 定义进程内 LLM 调用入口，供 service 层依赖。 实现方通过 llm.Manager 从 DB 解析模型配置，通过 llm.Caller 发起 HTTP 调用。

type ModelRouter added in v0.3.0 

type ModelRouter struct {
	// contains filtered or unexported fields
}

ModelRouter 是 Invoker 的默认实现，持有进程级不变的 Manager 和 Caller。

func NewModelRouter added in v0.3.0 

func NewModelRouter(manager llm.Manager, caller llm.Caller) *ModelRouter

NewModelRouter creates a ModelRouter with the given manager and caller.

func (*ModelRouter) Call added in v0.3.0 

func (r *ModelRouter) Call(ctx context.Context, orgID uint, req *llm.CallRequest, opts ...InvokeOption) (*llm.CallResult, error)

Call 执行一次进程内结构化 LLM 调用，使用选项模式选择模型。 优先级：WithModelID > WithModelCode > 组织默认模型。

type ModelStore 

type ModelStore struct {
	// contains filtered or unexported fields
}

ModelStore holds UpstreamConfig and business ID entries keyed by model name. It is safe for concurrent use.

func NewModelStore added in v0.1.17 

func NewModelStore() *ModelStore

NewModelStore creates an isolated model routing store.

func (*ModelStore) GetBiz added in v0.3.0 

func (s *ModelStore) GetBiz(modelKey string) *BusinessKeys

GetBiz returns business identifiers for the given proxy model key, or nil.

func (*ModelStore) Put 

func (s *ModelStore) Put(cfg UpstreamConfig)

Put registers an upstream configuration for a model. If cfg.Protocol is not explicitly set and cfg.Provider is non-empty, the protocol is inferred from the provider.

func (*ModelStore) PutBiz added in v0.3.0 

func (s *ModelStore) PutBiz(modelKey string, biz BusinessKeys)

PutBiz stores business identifiers keyed by proxy model name (modelName:runID).

func (*ModelStore) RemoveBiz added in v0.3.0 

func (s *ModelStore) RemoveBiz(modelKey string)

RemoveBiz removes business identifiers for the given proxy model key.

func (*ModelStore) Resolve 

func (s *ModelStore) Resolve(model string) (*UpstreamConfig, error)

Resolve returns the UpstreamConfig for the given model name.

func (*ModelStore) SetCaller added in v0.3.0 

func (s *ModelStore) SetCaller(caller llm.Caller)

SetCaller sets the llm.Caller used for upstream LLM calls.

func (*ModelStore) SetOrgID added in v0.3.0 

func (s *ModelStore) SetOrgID(orgID uint)

SetOrgID sets the org ID used for call recording.

type UpstreamConfig 

type UpstreamConfig struct {
	ModelID      uint
	ModelName    string
	Provider     string
	BaseURL      string
	BaseURLHasV1 bool
	APIKey       string
	Protocol     llmprotocol.Protocol
	MaxTokens    int
	Temperature  float64
	TimeoutSec   int
}

func (UpstreamConfig) ToModelConfig added in v0.3.0 

func (c UpstreamConfig) ToModelConfig() *llm.ModelConfig

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.