Documentation
¶
Overview ¶
Package llm provides LLM client abstractions for OpenAI-compatible APIs.
This package enables semstreams to use any OpenAI-compatible LLM service (shimmy, OpenAI, Anthropic via proxy, Ollama, etc.) for:
- Community summarization
- Search answer generation
- General inference tasks
The package follows the same patterns as the embedding package, using the OpenAI SDK for consistency and compatibility.
Package llm provides LLM client and prompt templates for graph processing.
Package llm provides LLM client abstractions for OpenAI-compatible APIs.
This package enables semstreams to integrate with any OpenAI-compatible LLM service for:
- Community summarization (clustering package)
- Search answer generation (querymanager package)
- General inference tasks
Supported Backends ¶
The package uses the OpenAI SDK, so it works with any compatible backend:
- semshimmy + seminstruct (recommended for local inference)
- OpenAI cloud
- Ollama
- vLLM
- Any OpenAI-compatible API
Usage ¶
Create a client:
cfg := llm.OpenAIConfig{
BaseURL: "http://shimmy:8080/v1",
Model: "mistral-7b-instruct",
}
client, err := llm.NewOpenAIClient(cfg)
Make a chat completion request:
resp, err := client.ChatCompletion(ctx, llm.ChatRequest{
SystemPrompt: "You are a helpful assistant.",
UserPrompt: "Summarize this community...",
MaxTokens: 150,
})
Prompts ¶
The package provides prompt templates as package variables that can be used directly or overridden via JSON file:
// Use built-in prompt
rendered, err := llm.CommunityPrompt.Render(data)
// Override via file
llm.LoadPromptsFromFile("prompts.json")
Prompts understand the 6-part federated entity ID notation:
{org}.{platform}.{domain}.{system}.{type}.{instance}
Configuration ¶
LLM configuration is part of the graph processor config:
{
"llm": {
"provider": "openai",
"base_url": "http://shimmy:8080/v1",
"model": "mistral-7b-instruct"
}
}
Package llm provides LLM client and prompt templates for graph processing.
Package llm provides LLM client and prompt templates for graph processing.
Index ¶
- Constants
- Variables
- func LoadPromptsFromFile(path string) error
- type ChatRequest
- type ChatResponse
- type Client
- type CommunitySummaryData
- type CommunitySummaryInfo
- type Config
- type ContentFetcher
- type ContentFetcherOption
- type DomainGroup
- type EntityContent
- type EntityDescriptionData
- type EntityParts
- type EntitySample
- type NATSContentFetcher
- type OpenAIClient
- type OpenAIConfig
- type PromptTemplate
- type PropertyInfo
- type RelationshipInfo
- type RenderedPrompt
- type SearchAnswerData
- type SystemType
Constants ¶
const ( // MaxSystemPromptLength is the maximum allowed system prompt length in characters. MaxSystemPromptLength = 8000 // MaxUserPromptLength is the maximum allowed user prompt length in characters. MaxUserPromptLength = 32000 // DefaultTemperature is used when Temperature is nil. DefaultTemperature = 0.7 )
Variables ¶
var ( // CommunityPrompt is the prompt template for community summarization. CommunityPrompt = PromptTemplate{ System: `You are an analyst summarizing communities of related entities. Entity IDs follow a 6-part federated notation: {org}.{platform}.{domain}.{system}.{type}.{instance} Parts: - org: Organization identifier (multi-tenancy) - platform: Platform/product within organization - domain: Business domain (e.g., environmental, content, logistics) - system: System or subsystem (e.g., sensor, document, device) - type: Entity type within system (e.g., temperature, manual, humidity) - instance: Unique instance identifier Generate concise summaries (1-2 sentences) that leverage this structure. For environmental domains: emphasize monitoring scope and measurements. For content domains: emphasize topics, themes, and knowledge areas. For mixed domains: describe relationships between different entity types.`, UserFormat: `Summarize this community of {{.EntityCount}} entities: {{if .OrgPlatform}}Organization/Platform: {{.OrgPlatform}} {{end}}Dominant domain: {{.DominantDomain}} Entities by domain: {{range .Domains}}- {{.Domain}} ({{.Count}} entities): {{range .SystemTypes}} - {{.Name}}: {{.Count}} {{end}}{{end}} Key themes: {{.Keywords}} Sample entities (parsed): {{range .SampleEntities}}- {{.Full}} org={{.Org}} platform={{.Platform}} domain={{.Domain}} system={{.System}} type={{.Type}} instance={{.Instance}} {{if .Title}} title: {{.Title}}{{end}} {{if .Abstract}} description: {{.Abstract}}{{end}} {{end}} Generate a concise summary describing what this community represents.`, } // SearchPrompt is the prompt template for GraphRAG search answer generation. SearchPrompt = PromptTemplate{ System: `You are a helpful assistant that answers questions based on entity graph context. Use the provided community summaries and entity information to answer the user's question. Be concise and factual. If the information is insufficient, say so.`, UserFormat: `Question: {{.Query}} Relevant communities: {{range .Communities}}- {{.Summary}} ({{.EntityCount}} entities, keywords: {{.Keywords}}) {{end}} Top matching entities: {{range .Entities}}- {{.ID}} ({{.Type}}){{if .Name}}: {{.Name}}{{end}} {{if .Description}} {{.Description}}{{end}} {{end}} Based on the above context, answer the question concisely.`, } // EntityPrompt is the prompt template for single entity descriptions. EntityPrompt = PromptTemplate{ System: `You are a helpful assistant that describes entities in a knowledge graph. Generate clear, informative descriptions based on the entity's properties and relationships.`, UserFormat: `Describe this entity: ID: {{.ID}} Type: {{.Type}} Properties: {{range .Properties}}- {{.Predicate}}: {{.Value}} {{end}} Relationships: {{range .Relationships}}- {{.Predicate}} -> {{.Target}} {{end}} Generate a brief description of this entity.`, } )
Prompt templates - package variables, directly accessible. These can be overridden via LoadPromptsFromFile.
Functions ¶
func LoadPromptsFromFile ¶
LoadPromptsFromFile overrides prompts from a JSON file. File format: {"community_summary": {...}, "search_answer": {...}, "entity_description": {...}}
Types ¶
type ChatRequest ¶
type ChatRequest struct {
// SystemPrompt is the system message that sets the assistant's behavior.
SystemPrompt string
// UserPrompt is the user's message/question.
UserPrompt string
// MaxTokens limits the response length (default: 256).
MaxTokens int
// Temperature controls randomness (0.0-2.0, default: 0.7).
// Use nil for default, or pointer to 0.0 for deterministic output.
Temperature *float64
}
ChatRequest represents a chat completion request.
type ChatResponse ¶
type ChatResponse struct {
// Content is the generated text response.
Content string
// PromptTokens is the number of tokens in the prompt.
PromptTokens int
// CompletionTokens is the number of tokens in the response.
CompletionTokens int
// TotalTokens is the total tokens used.
TotalTokens int
// Model is the model that generated the response.
Model string
// FinishReason indicates why generation stopped ("stop", "length", etc.).
FinishReason string
}
ChatResponse represents a chat completion response.
type Client ¶
type Client interface {
// ChatCompletion sends a chat completion request and returns the response.
ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)
// Model returns the model identifier being used.
Model() string
// Close releases any resources held by the client.
Close() error
}
Client defines the interface for LLM operations. Implementations can connect to any OpenAI-compatible API.
type CommunitySummaryData ¶
type CommunitySummaryData struct {
EntityCount int
Domains []DomainGroup // Grouped by domain from entity ID part[2]
DominantDomain string // Most common domain, or "mixed"
OrgPlatform string // Common org.platform if uniform
Keywords string
SampleEntities []EntityParts // Parsed entity samples
}
CommunitySummaryData is the data structure for community_summary prompts.
type CommunitySummaryInfo ¶
CommunitySummaryInfo contains community info for search prompts.
type Config ¶
type Config struct {
// Provider specifies which LLM backend to use.
// Values: "openai" (any OpenAI-compatible API), "none" (disabled)
Provider string `json:"provider"`
// BaseURL is the base URL of the LLM service.
// Examples:
// - "http://shimmy:8080/v1" (local shimmy)
// - "https://api.openai.com/v1" (OpenAI cloud)
BaseURL string `json:"base_url"`
// Model is the model identifier to use.
// Examples:
// - "mistral-7b-instruct" (shimmy)
// - "gpt-4" (OpenAI)
Model string `json:"model"`
// APIKey for authentication (optional for local services).
APIKey string `json:"api_key,omitempty"`
// TimeoutSeconds for HTTP requests (default: 60).
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
// MaxRetries for transient failures (default: 3).
MaxRetries int `json:"max_retries,omitempty"`
// PromptsFile is the path to a JSON file with custom prompts.
// If not specified, uses embedded default prompts.
PromptsFile string `json:"prompts_file,omitempty"`
// Domain specifies which prompt domain to use (e.g., "iot", "default").
// Allows domain-specific prompts registered by processors.
Domain string `json:"domain,omitempty"`
}
Config holds the configuration for LLM services.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns a config with sensible defaults for local development.
func (Config) GetAPIKey ¶
GetAPIKey returns the API key from config or falls back to LLM_API_KEY env var. This allows secure configuration via environment variables in production.
func (Config) String ¶
String returns a string representation with the API key redacted. This prevents accidental logging of sensitive credentials.
func (Config) ToOpenAIConfig ¶
func (c Config) ToOpenAIConfig() OpenAIConfig
ToOpenAIConfig converts to OpenAIConfig for client creation.
type ContentFetcher ¶
type ContentFetcher interface {
// FetchEntityContent retrieves title/abstract for entities with StorageRefs.
// Returns map[entityID]*EntityContent for successful fetches.
// Missing content is not an error - returns partial results gracefully.
// Entities without StorageRef are skipped silently.
FetchEntityContent(ctx context.Context, entities []*gtypes.EntityState) (map[string]*EntityContent, error)
}
ContentFetcher abstracts content retrieval for LLM prompts. Implementations may use NATS request/reply, direct store access, or mocks.
type ContentFetcherOption ¶
type ContentFetcherOption func(*NATSContentFetcher) error
ContentFetcherOption configures a NATSContentFetcher. Options return errors for validation (following natsclient pattern).
func WithAbstractFallback ¶
func WithAbstractFallback(enabled bool, maxChars int) ContentFetcherOption
WithAbstractFallback configures abstract generation from body text. When enabled and abstract role is empty, extracts first N chars from body. Default: enabled=true, maxChars=250
func WithContentLogger ¶
func WithContentLogger(logger *slog.Logger) ContentFetcherOption
WithContentLogger sets the logger for the content fetcher. Default: slog.Default()
func WithContentSubject ¶
func WithContentSubject(subject string) ContentFetcherOption
WithContentSubject sets the ObjectStore API subject. Default: "storage.objectstore.api"
func WithContentTimeout ¶
func WithContentTimeout(timeout time.Duration) ContentFetcherOption
WithContentTimeout sets the timeout for content fetch requests. Default: 2 seconds
type DomainGroup ¶
type DomainGroup struct {
Domain string // e.g., "environmental", "content"
Count int // Total entities in domain
SystemTypes []SystemType // system.type breakdown
}
DomainGroup groups entities by their domain (part[2] of entity ID).
type EntityContent ¶
type EntityContent struct {
Title string // From ContentRole "title"
Abstract string // From ContentRole "abstract" or description
}
EntityContent contains description fields fetched via NATS from ObjectStore. Used to enrich LLM prompts with actual entity titles and descriptions.
type EntityDescriptionData ¶
type EntityDescriptionData struct {
ID string
Type string
Properties []PropertyInfo
Relationships []RelationshipInfo
}
EntityDescriptionData is the data structure for entity_description prompts.
type EntityParts ¶
type EntityParts struct {
Full string // Complete entity ID
Org string // Part 0: Organization
Platform string // Part 1: Platform
Domain string // Part 2: Business domain
System string // Part 3: System/subsystem
Type string // Part 4: Entity type
Instance string // Part 5: Instance ID
Title string // Entity title from content store (optional)
Abstract string // Entity abstract from content store (optional)
}
EntityParts represents parsed 6-part entity ID components. Entity IDs follow the pattern: {org}.{platform}.{domain}.{system}.{type}.{instance}
type EntitySample ¶
type EntitySample struct {
ID string
Type string
Name string
Description string // From content store abstract (optional)
}
EntitySample represents a sample entity for search prompts.
type NATSContentFetcher ¶
type NATSContentFetcher struct {
// contains filtered or unexported fields
}
NATSContentFetcher implements ContentFetcher using NATS request/reply to fetch content from ObjectStore via its API subject.
func NewNATSContentFetcher ¶
func NewNATSContentFetcher( natsClient *natsclient.Client, opts ...ContentFetcherOption, ) (*NATSContentFetcher, error)
NewNATSContentFetcher creates a new ContentFetcher that uses NATS request/reply to fetch content from ObjectStore.
func (*NATSContentFetcher) FetchEntityContent ¶
func (f *NATSContentFetcher) FetchEntityContent( ctx context.Context, entities []*gtypes.EntityState, ) (map[string]*EntityContent, error)
FetchEntityContent retrieves title and abstract for entities with StorageRefs. Uses ObjectStore's "get" action to fetch StoredContent, then extracts title and abstract fields. This is where LLM-specific filtering happens, keeping ObjectStore generic.
Returns partial results - entities without StorageRef or fetch errors are skipped.
type OpenAIClient ¶
type OpenAIClient struct {
// contains filtered or unexported fields
}
OpenAIClient implements Client using the OpenAI SDK.
This implementation works with:
- shimmy (local inference server) - recommended
- OpenAI (cloud)
- Any OpenAI-compatible API (Ollama, LocalAI, vLLM, etc.)
Uses the standard OpenAI SDK for consistency with the embedding package.
func NewOpenAIClient ¶
func NewOpenAIClient(cfg OpenAIConfig) (*OpenAIClient, error)
NewOpenAIClient creates a new OpenAI-compatible LLM client.
func (*OpenAIClient) ChatCompletion ¶
func (c *OpenAIClient) ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)
ChatCompletion sends a chat completion request to the LLM service.
func (*OpenAIClient) Close ¶
func (c *OpenAIClient) Close() error
Close releases resources (no-op for HTTP client).
func (*OpenAIClient) Model ¶
func (c *OpenAIClient) Model() string
Model returns the model identifier.
type OpenAIConfig ¶
type OpenAIConfig struct {
// BaseURL is the base URL of the LLM service.
// Examples:
// - "http://shimmy:8080/v1" (shimmy local inference)
// - "http://localhost:8080/v1" (local development)
// - "https://api.openai.com/v1" (OpenAI cloud)
BaseURL string
// Model is the model to use for chat completions.
// Examples:
// - "mistral-7b-instruct" (shimmy with Mistral)
// - "gpt-4" (OpenAI)
// - "llama2" (Ollama)
Model string
// APIKey for authentication (optional for local services).
// Required for OpenAI, optional for shimmy/Ollama.
APIKey string
// Timeout for HTTP requests (default: 60s for LLM inference).
Timeout time.Duration
// MaxRetries for transient failures (default: 3).
MaxRetries int
// Logger for error logging (optional, defaults to slog.Default()).
Logger *slog.Logger
}
OpenAIConfig configures the OpenAI client.
type PromptTemplate ¶
type PromptTemplate struct {
// System is the system message that sets assistant behavior.
System string `json:"system"`
// UserFormat is a Go text/template for the user message.
UserFormat string `json:"user_format"`
}
PromptTemplate defines a reusable prompt template.
func (PromptTemplate) Render ¶
func (p PromptTemplate) Render(data any) (*RenderedPrompt, error)
Render executes the template with the given data.
type PropertyInfo ¶
PropertyInfo represents a property for prompts.
type RelationshipInfo ¶
RelationshipInfo represents a relationship for prompts.
type RenderedPrompt ¶
RenderedPrompt contains the rendered system and user messages.
type SearchAnswerData ¶
type SearchAnswerData struct {
Query string
Communities []CommunitySummaryInfo
Entities []EntitySample
}
SearchAnswerData is the data structure for search_answer prompts.
type SystemType ¶
SystemType represents a system.type combination count.