llm

package
v1.0.0-alpha.39 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 13, 2026 License: MIT Imports: 16 Imported by: 0

README

LLM Package

OpenAI-compatible LLM client and prompt templates for graph processing.

Purpose

This package provides LLM integration for:

  • Community Summarization - Generate natural language descriptions of entity communities
  • GraphRAG Search - Answer questions using community context
  • Entity Description - Generate descriptions for individual entities

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        LLM Package                          │
├─────────────────────────────────────────────────────────────┤
│  Prompt Templates (package variables)                       │
│  ┌─────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│  │ CommunityPrompt │ │ SearchPrompt │ │   EntityPrompt   │ │
│  └─────────────────┘ └──────────────┘ └──────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│  OpenAI Client                                              │
│  - Chat completions                                         │
│  - Model listing                                            │
│  - Health checks                                            │
├─────────────────────────────────────────────────────────────┤
│  External Services (OpenAI-compatible)                      │
│  ┌─────────────┐ ┌────────────┐ ┌───────────┐              │
│  │ semshimmy   │ │ seminstruct│ │  OpenAI   │              │
│  │(inference)  │ │  (proxy)   │ │  (cloud)  │              │
│  └─────────────┘ └────────────┘ └───────────┘              │
└─────────────────────────────────────────────────────────────┘

6-Part Entity ID Awareness

Prompts understand the federated entity ID notation:

{org}.{platform}.{domain}.{system}.{type}.{instance}
Part Index Example
org 0 acme
platform 1 logistics
domain 2 environmental
system 3 sensor
type 4 temperature
instance 5 sensor-042

The LLM is taught this structure in the system prompt, enabling intelligent summarization based on domain context. For example, environmental domains emphasize monitoring scope and measurements, while content domains focus on topics and knowledge areas.

Usage

Direct Prompt Usage
// Render community summary prompt
data := llm.CommunitySummaryData{
    EntityCount:    10,
    DominantDomain: "environmental",
    Domains: []llm.DomainGroup{
        {Domain: "environmental", Count: 7, SystemTypes: []llm.SystemType{
            {Name: "sensor.temperature", Count: 4},
            {Name: "sensor.humidity", Count: 3},
        }},
    },
    Keywords: "temperature, humidity, monitoring",
}

rendered, err := llm.CommunityPrompt.Render(data)
// rendered.System = "You are an analyst..."
// rendered.User = "Summarize this community..."
OpenAI Client
// Create client
cfg := llm.OpenAIConfig{
    BaseURL: "http://seminstruct:8083/v1",
    Model:   "default",
}
client, err := llm.NewOpenAIClient(cfg)

// Chat completion
resp, err := client.ChatCompletion(ctx, llm.ChatRequest{
    SystemPrompt: rendered.System,
    UserPrompt:   rendered.User,
    MaxTokens:    150,
})

fmt.Println(resp.Content) // "This community monitors..."
Override Prompts via File
// Load custom prompts from JSON
err := llm.LoadPromptsFromFile("custom-prompts.json")

File format:

{
  "community_summary": {
    "system": "Custom system prompt...",
    "user_format": "Custom template with {{.EntityCount}}..."
  },
  "search_answer": {
    "system": "...",
    "user_format": "..."
  }
}

Prompt Templates

CommunityPrompt

Used by clustering.LLMSummarizer to generate community descriptions.

Data Structure: CommunitySummaryData

Field Type Description
EntityCount int Total entities in community
Domains []DomainGroup Grouped by domain from entity ID part[2]
DominantDomain string Most common domain or "mixed"
OrgPlatform string Common org.platform if uniform
Keywords string Comma-separated key terms
SampleEntities []EntityParts Parsed entity samples with 6-part breakdown
SearchPrompt

Used by querymanager.GlobalSearch to generate GraphRAG answers.

Data Structure: SearchAnswerData

Field Type Description
Query string User's question
Communities []CommunitySummaryInfo Relevant community summaries
Entities []EntitySample Top matching entities
EntityPrompt

Used for single entity descriptions.

Data Structure: EntityDescriptionData

Field Type Description
ID string Entity identifier
Type string Entity type
Properties []PropertyInfo Property predicates and values
Relationships []RelationshipInfo Outgoing relationships

Supported Backends

Any OpenAI-compatible API:

Backend Description Use Case
semshimmy + seminstruct Local inference stack Development, edge deployment
OpenAI Cloud API Production with API key
Ollama Local models Development, privacy
vLLM High-performance serving Production self-hosted

Configuration

JSON configuration:

{
  "llm": {
    "provider": "openai",
    "base_url": "http://seminstruct:8083/v1",
    "model": "default"
  }
}

Docker services (Tier 2):

services:
  semshimmy:
    image: ghcr.io/c360studio/semshimmy:latest
  seminstruct:
    image: ghcr.io/c360studio/seminstruct:latest
    environment:
      - SEMINSTRUCT_SHIMMY_URL=http://semshimmy:8080

Package Files

File Description
client.go LLM client interface
openai_client.go OpenAI SDK implementation
config.go Configuration types
prompts.go Prompt templates and data types
doc.go Package documentation

Package Location

processor/graph/llm/ - Part of the graph processing package family:

  • processor/graph/clustering/ - Uses LLM for community summaries
  • processor/graph/querymanager/ - Uses LLM for GraphRAG answers
  • processor/graph/embedding/ - Vector embeddings (separate from LLM)

Documentation

Overview

Package llm provides LLM client abstractions for OpenAI-compatible APIs.

This package enables semstreams to use any OpenAI-compatible LLM service (shimmy, OpenAI, Anthropic via proxy, Ollama, etc.) for:

  • Community summarization
  • Search answer generation
  • General inference tasks

The package follows the same patterns as the embedding package, using the OpenAI SDK for consistency and compatibility.

Package llm provides LLM client and prompt templates for graph processing.

Package llm provides LLM client abstractions for OpenAI-compatible APIs.

This package enables semstreams to integrate with any OpenAI-compatible LLM service for:

  • Community summarization (clustering package)
  • Search answer generation (querymanager package)
  • General inference tasks

Supported Backends

The package uses the OpenAI SDK, so it works with any compatible backend:

  • semshimmy + seminstruct (recommended for local inference)
  • OpenAI cloud
  • Ollama
  • vLLM
  • Any OpenAI-compatible API

Usage

Create a client:

cfg := llm.OpenAIConfig{
    BaseURL: "http://shimmy:8080/v1",
    Model:   "mistral-7b-instruct",
}
client, err := llm.NewOpenAIClient(cfg)

Make a chat completion request:

resp, err := client.ChatCompletion(ctx, llm.ChatRequest{
    SystemPrompt: "You are a helpful assistant.",
    UserPrompt:   "Summarize this community...",
    MaxTokens:    150,
})

Prompts

The package provides prompt templates as package variables that can be used directly or overridden via JSON file:

// Use built-in prompt
rendered, err := llm.CommunityPrompt.Render(data)

// Override via file
llm.LoadPromptsFromFile("prompts.json")

Prompts understand the 6-part federated entity ID notation:

{org}.{platform}.{domain}.{system}.{type}.{instance}

Configuration

LLM configuration is part of the graph processor config:

{
    "llm": {
        "provider": "openai",
        "base_url": "http://shimmy:8080/v1",
        "model": "mistral-7b-instruct"
    }
}

Package llm provides LLM client and prompt templates for graph processing.

Package llm provides LLM client and prompt templates for graph processing.

Index

Constants

View Source
const (
	// MaxSystemPromptLength is the maximum allowed system prompt length in characters.
	MaxSystemPromptLength = 8000

	// MaxUserPromptLength is the maximum allowed user prompt length in characters.
	MaxUserPromptLength = 32000

	// DefaultTemperature is used when Temperature is nil.
	DefaultTemperature = 0.7
)

Variables

View Source
var (
	// CommunityPrompt is the prompt template for community summarization.
	CommunityPrompt = PromptTemplate{
		System: `You are an analyst summarizing communities of related entities.

Entity IDs follow a 6-part federated notation:
  {org}.{platform}.{domain}.{system}.{type}.{instance}

Parts:
- org: Organization identifier (multi-tenancy)
- platform: Platform/product within organization
- domain: Business domain (e.g., environmental, content, logistics)
- system: System or subsystem (e.g., sensor, document, device)
- type: Entity type within system (e.g., temperature, manual, humidity)
- instance: Unique instance identifier

Generate concise summaries (1-2 sentences) that leverage this structure.
For environmental domains: emphasize monitoring scope and measurements.
For content domains: emphasize topics, themes, and knowledge areas.
For mixed domains: describe relationships between different entity types.`,

		UserFormat: `Summarize this community of {{.EntityCount}} entities:

{{if .OrgPlatform}}Organization/Platform: {{.OrgPlatform}}
{{end}}Dominant domain: {{.DominantDomain}}

Entities by domain:
{{range .Domains}}- {{.Domain}} ({{.Count}} entities):
{{range .SystemTypes}}  - {{.Name}}: {{.Count}}
{{end}}{{end}}
Key themes: {{.Keywords}}

Sample entities (parsed):
{{range .SampleEntities}}- {{.Full}}
  org={{.Org}} platform={{.Platform}} domain={{.Domain}} system={{.System}} type={{.Type}} instance={{.Instance}}
{{if .Title}}  title: {{.Title}}{{end}}
{{if .Abstract}}  description: {{.Abstract}}{{end}}
{{end}}
Generate a concise summary describing what this community represents.`,
	}

	// SearchPrompt is the prompt template for GraphRAG search answer generation.
	SearchPrompt = PromptTemplate{
		System: `You are a helpful assistant that answers questions based on entity graph context.
Use the provided community summaries and entity information to answer the user's question.
Be concise and factual. If the information is insufficient, say so.`,

		UserFormat: `Question: {{.Query}}

Relevant communities:
{{range .Communities}}- {{.Summary}} ({{.EntityCount}} entities, keywords: {{.Keywords}})
{{end}}
Top matching entities:
{{range .Entities}}- {{.ID}} ({{.Type}}){{if .Name}}: {{.Name}}{{end}}
{{if .Description}}  {{.Description}}{{end}}
{{end}}
Based on the above context, answer the question concisely.`,
	}

	// EntityPrompt is the prompt template for single entity descriptions.
	EntityPrompt = PromptTemplate{
		System: `You are a helpful assistant that describes entities in a knowledge graph.
Generate clear, informative descriptions based on the entity's properties and relationships.`,

		UserFormat: `Describe this entity:

ID: {{.ID}}
Type: {{.Type}}

Properties:
{{range .Properties}}- {{.Predicate}}: {{.Value}}
{{end}}
Relationships:
{{range .Relationships}}- {{.Predicate}} -> {{.Target}}
{{end}}
Generate a brief description of this entity.`,
	}
)

Prompt templates - package variables, directly accessible. These can be overridden via LoadPromptsFromFile.

Functions

func LoadPromptsFromFile

func LoadPromptsFromFile(path string) error

LoadPromptsFromFile overrides prompts from a JSON file. File format: {"community_summary": {...}, "search_answer": {...}, "entity_description": {...}}

Types

type ChatRequest

type ChatRequest struct {
	// SystemPrompt is the system message that sets the assistant's behavior.
	SystemPrompt string

	// UserPrompt is the user's message/question.
	UserPrompt string

	// MaxTokens limits the response length (default: 256).
	MaxTokens int

	// Temperature controls randomness (0.0-2.0, default: 0.7).
	// Use nil for default, or pointer to 0.0 for deterministic output.
	Temperature *float64
}

ChatRequest represents a chat completion request.

type ChatResponse

type ChatResponse struct {
	// Content is the generated text response.
	Content string

	// PromptTokens is the number of tokens in the prompt.
	PromptTokens int

	// CompletionTokens is the number of tokens in the response.
	CompletionTokens int

	// TotalTokens is the total tokens used.
	TotalTokens int

	// Model is the model that generated the response.
	Model string

	// FinishReason indicates why generation stopped ("stop", "length", etc.).
	FinishReason string
}

ChatResponse represents a chat completion response.

type Client

type Client interface {
	// ChatCompletion sends a chat completion request and returns the response.
	ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)

	// Model returns the model identifier being used.
	Model() string

	// Close releases any resources held by the client.
	Close() error
}

Client defines the interface for LLM operations. Implementations can connect to any OpenAI-compatible API.

type CommunitySummaryData

type CommunitySummaryData struct {
	EntityCount    int
	Domains        []DomainGroup // Grouped by domain from entity ID part[2]
	DominantDomain string        // Most common domain, or "mixed"
	OrgPlatform    string        // Common org.platform if uniform
	Keywords       string
	SampleEntities []EntityParts // Parsed entity samples
}

CommunitySummaryData is the data structure for community_summary prompts.

type CommunitySummaryInfo

type CommunitySummaryInfo struct {
	Summary     string
	EntityCount int
	Keywords    string
}

CommunitySummaryInfo contains community info for search prompts.

type Config

type Config struct {
	// Provider specifies which LLM backend to use.
	// Values: "openai" (any OpenAI-compatible API), "none" (disabled)
	Provider string `json:"provider"`

	// BaseURL is the base URL of the LLM service.
	// Examples:
	//   - "http://shimmy:8080/v1" (local shimmy)
	//   - "https://api.openai.com/v1" (OpenAI cloud)
	BaseURL string `json:"base_url"`

	// Model is the model identifier to use.
	// Examples:
	//   - "mistral-7b-instruct" (shimmy)
	//   - "gpt-4" (OpenAI)
	Model string `json:"model"`

	// APIKey for authentication (optional for local services).
	APIKey string `json:"api_key,omitempty"`

	// TimeoutSeconds for HTTP requests (default: 60).
	TimeoutSeconds int `json:"timeout_seconds,omitempty"`

	// MaxRetries for transient failures (default: 3).
	MaxRetries int `json:"max_retries,omitempty"`

	// PromptsFile is the path to a JSON file with custom prompts.
	// If not specified, uses embedded default prompts.
	PromptsFile string `json:"prompts_file,omitempty"`

	// Domain specifies which prompt domain to use (e.g., "iot", "default").
	// Allows domain-specific prompts registered by processors.
	Domain string `json:"domain,omitempty"`
}

Config holds the configuration for LLM services.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a config with sensible defaults for local development.

func (Config) GetAPIKey

func (c Config) GetAPIKey() string

GetAPIKey returns the API key from config or falls back to LLM_API_KEY env var. This allows secure configuration via environment variables in production.

func (Config) IsEnabled

func (c Config) IsEnabled() bool

IsEnabled returns true if LLM is configured and enabled.

func (Config) String

func (c Config) String() string

String returns a string representation with the API key redacted. This prevents accidental logging of sensitive credentials.

func (Config) Timeout

func (c Config) Timeout() time.Duration

Timeout returns the timeout as a time.Duration.

func (Config) ToOpenAIConfig

func (c Config) ToOpenAIConfig() OpenAIConfig

ToOpenAIConfig converts to OpenAIConfig for client creation.

type ContentFetcher

type ContentFetcher interface {
	// FetchEntityContent retrieves title/abstract for entities with StorageRefs.
	// Returns map[entityID]*EntityContent for successful fetches.
	// Missing content is not an error - returns partial results gracefully.
	// Entities without StorageRef are skipped silently.
	FetchEntityContent(ctx context.Context, entities []*gtypes.EntityState) (map[string]*EntityContent, error)
}

ContentFetcher abstracts content retrieval for LLM prompts. Implementations may use NATS request/reply, direct store access, or mocks.

type ContentFetcherOption

type ContentFetcherOption func(*NATSContentFetcher) error

ContentFetcherOption configures a NATSContentFetcher. Options return errors for validation (following natsclient pattern).

func WithAbstractFallback

func WithAbstractFallback(enabled bool, maxChars int) ContentFetcherOption

WithAbstractFallback configures abstract generation from body text. When enabled and abstract role is empty, extracts first N chars from body. Default: enabled=true, maxChars=250

func WithContentLogger

func WithContentLogger(logger *slog.Logger) ContentFetcherOption

WithContentLogger sets the logger for the content fetcher. Default: slog.Default()

func WithContentSubject

func WithContentSubject(subject string) ContentFetcherOption

WithContentSubject sets the ObjectStore API subject. Default: "storage.objectstore.api"

func WithContentTimeout

func WithContentTimeout(timeout time.Duration) ContentFetcherOption

WithContentTimeout sets the timeout for content fetch requests. Default: 2 seconds

type DomainGroup

type DomainGroup struct {
	Domain      string       // e.g., "environmental", "content"
	Count       int          // Total entities in domain
	SystemTypes []SystemType // system.type breakdown
}

DomainGroup groups entities by their domain (part[2] of entity ID).

type EntityContent

type EntityContent struct {
	Title    string // From ContentRole "title"
	Abstract string // From ContentRole "abstract" or description
}

EntityContent contains description fields fetched via NATS from ObjectStore. Used to enrich LLM prompts with actual entity titles and descriptions.

type EntityDescriptionData

type EntityDescriptionData struct {
	ID            string
	Type          string
	Properties    []PropertyInfo
	Relationships []RelationshipInfo
}

EntityDescriptionData is the data structure for entity_description prompts.

type EntityParts

type EntityParts struct {
	Full     string // Complete entity ID
	Org      string // Part 0: Organization
	Platform string // Part 1: Platform
	Domain   string // Part 2: Business domain
	System   string // Part 3: System/subsystem
	Type     string // Part 4: Entity type
	Instance string // Part 5: Instance ID
	Title    string // Entity title from content store (optional)
	Abstract string // Entity abstract from content store (optional)
}

EntityParts represents parsed 6-part entity ID components. Entity IDs follow the pattern: {org}.{platform}.{domain}.{system}.{type}.{instance}

type EntitySample

type EntitySample struct {
	ID          string
	Type        string
	Name        string
	Description string // From content store abstract (optional)
}

EntitySample represents a sample entity for search prompts.

type NATSContentFetcher

type NATSContentFetcher struct {
	// contains filtered or unexported fields
}

NATSContentFetcher implements ContentFetcher using NATS request/reply to fetch content from ObjectStore via its API subject.

func NewNATSContentFetcher

func NewNATSContentFetcher(
	natsClient *natsclient.Client,
	opts ...ContentFetcherOption,
) (*NATSContentFetcher, error)

NewNATSContentFetcher creates a new ContentFetcher that uses NATS request/reply to fetch content from ObjectStore.

func (*NATSContentFetcher) FetchEntityContent

func (f *NATSContentFetcher) FetchEntityContent(
	ctx context.Context,
	entities []*gtypes.EntityState,
) (map[string]*EntityContent, error)

FetchEntityContent retrieves title and abstract for entities with StorageRefs. Uses ObjectStore's "get" action to fetch StoredContent, then extracts title and abstract fields. This is where LLM-specific filtering happens, keeping ObjectStore generic.

Returns partial results - entities without StorageRef or fetch errors are skipped.

type OpenAIClient

type OpenAIClient struct {
	// contains filtered or unexported fields
}

OpenAIClient implements Client using the OpenAI SDK.

This implementation works with:

  • shimmy (local inference server) - recommended
  • OpenAI (cloud)
  • Any OpenAI-compatible API (Ollama, LocalAI, vLLM, etc.)

Uses the standard OpenAI SDK for consistency with the embedding package.

func NewOpenAIClient

func NewOpenAIClient(cfg OpenAIConfig) (*OpenAIClient, error)

NewOpenAIClient creates a new OpenAI-compatible LLM client.

func (*OpenAIClient) ChatCompletion

func (c *OpenAIClient) ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)

ChatCompletion sends a chat completion request to the LLM service.

func (*OpenAIClient) Close

func (c *OpenAIClient) Close() error

Close releases resources (no-op for HTTP client).

func (*OpenAIClient) Model

func (c *OpenAIClient) Model() string

Model returns the model identifier.

type OpenAIConfig

type OpenAIConfig struct {
	// BaseURL is the base URL of the LLM service.
	// Examples:
	//   - "http://shimmy:8080/v1" (shimmy local inference)
	//   - "http://localhost:8080/v1" (local development)
	//   - "https://api.openai.com/v1" (OpenAI cloud)
	BaseURL string

	// Model is the model to use for chat completions.
	// Examples:
	//   - "mistral-7b-instruct" (shimmy with Mistral)
	//   - "gpt-4" (OpenAI)
	//   - "llama2" (Ollama)
	Model string

	// APIKey for authentication (optional for local services).
	// Required for OpenAI, optional for shimmy/Ollama.
	APIKey string

	// Timeout for HTTP requests (default: 60s for LLM inference).
	Timeout time.Duration

	// MaxRetries for transient failures (default: 3).
	MaxRetries int

	// Logger for error logging (optional, defaults to slog.Default()).
	Logger *slog.Logger
}

OpenAIConfig configures the OpenAI client.

type PromptTemplate

type PromptTemplate struct {
	// System is the system message that sets assistant behavior.
	System string `json:"system"`

	// UserFormat is a Go text/template for the user message.
	UserFormat string `json:"user_format"`
}

PromptTemplate defines a reusable prompt template.

func (PromptTemplate) Render

func (p PromptTemplate) Render(data any) (*RenderedPrompt, error)

Render executes the template with the given data.

type PropertyInfo

type PropertyInfo struct {
	Predicate string
	Value     string
}

PropertyInfo represents a property for prompts.

type RelationshipInfo

type RelationshipInfo struct {
	Predicate string
	Target    string
}

RelationshipInfo represents a relationship for prompts.

type RenderedPrompt

type RenderedPrompt struct {
	System string
	User   string
}

RenderedPrompt contains the rendered system and user messages.

type SearchAnswerData

type SearchAnswerData struct {
	Query       string
	Communities []CommunitySummaryInfo
	Entities    []EntitySample
}

SearchAnswerData is the data structure for search_answer prompts.

type SystemType

type SystemType struct {
	Name  string // e.g., "sensor.temperature"
	Count int
}

SystemType represents a system.type combination count.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL