Documentation
¶
Overview ¶
Package tools provides the interface-based tool system for the Sprout AI agent.
Tools are capabilities the LLM can invoke — reading files, executing shell commands, searching code, delegating to subagents, automating browsers, and more. Each tool implements the ToolHandler interface and is registered with the ToolRegistry.
ToolHandler interface ¶
The ToolHandler interface replaced the legacy `type ToolHandler func(ctx, args, agent) (images, output, error)` func type. The old func-based system tightly coupled every tool to the *Agent type, making tools hard to test in isolation and difficult to share across different execution contexts. The new interface-based system provides explicit dependencies through ToolEnv, enabling clean separation of concerns:
type ToolHandler interface {
Name() string
Definition() ToolDefinition
Validate(args map[string]any) error
Execute(ctx context.Context, env ToolEnv, args map[string]any) (ToolResult, error)
}
Adding a new tool ¶
1. Create a new file in this package (e.g., `my_tool_handler.go`). 2. Define a struct and implement the four ToolHandler methods. 3. Register it in `AllTools()` in `all.go`.
See AGENTS.md for tool documentation and conventions.
Migration from legacy func-style handlers ¶
The legacy tool system used function types directly coupled to *Agent. The new interface-based system decouples tools from the agent via ToolEnv, which provides explicit dependencies (EventBus, WorkspaceRoot, OutputWriter, etc.).
During the migration period, a dual-dispatch shim in pkg/agent/tool_definitions.go bridges both systems: when ExecuteTool() is called, it first checks the new registry via tools.GetNewToolRegistry().Lookup(name). If a handler is found there, it builds a ToolEnv from the agent context and dispatches through the new interface. If no handler exists in the new registry, it falls back to the legacy func-style handlers. This allows incremental migration without breaking existing functionality.
Some current tools (e.g., browseURLHandler, runSubagentHandler) are thin wrappers around legacy agent methods, pending full refactoring. These are marked with comments in all.go.
Shell command and file path security classifier.
This module provides string-based heuristics for classifying tool calls by risk level (SAFE, CAUTION, DANGEROUS). It is designed as a lightweight defense-in-depth layer that operates on raw command strings and path arguments WITHOUT accessing the filesystem.
Important Limitations ¶
This classifier intentionally performs NO filesystem operations (no stat, no resolve, no symlink following). This keeps it fast and concurrency-safe, but means:
- Symlink attacks are not detected. For example, "rm -rf build/" is classified as safe even if "build" is a symlink to "/etc" or "$HOME".
- Relative path traversal is not resolved. "rm -rf ../important-project" bypasses all safe-directory checks because the classifier only matches the first path component literally (".." has no special meaning here).
- Path normalization is not performed. Multiple slashes ("//"), "." segments, and case variations on case-insensitive filesystems are not normalized.
- Environment variable expansion, glob expansion, and shell aliases are not considered. "rm -rf $BUILD_DIR" is classified as CAUTION (command substitution), not DANGEROUS, because the classifier cannot resolve the variable.
- The classifier is prefix-based, not semantic. "rm -rf node_modules-new" is safe because it matches "rm -rf node_modules " prefix, even though the actual target is a different directory.
These limitations are acceptable because the classifier's purpose is gate-keeping for LLM-initiated operations in a workspace context — NOT a security boundary. Actual enforcement (filesystem permissions, user approval, interactive confirmation) should be handled by separate layers.
Package tools provides platform-specific JSON encoding for native builds.
Index ¶
- Constants
- Variables
- func AnalyzeImage(imagePath string, analysisPrompt string, analysisMode string) (string, error)
- func AskUser(question string) (string, error)
- func AskUserWithEventBus(ctx context.Context, question string, eventBus *events.EventBus, ...) (string, error)
- func CheckBackgroundOutput(ctx context.Context, sessionID string) (string, error)
- func CheckPDFPython3Available() error
- func ClearLastVisionUsage()
- func CreateOllamaClient(model string) (api.ClientInterface, error)
- func CreateVisionClient() (api.ClientInterface, error)
- func CreateVisionClientWithModel(modelName string) (api.ClientInterface, error)
- func CreateVisionClientWithProvider(providerType api.ClientType) (api.ClientInterface, error)
- func DefaultTaskQueuePath() string
- func EditFile(ctx context.Context, filePath, oldString, newString string) (string, error)
- func EnsureOllamaModelTag(model string) string
- func ExecuteGitOperation(ctx context.Context, op GitOperation, sessionID string, ...) (string, error)
- func ExecuteShellCommand(ctx context.Context, command string) (string, error)
- func ExecuteShellCommandBackground(ctx context.Context, command string, sessionID string) (string, error)
- func ExecuteShellCommandWithSafety(ctx context.Context, command string, interactiveMode bool, sessionID string, ...) (string, error)
- func FetchURL(url string, cfg *configuration.Manager) (string, error)
- func FormatTodoPriorityError(priority string) string
- func FormatTodoStatusError(status string) string
- func GeneratePromptForMode(mode string) string
- func GenerateRepoMap(ctx context.Context, rootDir string) (string, error)
- func GetBaseName(path string) string
- func GetCustomProviderConfig(providerType api.ClientType) (configuration.CustomProviderConfig, bool)
- func GetCustomVisionFallback(providerType api.ClientType) (api.ClientType, string, bool)
- func GetCustomVisionProviders() []api.ClientType
- func GetDefaultModelForProvider(providerType api.ClientType) string
- func GetFileExtension(path string) string
- func GetOCRPrompt() string
- func GetPDFOCRPrompt() string
- func GetPDFPythonExecutable() (string, error)
- func GetUIElementPrompt() string
- func GetVisionCacheStats() map[string]interface{}
- func GetVisionModelForProvider(providerType api.ClientType) string
- func HasVisionCapability() bool
- func IsFileDeletionCommand(command string) bool
- func IsHTMLInput(path string) bool
- func IsValidPriority(priority string) bool
- func IsValidStatus(status string) bool
- func NormalizeTodoID(id interface{}) string
- func OptimizeImageData(imagePath string, data []byte) ([]byte, string, error)
- func ProcessPDFForTextOnly(pdfPath string) (string, error)
- func ProcessPDFWithVision(pdfPath string) (string, error)
- func PromptForGitApprovalStdin(command string) (bool, error)
- func ReadFile(ctx context.Context, filePath string) (string, error)
- func ReadFileWithRange(ctx context.Context, filePath string, startLine, endLine int) (string, error)
- func ResolvePDFInputPath(inputPath string) (string, func(), error)
- func SetAuditLogger(l *AuditLogger)
- func SetGlobalAskUserManager(mgr *AskUserManager)deprecated
- func SimplePDFInfo(pdfPath string) (map[string]interface{}, error)
- func TodoWrite(todos []TodoItem) string
- func ValidTodoPriorityList() []string
- func ValidTodoStatuses() []string
- func ValidateGitArgs(args string) error
- func WebSearch(query string, cfg *configuration.Manager) (string, error)
- func WithBackgroundProcessManager(ctx context.Context, bpm *BackgroundProcessManager) context.Context
- func WithTerminalManager(ctx context.Context, tm TerminalAccess) context.Context
- func WriteFile(ctx context.Context, filePath, content string) (string, error)
- type ApprovalManager
- type ApprovalResult
- type AskUserManager
- type AuditEntry
- type AuditLogger
- type BackgroundProcess
- type BackgroundProcessManager
- func (m *BackgroundProcessManager) AdoptProcess(cmd *exec.Cmd, outputPath string, command string, dir string, ...) (string, error)
- func (m *BackgroundProcessManager) CheckOutput(sessionID string) (string, string, error)
- func (m *BackgroundProcessManager) Close()
- func (m *BackgroundProcessManager) IsActive(sessionID string) bool
- func (m *BackgroundProcessManager) SessionIDs() []string
- func (m *BackgroundProcessManager) Start(ctx context.Context, command string, dir string) (string, error)
- func (m *BackgroundProcessManager) Stop(sessionID string) error
- func (m *BackgroundProcessManager) StopAll()
- type BinaryFetchResult
- type GitApprovalPrompter
- type GitCommitFlowExecutor
- type GitOperation
- type GitOperationType
- type ImageAnalysisResponse
- type ImageAnalysisSupported
- type ImageData
- type PDFPipelineResult
- type ParameterDef
- type ResponseKind
- type RiskCategory
- type RollbackResult
- type SecurityResult
- type SecurityRisk
- type SubtaskInput
- type Task
- type TaskQueue
- func (tq *TaskQueue) AddTask(title, description, priority, workingDir, persona string) (*Task, error)
- func (tq *TaskQueue) Load() error
- func (tq *TaskQueue) PublishTask(taskID, status, result string, subtasks []SubtaskInput) ([]Task, error)
- func (tq *TaskQueue) ReadTasks(status string, limit int) ([]Task, error)
- func (tq *TaskQueue) Save() error
- type TerminalAccess
- type TodoItem
- type TodoManager
- type ToolDefinition
- type ToolEnv
- type ToolHandler
- type ToolRegistry
- func (r *ToolRegistry) All() map[string]ToolHandler
- func (r *ToolRegistry) ForPersona(persona string) map[string]ToolHandler
- func (r *ToolRegistry) Lookup(name string) (ToolHandler, bool)
- func (r *ToolRegistry) Names() []string
- func (r *ToolRegistry) Register(handler ToolHandler) error
- func (r *ToolRegistry) Unregister(name string) bool
- type ToolResult
- type UIElement
- type ViewHistoryResult
- type VisionAnalysis
- type VisionProcessor
- func (vp *VisionProcessor) AnalyzeImage(imagePath string, optionalPrompt ...string) (VisionAnalysis, error)
- func (vp *VisionProcessor) CreateVisionPrompt(imagePath string) string
- func (vp *VisionProcessor) DownloadImage(url string) ([]byte, error)
- func (vp *VisionProcessor) EnhanceTextWithAnalysis(text, imagePath string, analysis VisionAnalysis) string
- func (vp *VisionProcessor) ExtractPosition(line string) string
- func (vp *VisionProcessor) ExtractUIElements(description string) []UIElement
- func (vp *VisionProcessor) GetImageData(imagePath string) (string, string, error)
- func (vp *VisionProcessor) LooksLikeUI(description string) bool
- func (vp *VisionProcessor) ParseUIElementFromLine(line string) UIElement
- func (vp *VisionProcessor) ProcessImagesInText(text string) (string, []VisionAnalysis, error)
- func (vp *VisionProcessor) ProcessPDFForVision(pdfPath string) (VisionAnalysis, error)
- type VisionUsageInfo
Constants ¶
const ( // MaxNumberTodosToShowFull defines maximum todos to display fully in summaries MaxNumberTodosToShowFull = 3 // TidyMaxTodos is the maximum to show when many todos exist TidyMaxTodos = 2 )
const ( ErrCodeInputUnsupported = "INPUT_UNSUPPORTED_TYPE" ErrCodeRemoteFetchFailed = "REMOTE_FETCH_FAILED" ErrCodeLocalFileNotFound = "LOCAL_FILE_NOT_FOUND" ErrCodeOCRNoTextDetected = "OCR_NO_TEXT_DETECTED" ErrCodePDFProcessingFailed = "PDF_PROCESSING_FAILED" ErrCodeVisionNotAvailable = "VISION_NOT_AVAILABLE" ErrCodeVisionRequestFailed = "VISION_REQUEST_FAILED" ErrCodeInvalidResponse = "INVALID_RESPONSE" // Special error for model download needed ErrCodeModelDownloadNeeded = "MODEL_DOWNLOAD_NEEDED" ErrModelDownloadNeeded = "PDF_OCR_MODEL_NEEDS_DOWNLOAD:" )
Error codes for analyze_image_content tool
const DefaultAskUserTimeout = 10 * time.Minute
Variables ¶
ValidTodoPriorities contains all allowable todo priority values
var ValidTodos = map[string]bool{ "pending": true, "in_progress": true, "completed": true, "cancelled": true, }
ValidTodos contains all allowable todo status values
Functions ¶
func AnalyzeImage ¶
AnalyzeImage is the tool function called by the agent for image analysis Returns a structured JSON response with metadata for robust error handling
func AskUser ¶
AskUser prompts the user with a question and reads input from stdin. This is the legacy CLI-only implementation kept for backward compatibility.
func AskUserWithEventBus ¶
func AskUserWithEventBus(ctx context.Context, question string, eventBus *events.EventBus, clientID, userID, chatID string, mgr *AskUserManager) (string, error)
AskUserWithEventBus prompts the user with a question using the event bus for WebUI mode, falling back to stdin for CLI mode.
func CheckBackgroundOutput ¶
CheckBackgroundOutput retrieves accumulated output for a background session. Returns JSON with session_id, status, and output fields. Works in WebUI mode (TerminalManager) and CLI mode (BackgroundProcessManager).
func CheckPDFPython3Available ¶
func CheckPDFPython3Available() error
CheckPDFPython3Available validates that a compatible Python runtime is available for PDF processing.
func ClearLastVisionUsage ¶
func ClearLastVisionUsage()
ClearLastVisionUsage clears the stored vision usage information
func CreateOllamaClient ¶
func CreateOllamaClient(model string) (api.ClientInterface, error)
CreateOllamaClient creates an Ollama client with the specified model
func CreateVisionClient ¶
func CreateVisionClient() (api.ClientInterface, error)
CreateVisionClient creates a client capable of vision analysis
func CreateVisionClientWithModel ¶
func CreateVisionClientWithModel(modelName string) (api.ClientInterface, error)
CreateVisionClientWithModel creates a vision client using a specific model
func CreateVisionClientWithProvider ¶
func CreateVisionClientWithProvider(providerType api.ClientType) (api.ClientInterface, error)
CreateVisionClientWithProvider creates a vision client using the specified provider
func DefaultTaskQueuePath ¶
func DefaultTaskQueuePath() string
DefaultTaskQueuePath returns the default path for the task queue file.
func EnsureOllamaModelTag ¶
EnsureOllamaModelTag ensures the model has a tag suffix
func ExecuteGitOperation ¶
func ExecuteGitOperation(ctx context.Context, op GitOperation, sessionID string, commitFlowExecutor GitCommitFlowExecutor, approvalPrompter GitApprovalPrompter) (string, error)
ExecuteGitOperation executes a git operation with approval (all git operations require approval)
func ExecuteShellCommand ¶
ExecuteShellCommand executes a shell command with safety checks
func ExecuteShellCommandBackground ¶
func ExecuteShellCommandBackground(ctx context.Context, command string, sessionID string) (string, error)
ExecuteShellCommandBackground runs a command in a background hidden PTY session and returns a JSON result with the session ID. Works in WebUI mode (TerminalManager) and CLI mode (BackgroundProcessManager). This is for commands that should run asynchronously without waiting for completion.
func ExecuteShellCommandWithSafety ¶
func ExecuteShellCommandWithSafety(ctx context.Context, command string, interactiveMode bool, sessionID string, streamOutput bool) (string, error)
ExecuteShellCommandWithSafety executes a shell command with configurable safety checks. The streamOutput parameter controls whether output streams to terminal in real-time (true) or is captured silently (false, for LLM tool calls).
Native builds use os/exec; the js/wasm build routes through pkg/wasmshell. The platform-specific implementation lives in shell_native.go / shell_js.go.
func FetchURL ¶
func FetchURL(url string, cfg *configuration.Manager) (string, error)
FetchURL fetches content from a specific URL using the webcontent fetcher. This provides direct URL access as an agent tool.
func FormatTodoPriorityError ¶
FormatTodoPriorityError returns a standardized error message for invalid priority values.
func FormatTodoStatusError ¶
FormatTodoStatusError returns a standardized error message for invalid status values.
func GeneratePromptForMode ¶
GeneratePromptForMode creates appropriate prompts based on analysis mode
func GenerateRepoMap ¶
GenerateRepoMap walks the directory tree rooted at rootDir and produces a lightweight overview of the codebase showing file paths and top-level symbols. For Go files it uses go/ast; for TS/JS/Python it uses tree-sitter via pkg/ast. Output is truncated to ~1024 tokens.
func GetBaseName ¶
GetBaseName returns the base name of a file path
func GetCustomProviderConfig ¶
func GetCustomProviderConfig(providerType api.ClientType) (configuration.CustomProviderConfig, bool)
GetCustomProviderConfig returns the custom provider configuration for a given type
func GetCustomVisionFallback ¶
func GetCustomVisionFallback(providerType api.ClientType) (api.ClientType, string, bool)
GetCustomVisionFallback returns the fallback provider and model for vision
func GetCustomVisionProviders ¶
func GetCustomVisionProviders() []api.ClientType
GetCustomVisionProviders returns a list of custom providers that support vision
func GetDefaultModelForProvider ¶
func GetDefaultModelForProvider(providerType api.ClientType) string
GetDefaultModelForProvider returns the default model for a given provider type
func GetFileExtension ¶
GetFileExtension returns the file extension (with dot) in lowercase
func GetOCRPrompt ¶
func GetOCRPrompt() string
GetOCRPrompt returns a prompt for OCR text extraction
func GetPDFPythonExecutable ¶
GetPDFPythonExecutable ensures a consistent per-user Python environment for PDF extraction.
func GetUIElementPrompt ¶
func GetUIElementPrompt() string
GetUIElementPrompt returns a prompt for extracting UI elements
func GetVisionCacheStats ¶
func GetVisionCacheStats() map[string]interface{}
GetVisionCacheStats returns statistics about vision result caching
func GetVisionModelForProvider ¶
func GetVisionModelForProvider(providerType api.ClientType) string
GetVisionModelForProvider returns the appropriate vision model for a given provider Vision models are configured in the provider JSON config files in pkg/agent_providers/configs/ This function creates a temporary provider client to get the configured vision model
func HasVisionCapability ¶
func HasVisionCapability() bool
HasVisionCapability checks if vision processing is available
func IsFileDeletionCommand ¶
IsFileDeletionCommand checks if a command will delete files This is used for change tracking (not security validation) Security validation is handled by the static classifier in security_classifier.go
func IsHTMLInput ¶
IsHTMLInput checks if the input path appears to be HTML content. For URLs, it does a HEAD request to check Content-Type. For local files, it checks the file extension.
func IsValidPriority ¶
IsValidPriority checks if the given priority string is valid. Empty string is accepted (priority is optional).
func IsValidStatus ¶
IsValidStatus checks if the given status string is valid.
func NormalizeTodoID ¶
func NormalizeTodoID(id interface{}) string
NormalizeTodoID converts various ID formats to the internal "todo_X" format. Accepted inputs:
- string: "todo_1" -> "todo_1", "1" -> "todo_1"
- float64: 1.0 -> "todo_1"
- int: 1 -> "todo_1"
Returns empty string for unsupported types.
func OptimizeImageData ¶
func ProcessPDFForTextOnly ¶
ProcessPDFForTextOnly processes a PDF to extract text content. First tries pypdf text extraction (no vision client required). If text extraction fails and a vision client is available, falls back to OCR.
func ProcessPDFWithVision ¶
ProcessPDFWithVision processes a PDF file using Ollama with glm-ocr model
func PromptForGitApprovalStdin ¶
PromptForGitApprovalStdin prompts for git approval using stdin. Fires mid-turn during git tool execution, so it pauses the SteerInputReader to release stdin back to cooked mode (otherwise the bufio.Reader hits EOF immediately while steer holds the raw- mode fd).
func ReadFileWithRange ¶
func ResolvePDFInputPath ¶
The caller must invoke the returned cleanup function when done with the resolved path.
func SetAuditLogger ¶
func SetAuditLogger(l *AuditLogger)
SetAuditLogger sets the package-level audit logger for recording security decisions. Must be called during initialization before concurrent goroutines begin calling ClassifyToolCall.
func SetGlobalAskUserManager
deprecated
func SetGlobalAskUserManager(mgr *AskUserManager)
SetGlobalAskUserManager sets the global singleton (called by webui setup).
Deprecated: use dependency injection via Agent.InjectWebUIManagers instead.
func SimplePDFInfo ¶
SimplePDFInfo returns basic info about PDF file
func ValidTodoPriorityList ¶
func ValidTodoPriorityList() []string
ValidTodoPriorityList returns a slice of all valid priority values for error messages
func ValidTodoStatuses ¶
func ValidTodoStatuses() []string
ValidTodoStatuses returns a slice of all valid status values for error messages
func ValidateGitArgs ¶
ValidateGitArgs validates that the provided git arguments string does not contain any dangerous flags or patterns.
It uses a combination of matching strategies:
- Field prefix matching: splits args into whitespace-delimited tokens and checks if any token starts with a blocklisted prefix. Catches abbreviations.
- Substring matching: for multi-token patterns like "-c core.", checks containment in the full args string.
Returns nil if all arguments are safe, or an error describing which flag was blocked and why.
func WebSearch ¶
func WebSearch(query string, cfg *configuration.Manager) (string, error)
WebSearch performs a web search and returns raw search results. The agent can then decide which URLs to fetch and process.
func WithBackgroundProcessManager ¶
func WithBackgroundProcessManager(ctx context.Context, bpm *BackgroundProcessManager) context.Context
WithBackgroundProcessManager returns a new context that carries the BackgroundProcessManager. Use BackgroundProcessManagerFromContext to retrieve it.
func WithTerminalManager ¶
func WithTerminalManager(ctx context.Context, tm TerminalAccess) context.Context
WithTerminalManager returns a new context that carries the TerminalAccess. Use TerminalManagerFromContext to retrieve it.
Types ¶
type ApprovalManager ¶
type ApprovalManager interface {
// RequestApproval asks the user to approve a tool execution.
// Returns an ApprovalResult with the outcome and optional context.
RequestApproval(requestID, toolName, riskLevel, prompt string, extras map[string]string) ApprovalResult
}
ApprovalManager handles security approval requests for tool execution.
type ApprovalResult ¶
type ApprovalResult struct {
Approved bool `json:"approved"`
Reason string `json:"reason,omitempty"` // "rejected", "timed_out", "cancelled"
UserComment string `json:"user_comment,omitempty"` // Optional feedback from user
}
ApprovalResult contains the outcome of an approval request.
type AskUserManager ¶
type AskUserManager struct {
// contains filtered or unexported fields
}
AskUserManager coordinates ask_user requests between the agent and the webui. It follows the same pattern as security.ApprovalManager but returns string responses instead of bool.
func GetGlobalAskUserManager
deprecated
func GetGlobalAskUserManager() *AskUserManager
GetGlobalAskUserManager returns the global singleton.
Deprecated: use dependency injection via Agent.InjectWebUIManagers instead.
func NewAskUserManager ¶
func NewAskUserManager() *AskUserManager
NewAskUserManager creates a new AskUserManager with the default timeout.
func (*AskUserManager) RequestAskUser ¶
func (m *AskUserManager) RequestAskUser(ctx context.Context, eventBus *events.EventBus, question, clientID, userID, chatID string) (string, error)
RequestAskUser publishes an ask_user_request event and blocks until the webui responds, a timeout elapses, the context is cancelled, or the event bus is nil. Returns the user's text response.
func (*AskUserManager) RespondToAskUser ¶
func (m *AskUserManager) RespondToAskUser(requestID string, response string) bool
RespondToAskUser resolves a pending ask_user request with the user's text response. Returns true if the request existed and was responded to, false otherwise.
func (*AskUserManager) SetTimeout ¶
func (m *AskUserManager) SetTimeout(d time.Duration)
SetTimeout sets the maximum duration requests will block. A zero or negative value resets to the default.
type AuditEntry ¶
type AuditEntry struct {
Timestamp time.Time `json:"timestamp"`
Tool string `json:"tool"`
Args string `json:"args,omitempty"`
RiskLevel string `json:"risk_level"`
Category string `json:"category"`
Action string `json:"action"` // "allowed", "denied", "prompted"
Reasoning string `json:"reasoning,omitempty"`
Source string `json:"source,omitempty"` // "classifier", "policy", "user_override"
SessionID string `json:"session_id,omitempty"`
Workspace string `json:"workspace,omitempty"`
}
AuditEntry represents a single security audit log entry.
type AuditLogger ¶
type AuditLogger struct {
// contains filtered or unexported fields
}
AuditLogger provides thread-safe JSONL audit logging for security decisions.
func NewAuditLogger ¶
func NewAuditLogger(logPath string) (*AuditLogger, error)
NewAuditLogger creates or opens a log file at the given path, automatically creating parent directories as needed.
func (*AuditLogger) Close ¶
func (l *AuditLogger) Close() error
Close closes the underlying log file.
func (*AuditLogger) Log ¶
func (l *AuditLogger) Log(entry AuditEntry) error
Log marshals the entry to JSON and appends it as a single line (JSONL/NDJSON format) followed by a newline.
func (*AuditLogger) LogEntry ¶
func (l *AuditLogger) LogEntry(entry AuditEntry) error
LogEntry is an alias for Log, named for call-site clarity. Nil-receiver safe via Log's internal nil guard.
type BackgroundProcess ¶
type BackgroundProcess struct {
ID string // "bg-<sanitized-prefix>-<random-hex>"
Cmd *exec.Cmd // the running process (nil after exit)
Process *os.Process
OutputPath string // path to accumulated output temp file
Dir string // working directory
Command string // original command string
StartedAt time.Time
LastPolled time.Time
// contains filtered or unexported fields
}
BackgroundProcess represents a tracked background process for CLI mode. Unlike WebUI background sessions (PTY-based), these use os/exec with output piped to a temp file for polling via check_background.
type BackgroundProcessManager ¶
type BackgroundProcessManager struct {
// contains filtered or unexported fields
}
BackgroundProcessManager manages background processes for CLI mode. Provides the same lifecycle as the WebUI's TerminalManager background sessions but without PTY support.
func BackgroundProcessManagerFromContext ¶
func BackgroundProcessManagerFromContext(ctx context.Context) *BackgroundProcessManager
BackgroundProcessManagerFromContext extracts the BackgroundProcessManager from the context. Returns nil if no manager is available.
func NewBackgroundProcessManager ¶
func NewBackgroundProcessManager() *BackgroundProcessManager
NewBackgroundProcessManager creates a new BackgroundProcessManager and starts the cleanup goroutine.
func (*BackgroundProcessManager) AdoptProcess ¶
func (m *BackgroundProcessManager) AdoptProcess(cmd *exec.Cmd, outputPath string, command string, dir string, waitCh <-chan error) (string, error)
AdoptProcess takes an already-started exec.Cmd (from timeout promotion) and registers it into the background process manager. The output file is already created by the caller.
If waitCh is non-nil, AdoptProcess assumes the caller has already started a goroutine calling cmd.Wait() and reads its result from waitCh instead of calling cmd.Wait() itself. Calling cmd.Wait() concurrently from two goroutines on the same exec.Cmd is undefined behavior and trips the race detector. The shell-promotion path uses this to hand off its existing Wait goroutine. Callers that haven't yet started a Wait (e.g. tests) pass nil and AdoptProcess starts one internally.
func (*BackgroundProcessManager) CheckOutput ¶
func (m *BackgroundProcessManager) CheckOutput(sessionID string) (string, string, error)
CheckOutput reads accumulated output from a background session. Returns the raw output string, status ("running" or "exited"), and any error.
func (*BackgroundProcessManager) Close ¶
func (m *BackgroundProcessManager) Close()
Close stops the cleanup goroutine and terminates all background processes.
func (*BackgroundProcessManager) IsActive ¶
func (m *BackgroundProcessManager) IsActive(sessionID string) bool
IsActive checks whether a session is still running.
func (*BackgroundProcessManager) SessionIDs ¶
func (m *BackgroundProcessManager) SessionIDs() []string
SessionIDs returns all tracked session IDs.
func (*BackgroundProcessManager) Start ¶
func (m *BackgroundProcessManager) Start(ctx context.Context, command string, dir string) (string, error)
Start creates a new background process, pipes its output to a temp file, and returns a session ID for later polling.
func (*BackgroundProcessManager) Stop ¶
func (m *BackgroundProcessManager) Stop(sessionID string) error
Stop terminates a background session by sending SIGINT, waiting briefly, then SIGKILL if still running. Cleans up the process tracking.
func (*BackgroundProcessManager) StopAll ¶
func (m *BackgroundProcessManager) StopAll()
StopAll terminates all managed background processes.
type BinaryFetchResult ¶
type BinaryFetchResult struct {
Images []api.ImageData // Populated for image URLs (and scanned PDFs)
Text string // Populated for text-based PDFs
Source string // Description of how content was obtained
EffectiveURL string // Post-redirect URL (differs from input if redirected)
}
BinaryFetchResult holds the result of fetching binary content from a URL. Exactly one of Images or Text will be meaningfully populated.
func FetchBinaryURL ¶
func FetchBinaryURL(url string, kind ResponseKind) (*BinaryFetchResult, error)
FetchBinaryURL downloads binary content from a URL and processes it for multimodal consumption based on the detected content type.
type GitApprovalPrompter ¶
GitApprovalPrompter is an interface for prompting the user for approval This avoids importing the agent package and creating import cycles
type GitCommitFlowExecutor ¶
GitCommitFlowExecutor is an interface for executing the commit flow This allows the git tool to delegate commit operations without creating import cycles
type GitOperation ¶
type GitOperation struct {
Operation GitOperationType `json:"operation"`
Args string `json:"args,omitempty"`
}
GitOperation defines a git operation request
type GitOperationType ¶
type GitOperationType string
GitOperationType defines the type of git operation
const ( GitOpCommit GitOperationType = "commit" GitOpPush GitOperationType = "push" GitOpAdd GitOperationType = "add" GitOpRm GitOperationType = "rm" GitOpMv GitOperationType = "mv" GitOpReset GitOperationType = "reset" GitOpRebase GitOperationType = "rebase" GitOpMerge GitOperationType = "merge" GitOpCheckout GitOperationType = "checkout" GitOpBranchDelete GitOperationType = "branch_delete" GitOpTag GitOperationType = "tag" GitOpClean GitOperationType = "clean" GitOpStash GitOperationType = "stash" GitOpAm GitOperationType = "am" GitOpApply GitOperationType = "apply" GitOpCherryPick GitOperationType = "cherry_pick" GitOpRevert GitOperationType = "revert" GitOpPull GitOperationType = "pull" GitOpFetch GitOperationType = "fetch" GitOpRestore GitOperationType = "restore" )
type ImageAnalysisResponse ¶
type ImageAnalysisResponse struct {
Success bool `json:"success"`
ToolInvoked bool `json:"tool_invoked"`
InputResolved bool `json:"input_resolved"`
OCRAttempted bool `json:"ocr_attempted"`
InputType string `json:"input_type"` // "local_file", "remote_url", "unknown"
InputPath string `json:"input_path"`
ErrorCode string `json:"error_code,omitempty"`
ErrorMessage string `json:"error_message,omitempty"`
ExtractedText string `json:"extracted_text,omitempty"`
OutputTruncated bool `json:"output_truncated,omitempty"`
OriginalChars int `json:"original_chars,omitempty"`
ReturnedChars int `json:"returned_chars,omitempty"`
FullOutputPath string `json:"full_output_path,omitempty"` // Path to full OCR/analysis text when truncated
Analysis *VisionAnalysis `json:"analysis,omitempty"`
SupportedInput ImageAnalysisSupported `json:"supported_input"`
}
ImageAnalysisResponse represents a structured response for the analyze_image_content tool
type ImageAnalysisSupported ¶
type ImageAnalysisSupported struct {
RemoteURL bool `json:"remote_url"`
LocalFile bool `json:"local_file"`
ImageFormats bool `json:"image_formats"` // jpg, png, gif, webp, etc.
PDFSupport bool `json:"pdf_support"` // PDF support status
PDFWorkaround string `json:"pdf_workaround"` // Instructions for PDF handling
MaxFileSizeMB int `json:"max_file_size_mb"`
}
ImageAnalysisSupported describes what input types are supported
type ImageData ¶
type ImageData struct {
// URI is the path or data URI of the image
URI string `json:"uri"`
// MIMEType is the image MIME type (e.g., "image/png")
MIMEType string `json:"mime_type"`
}
ImageData represents an image returned by a vision-capable tool.
type PDFPipelineResult ¶
type PDFPipelineResult struct {
Text string // Extracted text from pypdf (if available)
Images []api.ImageData // Page images for multimodal models (if text extraction failed)
Source string // "pypdf" or "page_images"
}
PDFPipelineResult holds the output of PDF processing for multimodal consumption.
func ProcessPDFForMultimodal ¶
func ProcessPDFForMultimodal(pdfPath string) (*PDFPipelineResult, error)
ProcessPDFForMultimodal processes a PDF file for multimodal consumption. Step 1: Try pypdf text extraction (works for text-based PDFs). Step 2: If no text found, render pages to optimized images for the model to see directly.
type ParameterDef ¶
type ParameterDef struct {
Name string `json:"name"`
Type string `json:"type"`
Required bool `json:"required"`
Description string `json:"description"`
}
ParameterDef defines a single tool parameter's schema.
type ResponseKind ¶
type ResponseKind int
ResponseKind classifies what kind of content a URL serves.
const ( ResponseKindUnknown ResponseKind = iota // Unable to determine or unsupported ResponseKindText // HTML, JSON, XML, plain text, etc. ResponseKindImage // PNG, JPEG, GIF, WebP, BMP, AVIF ResponseKindPDF // application/pdf )
func ClassifyContentType ¶
func ClassifyContentType(contentType string, urlPath string) ResponseKind
ClassifyContentType maps a Content-Type header value to a ResponseKind. Falls back to URL path extension when the header is ambiguous.
func ProbeURLContentType ¶
func ProbeURLContentType(url string) (ResponseKind, string)
ProbeURLContentType sends a HEAD request to determine the kind of content a URL serves. Falls back to URL path extension if the HEAD request fails. Returns both the ResponseKind and the effective URL (after redirects).
func (ResponseKind) IsBinary ¶
func (k ResponseKind) IsBinary() bool
IsBinary returns true if the ResponseKind represents binary content that should go through the multimodal pipeline rather than text extraction.
func (ResponseKind) String ¶
func (k ResponseKind) String() string
String returns a human-readable name for debugging.
type RiskCategory ¶
type RiskCategory string
RiskCategory represents the specific category of risk for a classified tool call.
const ( // RiskCategoryReadOnly — commands that only read data (cat, ls, head, grep, etc.) RiskCategoryReadOnly RiskCategory = "read-only" // RiskCategoryFileWrite — commands that modify files (write_file, edit_file, mkdir, cp, mv) RiskCategoryFileWrite RiskCategory = "file-write" // RiskCategoryNetwork — commands that access network (curl, wget, fetch) RiskCategoryNetwork RiskCategory = "network" // RiskCategoryProcessManagement — commands that manage processes (kill, pkill, docker start/stop) RiskCategoryProcessManagement RiskCategory = "process-management" // RiskCategoryDestructive — commands that destroy data (rm -rf, git reset --hard) RiskCategoryDestructive RiskCategory = "destructive" // RiskCategoryPrivileged — commands requiring elevated permissions (sudo, chmod, chown) RiskCategoryPrivileged RiskCategory = "privileged" // RiskCategoryUnknown — default when category cannot be determined RiskCategoryUnknown RiskCategory = "unknown" )
type RollbackResult ¶
RollbackResult captures the output, metadata, and success state for rollback operations.
func RollbackChanges ¶
func RollbackChanges(revisionID string, filePath string, confirm bool) (RollbackResult, error)
RollbackChanges previews or performs a rollback for a revision or file.
type SecurityResult ¶
type SecurityResult struct {
Risk SecurityRisk
Reasoning string
ShouldBlock bool
ShouldPrompt bool
IsHardBlock bool
RiskType string // Deprecated: Use Category instead. Risk category for user-facing messages
Category RiskCategory // Granular risk category for the classified operation
// IntentConfirmation marks a tool call as requiring explicit user
// confirmation before proceeding, but NOT because it's dangerous.
// Used for operations that are safe but consequential — like launching
// a long-running autonomous workflow. The approval prompt uses
// intent-focused framing instead of security-warning framing.
IntentConfirmation bool
}
SecurityResult contains the classification result for a tool call
func ClassifyToolCall ¶
func ClassifyToolCall(toolName string, args map[string]interface{}) SecurityResult
ClassifyToolCall classifies a tool call for security purposes based on the tool name and its arguments. It returns a SecurityResult indicating the risk level, reasoning, and whether the operation should be blocked or prompt the user.
Classification is purely string-based (no filesystem access). See the package-level documentation for known limitations of this approach.
Only tools whose arguments carry risk (shell commands, file writes, git ops) need explicit classification. All other registered tools default to SAFE — if a tool is in the registry, it's already vetted. The only real security value is inspecting the *arguments* to those risky tools.
func (SecurityResult) IsDestructive ¶
func (r SecurityResult) IsDestructive() bool
IsDestructive returns true if the operation's risk category is destructive.
type SecurityRisk ¶
type SecurityRisk int
SecurityRisk represents the risk level of a tool call
const ( SecuritySafe SecurityRisk = 0 SecurityCaution SecurityRisk = 1 SecurityDangerous SecurityRisk = 2 )
func (SecurityRisk) String ¶
func (r SecurityRisk) String() string
String returns a human-readable risk level
type SubtaskInput ¶
type SubtaskInput struct {
Title string `json:"title"`
WorkingDir string `json:"working_dir,omitempty"`
Persona string `json:"persona,omitempty"`
Priority string `json:"priority,omitempty"`
}
SubtaskInput represents input for creating subtasks
type Task ¶
type Task struct {
ID string `json:"id"`
Title string `json:"title"`
Description string `json:"description,omitempty"`
Status string `json:"status"` // pending, in_progress, completed, failed, blocked
Priority string `json:"priority"` // high, medium, low
AssignedTo string `json:"assigned_to,omitempty"`
WorkingDir string `json:"working_dir,omitempty"`
Persona string `json:"persona,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
Result string `json:"result,omitempty"`
ParentTaskID string `json:"parent_task_id,omitempty"`
}
Task represents a single task in the queue
type TaskQueue ¶
type TaskQueue struct {
// contains filtered or unexported fields
}
TaskQueue manages a persistent file-based task queue.
Public mutation methods (ReadTasks, PublishTask, AddTask) each acquire an exclusive or shared file lock, read the file fresh from disk, perform their operation, and write back atomically. This prevents the race condition that occurs when callers invoke Load() then PublishTask() as separate steps.
Load() and Save() are retained as convenience wrappers for advanced usage.
func NewTaskQueue ¶
NewTaskQueue creates a new TaskQueue instance.
func (*TaskQueue) AddTask ¶
func (tq *TaskQueue) AddTask(title, description, priority, workingDir, persona string) (*Task, error)
AddTask atomically adds a new task. Reads fresh from disk under an exclusive file lock, so concurrent adds from separate processes do not overwrite each other.
func (*TaskQueue) Load ¶
Load reads tasks from disk into the in-memory cache. Useful for initial load followed by multiple operations before Save(), but prefer the self-contained ReadTasks / PublishTask / AddTask methods for safety.
func (*TaskQueue) PublishTask ¶
func (tq *TaskQueue) PublishTask(taskID, status, result string, subtasks []SubtaskInput) ([]Task, error)
PublishTask atomically updates a task's status and result and optionally creates subtasks. Reads fresh from disk under an exclusive file lock, so it never overwrites work done by another process between Load and Publish.
type TerminalAccess ¶
type TerminalAccess interface {
// ExecuteCommandInHidden runs a command synchronously on a hidden PTY session
// and returns the output and exit code.
ExecuteCommandInHidden(ctx context.Context, sessionID string, command string) (output string, exitCode int, err error)
// GetOrCreateHiddenSessionForChat returns the session ID of an existing hidden session
// for the given chat, or creates a new one. Returns the session ID.
GetOrCreateHiddenSessionForChat(ctx context.Context, chatID string) (sessionID string, err error)
// ExecuteCommandInBackground writes a command to a new hidden PTY session
// and returns immediately with the session ID. Does NOT wait for completion.
// Background sessions get a descriptive name and longer cleanup timeout.
ExecuteCommandInBackground(ctx context.Context, chatID, command string) (sessionID string, err error)
// GetBackgroundOutput returns accumulated output for a background session.
GetBackgroundOutput(sessionID string) (output string, err error)
// StopBackgroundSession terminates a background session by session ID.
// Sends Ctrl+C to the PTY and closes the session. Returns an error if the
// session is not found or is not a background session.
StopBackgroundSession(sessionID string) error
// IsSessionActive checks whether a session (by ID) is still active.
// Returns false if the session doesn't exist or has terminated.
IsSessionActive(sessionID string) bool
}
TerminalAccess abstracts the operations that shell command execution needs from a terminal manager. This interface is satisfied by the webui's TerminalManager struct (pkg/webui/terminal_types.go) — no explicit import is needed; Go satisfies interfaces structurally.
When a TerminalAccess is available in the context (WebUI mode), shell commands can route through hidden PTY sessions. When absent (CLI mode), commands use the existing os/exec path unchanged.
func TerminalManagerFromContext ¶
func TerminalManagerFromContext(ctx context.Context) TerminalAccess
TerminalManagerFromContext extracts the TerminalAccess from the context. Returns nil if no terminal manager is available (CLI mode).
type TodoItem ¶
type TodoItem struct {
ID string `json:"id"`
Content string `json:"content"`
Status string `json:"status"` // pending, in_progress, completed
Priority string `json:"priority"` // high, medium, low
}
TodoItem represents a single todo item matching Claude Code's TodoWrite/TodoRead schema
type TodoManager ¶
type TodoManager struct {
// contains filtered or unexported fields
}
TodoManager manages the todo list for the current session
func NewTodoManager ¶
func NewTodoManager() *TodoManager
NewTodoManager creates a new TodoManager instance.
func (*TodoManager) Read ¶
func (tm *TodoManager) Read() []TodoItem
Read returns a copy of the current todo list.
func (*TodoManager) Write ¶
func (tm *TodoManager) Write(todos []TodoItem) string
Write replaces all todo items with the new list and returns a status message.
type ToolDefinition ¶
type ToolDefinition struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters []ParameterDef `json:"parameters"`
Required []string `json:"required,omitempty"` // Required parameter names
}
ToolDefinition describes a tool's schema for LLM consumption.
type ToolEnv ¶
type ToolEnv struct {
// EventBus for publishing events (tool_start, tool_end, etc.)
EventBus *events.EventBus
// WorkspaceRoot is the working directory root for path resolution
WorkspaceRoot string
// OutputWriter for writing tool output (stdout, logs, etc.)
OutputWriter io.Writer
// ApprovalManager for security approvals; nil if approvals are not supported
ApprovalManager ApprovalManager
// MaxTokensFunc returns the current token budget limit
MaxTokensFunc func() int
// ConfigManager provides configuration access for tools that need it (e.g., API keys for web fetching)
ConfigManager *configuration.Manager
// EmbeddingMgr is the agent's long-lived embedding manager. When set, tools
// must reuse it instead of constructing their own — the manager holds the
// loaded ONNX model and an open HNSW handle, so per-call construction is
// both slow and unsafe under concurrent writes.
EmbeddingMgr *embedding.EmbeddingManager
}
ToolEnv provides the execution context for a tool without coupling to *Agent.
type ToolHandler ¶
type ToolHandler interface {
// Name returns the unique tool identifier (e.g., "read_file").
Name() string
// Definition returns the JSON schema definition for the LLM to understand the tool.
Definition() ToolDefinition
// Validate checks arguments before execution. Returns error if invalid.
Validate(args map[string]any) error
// Execute runs the tool with the given context, environment, and arguments.
Execute(ctx context.Context, env ToolEnv, args map[string]any) (ToolResult, error)
}
ToolHandler defines the interface for a tool that can be invoked by the agent.
func AllTools ¶
func AllTools() []ToolHandler
AllTools returns all available tool handlers for registration. This is the central registration point for the new interface-based tool system. Currently includes: read_file, list_directory, fetch_url, search_files, repo_map, list_memories, read_memory, rollback_changes, view_history, list_skills, embedding_index, write_file, write_structured_file, edit_file, shell_command, save_memory, search_memories, task_queue_add, task_queue_publish, task_queue_read, todo_write, todo_read, ask_user, patch_structured_file, self_review, commit, git, activate_skill, add_memory, delete_memory, browse_url, web_search, semantic_search, analyze_image_content, and analyze_ui_screenshot.
Subagent tools (run_subagent / run_parallel_subagents) are NOT in this list — they live exclusively in the seed registry under pkg/agent because they require *Agent access for nested runner orchestration. SP-059 Phase 3b removed earlier stub entries that returned hardcoded errors; the seed registry's dual-dispatch path is canonical.
To register all tools with a registry:
registry := tools.NewToolRegistry()
for _, h := range tools.AllTools() {
registry.Register(h)
}
type ToolRegistry ¶
type ToolRegistry struct {
// contains filtered or unexported fields
}
ToolRegistry provides thread-safe registration and lookup of ToolHandlers.
func GetNewToolRegistry ¶
func GetNewToolRegistry() *ToolRegistry
GetNewToolRegistry returns the global new-style tool registry singleton.
func NewToolRegistry ¶
func NewToolRegistry() *ToolRegistry
NewToolRegistry creates an empty ToolRegistry.
func (*ToolRegistry) All ¶
func (r *ToolRegistry) All() map[string]ToolHandler
All returns a copy of all registered tools.
func (*ToolRegistry) ForPersona ¶
func (r *ToolRegistry) ForPersona(persona string) map[string]ToolHandler
ForPersona returns tools available for a given persona. Initially returns all tools. TODO(SP-038): Implement per-persona tool filtering
func (*ToolRegistry) Lookup ¶
func (r *ToolRegistry) Lookup(name string) (ToolHandler, bool)
Lookup finds a tool by name. Returns (handler, true) if found, (nil, false) otherwise.
func (*ToolRegistry) Names ¶
func (r *ToolRegistry) Names() []string
Names returns a sorted list of all registered tool names.
func (*ToolRegistry) Register ¶
func (r *ToolRegistry) Register(handler ToolHandler) error
Register adds a tool handler. Returns error if name is already registered.
func (*ToolRegistry) Unregister ¶
func (r *ToolRegistry) Unregister(name string) bool
Unregister removes a tool handler by name. Returns true if the tool was found and removed.
type ToolResult ¶
type ToolResult struct {
// Output is the primary text result of the tool execution.
Output string `json:"output"`
// StructuredOut holds optional structured data (maps, slices, etc.)
StructuredOut any `json:"structured_out,omitempty"`
// Images contains optional image data for vision-capable tools.
Images []ImageData `json:"images,omitempty"`
// TokenUsage tracks tokens consumed during execution.
TokenUsage int64 `json:"token_usage"`
// IsError indicates whether this result represents an error state.
IsError bool `json:"is_error"`
}
ToolResult is the return value from a tool's Execute method.
type UIElement ¶
type UIElement struct {
Type string `json:"type"` // button, input, text, etc.
Description string `json:"description"` // what it looks like
Position string `json:"position"` // approximate location
Issues string `json:"issues,omitempty"` // any problems noted
}
UIElement represents a UI element detected in an image
type ViewHistoryResult ¶
ViewHistoryResult captures the output and metadata for history views.
func ViewHistory ¶
func ViewHistory(limit int, fileFilter string, since *time.Time, showContent bool) (ViewHistoryResult, error)
ViewHistory returns a formatted history view based on the provided filters.
type VisionAnalysis ¶
type VisionAnalysis struct {
ImagePath string `json:"image_path"`
Description string `json:"description"`
Elements []UIElement `json:"elements,omitempty"`
Issues []string `json:"issues,omitempty"`
Suggestions []string `json:"suggestions,omitempty"`
}
VisionAnalysis represents the result of vision model analysis
type VisionProcessor ¶
type VisionProcessor struct {
// contains filtered or unexported fields
}
VisionProcessor handles image analysis using vision-capable models
func NewVisionProcessor ¶
func NewVisionProcessor(client api.ClientInterface, logger *utils.Logger, debug bool) *VisionProcessor
NewVisionProcessor creates a vision processor with the given client
func NewVisionProcessorWithMode ¶
func NewVisionProcessorWithMode(debug bool, _ string) (*VisionProcessor, error)
NewVisionProcessorWithMode creates a vision processor for image/OCR workflows. Client selection is intentionally deterministic and does not vary by mode: provider-vision list first, local Ollama fallback last.
func NewVisionProcessorWithProvider ¶
func NewVisionProcessorWithProvider(debug bool, providerType api.ClientType) (*VisionProcessor, error)
NewVisionProcessorWithProvider creates a vision processor using the specified provider
func (*VisionProcessor) AnalyzeImage ¶
func (vp *VisionProcessor) AnalyzeImage(imagePath string, optionalPrompt ...string) (VisionAnalysis, error)
AnalyzeImage processes a single image with the vision model. If optionalPrompt is provided and non-empty, it is used as the prompt; otherwise the default vision prompt for imagePath is created.
func (*VisionProcessor) CreateVisionPrompt ¶
func (vp *VisionProcessor) CreateVisionPrompt(imagePath string) string
CreateVisionPrompt creates an appropriate prompt based on image context
func (*VisionProcessor) DownloadImage ¶
func (vp *VisionProcessor) DownloadImage(url string) ([]byte, error)
DownloadImage downloads an image from URL
func (*VisionProcessor) EnhanceTextWithAnalysis ¶
func (vp *VisionProcessor) EnhanceTextWithAnalysis(text, imagePath string, analysis VisionAnalysis) string
EnhanceTextWithAnalysis replaces image references with detailed analysis
func (*VisionProcessor) ExtractPosition ¶
func (vp *VisionProcessor) ExtractPosition(line string) string
ExtractPosition attempts to extract position information from a description
func (*VisionProcessor) ExtractUIElements ¶
func (vp *VisionProcessor) ExtractUIElements(description string) []UIElement
ExtractUIElements attempts to extract structured UI elements from the description
func (*VisionProcessor) GetImageData ¶
func (vp *VisionProcessor) GetImageData(imagePath string) (string, string, error)
GetImageData reads image data from file or URL
func (*VisionProcessor) LooksLikeUI ¶
func (vp *VisionProcessor) LooksLikeUI(description string) bool
LooksLikeUI determines if the description suggests a UI interface
func (*VisionProcessor) ParseUIElementFromLine ¶
func (vp *VisionProcessor) ParseUIElementFromLine(line string) UIElement
ParseUIElementFromLine attempts to extract a UI element from a description line
func (*VisionProcessor) ProcessImagesInText ¶
func (vp *VisionProcessor) ProcessImagesInText(text string) (string, []VisionAnalysis, error)
ProcessImagesInText detects images in text and processes them with vision models
func (*VisionProcessor) ProcessPDFForVision ¶
func (vp *VisionProcessor) ProcessPDFForVision(pdfPath string) (VisionAnalysis, error)
ProcessPDFForVision processes PDF using Ollama with glm-ocr model
type VisionUsageInfo ¶
type VisionUsageInfo struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
EstimatedCost float64 `json:"estimated_cost"`
}
VisionUsageInfo contains token usage and cost information from vision model calls
func GetLastVisionUsage ¶
func GetLastVisionUsage() *VisionUsageInfo
GetLastVisionUsage returns the usage information from the last vision model call
Source Files
¶
- activate_skill_handler.go
- add_memory_handler.go
- all.go
- analyze_image_content_handler.go
- analyze_ui_screenshot_handler.go
- ask_user.go
- ask_user_handler.go
- background_process.go
- background_process_signal_unix.go
- binary_fetch.go
- browse_url_handler.go
- commit_handler.go
- common.go
- delete_memory_handler.go
- edit.go
- edit_handler.go
- embedding_index_handler.go
- fetch_url.go
- fetch_url_handler.go
- git.go
- git_args_validate.go
- git_handler.go
- handler.go
- handlers_common.go
- history.go
- list_dir.go
- list_memories_handler.go
- list_skills_handler.go
- normalization.go
- patch_structured_file_handler.go
- pdf_python_env.go
- read.go
- read_file.go
- read_memory_handler.go
- registry.go
- repo_map.go
- repo_map_handler.go
- rollback_changes_handler.go
- safety.go
- save_memory_handler.go
- search_files_handler.go
- search_memories_handler.go
- security_audit.go
- security_classifier.go
- self_review_handler.go
- semantic_search_handler.go
- shell.go
- shell_handler.go
- shell_native.go
- shell_patterns.go
- shell_utils.go
- structured_helpers.go
- structured_json_native.go
- task_queue.go
- task_queue_add_handler.go
- task_queue_publish_handler.go
- task_queue_read_handler.go
- terminal.go
- todo.go
- todo_read_handler.go
- todo_write_handler.go
- url_content_type.go
- view_history_handler.go
- vision.go
- vision_analyze.go
- vision_pdf.go
- vision_pdf_pipeline.go
- vision_prompts.go
- vision_types.go
- web_search.go
- web_search_handler.go
- write.go
- write_handler.go
- write_structured_handler.go