training

package
v0.17.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 23, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package training provides utilities for exporting session data into training-ready formats (ShareGPT, OpenAI fine-tuning JSONL, Alpaca).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CollectRemoteUsernames added in v0.17.5

func CollectRemoteUsernames(strings []string) []string

CollectRemoteUsernames scans strings for home directory paths and returns any usernames that differ from the local username. This enables redaction of usernames from remote machines where /home/<user> doesn't match the local /Users/<name> or /home/<name> pattern.

func PushSession added in v0.17.5

func PushSession(state agent.ConversationState, endpoint string, excludePaths []string) error

PushSession pushes a PII-redacted conversation state to the training endpoint. It applies the same PII redaction pipeline as the export command (RedactContent), so no raw PII or secrets leave the machine.

This is a fire-and-forget operation — errors are returned (for optional logging by the caller) but must never cause a session save to fail or panic. The caller is expected to invoke this in a goroutine.

If the session's working directory matches any excludePath prefix, the push is skipped silently (returns nil).

func RedactContent added in v0.17.5

func RedactContent(s string) string

RedactContent applies both PII and secret redaction to a content string. This is the single entry point for all content sanitization in exports. Call SetRemoteUsernames before calling this to enable redaction of usernames from remote machines discovered during a pre-scan.

func SetRemoteUsernames added in v0.17.5

func SetRemoteUsernames(users []string)

SetRemoteUsernames sets the additional usernames that RedactContent will redact as $USER in addition to the local username.

Types

type AlpacaExample

type AlpacaExample struct {
	Instruction string `json:"instruction"`
	Input       string `json:"input"`
	Output      string `json:"output"`
}

AlpacaExample is one training example in Alpaca format.

type ExportOptions

type ExportOptions struct {
	// Format is one of "sharegpt", "openai", "alpaca".
	Format string

	// Output is the destination file path.
	Output string

	// All, when true, exports sessions from every directory scope.
	All bool

	// MinTurns is the minimum number of user+assistant exchanges required.
	MinTurns int

	// MinActions is the minimum number of TaskActions required.
	MinActions int

	// NoToolResults, when true, replaces tool-result messages with short
	// placeholders instead of including the raw content.
	NoToolResults bool

	// IncludeSystem includes system-prompt messages in the output.
	IncludeSystem bool

	// Session, when non-empty, exports only the session with this ID.
	Session string

	// StructuredTools, when true, preserves the OpenAI function-calling
	// schema: assistant messages retain their ToolCalls arrays and tool
	// results keep role:"tool" with tool_call_id. When false (default),
	// tool calls are flattened to text and tool results are converted to
	// user-role messages with a "[Tool Result]" prefix.
	StructuredTools bool

	// IncludeSubagents, when true, extracts single-task examples from
	// run_subagent and run_parallel_subagents tool calls within each
	// session and appends them to the output alongside the regular
	// conversation examples. Only meaningful for the "openai" format.
	IncludeSubagents bool

	// ExcludePaths is a list of absolute path prefixes. Sessions whose
	// WorkingDirectory starts with any of these paths are excluded.
	ExcludePaths []string
}

ExportOptions configures what sessions to export and how to format them.

type ExportResult

type ExportResult struct {
	SessionsScanned   int    `json:"sessions_scanned"`
	SessionsExported  int    `json:"sessions_exported"`
	ExamplesGenerated int    `json:"examples_generated"`
	SessionsFiltered  int    `json:"sessions_filtered"`
	OutputPath        string `json:"output_path"`
}

ExportResult contains statistics about an export run.

func ExportSessions

func ExportSessions(opts ExportOptions) (*ExportResult, error)

ExportSessions reads sessions according to opts and writes the result to opts.Output. It returns a summary of the operation.

type FileChangeExample added in v0.17.5

type FileChangeExample struct {
	Messages []OpenAIMessage           `json:"messages"`
	Metadata FileChangeExampleMetadata `json:"metadata"`
}

FileChangeExample is one training example built from a file diff pair.

type FileChangeExampleMetadata added in v0.17.5

type FileChangeExampleMetadata struct {
	Source      string `json:"source"`
	Description string `json:"description"`
	Model       string `json:"model,omitempty"`
	File        string `json:"file"`
}

FileChangeExampleMetadata holds metadata about a single file-change example.

type FileChangeExportOptions added in v0.17.5

type FileChangeExportOptions struct {
	// Output is the destination file path (OpenAI JSONL).
	Output string

	// MaxSize is the maximum number of characters per file content (before
	// or after). Changes where either side exceeds this are skipped.
	// Defaults to 50000 when zero.
	MaxSize int

	// ExcludePaths is a list of absolute path prefixes. Working directories
	// starting with any of these paths are excluded from the export.
	ExcludePaths []string
}

FileChangeExportOptions configures a file-change diff-pair export run.

type FileChangeExportResult added in v0.17.5

type FileChangeExportResult struct {
	ChangesScanned  int    `json:"changes_scanned"`
	ChangesExported int    `json:"changes_exported"`
	ChangesFiltered int    `json:"changes_filtered"`
	OutputPath      string `json:"output_path"`
}

FileChangeExportResult contains statistics about a file-change export run.

func ExportFileChanges added in v0.17.5

func ExportFileChanges(opts FileChangeExportOptions) (*FileChangeExportResult, error)

ExportFileChanges scans all .sprout/changes directories discovered via the session registry (agent.ListAllSessionsWithTimestamps), reads each file-change diff pair, filters out noise, and writes OpenAI JSONL training examples to opts.Output.

type OpenAIMessage

type OpenAIMessage struct {
	Role       string         `json:"role"`
	Content    string         `json:"content"`
	ToolCalls  []api.ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string         `json:"tool_call_id,omitempty"`
}

OpenAIMessage is a single message within an OpenAI training example.

type OpenAITrainingExample

type OpenAITrainingExample struct {
	Messages []OpenAIMessage `json:"messages"`
}

OpenAITrainingExample is one training example for OpenAI fine-tuning JSONL.

type PIIRedactionConfig added in v0.17.5

type PIIRedactionConfig struct {
	// HomeDir is the user's home directory to redact (e.g. /Users/alan).
	// When set, all occurrences are replaced with the placeholder.
	HomeDir string

	// Username is the OS username to redact.
	Username string

	// Hostname is the machine hostname to redact.
	Hostname string
}

PIIRedactionConfig controls what personally identifiable information is scrubbed from training data exports.

func DefaultPIIConfig added in v0.17.5

func DefaultPIIConfig() PIIRedactionConfig

DefaultPIIConfig builds a PII config from the current environment.

type ShareGPTConversation

type ShareGPTConversation struct {
	ID       string            `json:"id"`
	Messages []ShareGPTMessage `json:"messages"`
	Metadata ShareGPTMetadata  `json:"metadata"`
}

ShareGPTConversation represents one conversation in ShareGPT format.

type ShareGPTMessage

type ShareGPTMessage struct {
	Role       string         `json:"role"` // "system", "user", "assistant"
	Content    string         `json:"content"`
	ToolCalls  []api.ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string         `json:"tool_call_id,omitempty"`
}

ShareGPTMessage is a single message in a ShareGPT conversation.

type ShareGPTMetadata

type ShareGPTMetadata struct {
	SessionID   string  `json:"session_id"`
	SessionName string  `json:"session_name"`
	Source      string  `json:"source"`
	Model       string  `json:"model,omitempty"`
	Provider    string  `json:"provider,omitempty"`
	TotalCost   float64 `json:"total_cost"`
	WorkingDir  string  `json:"working_directory"`
}

ShareGPTMetadata holds extra information about a ShareGPT conversation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL