Documentation
¶
Overview ¶
Package training provides utilities for exporting session data into training-ready formats (ShareGPT, OpenAI fine-tuning JSONL, Alpaca).
Index ¶
- func CollectRemoteUsernames(strings []string) []string
- func PushSession(state agent.ConversationState, endpoint string, excludePaths []string) error
- func RedactContent(s string) string
- func SetRemoteUsernames(users []string)
- type AlpacaExample
- type ExportOptions
- type ExportResult
- type FileChangeExample
- type FileChangeExampleMetadata
- type FileChangeExportOptions
- type FileChangeExportResult
- type OpenAIMessage
- type OpenAITrainingExample
- type PIIRedactionConfig
- type ShareGPTConversation
- type ShareGPTMessage
- type ShareGPTMetadata
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CollectRemoteUsernames ¶ added in v0.17.5
CollectRemoteUsernames scans strings for home directory paths and returns any usernames that differ from the local username. This enables redaction of usernames from remote machines where /home/<user> doesn't match the local /Users/<name> or /home/<name> pattern.
func PushSession ¶ added in v0.17.5
func PushSession(state agent.ConversationState, endpoint string, excludePaths []string) error
PushSession pushes a PII-redacted conversation state to the training endpoint. It applies the same PII redaction pipeline as the export command (RedactContent), so no raw PII or secrets leave the machine.
This is a fire-and-forget operation — errors are returned (for optional logging by the caller) but must never cause a session save to fail or panic. The caller is expected to invoke this in a goroutine.
If the session's working directory matches any excludePath prefix, the push is skipped silently (returns nil).
func RedactContent ¶ added in v0.17.5
RedactContent applies both PII and secret redaction to a content string. This is the single entry point for all content sanitization in exports. Call SetRemoteUsernames before calling this to enable redaction of usernames from remote machines discovered during a pre-scan.
func SetRemoteUsernames ¶ added in v0.17.5
func SetRemoteUsernames(users []string)
SetRemoteUsernames sets the additional usernames that RedactContent will redact as $USER in addition to the local username.
Types ¶
type AlpacaExample ¶
type AlpacaExample struct {
Instruction string `json:"instruction"`
Input string `json:"input"`
Output string `json:"output"`
}
AlpacaExample is one training example in Alpaca format.
type ExportOptions ¶
type ExportOptions struct {
// Format is one of "sharegpt", "openai", "alpaca".
Format string
// Output is the destination file path.
Output string
// All, when true, exports sessions from every directory scope.
All bool
// MinTurns is the minimum number of user+assistant exchanges required.
MinTurns int
// MinActions is the minimum number of TaskActions required.
MinActions int
// NoToolResults, when true, replaces tool-result messages with short
// placeholders instead of including the raw content.
NoToolResults bool
// IncludeSystem includes system-prompt messages in the output.
IncludeSystem bool
// Session, when non-empty, exports only the session with this ID.
Session string
// StructuredTools, when true, preserves the OpenAI function-calling
// schema: assistant messages retain their ToolCalls arrays and tool
// results keep role:"tool" with tool_call_id. When false (default),
// tool calls are flattened to text and tool results are converted to
// user-role messages with a "[Tool Result]" prefix.
StructuredTools bool
// IncludeSubagents, when true, extracts single-task examples from
// run_subagent and run_parallel_subagents tool calls within each
// session and appends them to the output alongside the regular
// conversation examples. Only meaningful for the "openai" format.
IncludeSubagents bool
// ExcludePaths is a list of absolute path prefixes. Sessions whose
// WorkingDirectory starts with any of these paths are excluded.
ExcludePaths []string
}
ExportOptions configures what sessions to export and how to format them.
type ExportResult ¶
type ExportResult struct {
SessionsScanned int `json:"sessions_scanned"`
SessionsExported int `json:"sessions_exported"`
ExamplesGenerated int `json:"examples_generated"`
SessionsFiltered int `json:"sessions_filtered"`
OutputPath string `json:"output_path"`
}
ExportResult contains statistics about an export run.
func ExportSessions ¶
func ExportSessions(opts ExportOptions) (*ExportResult, error)
ExportSessions reads sessions according to opts and writes the result to opts.Output. It returns a summary of the operation.
type FileChangeExample ¶ added in v0.17.5
type FileChangeExample struct {
Messages []OpenAIMessage `json:"messages"`
Metadata FileChangeExampleMetadata `json:"metadata"`
}
FileChangeExample is one training example built from a file diff pair.
type FileChangeExampleMetadata ¶ added in v0.17.5
type FileChangeExampleMetadata struct {
Source string `json:"source"`
Description string `json:"description"`
Model string `json:"model,omitempty"`
File string `json:"file"`
}
FileChangeExampleMetadata holds metadata about a single file-change example.
type FileChangeExportOptions ¶ added in v0.17.5
type FileChangeExportOptions struct {
// Output is the destination file path (OpenAI JSONL).
Output string
// MaxSize is the maximum number of characters per file content (before
// or after). Changes where either side exceeds this are skipped.
// Defaults to 50000 when zero.
MaxSize int
// ExcludePaths is a list of absolute path prefixes. Working directories
// starting with any of these paths are excluded from the export.
ExcludePaths []string
}
FileChangeExportOptions configures a file-change diff-pair export run.
type FileChangeExportResult ¶ added in v0.17.5
type FileChangeExportResult struct {
ChangesScanned int `json:"changes_scanned"`
ChangesExported int `json:"changes_exported"`
ChangesFiltered int `json:"changes_filtered"`
OutputPath string `json:"output_path"`
}
FileChangeExportResult contains statistics about a file-change export run.
func ExportFileChanges ¶ added in v0.17.5
func ExportFileChanges(opts FileChangeExportOptions) (*FileChangeExportResult, error)
ExportFileChanges scans all .sprout/changes directories discovered via the session registry (agent.ListAllSessionsWithTimestamps), reads each file-change diff pair, filters out noise, and writes OpenAI JSONL training examples to opts.Output.
type OpenAIMessage ¶
type OpenAIMessage struct {
Role string `json:"role"`
Content string `json:"content"`
ToolCalls []api.ToolCall `json:"tool_calls,omitempty"`
ToolCallID string `json:"tool_call_id,omitempty"`
}
OpenAIMessage is a single message within an OpenAI training example.
type OpenAITrainingExample ¶
type OpenAITrainingExample struct {
Messages []OpenAIMessage `json:"messages"`
}
OpenAITrainingExample is one training example for OpenAI fine-tuning JSONL.
type PIIRedactionConfig ¶ added in v0.17.5
type PIIRedactionConfig struct {
// HomeDir is the user's home directory to redact (e.g. /Users/alan).
// When set, all occurrences are replaced with the placeholder.
HomeDir string
// Username is the OS username to redact.
Username string
// Hostname is the machine hostname to redact.
Hostname string
}
PIIRedactionConfig controls what personally identifiable information is scrubbed from training data exports.
func DefaultPIIConfig ¶ added in v0.17.5
func DefaultPIIConfig() PIIRedactionConfig
DefaultPIIConfig builds a PII config from the current environment.
type ShareGPTConversation ¶
type ShareGPTConversation struct {
}
ShareGPTConversation represents one conversation in ShareGPT format.
type ShareGPTMessage ¶
type ShareGPTMessage struct {
}
ShareGPTMessage is a single message in a ShareGPT conversation.
type ShareGPTMetadata ¶
type ShareGPTMetadata struct {
}
ShareGPTMetadata holds extra information about a ShareGPT conversation.