Documentation
¶
Overview ¶
Package overwatch provides real-time execution monitoring capabilities for the Superbrain system. It implements the Overwatch Layer which wraps CLI executions with monitoring, heartbeat detection, and silence threshold detection to identify hung processes.
Index ¶
- func DefaultOverwatchConfig() *config.OverwatchConfig
- type Monitor
- func (m *Monitor) ActiveContextCount() int
- func (m *Monitor) GetContext(requestID string) *OverwatchContext
- func (m *Monitor) StartMonitoring(processID int, provider, model, requestID string) *OverwatchContext
- func (m *Monitor) Stop()
- func (m *Monitor) StopMonitoring(requestID string)
- func (m *Monitor) WrapReader(reader io.Reader, ctx *OverwatchContext) *MonitoredReader
- func (m *Monitor) WrapWriter(writer io.Writer, ctx *OverwatchContext) *MonitoredWriter
- type MonitoredReader
- type MonitoredWriter
- type Overwatch
- type OverwatchContext
- func (ctx *OverwatchContext) CanRestart() bool
- func (ctx *OverwatchContext) CaptureSnapshot(processState string) *types.DiagnosticSnapshot
- func (ctx *OverwatchContext) CheckHeartbeat() bool
- func (ctx *OverwatchContext) GetElapsedTime() time.Duration
- func (ctx *OverwatchContext) GetElapsedTimeMs() int64
- func (ctx *OverwatchContext) GetHealingActions() []types.HealingAction
- func (ctx *OverwatchContext) GetHealingMetadata() *types.HealingMetadata
- func (ctx *OverwatchContext) GetLastLogLines(n int) []string
- func (ctx *OverwatchContext) GetSilenceDuration() time.Duration
- func (ctx *OverwatchContext) IncrementDiagnosticCount() int
- func (ctx *OverwatchContext) IncrementRestartCount() int
- func (ctx *OverwatchContext) MarkSilent()
- func (ctx *OverwatchContext) RecordHealingAction(action types.HealingAction)
- func (ctx *OverwatchContext) RecordOutput(line string)
- func (ctx *OverwatchContext) ResetForRestart(newPID int)
- func (ctx *OverwatchContext) UpdateProcessID(pid int)
- type RingBuffer
- type SnapshotHandler
- type StreamMonitor
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultOverwatchConfig ¶
func DefaultOverwatchConfig() *config.OverwatchConfig
DefaultOverwatchConfig returns the default Overwatch configuration.
Types ¶
type Monitor ¶
type Monitor struct {
// contains filtered or unexported fields
}
Monitor provides real-time monitoring of CLI process execution. It wraps stdout/stderr streams, detects silence, and emits diagnostic snapshots.
func NewMonitor ¶
func NewMonitor(cfg *config.OverwatchConfig, onSnapshot SnapshotHandler) *Monitor
NewMonitor creates a new Monitor with the given configuration.
func (*Monitor) ActiveContextCount ¶
ActiveContextCount returns the number of currently monitored executions.
func (*Monitor) GetContext ¶
func (m *Monitor) GetContext(requestID string) *OverwatchContext
GetContext returns the monitoring context for a request ID.
func (*Monitor) StartMonitoring ¶
func (m *Monitor) StartMonitoring(processID int, provider, model, requestID string) *OverwatchContext
StartMonitoring begins monitoring a process execution. It returns an OverwatchContext that tracks the execution state.
func (*Monitor) StopMonitoring ¶
StopMonitoring stops monitoring for a specific request.
func (*Monitor) WrapReader ¶
func (m *Monitor) WrapReader(reader io.Reader, ctx *OverwatchContext) *MonitoredReader
WrapReader creates a MonitoredReader that tracks output from the given reader.
func (*Monitor) WrapWriter ¶
func (m *Monitor) WrapWriter(writer io.Writer, ctx *OverwatchContext) *MonitoredWriter
WrapWriter creates a MonitoredWriter that tracks output written to the given writer.
type MonitoredReader ¶
type MonitoredReader struct {
// contains filtered or unexported fields
}
MonitoredReader wraps an io.Reader to track output and update the monitoring context.
type MonitoredWriter ¶
type MonitoredWriter struct {
// contains filtered or unexported fields
}
MonitoredWriter wraps an io.Writer to track output and update the monitoring context.
type Overwatch ¶
type Overwatch interface {
// StartMonitoring begins watching an execution.
StartMonitoring(processID int, provider, model, requestID string) *OverwatchContext
// StopMonitoring stops watching an execution.
StopMonitoring(requestID string)
// GetContext returns the monitoring context for a request.
GetContext(requestID string) *OverwatchContext
// WrapReader creates a monitored reader.
WrapReader(reader io.Reader, ctx *OverwatchContext) *MonitoredReader
// WrapWriter creates a monitored writer.
WrapWriter(writer io.Writer, ctx *OverwatchContext) *MonitoredWriter
// Stop stops all monitoring.
Stop()
}
Overwatch is the main interface for the Overwatch Layer. It provides methods for starting monitoring, checking health, and capturing snapshots.
type OverwatchContext ¶
type OverwatchContext struct {
// ProcessID is the OS process ID of the monitored process.
ProcessID int
// StartTime is when the process was spawned.
StartTime time.Time
// LastOutputTime is the timestamp of the most recent stdout/stderr output.
// Used for heartbeat detection and silence threshold calculation.
LastOutputTime time.Time
// LogBuffer stores recent log lines for diagnostic snapshots.
LogBuffer *RingBuffer
// RestartCount tracks how many times this execution has been restarted.
RestartCount int
// DiagnosticCount tracks how many diagnostic snapshots have been captured.
DiagnosticCount int
// HealingActions records all autonomous interventions taken during this execution.
HealingActions []types.HealingAction
// Config holds the Overwatch configuration parameters.
Config *config.OverwatchConfig
// Provider is the provider being executed (e.g., "claudecli", "geminicli").
Provider string
// Model is the model being used for this execution.
Model string
// RequestID uniquely identifies the request that triggered this execution.
RequestID string
// IsSilent indicates whether the process is currently in a silent state.
IsSilent bool
// SilenceStartTime is when the current silence period began (if IsSilent is true).
SilenceStartTime time.Time
// contains filtered or unexported fields
}
OverwatchContext holds the monitoring state for a single CLI execution. It tracks process health, output timing, and healing actions taken during the execution lifecycle.
func NewOverwatchContext ¶
func NewOverwatchContext(processID int, cfg *config.OverwatchConfig, provider, model, requestID string) *OverwatchContext
NewOverwatchContext creates a new monitoring context for a process execution. It initializes the log buffer and sets the start time.
func (*OverwatchContext) CanRestart ¶
func (ctx *OverwatchContext) CanRestart() bool
CanRestart returns true if another restart attempt is allowed.
func (*OverwatchContext) CaptureSnapshot ¶
func (ctx *OverwatchContext) CaptureSnapshot(processState string) *types.DiagnosticSnapshot
CaptureSnapshot creates a diagnostic snapshot of the current execution state. This is called when the silence threshold is exceeded to gather diagnostic information.
func (*OverwatchContext) CheckHeartbeat ¶
func (ctx *OverwatchContext) CheckHeartbeat() bool
CheckHeartbeat checks if the process is healthy based on output timing. Returns true if output was received within the silence threshold, false otherwise.
func (*OverwatchContext) GetElapsedTime ¶
func (ctx *OverwatchContext) GetElapsedTime() time.Duration
GetElapsedTime returns the total time elapsed since process start.
func (*OverwatchContext) GetElapsedTimeMs ¶
func (ctx *OverwatchContext) GetElapsedTimeMs() int64
GetElapsedTimeMs returns the total time elapsed since process start in milliseconds.
func (*OverwatchContext) GetHealingActions ¶
func (ctx *OverwatchContext) GetHealingActions() []types.HealingAction
GetHealingActions returns a copy of all healing actions recorded.
func (*OverwatchContext) GetHealingMetadata ¶
func (ctx *OverwatchContext) GetHealingMetadata() *types.HealingMetadata
GetHealingMetadata creates a HealingMetadata struct from the context's recorded actions.
func (*OverwatchContext) GetLastLogLines ¶
func (ctx *OverwatchContext) GetLastLogLines(n int) []string
GetLastLogLines returns the last n log lines from the buffer.
func (*OverwatchContext) GetSilenceDuration ¶
func (ctx *OverwatchContext) GetSilenceDuration() time.Duration
GetSilenceDuration returns how long the process has been silent.
func (*OverwatchContext) IncrementDiagnosticCount ¶
func (ctx *OverwatchContext) IncrementDiagnosticCount() int
IncrementDiagnosticCount increments the diagnostic counter and returns the new count.
func (*OverwatchContext) IncrementRestartCount ¶
func (ctx *OverwatchContext) IncrementRestartCount() int
IncrementRestartCount increments the restart counter and returns the new count.
func (*OverwatchContext) MarkSilent ¶
func (ctx *OverwatchContext) MarkSilent()
MarkSilent marks the process as being in a silent state. This is called when the silence threshold is first exceeded.
func (*OverwatchContext) RecordHealingAction ¶
func (ctx *OverwatchContext) RecordHealingAction(action types.HealingAction)
RecordHealingAction adds a healing action to the context's history.
func (*OverwatchContext) RecordOutput ¶
func (ctx *OverwatchContext) RecordOutput(line string)
RecordOutput records that output was received from the process. This updates the LastOutputTime for heartbeat tracking and clears the silent state.
func (*OverwatchContext) ResetForRestart ¶
func (ctx *OverwatchContext) ResetForRestart(newPID int)
ResetForRestart resets timing state for a new restart attempt. Preserves restart count, healing actions, and configuration.
func (*OverwatchContext) UpdateProcessID ¶
func (ctx *OverwatchContext) UpdateProcessID(pid int)
UpdateProcessID updates the process ID (used after restart).
type RingBuffer ¶
type RingBuffer struct {
// contains filtered or unexported fields
}
RingBuffer is a thread-safe circular buffer for storing log lines. It maintains a fixed-size buffer that overwrites the oldest entries when full, ensuring memory-bounded log retention for diagnostic purposes.
func NewRingBuffer ¶
func NewRingBuffer(size int) *RingBuffer
NewRingBuffer creates a new RingBuffer with the specified capacity. If size is <= 0, a default size of 50 is used.
func (*RingBuffer) Cap ¶
func (rb *RingBuffer) Cap() int
Cap returns the maximum capacity of the buffer.
func (*RingBuffer) Clear ¶
func (rb *RingBuffer) Clear()
Clear removes all lines from the buffer. This operation is thread-safe.
func (*RingBuffer) GetAll ¶
func (rb *RingBuffer) GetAll() []string
GetAll returns all lines currently in the buffer in chronological order. This operation is thread-safe.
func (*RingBuffer) GetLast ¶
func (rb *RingBuffer) GetLast(n int) []string
GetLast returns the last n lines from the buffer in chronological order. If n is greater than the number of stored lines, all stored lines are returned. If n is <= 0, an empty slice is returned. This operation is thread-safe.
func (*RingBuffer) IsFull ¶
func (rb *RingBuffer) IsFull() bool
IsFull returns true if the buffer has reached capacity and is overwriting old entries. This operation is thread-safe.
func (*RingBuffer) Len ¶
func (rb *RingBuffer) Len() int
Len returns the current number of lines in the buffer. This operation is thread-safe.
func (*RingBuffer) Write ¶
func (rb *RingBuffer) Write(line string)
Write adds a line to the buffer. If the buffer is full, the oldest line is overwritten. This operation is thread-safe.
type SnapshotHandler ¶
type SnapshotHandler func(ctx *OverwatchContext, snapshot *types.DiagnosticSnapshot)
SnapshotHandler is called when a diagnostic snapshot is captured due to silence detection.
type StreamMonitor ¶
type StreamMonitor struct {
// contains filtered or unexported fields
}
StreamMonitor provides line-by-line monitoring of stdout and stderr streams.
func NewStreamMonitor ¶
func NewStreamMonitor(ctx *OverwatchContext) *StreamMonitor
NewStreamMonitor creates a StreamMonitor for the given context.
func (*StreamMonitor) MonitorStreams ¶
func (sm *StreamMonitor) MonitorStreams( bgCtx context.Context, stdout, stderr io.Reader, onStdout, onStderr func(line string), )
MonitorStreams starts monitoring stdout and stderr streams concurrently. It reads lines from each stream and records them in the context. The provided callbacks are called for each line read.
func (*StreamMonitor) Wait ¶
func (sm *StreamMonitor) Wait()
Wait blocks until both stdout and stderr monitoring complete.