Documentation
¶
Overview ¶
Package cancel provides hierarchical cancellation for the CRS algorithm system.
Overview ¶
This package implements a first-class cancellation framework that enables graceful shutdown of algorithms at any level of the hierarchy: session, activity, or individual algorithm. It is designed to prevent hung processes and enable clean resource cleanup.
Architecture ¶
The cancellation system uses a hierarchical context tree:
SessionContext (root) ├── ActivityContext (Search) │ ├── AlgorithmContext (PN-MCTS) │ ├── AlgorithmContext (Zobrist) │ └── AlgorithmContext (UnitProp) ├── ActivityContext (Constraint) │ ├── AlgorithmContext (TMS) │ └── ... └── ...
Cancelling a parent context automatically cancels all children, but children can be cancelled independently without affecting siblings or parents.
Cancellation Triggers ¶
Four types of cancellation are supported:
- User-initiated: Explicit cancel via API, Ctrl+C, or stop button
- Timeout: Algorithm exceeds its configured Timeout() duration
- Deadlock: No progress reported for 3x the ProgressInterval
- Resource limit: Memory or CPU threshold exceeded
Graceful Shutdown Protocol ¶
When cancellation is triggered, the following timeline applies:
T+0ms Signal cancel (set cancellation flag) T+100ms Algorithms should detect ctx.Done() T+500ms Collect partial results from cooperative algorithms T+2000ms Force kill algorithms that haven't responded T+5000ms Generate final report and release resources
Algorithm Contract ¶
Algorithms MUST adhere to the cancellation contract:
- Check ctx.Done() at least every 100ms
- Call ReportProgress() to reset the deadlock timer
- Return partial results when cancelled (if supported)
- Never block indefinitely without checking cancellation
Example algorithm implementation:
func (a *MyAlgorithm) Process(ctx context.Context, snapshot CRSSnapshot, input *Input) (*Output, Delta, error) {
for i := 0; i < iterations; i++ {
// Check for cancellation
select {
case <-ctx.Done():
return a.partialResult(), a.partialDelta(), ctx.Err()
default:
}
// Report progress to avoid deadlock detection
cancel.ReportProgress(ctx)
// Do work...
}
return result, delta, nil
}
Thread Safety ¶
All exported types in this package are safe for concurrent use. The CancellationController uses fine-grained locking to minimize contention.
Metrics ¶
The package exports Prometheus metrics:
- cancel_total: Counter of cancellations by type, level, and reason
- cancel_duration_seconds: Histogram of time from signal to completion
- deadlock_detected_total: Counter of deadlock detections by component
- resource_limit_exceeded_total: Counter of resource violations
- partial_results_collected: Counter of partial results saved
Usage ¶
// Create controller
ctrl := cancel.NewController(cancel.ControllerConfig{
DefaultTimeout: 30 * time.Second,
DeadlockMultiplier: 3,
GracePeriod: 500 * time.Millisecond,
ForceKillTimeout: 2 * time.Second,
})
// Create session
session := ctrl.NewSession(ctx, cancel.SessionConfig{
ID: "session-123",
ResourceLimits: cancel.ResourceLimits{MaxMemoryBytes: 1 << 30},
})
// Create activity context
activityCtx := session.NewActivity("search")
// Create algorithm context
algoCtx := activityCtx.NewAlgorithm("pnmcts", 5*time.Second)
// Cancel at any level
ctrl.Cancel("pnmcts", cancel.CancelReason{Type: cancel.CancelUser, Message: "User requested"})
// or
ctrl.Cancel("search", cancel.CancelReason{Type: cancel.CancelTimeout})
// or
ctrl.CancelAll(cancel.CancelReason{Type: cancel.CancelResourceLimit})
Index ¶
- Variables
- func GetContextID(ctx context.Context) string
- func ReportProgress(ctx context.Context)
- type ActivityContext
- func (a *ActivityContext) Algorithm(name string) *AlgorithmContext
- func (a *ActivityContext) Algorithms() []*AlgorithmContext
- func (a *ActivityContext) Cancel(reason CancelReason)
- func (b *ActivityContext) Context() context.Context
- func (b *ActivityContext) Done() <-chan struct{}
- func (b *ActivityContext) Err() error
- func (b *ActivityContext) ID() string
- func (b *ActivityContext) LastProgress() int64
- func (b *ActivityContext) Level() Level
- func (a *ActivityContext) Name() string
- func (a *ActivityContext) NewAlgorithm(name string, timeout time.Duration) *AlgorithmContext
- func (b *ActivityContext) PartialResult() any
- func (b *ActivityContext) ReportProgress()
- func (a *ActivityContext) Session() *SessionContext
- func (b *ActivityContext) SetPartialCollector(collector PartialResultCollector)
- func (b *ActivityContext) State() State
- func (a *ActivityContext) Status() Status
- type AlgorithmContext
- func (a *AlgorithmContext) Activity() *ActivityContext
- func (b *AlgorithmContext) Cancel(reason CancelReason)
- func (b *AlgorithmContext) Context() context.Context
- func (b *AlgorithmContext) Done() <-chan struct{}
- func (b *AlgorithmContext) Err() error
- func (b *AlgorithmContext) ID() string
- func (b *AlgorithmContext) LastProgress() int64
- func (b *AlgorithmContext) Level() Level
- func (a *AlgorithmContext) MarkDone()
- func (a *AlgorithmContext) Name() string
- func (b *AlgorithmContext) PartialResult() any
- func (b *AlgorithmContext) ReportProgress()
- func (b *AlgorithmContext) SetPartialCollector(collector PartialResultCollector)
- func (b *AlgorithmContext) State() State
- func (a *AlgorithmContext) Status() Status
- func (a *AlgorithmContext) Timeout() time.Duration
- type CancelReason
- type CancelType
- type Cancellable
- type CancellationController
- func (c *CancellationController) Cancel(id string, reason CancelReason) error
- func (c *CancellationController) CancelAll(reason CancelReason)
- func (c *CancellationController) Close() error
- func (c *CancellationController) GetContext(id string) (Cancellable, bool)
- func (c *CancellationController) GetSession(id string) (*SessionContext, bool)
- func (c *CancellationController) NewSession(parent context.Context, config SessionConfig) (*SessionContext, error)
- func (c *CancellationController) Shutdown(ctx context.Context) (*ShutdownResult, error)
- func (c *CancellationController) Status() *ControllerStatus
- type Controller
- type ControllerConfig
- type ControllerStatus
- type DeadlockDetector
- type Level
- type Metrics
- type MetricsConfig
- type PartialResultCollector
- type ProgressReporter
- type ResourceLimits
- type ResourceMonitor
- type SessionConfig
- type SessionContext
- func (s *SessionContext) Activities() []*ActivityContext
- func (s *SessionContext) Activity(name string) *ActivityContext
- func (s *SessionContext) Cancel(reason CancelReason)
- func (b *SessionContext) Context() context.Context
- func (s *SessionContext) Done() <-chan struct{}
- func (b *SessionContext) Err() error
- func (b *SessionContext) ID() string
- func (b *SessionContext) LastProgress() int64
- func (b *SessionContext) Level() Level
- func (s *SessionContext) NewActivity(name string) *ActivityContext
- func (b *SessionContext) PartialResult() any
- func (s *SessionContext) ProgressInterval() time.Duration
- func (b *SessionContext) ReportProgress()
- func (b *SessionContext) SetPartialCollector(collector PartialResultCollector)
- func (b *SessionContext) State() State
- func (s *SessionContext) Status() Status
- type ShutdownCoordinator
- type ShutdownPhase
- type ShutdownResult
- type State
- type Status
- type TimeoutEnforcer
Constants ¶
This section is empty.
Variables ¶
var ( // ErrControllerClosed is returned when operations are attempted on a closed controller. ErrControllerClosed = errors.New("cancellation controller is closed") // ErrSessionNotFound is returned when a session ID is not found. ErrSessionNotFound = errors.New("session not found") // ErrActivityNotFound is returned when an activity is not found. ErrActivityNotFound = errors.New("activity not found") // ErrAlgorithmNotFound is returned when an algorithm is not found. ErrAlgorithmNotFound = errors.New("algorithm not found") // ErrAlreadyCancelled is returned when attempting to cancel an already cancelled context. ErrAlreadyCancelled = errors.New("context already cancelled") // ErrInvalidConfig is returned when configuration is invalid. ErrInvalidConfig = errors.New("invalid configuration") // ErrNilContext is returned when a nil context is provided. ErrNilContext = errors.New("context must not be nil") )
Functions ¶
func GetContextID ¶
GetContextID returns the context ID from the context, if available.
func ReportProgress ¶
ReportProgress reports progress from within an algorithm. This resets the deadlock detection timer.
Description:
Algorithms should call this function periodically to indicate they are making progress. If no progress is reported within the deadlock threshold, the algorithm will be automatically cancelled.
Inputs:
- ctx: The context passed to the algorithm's Process method.
Example:
func (a *MyAlgo) Process(ctx context.Context, ...) {
for {
cancel.ReportProgress(ctx) // Report progress
// ... do work ...
}
}
Types ¶
type ActivityContext ¶
type ActivityContext struct {
// contains filtered or unexported fields
}
ActivityContext groups related algorithms within a session.
Thread Safety: Safe for concurrent use.
func (*ActivityContext) Algorithm ¶
func (a *ActivityContext) Algorithm(name string) *AlgorithmContext
Algorithm returns the algorithm with the given name, or nil if not found.
func (*ActivityContext) Algorithms ¶
func (a *ActivityContext) Algorithms() []*AlgorithmContext
Algorithms returns all algorithm contexts.
func (*ActivityContext) Cancel ¶
func (a *ActivityContext) Cancel(reason CancelReason)
Cancel cancels this activity and all its algorithms.
func (*ActivityContext) Done ¶
func (b *ActivityContext) Done() <-chan struct{}
Done returns a channel that is closed when this context is cancelled or done.
func (*ActivityContext) Err ¶
func (b *ActivityContext) Err() error
Err returns the error after Done is closed.
func (*ActivityContext) ID ¶
func (b *ActivityContext) ID() string
ID returns the unique identifier for this context.
func (*ActivityContext) LastProgress ¶
func (b *ActivityContext) LastProgress() int64
LastProgress returns the last progress timestamp.
func (*ActivityContext) Level ¶
func (b *ActivityContext) Level() Level
Level returns the hierarchical level.
func (*ActivityContext) Name ¶
func (a *ActivityContext) Name() string
Name returns the activity name.
func (*ActivityContext) NewAlgorithm ¶
func (a *ActivityContext) NewAlgorithm(name string, timeout time.Duration) *AlgorithmContext
NewAlgorithm creates a new algorithm context within this activity.
Description:
Creates an algorithm context for a specific algorithm execution. The algorithm inherits cancellation from this activity.
Inputs:
- name: Unique name for the algorithm within this activity.
- timeout: Maximum execution time. Zero uses the controller default.
Outputs:
- *AlgorithmContext: The created algorithm context. Never nil.
Thread Safety: Safe for concurrent use.
func (*ActivityContext) PartialResult ¶
func (b *ActivityContext) PartialResult() any
PartialResult returns any collected partial result.
func (*ActivityContext) ReportProgress ¶
func (b *ActivityContext) ReportProgress()
ReportProgress updates the last progress timestamp.
func (*ActivityContext) Session ¶
func (a *ActivityContext) Session() *SessionContext
Session returns the parent session.
func (*ActivityContext) SetPartialCollector ¶
func (b *ActivityContext) SetPartialCollector(collector PartialResultCollector)
SetPartialCollector sets the function to collect partial results.
func (*ActivityContext) State ¶
func (b *ActivityContext) State() State
State returns the current state.
func (*ActivityContext) Status ¶
func (a *ActivityContext) Status() Status
Status returns the current status including all children.
type AlgorithmContext ¶
type AlgorithmContext struct {
// contains filtered or unexported fields
}
AlgorithmContext represents a single algorithm execution.
Thread Safety: Safe for concurrent use.
func (*AlgorithmContext) Activity ¶
func (a *AlgorithmContext) Activity() *ActivityContext
Activity returns the parent activity.
func (*AlgorithmContext) Cancel ¶
func (b *AlgorithmContext) Cancel(reason CancelReason)
Cancel initiates cancellation with the given reason.
func (*AlgorithmContext) Done ¶
func (b *AlgorithmContext) Done() <-chan struct{}
Done returns a channel that is closed when this context is cancelled or done.
func (*AlgorithmContext) Err ¶
func (b *AlgorithmContext) Err() error
Err returns the error after Done is closed.
func (*AlgorithmContext) ID ¶
func (b *AlgorithmContext) ID() string
ID returns the unique identifier for this context.
func (*AlgorithmContext) LastProgress ¶
func (b *AlgorithmContext) LastProgress() int64
LastProgress returns the last progress timestamp.
func (*AlgorithmContext) Level ¶
func (b *AlgorithmContext) Level() Level
Level returns the hierarchical level.
func (*AlgorithmContext) MarkDone ¶
func (a *AlgorithmContext) MarkDone()
MarkDone marks the algorithm as normally completed.
func (*AlgorithmContext) Name ¶
func (a *AlgorithmContext) Name() string
Name returns the algorithm name.
func (*AlgorithmContext) PartialResult ¶
func (b *AlgorithmContext) PartialResult() any
PartialResult returns any collected partial result.
func (*AlgorithmContext) ReportProgress ¶
func (b *AlgorithmContext) ReportProgress()
ReportProgress updates the last progress timestamp.
func (*AlgorithmContext) SetPartialCollector ¶
func (b *AlgorithmContext) SetPartialCollector(collector PartialResultCollector)
SetPartialCollector sets the function to collect partial results.
func (*AlgorithmContext) State ¶
func (b *AlgorithmContext) State() State
State returns the current state.
func (*AlgorithmContext) Status ¶
func (a *AlgorithmContext) Status() Status
Status returns the current status.
func (*AlgorithmContext) Timeout ¶
func (a *AlgorithmContext) Timeout() time.Duration
Timeout returns the configured timeout.
type CancelReason ¶
type CancelReason struct {
// Type indicates the category of cancellation.
Type CancelType
// Message provides a human-readable description.
Message string
// Threshold describes the limit that was exceeded (for timeout/resource cancellations).
// Example: "memory > 1GB" or "timeout > 5s"
Threshold string
// Component identifies which component triggered the cancellation.
Component string
// Timestamp is when the cancellation was triggered (Unix milliseconds UTC).
Timestamp int64
}
CancelReason describes why cancellation occurred.
type CancelType ¶
type CancelType int
CancelType indicates why cancellation occurred.
const ( // CancelUser indicates user-initiated cancellation (API, Ctrl+C, stop button). CancelUser CancelType = iota // CancelTimeout indicates the algorithm exceeded its configured timeout. CancelTimeout // CancelDeadlock indicates no progress was reported within the deadlock threshold. CancelDeadlock // CancelResourceLimit indicates memory or CPU limits were exceeded. CancelResourceLimit // CancelParent indicates the parent context was cancelled. CancelParent // CancelShutdown indicates system shutdown is in progress. CancelShutdown )
func (CancelType) String ¶
func (t CancelType) String() string
String returns the string representation of the cancel type.
type Cancellable ¶
type Cancellable interface {
// ID returns the unique identifier for this context.
ID() string
// Level returns the hierarchical level (session, activity, algorithm).
Level() Level
// State returns the current state.
State() State
// Context returns the underlying context.Context.
Context() context.Context
// Cancel initiates cancellation with the given reason.
Cancel(reason CancelReason)
// Done returns a channel that is closed when this context is cancelled or done.
Done() <-chan struct{}
// Err returns the error after Done is closed.
Err() error
// Status returns the current status.
Status() Status
}
Cancellable represents any context that can be cancelled.
Thread Safety: Safe for concurrent use.
type CancellationController ¶
type CancellationController struct {
// contains filtered or unexported fields
}
CancellationController manages hierarchical cancellation for the CRS system.
Thread Safety: Safe for concurrent use.
func GetController ¶
func GetController(ctx context.Context) *CancellationController
GetController returns the CancellationController from the context, if available.
func NewController ¶
func NewController(config ControllerConfig, logger *slog.Logger) (*CancellationController, error)
NewController creates a new CancellationController.
Description:
Creates and initializes a new cancellation controller with the given configuration. The controller starts background goroutines for deadlock detection and resource monitoring.
Inputs:
- config: Controller configuration. Zero values use defaults.
- logger: Logger for cancellation events. If nil, uses slog.Default().
Outputs:
- *CancellationController: The created controller. Never nil.
- error: Non-nil if configuration is invalid.
Example:
ctrl, err := cancel.NewController(cancel.ControllerConfig{
DefaultTimeout: 30 * time.Second,
}, slog.Default())
if err != nil {
return err
}
defer ctrl.Close()
Thread Safety: The returned controller is safe for concurrent use.
func (*CancellationController) Cancel ¶
func (c *CancellationController) Cancel(id string, reason CancelReason) error
Cancel initiates cancellation for the specified target.
Description:
Cancels the context with the given ID. The ID can refer to a session, activity, or algorithm. Cancellation propagates to all children.
Inputs:
- id: The ID of the context to cancel.
- reason: The reason for cancellation.
Outputs:
- error: Non-nil if the ID is not found or controller is closed.
Thread Safety: Safe for concurrent use.
func (*CancellationController) CancelAll ¶
func (c *CancellationController) CancelAll(reason CancelReason)
CancelAll cancels all active contexts immediately.
Description:
Emergency stop for all sessions, activities, and algorithms.
Inputs:
- reason: The reason for cancellation.
Thread Safety: Safe for concurrent use.
func (*CancellationController) Close ¶
func (c *CancellationController) Close() error
Close releases all resources held by the controller.
Description:
Should be called when the controller is no longer needed. Calls Shutdown internally if not already called.
Outputs:
- error: Non-nil if shutdown encountered errors.
Thread Safety: Safe for concurrent use. Idempotent.
func (*CancellationController) GetContext ¶
func (c *CancellationController) GetContext(id string) (Cancellable, bool)
GetContext returns the context with the given ID.
func (*CancellationController) GetSession ¶
func (c *CancellationController) GetSession(id string) (*SessionContext, bool)
GetSession returns the session with the given ID.
func (*CancellationController) NewSession ¶
func (c *CancellationController) NewSession(parent context.Context, config SessionConfig) (*SessionContext, error)
NewSession creates a new cancellable session context.
Description:
Creates a top-level session context that can contain activities and algorithms. The session inherits cancellation from the parent context.
Inputs:
- parent: Parent context. Must not be nil.
- config: Session configuration. ID is required.
Outputs:
- *SessionContext: The created session context. Never nil on success.
- error: Non-nil if parent is nil, controller is closed, or config is invalid.
Thread Safety: Safe for concurrent use.
func (*CancellationController) Shutdown ¶
func (c *CancellationController) Shutdown(ctx context.Context) (*ShutdownResult, error)
Shutdown gracefully shuts down the controller.
Description:
Cancels all contexts, waits for graceful shutdown, then force kills any remaining contexts. Blocks until complete or context is cancelled.
Inputs:
- ctx: Context for the shutdown operation itself.
Outputs:
- *ShutdownResult: Results of the shutdown operation.
- error: Non-nil if ctx was cancelled before shutdown completed.
Thread Safety: Safe for concurrent use. Only the first call performs shutdown.
func (*CancellationController) Status ¶
func (c *CancellationController) Status() *ControllerStatus
Status returns the current status of all contexts.
Description:
Returns a snapshot of the current state of all sessions and their children.
Outputs:
- *ControllerStatus: Current status. Never nil.
Thread Safety: Safe for concurrent use. Returns a snapshot.
type Controller ¶
type Controller interface {
// NewSession creates a new cancellable session context.
//
// Description:
// Creates a top-level session context that can contain activities and algorithms.
// The session inherits cancellation from the parent context.
//
// Inputs:
// - parent: Parent context. Must not be nil.
// - config: Session configuration. ID is required.
//
// Outputs:
// - *SessionContext: The created session context. Never nil on success.
// - error: Non-nil if parent is nil or config is invalid.
//
// Thread Safety: Safe for concurrent use.
NewSession(parent context.Context, config SessionConfig) (*SessionContext, error)
// Cancel initiates cancellation for the specified target.
//
// Description:
// Cancels the context with the given ID. The ID can refer to a session,
// activity, or algorithm. Cancellation propagates to all children.
//
// Inputs:
// - id: The ID of the context to cancel.
// - reason: The reason for cancellation.
//
// Outputs:
// - error: Non-nil if the ID is not found.
//
// Thread Safety: Safe for concurrent use.
Cancel(id string, reason CancelReason) error
// CancelAll cancels all active contexts immediately.
//
// Description:
// Emergency stop for all sessions, activities, and algorithms.
//
// Inputs:
// - reason: The reason for cancellation.
//
// Thread Safety: Safe for concurrent use.
CancelAll(reason CancelReason)
// Status returns the current status of all contexts.
//
// Description:
// Returns a snapshot of the current state of all sessions and their children.
//
// Outputs:
// - *ControllerStatus: Current status. Never nil.
//
// Thread Safety: Safe for concurrent use. Returns a snapshot.
Status() *ControllerStatus
// Shutdown gracefully shuts down the controller.
//
// Description:
// Cancels all contexts, waits for graceful shutdown, then force kills
// any remaining contexts. Blocks until complete or context is cancelled.
//
// Inputs:
// - ctx: Context for the shutdown operation itself.
//
// Outputs:
// - *ShutdownResult: Results of the shutdown operation.
// - error: Non-nil if ctx was cancelled before shutdown completed.
//
// Thread Safety: Safe for concurrent use. Only the first call performs shutdown.
Shutdown(ctx context.Context) (*ShutdownResult, error)
// Close releases all resources held by the controller.
//
// Description:
// Should be called when the controller is no longer needed.
// Calls Shutdown internally if not already called.
//
// Outputs:
// - error: Non-nil if shutdown encountered errors.
//
// Thread Safety: Safe for concurrent use. Idempotent.
Close() error
}
Controller manages hierarchical cancellation across sessions, activities, and algorithms.
Thread Safety: Safe for concurrent use.
type ControllerConfig ¶
type ControllerConfig struct {
// DefaultTimeout is applied to algorithms that don't specify their own.
// Must be > 0. Default: 30 seconds.
DefaultTimeout time.Duration
// DeadlockMultiplier determines when deadlock is detected.
// Deadlock is detected after DeadlockMultiplier * ProgressInterval with no progress.
// Must be >= 2. Default: 3.
DeadlockMultiplier int
// GracePeriod is how long to wait for graceful shutdown before force kill.
// Must be > 0. Default: 500 milliseconds.
GracePeriod time.Duration
// ForceKillTimeout is the maximum time to wait for force kill to complete.
// Must be > GracePeriod. Default: 2 seconds.
ForceKillTimeout time.Duration
// ProgressCheckInterval is how often the deadlock detector checks for progress.
// Must be > 0. Default: 100 milliseconds.
ProgressCheckInterval time.Duration
// EnableMetrics enables Prometheus metrics collection.
// Default: true.
EnableMetrics bool
}
ControllerConfig configures the CancellationController.
func (*ControllerConfig) ApplyDefaults ¶
func (c *ControllerConfig) ApplyDefaults()
ApplyDefaults fills in zero values with sensible defaults.
func (*ControllerConfig) Validate ¶
func (c *ControllerConfig) Validate() error
Validate checks if the configuration is valid.
Description:
Validates all configuration fields and applies defaults where needed.
Outputs:
- error: Non-nil if configuration is invalid.
type ControllerStatus ¶
type ControllerStatus struct {
// Sessions contains the status of all active sessions.
Sessions []Status
// TotalActive is the count of non-terminal contexts.
TotalActive int
// TotalCancelled is the count of cancelled contexts.
TotalCancelled int
// TotalCompleted is the count of normally completed contexts.
TotalCompleted int
}
ControllerStatus provides the overall status of the cancellation controller.
type DeadlockDetector ¶
type DeadlockDetector struct {
// contains filtered or unexported fields
}
DeadlockDetector monitors contexts for progress and detects deadlocks.
A deadlock is detected when a context has not reported progress for DeadlockMultiplier * ProgressInterval.
Thread Safety: Safe for concurrent use.
func NewDeadlockDetector ¶
func NewDeadlockDetector(controller *CancellationController, checkInterval time.Duration, multiplier int) *DeadlockDetector
NewDeadlockDetector creates a new deadlock detector.
Description:
Creates a deadlock detector that monitors all contexts registered with the controller and cancels any that haven't reported progress.
Inputs:
- controller: The cancellation controller to monitor.
- checkInterval: How often to check for deadlocks.
- multiplier: Deadlock threshold = multiplier * context's ProgressInterval.
Outputs:
- *DeadlockDetector: The created detector. Never nil.
func (*DeadlockDetector) Run ¶
func (d *DeadlockDetector) Run(stopCh <-chan struct{}, wg *sync.WaitGroup)
Run starts the deadlock detection loop.
Description:
Periodically checks all contexts for progress. If a context hasn't reported progress within the deadlock threshold, it is cancelled.
Inputs:
- stopCh: Channel that signals the detector to stop.
- wg: WaitGroup to signal when the detector has stopped.
Thread Safety: Should only be called once. Safe when called from a goroutine.
type Metrics ¶
type Metrics struct {
// CancelTotal counts cancellations by type, level, and component.
CancelTotal *prometheus.CounterVec
// CancelAllTotal counts emergency cancel-all operations.
CancelAllTotal prometheus.Counter
// CancelDurationSeconds measures the time from signal to completion.
CancelDurationSeconds *prometheus.HistogramVec
// DeadlockDetectedTotal counts deadlock detections by component.
DeadlockDetectedTotal *prometheus.CounterVec
// ResourceLimitExceededTotal counts resource limit violations.
ResourceLimitExceededTotal *prometheus.CounterVec
// TimeoutTotal counts algorithm timeouts by component.
TimeoutTotal *prometheus.CounterVec
// PartialResultsCollected counts partial results collected during shutdown.
PartialResultsCollected prometheus.Counter
// ForceKilledTotal counts contexts that had to be force-killed.
ForceKilledTotal prometheus.Counter
// SessionsCreated counts sessions created.
SessionsCreated prometheus.Counter
// ActiveContexts is a gauge of currently active (non-terminal) contexts.
ActiveContexts *prometheus.GaugeVec
// GracefulShutdownDurationSeconds measures total shutdown duration.
GracefulShutdownDurationSeconds prometheus.Histogram
// ProgressReportsTotal counts progress reports by component.
ProgressReportsTotal *prometheus.CounterVec
}
Metrics holds all Prometheus metrics for the cancellation framework.
Thread Safety: Safe for concurrent use (Prometheus metrics are thread-safe).
func NewMetrics ¶
func NewMetrics() *Metrics
NewMetrics creates and registers all cancellation metrics.
Description:
Creates a new Metrics instance with all Prometheus metrics registered. Uses promauto for automatic registration with the default registerer.
Outputs:
- *Metrics: The created metrics. Never nil.
Thread Safety: Safe for concurrent use.
type MetricsConfig ¶
type MetricsConfig struct {
// Enabled determines if metrics are collected at all.
Enabled bool
// Namespace is the Prometheus namespace prefix.
// Default: "trace"
Namespace string
// Subsystem is the Prometheus subsystem prefix.
// Default: "cancel"
Subsystem string
}
MetricsConfig configures which metrics to enable.
func DefaultMetricsConfig ¶
func DefaultMetricsConfig() MetricsConfig
DefaultMetricsConfig returns the default metrics configuration.
type PartialResultCollector ¶
PartialResultCollector is called during graceful shutdown to collect partial results.
type ProgressReporter ¶
type ProgressReporter func()
ProgressReporter is called by algorithms to report progress. This resets the deadlock detection timer.
type ResourceLimits ¶
type ResourceLimits struct {
// MaxMemoryBytes is the maximum memory usage before triggering cancellation.
// Zero means no limit.
MaxMemoryBytes int64
// MaxCPUPercent is the maximum CPU usage (0-100) before triggering cancellation.
// Zero means no limit.
MaxCPUPercent float64
// MaxGoroutines is the maximum number of goroutines before triggering cancellation.
// Zero means no limit.
MaxGoroutines int
}
ResourceLimits defines resource constraints for cancellation.
func (*ResourceLimits) HasLimits ¶
func (r *ResourceLimits) HasLimits() bool
HasLimits returns true if any limits are configured.
func (*ResourceLimits) Validate ¶
func (r *ResourceLimits) Validate() error
Validate checks if resource limits are valid.
type ResourceMonitor ¶
type ResourceMonitor struct {
// contains filtered or unexported fields
}
ResourceMonitor monitors resource usage and cancels contexts that exceed limits.
Thread Safety: Safe for concurrent use.
func NewResourceMonitor ¶
func NewResourceMonitor(controller *CancellationController) *ResourceMonitor
NewResourceMonitor creates a new resource monitor.
Description:
Creates a resource monitor that watches for memory, CPU, and goroutine limit violations.
Inputs:
- controller: The cancellation controller to use for cancellation.
Outputs:
- *ResourceMonitor: The created monitor. Never nil.
func (*ResourceMonitor) MonitorSession ¶
func (m *ResourceMonitor) MonitorSession(session *SessionContext, stopCh <-chan struct{}, wg *sync.WaitGroup)
MonitorSession monitors a session's resource usage.
Description:
Periodically checks if the session exceeds its configured resource limits. If limits are exceeded, the session is cancelled.
Inputs:
- session: The session to monitor.
- stopCh: Channel that signals the monitor to stop.
- wg: WaitGroup to signal when the monitor has stopped.
Thread Safety: Should only be called once per session.
type SessionConfig ¶
type SessionConfig struct {
// ID is the unique identifier for this session.
// Required.
ID string
// ResourceLimits defines resource constraints for this session.
// Optional. Zero values mean no limit.
ResourceLimits ResourceLimits
// Timeout overrides the controller's default timeout for this session.
// Zero means use the controller default.
Timeout time.Duration
// ProgressInterval is how often algorithms should report progress.
// Zero means use the default (1 second).
ProgressInterval time.Duration
}
SessionConfig configures a new session context.
func (*SessionConfig) ApplyDefaults ¶
func (c *SessionConfig) ApplyDefaults()
ApplyDefaults fills in zero values with sensible defaults.
func (*SessionConfig) Validate ¶
func (c *SessionConfig) Validate() error
Validate checks if the session configuration is valid.
type SessionContext ¶
type SessionContext struct {
// contains filtered or unexported fields
}
SessionContext is the top-level cancellable context for an MCTS session.
Thread Safety: Safe for concurrent use.
func (*SessionContext) Activities ¶
func (s *SessionContext) Activities() []*ActivityContext
Activities returns all activity contexts.
func (*SessionContext) Activity ¶
func (s *SessionContext) Activity(name string) *ActivityContext
Activity returns the activity with the given name, or nil if not found.
func (*SessionContext) Cancel ¶
func (s *SessionContext) Cancel(reason CancelReason)
Cancel cancels this session and all its activities and algorithms.
func (*SessionContext) Done ¶
func (s *SessionContext) Done() <-chan struct{}
Done marks the session as normally completed.
func (*SessionContext) Err ¶
func (b *SessionContext) Err() error
Err returns the error after Done is closed.
func (*SessionContext) ID ¶
func (b *SessionContext) ID() string
ID returns the unique identifier for this context.
func (*SessionContext) LastProgress ¶
func (b *SessionContext) LastProgress() int64
LastProgress returns the last progress timestamp.
func (*SessionContext) Level ¶
func (b *SessionContext) Level() Level
Level returns the hierarchical level.
func (*SessionContext) NewActivity ¶
func (s *SessionContext) NewActivity(name string) *ActivityContext
NewActivity creates a new activity context within this session.
Description:
Creates an activity context that groups related algorithms. The activity inherits cancellation from this session.
Inputs:
- name: Unique name for the activity within this session.
Outputs:
- *ActivityContext: The created activity context. Never nil.
Thread Safety: Safe for concurrent use.
func (*SessionContext) PartialResult ¶
func (b *SessionContext) PartialResult() any
PartialResult returns any collected partial result.
func (*SessionContext) ProgressInterval ¶
func (s *SessionContext) ProgressInterval() time.Duration
ProgressInterval returns the configured progress interval.
func (*SessionContext) ReportProgress ¶
func (b *SessionContext) ReportProgress()
ReportProgress updates the last progress timestamp.
func (*SessionContext) SetPartialCollector ¶
func (b *SessionContext) SetPartialCollector(collector PartialResultCollector)
SetPartialCollector sets the function to collect partial results.
func (*SessionContext) State ¶
func (b *SessionContext) State() State
State returns the current state.
func (*SessionContext) Status ¶
func (s *SessionContext) Status() Status
Status returns the current status including all children.
type ShutdownCoordinator ¶
type ShutdownCoordinator struct {
// contains filtered or unexported fields
}
ShutdownCoordinator manages the graceful shutdown protocol.
The protocol follows this timeline:
T+0ms Signal cancel (set cancellation flag) T+100ms Algorithms should detect ctx.Done() T+500ms Collect partial results from cooperative algorithms T+2000ms Force kill algorithms that haven't responded T+5000ms Generate final report and release resources
Thread Safety: Safe for concurrent use.
func NewShutdownCoordinator ¶
func NewShutdownCoordinator(controller *CancellationController, gracePeriod, forceKillTimeout time.Duration) *ShutdownCoordinator
NewShutdownCoordinator creates a new shutdown coordinator.
Description:
Creates a coordinator that manages the graceful shutdown protocol with configurable timeouts.
Inputs:
- controller: The cancellation controller.
- gracePeriod: Time to wait for graceful shutdown before collecting partial results.
- forceKillTimeout: Time to wait after grace period before force killing.
Outputs:
- *ShutdownCoordinator: The created coordinator. Never nil.
func (*ShutdownCoordinator) Execute ¶
func (s *ShutdownCoordinator) Execute(ctx context.Context, reason CancelReason) (*ShutdownResult, error)
Execute runs the graceful shutdown protocol.
Description:
Executes all phases of the shutdown protocol in order: 1. Signal all contexts to cancel 2. Wait for grace period, collecting partial results 3. Force kill remaining contexts 4. Generate and return the shutdown report
Inputs:
- ctx: Context for the shutdown operation. If cancelled, shutdown aborts.
- reason: The reason for shutdown.
Outputs:
- *ShutdownResult: The results of the shutdown operation.
- error: Non-nil if ctx was cancelled or another error occurred.
Thread Safety: Safe for concurrent use. Only the first call executes shutdown.
func (*ShutdownCoordinator) Phase ¶
func (s *ShutdownCoordinator) Phase() ShutdownPhase
Phase returns the current shutdown phase.
func (*ShutdownCoordinator) WaitForCompletion ¶
func (s *ShutdownCoordinator) WaitForCompletion(ctx context.Context, checkInterval time.Duration) error
WaitForCompletion waits for all contexts to reach a terminal state.
Description:
Blocks until all contexts are either cancelled or done, or until the provided context is cancelled.
Inputs:
- ctx: Context for the wait operation.
- checkInterval: How often to check context states.
Outputs:
- error: Non-nil if ctx was cancelled before all contexts completed.
type ShutdownPhase ¶
type ShutdownPhase int
ShutdownPhase represents a phase in the graceful shutdown protocol.
const ( // PhaseSignal is the initial phase where cancel signals are sent. PhaseSignal ShutdownPhase = iota // PhaseCollect is when partial results are being collected. PhaseCollect // PhaseForceKill is when non-responsive contexts are force-terminated. PhaseForceKill // PhaseReport is when the shutdown report is being generated. PhaseReport // PhaseComplete indicates shutdown is complete. PhaseComplete )
func (ShutdownPhase) String ¶
func (p ShutdownPhase) String() string
String returns the string representation of the shutdown phase.
type ShutdownResult ¶
type ShutdownResult struct {
// Success is true if all contexts were cleanly shut down.
Success bool
// Duration is how long the shutdown took.
Duration time.Duration
// PartialResultsCollected is the count of algorithms that returned partial results.
PartialResultsCollected int
// ForceKilled is the count of algorithms that had to be force killed.
ForceKilled int
// Errors contains any errors encountered during shutdown.
Errors []error
}
ShutdownResult contains the results of a graceful shutdown.
type State ¶
type State int
State represents the current state of a cancellable context.
const ( // StateRunning indicates the context is actively running. StateRunning State = iota // StateCancelling indicates cancellation has been signaled but cleanup is in progress. StateCancelling // StateCancelled indicates cancellation and cleanup are complete. StateCancelled // StateDone indicates normal completion (not cancelled). StateDone )
func (State) IsTerminal ¶
IsTerminal returns true if this is a terminal state.
type Status ¶
type Status struct {
// ID is the unique identifier of this context.
ID string
// Level indicates whether this is a session, activity, or algorithm.
Level Level
// State is the current state.
State State
// CancelReason is set if State is Cancelling or Cancelled.
CancelReason *CancelReason
// StartTime is when this context was created (Unix milliseconds UTC).
StartTime int64
// LastProgress is the last time progress was reported (Unix milliseconds UTC).
LastProgress int64
// Duration is how long this context has been running.
Duration time.Duration
// Children contains the status of child contexts (if any).
Children []Status
// PartialResultsAvailable indicates if partial results were collected.
PartialResultsAvailable bool
}
Status provides the current status of a cancellable context.
type TimeoutEnforcer ¶
type TimeoutEnforcer struct {
// contains filtered or unexported fields
}
TimeoutEnforcer monitors algorithm timeouts and cancels expired algorithms. This is integrated into the context creation with context.WithTimeout, but we provide additional monitoring for algorithms that don't respect context cancellation.
Thread Safety: Safe for concurrent use.
func NewTimeoutEnforcer ¶
func NewTimeoutEnforcer(controller *CancellationController) *TimeoutEnforcer
NewTimeoutEnforcer creates a new timeout enforcer.
func (*TimeoutEnforcer) EnforceTimeout ¶
func (t *TimeoutEnforcer) EnforceTimeout(ctx *AlgorithmContext, timeout time.Duration, gracePeriod time.Duration)
EnforceTimeout monitors an algorithm and force-cancels it if the timeout is exceeded and the algorithm hasn't responded to context cancellation.
Description:
Waits for the algorithm to complete or timeout. If the algorithm doesn't respond to context cancellation within the grace period, it is force-cancelled.
Inputs:
- ctx: The algorithm context to monitor.
- timeout: The timeout duration.
- gracePeriod: How long to wait for graceful cancellation before force-kill.
Thread Safety: Should only be called once per algorithm context.