adapt

package
v0.26.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: GPL-3.0 Imports: 15 Imported by: 0

Documentation

Overview

Package adapt holds the sensor's adaptive controllers and the one table of their bounds (RFC-074 in openctemio/openctem api/docs/rfcs, RFC-074-adaptive-sensor-control.md).

A controller moves a value from a measured signal, inside a floor and a ceiling. The ceiling is always the lowest of the limits a human or a policy set: a controller may go below it, never above. Every number a controller uses is in this file; code elsewhere refers to the table, never to a literal.

Stability: Experimental (docs/STABILITY.md).

Index

Constants

View Source
const (
	ReasonTimeout    = "timeout"
	ReasonReset      = "reset"
	ReasonRefused    = "refused"
	ReasonTLS        = "tls_handshake"
	ReasonHTTP429    = "http_429"
	ReasonHTTP503    = "http_503"
	ReasonBlocked    = "blocked"
	ReasonCleanRun   = "clean_window"
	ReasonSlowStart  = "slow_start"
	ReasonOpened     = "breaker_open"
	ReasonHalfOpen   = "breaker_half_open"
	ReasonClosed     = "breaker_closed"
	ReasonGaveUp     = "breaker_gave_up"
	ReasonRetryAfter = "retry_after"
)

Reasons of an Event and of a Change.

View Source
const (
	// GapBreakerOpen: the host stopped answering and the breaker refused
	// connections to it (or gave it up).
	GapBreakerOpen = "breaker_open"
	// GapRateFloor: the host kept pushing back at the lowest rate, or asked
	// to wait longer than a connection can be held.
	GapRateFloor = "rate_floor"
	// GapBlocked: the host answered with a block (a run of 403s).
	GapBlocked = "blocked"
)

Gap reasons: why a host's coverage is incomplete.

Variables

View Source
var ErrHostUnavailable = errors.New("adapt: host is backed off")

ErrHostUnavailable is the error of a connection the breaker refused.

View Source
var TargetRate = TargetRateBounds{
	Floor:            1,
	DefaultCeiling:   100,
	StartFraction:    0.25,
	Window:           2 * time.Second,
	IncreaseFraction: 0.10,
	DecreaseFactor:   0.5,
	BreakerFailures:  5,
	BreakerOpen:      15 * time.Second,
	BreakerOpenMax:   2 * time.Minute,
	BreakerJitter:    0.2,
	BreakerMaxTrips:  3,
	ProbeTimeout:     30 * time.Second,
	RetryAfterMax:    5 * time.Minute,
	HoldMax:          10 * time.Second,
	FloorStrikes:     3,
	BlockRun:         20,
	MaxHosts:         65536,
	MaxOpenPorts:     64,
	MaxGaps:          100,
	MaxChanges:       20,
	ToolHostErrors:   30,
}

TargetRate is the table of target-rate bounds. Why each value:

  • Floor 1/s: the slowest pace still useful for a scan in a job's time; slower than that the result is reported as a gap, not stretched.
  • DefaultCeiling 100/s: the forwarder's default before this controller.
  • StartFraction 0.25: a new host is probed at a quarter of the ceiling; doubling per clean window reaches the ceiling in two windows.
  • Window 2s: several round trips to a distant host, and at least two requests at the floor, so a window has evidence before it moves.
  • IncreaseFraction 0.10, DecreaseFactor 0.5: additive increase of a tenth of the ceiling per clean window, halving on push-back; one decrease per window so the replies to a burst already in flight do not collapse the rate.
  • BreakerFailures 5, BreakerOpen 15s doubling to 2m, ±20 % jitter, 3 trips: a host that stops answering is left alone for a while and probed once; after three trips (about 1.5 minutes of silence) it is a coverage gap for the task.
  • RetryAfterMax 5m: the same cap the control client honors.
  • HoldMax 10s: within a tool's usual connect timeout; longer waits are refused instead of hanging the tool.
  • FloorStrikes 3, BlockRun 20: enough repetitions that one refusal or a handful of forbidden paths do not count as a block.
  • ToolHostErrors 30: a tool's own breaker counts every failed request of every concurrent check (not consecutive failures), so it needs a larger budget than BreakerFailures × BreakerMaxTrips (15); 30 lets a few slow checks fail without dropping a live host.

Functions

func Ceiling

func Ceiling(limits ...float64) float64

Ceiling is the lowest of the positive limits (a scan's rate_limit, a scan window's, a program rule's, the local policy's rate.max_rps, the sensor's own setting): the rate a controller never exceeds. Limits of zero or less are "not set"; with none set it is TargetRate.DefaultCeiling.

func FailureReason

func FailureReason(err error) string

FailureReason classifies a network error toward a target as a failure signal (ReasonTimeout, ReasonRefused, ReasonReset), or "" when it says nothing about the target (the task was canceled, or the error is not a network one).

func ParseRetryAfter

func ParseRetryAfter(v string, now time.Time) time.Duration

ParseRetryAfter reads a Retry-After value (delta seconds or an HTTP date), capped at TargetRate.RetryAfterMax; zero when absent or unreadable.

Types

type Change

type Change struct {
	Host   string  `json:"host"`
	From   float64 `json:"from"`
	To     float64 `json:"to"`
	Reason string  `json:"reason"`
}

Change is one move of a host's rate or breaker.

type Event

type Event struct {
	Signal     Signal
	Reason     string
	RetryAfter time.Duration
}

Event is one signal about a host.

type Gap

type Gap struct {
	Host   string `json:"host"`
	Reason string `json:"reason"`
	Detail string `json:"detail,omitempty"`
}

Gap is one host whose coverage is incomplete, and why.

type HostRate

type HostRate struct {

	// OnChange, when set, receives every rate or breaker change (called
	// without the controller's lock held).
	OnChange func(Change)
	// contains filtered or unexported fields
}

HostRate is one task's per-host rate controller: additive increase, multiplicative decrease (AIMD) on each destination host, with a breaker per host. Its state belongs to one task: a forwarder creates it and drops it with the task, so nothing a host taught one task (or one tenant) reaches another.

Signals: a connection or an answer is health; 429 and 503 (with their Retry-After) are congestion; a reset, a timeout, a refused connection on a port that answered before and a failed TLS handshake are failures; a run of 403 answers after other answers is a block. The rate never goes above the ceiling the caller gives (Ceiling of every limit) and never below TargetRate.Floor (or the ceiling, when that is lower).

A host the controller cannot cover (the breaker refused connections to it, it kept pushing back at the floor, or it blocked the scan) is a coverage gap: the task completes and reports it (Gaps), it is not read as "no findings".

func NewHostRate

func NewHostRate(ceiling float64) *HostRate

NewHostRate returns the controller of one task with the given ceiling (Ceiling of every limit that applies; zero or less: the default).

func (*HostRate) Acquire

func (c *HostRate) Acquire(ctx context.Context, host string) error

Acquire waits until a connection (or a request) to host may start, bounded by ctx. It returns ErrHostUnavailable when the host's breaker is open or the host asked to wait longer than TargetRate.HoldMax.

func (*HostRate) Ceiling

func (c *HostRate) Ceiling() float64

Ceiling is the rate the controller never exceeds.

func (*HostRate) Gaps

func (c *HostRate) Gaps() ([]Gap, int)

Gaps are the hosts whose coverage is incomplete (at most TargetRate.MaxGaps) and how many there are in all.

func (*HostRate) Rate

func (c *HostRate) Rate(host string) float64

Rate is host's current rate (the start rate for a host not seen yet).

func (*HostRate) Report

func (c *HostRate) Report(host string, e Event)

Report feeds one signal about host to the controller.

func (*HostRate) ReportDial

func (c *HostRate) ReportDial(host string, port int, err error)

ReportDial feeds the outcome of a connection attempt to host:port. A failure counts only on a port that accepted a connection before in this task: a closed or filtered port of a host is not push-back (a port scan meets thousands of them). Cancellation is not a signal.

func (*HostRate) ReportHTTP

func (c *HostRate) ReportHTTP(host string, status int, retryAfter string)

ReportHTTP feeds one HTTP answer of host (status and its Retry-After header value).

func (*HostRate) Summary

func (c *HostRate) Summary() Summary

Summary is the task's target-rate outcome so far.

type Signal

type Signal int

Signal is what one connection or answer said about its host.

const (
	// Connected: a connection was established.
	Connected Signal = iota + 1
	// Answered: the host answered (an HTTP response other than 403, 429
	// or 503, or a tunnel that ended cleanly after the host sent data).
	Answered
	// Congested: 429 or 503 (Event.RetryAfter set when the host sent one).
	Congested
	// Forbidden: a 403 answer.
	Forbidden
	// Failed: a reset, a timeout, a refused connection on a port that
	// answered before, or a failed TLS handshake (Event.Reason says which).
	Failed
)

type Summary

type Summary struct {
	Ceiling float64 `json:"ceiling"`
	Floor   float64 `json:"floor"`
	Start   float64 `json:"start"`
	// Min is the lowest rate any host reached; Final the lowest rate a
	// host ended at.
	Min   float64 `json:"min"`
	Final float64 `json:"final"`
	Hosts int     `json:"hosts"`
	// Throttled hosts were slowed at least once; Unreachable hosts were
	// given up by their breaker.
	Throttled   int `json:"hosts_throttled"`
	Unreachable int `json:"hosts_unreachable"`
	// Gaps lists at most TargetRate.MaxGaps hosts; GapsTotal counts all.
	Gaps      []Gap `json:"gaps,omitempty"`
	GapsTotal int   `json:"gaps_total,omitempty"`
}

Summary is a task's target-rate outcome, for its result and its log.

type TargetRateBounds

type TargetRateBounds struct {
	// Floor is the lowest rate the controller paces a host at. A host that
	// still pushes back at the floor is reported as a coverage gap
	// (GapRateFloor) instead of being slowed further.
	Floor float64
	// DefaultCeiling is the ceiling when no limit was given at all (the
	// forwarder's historical default; never "unlimited").
	DefaultCeiling float64
	// StartFraction of the ceiling is where a host starts (slow start).
	StartFraction float64
	// Window is how often a host's rate is re-evaluated: a window with
	// answers and no push-back raises the rate.
	Window time.Duration
	// IncreaseFraction of the ceiling is added per clean window once slow
	// start ended (additive increase).
	IncreaseFraction float64
	// DecreaseFactor multiplies the rate on push-back (multiplicative
	// decrease), at most once per window.
	DecreaseFactor float64
	// BreakerFailures consecutive failures open a host's breaker.
	BreakerFailures int
	// BreakerOpen is how long a breaker stays open the first time; it
	// doubles on each further trip up to BreakerOpenMax.
	BreakerOpen    time.Duration
	BreakerOpenMax time.Duration
	// BreakerJitter spreads the open time by ± this fraction.
	BreakerJitter float64
	// BreakerMaxTrips trips give a host up for the rest of the task.
	BreakerMaxTrips int
	// ProbeTimeout is how long a half-open probe may stay unanswered before
	// another one is allowed.
	ProbeTimeout time.Duration
	// RetryAfterMax caps a target's Retry-After.
	RetryAfterMax time.Duration
	// HoldMax is the longest a connection is held to honor a Retry-After;
	// a longer one refuses connections to the host until it passes.
	HoldMax time.Duration
	// FloorStrikes push-backs while at the floor make a host a coverage gap
	// (GapRateFloor).
	FloorStrikes int
	// BlockRun consecutive 403 answers, from a host that answered otherwise
	// before, read as a block.
	BlockRun int
	// MaxHosts bounds the hosts one task tracks; further hosts share one
	// conservative state.
	MaxHosts int
	// MaxOpenPorts bounds the reachable ports remembered per host.
	MaxOpenPorts int
	// MaxGaps bounds the coverage gaps one result lists (the total is
	// still counted).
	MaxGaps int
	// MaxChanges bounds the rate changes one task logs.
	MaxChanges int
	// ToolHostErrors is the per-host error budget handed to a tool that
	// keeps its own host breaker (nuclei -max-host-error): errors on one
	// host, over all its requests, before the tool skips the host.
	ToolHostErrors int
}

TargetRateBounds bound the per-host request rate toward scan targets (HostRate). Rates are requests (connections, and requests on a connection the forwarder reads) per second.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL