Help us shape the future of Go! Take the 2026 Go Developer Survey

adapt

package
v0.37.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package adapt holds the sensor's adaptive controllers and the one table of their bounds (RFC-074 in openctemio/openctem api/docs/rfcs, RFC-074-adaptive-sensor-control.md).

A controller moves a value from a measured signal, inside a floor and a ceiling. The ceiling is always the lowest of the limits a human or a policy set: a controller may go below it, never above. Every number a controller uses is in this file; code elsewhere refers to the table, never to a literal.

Stability: Experimental (docs/STABILITY.md).

Index

Constants

View Source
const (
	ReasonTimeout    = "timeout"
	ReasonReset      = "reset"
	ReasonRefused    = "refused"
	ReasonTLS        = "tls_handshake"
	ReasonHTTP429    = "http_429"
	ReasonHTTP503    = "http_503"
	ReasonBlocked    = "blocked"
	ReasonCleanRun   = "clean_window"
	ReasonSlowStart  = "slow_start"
	ReasonOpened     = "breaker_open"
	ReasonHalfOpen   = "breaker_half_open"
	ReasonClosed     = "breaker_closed"
	ReasonGaveUp     = "breaker_gave_up"
	ReasonRetryAfter = "retry_after"
)

Reasons of an Event and of a Change.

View Source
const (
	// GapBreakerOpen: the host stopped answering and the breaker refused
	// connections to it (or gave it up).
	GapBreakerOpen = "breaker_open"
	// GapRateFloor: the host kept pushing back at the lowest rate, or asked
	// to wait longer than a connection can be held.
	GapRateFloor = "rate_floor"
	// GapBlocked: the host answered with a block (a run of 403s).
	GapBlocked = "blocked"
)

Gap reasons: why a host's coverage is incomplete.

Variables

View Source
var ErrHostUnavailable = errors.New("adapt: host is backed off")

ErrHostUnavailable is the error of a connection the breaker refused.

View Source
var TargetRate = TargetRateBounds{
	Floor:            1,
	DefaultCeiling:   100,
	StartFraction:    0.25,
	Window:           2 * time.Second,
	IncreaseFraction: 0.10,
	DecreaseFactor:   0.5,
	BreakerFailures:  5,
	BreakerOpen:      15 * time.Second,
	BreakerOpenMax:   2 * time.Minute,
	BreakerJitter:    0.2,
	BreakerMaxTrips:  3,
	ProbeTimeout:     30 * time.Second,
	RetryAfterMax:    5 * time.Minute,
	HoldMax:          10 * time.Second,
	FloorStrikes:     3,
	BlockRun:         20,
	MaxHosts:         65536,
	MaxOpenPorts:     64,
	MaxGaps:          100,
	MaxChanges:       20,
	ToolHostErrors:   30,
}

TargetRate is the table of target-rate bounds. Why each value:

  • Floor 1/s: the slowest pace still useful for a scan in a job's time; slower than that the result is reported as a gap, not stretched.
  • DefaultCeiling 100/s: the forwarder's default before this controller.
  • StartFraction 0.25: a new host is probed at a quarter of the ceiling; doubling per clean window reaches the ceiling in two windows.
  • Window 2s: several round trips to a distant host, and at least two requests at the floor, so a window has evidence before it moves.
  • IncreaseFraction 0.10, DecreaseFactor 0.5: additive increase of a tenth of the ceiling per clean window, halving on push-back; one decrease per window so the replies to a burst already in flight do not collapse the rate.
  • BreakerFailures 5, BreakerOpen 15s doubling to 2m, ±20 % jitter, 3 trips: a host that stops answering is left alone for a while and probed once; after three trips (about 1.5 minutes of silence) it is a coverage gap for the task.
  • RetryAfterMax 5m: the same cap the control client honors.
  • HoldMax 10s: within a tool's usual connect timeout; longer waits are refused instead of hanging the tool.
  • FloorStrikes 3, BlockRun 20: enough repetitions that one refusal or a handful of forbidden paths do not count as a block.
  • ToolHostErrors 30: a tool's own breaker counts every failed request of every concurrent check (not consecutive failures), so it needs a larger budget than BreakerFailures × BreakerMaxTrips (15); 30 lets a few slow checks fail without dropping a live host.

Functions

func Ceiling

func Ceiling(limits ...float64) float64

Ceiling is the lowest of the positive limits (a scan's rate_limit, a scan window's, a program rule's, the local policy's rate.max_rps, the sensor's own setting): the rate a controller never exceeds. Limits of zero or less are "not set"; with none set it is TargetRate.DefaultCeiling.

func FailureReason

func FailureReason(err error) string

FailureReason classifies a network error toward a target as a failure signal (ReasonTimeout, ReasonRefused, ReasonReset), or "" when it says nothing about the target (the task was canceled, or the error is not a network one).

func ParseRetryAfter

func ParseRetryAfter(v string, now time.Time) time.Duration

ParseRetryAfter reads a Retry-After value (delta seconds or an HTTP date), capped at TargetRate.RetryAfterMax; zero when absent or unreadable.

Types

type Change

type Change struct {
	Host   string  `json:"host"`
	From   float64 `json:"from"`
	To     float64 `json:"to"`
	Reason string  `json:"reason"`
}

Change is one move of a host's rate or breaker.

type Event

type Event struct {
	Signal     Signal
	Reason     string
	RetryAfter time.Duration
}

Event is one signal about a host.

type Gap

type Gap struct {
	Host   string `json:"host"`
	Reason string `json:"reason"`
	Detail string `json:"detail,omitempty"`
}

Gap is one host whose coverage is incomplete, and why.

type HostRate

type HostRate struct {

	// OnChange, when set, receives every rate or breaker change (called
	// without the controller's lock held).
	OnChange func(Change)
	// contains filtered or unexported fields
}

HostRate is one task's per-host rate controller: additive increase, multiplicative decrease (AIMD) on each destination host, with a breaker per host. Its state belongs to one task: a forwarder creates it and drops it with the task, so nothing a host taught one task (or one tenant) reaches another.

Signals: a connection or an answer is health; 429 and 503 (with their Retry-After) are congestion; a reset, a timeout, a refused connection on a port that answered before and a failed TLS handshake are failures; a run of 403 answers after other answers is a block. The rate never goes above the ceiling the caller gives (Ceiling of every limit) and never below TargetRate.Floor (or the ceiling, when that is lower).

A host the controller cannot cover (the breaker refused connections to it, it kept pushing back at the floor, or it blocked the scan) is a coverage gap: the task completes and reports it (Gaps), it is not read as "no findings".

func NewHostRate

func NewHostRate(ceiling float64) *HostRate

NewHostRate returns the controller of one task with the given ceiling (Ceiling of every limit that applies; zero or less: the default).

func (*HostRate) Acquire

func (c *HostRate) Acquire(ctx context.Context, host string) error

Acquire waits until a connection (or a request) to host may start, bounded by ctx. It returns ErrHostUnavailable when the host's breaker is open or the host asked to wait longer than TargetRate.HoldMax.

func (*HostRate) Ceiling

func (c *HostRate) Ceiling() float64

Ceiling is the rate the controller never exceeds.

func (*HostRate) Gaps

func (c *HostRate) Gaps() ([]Gap, int)

Gaps are the hosts whose coverage is incomplete (at most TargetRate.MaxGaps) and how many there are in all.

func (*HostRate) Rate

func (c *HostRate) Rate(host string) float64

Rate is host's current rate (the start rate for a host not seen yet).

func (*HostRate) Report

func (c *HostRate) Report(host string, e Event)

Report feeds one signal about host to the controller.

func (*HostRate) ReportDial

func (c *HostRate) ReportDial(host string, port int, err error)

ReportDial feeds the outcome of a connection attempt to host:port. A failure counts only on a port that accepted a connection before in this task: a closed or filtered port of a host is not push-back (a port scan meets thousands of them). Cancellation is not a signal.

func (*HostRate) ReportHTTP

func (c *HostRate) ReportHTTP(host string, status int, retryAfter string)

ReportHTTP feeds one HTTP answer of host (status and its Retry-After header value).

func (*HostRate) Summary

func (c *HostRate) Summary() Summary

Summary is the task's target-rate outcome so far.

type Signal

type Signal int

Signal is what one connection or answer said about its host.

const (
	// Connected: a connection was established.
	Connected Signal = iota + 1
	// Answered: the host answered (an HTTP response other than 403, 429
	// or 503, or a tunnel that ended cleanly after the host sent data).
	Answered
	// Congested: 429 or 503 (Event.RetryAfter set when the host sent one).
	Congested
	// Forbidden: a 403 answer.
	Forbidden
	// Failed: a reset, a timeout, a refused connection on a port that
	// answered before, or a failed TLS handshake (Event.Reason says which).
	Failed
)

type Summary

type Summary struct {
	Ceiling float64 `json:"ceiling"`
	Floor   float64 `json:"floor"`
	Start   float64 `json:"start"`
	// Min is the lowest rate any host reached; Final the lowest rate a
	// host ended at.
	Min   float64 `json:"min"`
	Final float64 `json:"final"`
	Hosts int     `json:"hosts"`
	// Throttled hosts were slowed at least once; Unreachable hosts were
	// given up by their breaker.
	Throttled   int `json:"hosts_throttled"`
	Unreachable int `json:"hosts_unreachable"`
	// Gaps lists at most TargetRate.MaxGaps hosts; GapsTotal counts all.
	Gaps      []Gap `json:"gaps,omitempty"`
	GapsTotal int   `json:"gaps_total,omitempty"`
}

Summary is a task's target-rate outcome, for its result and its log.

type TargetRateBounds

type TargetRateBounds struct {
	// Floor is the lowest rate the controller paces a host at. A host that
	// still pushes back at the floor is reported as a coverage gap
	// (GapRateFloor) instead of being slowed further.
	Floor float64
	// DefaultCeiling is the ceiling when no limit was given at all (the
	// forwarder's historical default; never "unlimited").
	DefaultCeiling float64
	// StartFraction of the ceiling is where a host starts (slow start).
	StartFraction float64
	// Window is how often a host's rate is re-evaluated: a window with
	// answers and no push-back raises the rate.
	Window time.Duration
	// IncreaseFraction of the ceiling is added per clean window once slow
	// start ended (additive increase).
	IncreaseFraction float64
	// DecreaseFactor multiplies the rate on push-back (multiplicative
	// decrease), at most once per window.
	DecreaseFactor float64
	// BreakerFailures consecutive failures open a host's breaker.
	BreakerFailures int
	// BreakerOpen is how long a breaker stays open the first time; it
	// doubles on each further trip up to BreakerOpenMax.
	BreakerOpen    time.Duration
	BreakerOpenMax time.Duration
	// BreakerJitter spreads the open time by ± this fraction.
	BreakerJitter float64
	// BreakerMaxTrips trips give a host up for the rest of the task.
	BreakerMaxTrips int
	// ProbeTimeout is how long a half-open probe may stay unanswered before
	// another one is allowed.
	ProbeTimeout time.Duration
	// RetryAfterMax caps a target's Retry-After.
	RetryAfterMax time.Duration
	// HoldMax is the longest a connection is held to honor a Retry-After;
	// a longer one refuses connections to the host until it passes.
	HoldMax time.Duration
	// FloorStrikes push-backs while at the floor make a host a coverage gap
	// (GapRateFloor).
	FloorStrikes int
	// BlockRun consecutive 403 answers, from a host that answered otherwise
	// before, read as a block.
	BlockRun int
	// MaxHosts bounds the hosts one task tracks; further hosts share one
	// conservative state.
	MaxHosts int
	// MaxOpenPorts bounds the reachable ports remembered per host.
	MaxOpenPorts int
	// MaxGaps bounds the coverage gaps one result lists (the total is
	// still counted).
	MaxGaps int
	// MaxChanges bounds the rate changes one task logs.
	MaxChanges int
	// ToolHostErrors is the per-host error budget handed to a tool that
	// keeps its own host breaker (nuclei -max-host-error): errors on one
	// host, over all its requests, before the tool skips the host.
	ToolHostErrors int
}

TargetRateBounds bound the per-host request rate toward scan targets (HostRate). Rates are requests (connections, and requests on a connection the forwarder reads) per second.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL