Documentation
¶
Overview ¶
Package adapt holds the sensor's adaptive controllers and the one table of their bounds (RFC-074 in openctemio/openctem api/docs/rfcs, RFC-074-adaptive-sensor-control.md).
A controller moves a value from a measured signal, inside a floor and a ceiling. The ceiling is always the lowest of the limits a human or a policy set: a controller may go below it, never above. Every number a controller uses is in this file; code elsewhere refers to the table, never to a literal.
Stability: Experimental (docs/STABILITY.md).
Index ¶
- Constants
- Variables
- func Ceiling(limits ...float64) float64
- func FailureReason(err error) string
- func ParseRetryAfter(v string, now time.Time) time.Duration
- type Change
- type Event
- type Gap
- type HostRate
- func (c *HostRate) Acquire(ctx context.Context, host string) error
- func (c *HostRate) Ceiling() float64
- func (c *HostRate) Gaps() ([]Gap, int)
- func (c *HostRate) Rate(host string) float64
- func (c *HostRate) Report(host string, e Event)
- func (c *HostRate) ReportDial(host string, port int, err error)
- func (c *HostRate) ReportHTTP(host string, status int, retryAfter string)
- func (c *HostRate) Summary() Summary
- type Signal
- type Summary
- type TargetRateBounds
Constants ¶
const ( ReasonTimeout = "timeout" ReasonReset = "reset" ReasonRefused = "refused" ReasonTLS = "tls_handshake" ReasonHTTP429 = "http_429" ReasonHTTP503 = "http_503" ReasonBlocked = "blocked" ReasonCleanRun = "clean_window" ReasonSlowStart = "slow_start" ReasonOpened = "breaker_open" ReasonHalfOpen = "breaker_half_open" ReasonClosed = "breaker_closed" ReasonGaveUp = "breaker_gave_up" ReasonRetryAfter = "retry_after" )
Reasons of an Event and of a Change.
const ( // GapBreakerOpen: the host stopped answering and the breaker refused // connections to it (or gave it up). GapBreakerOpen = "breaker_open" // GapRateFloor: the host kept pushing back at the lowest rate, or asked // to wait longer than a connection can be held. GapRateFloor = "rate_floor" // GapBlocked: the host answered with a block (a run of 403s). GapBlocked = "blocked" )
Gap reasons: why a host's coverage is incomplete.
Variables ¶
ErrHostUnavailable is the error of a connection the breaker refused.
var TargetRate = TargetRateBounds{ Floor: 1, DefaultCeiling: 100, StartFraction: 0.25, Window: 2 * time.Second, IncreaseFraction: 0.10, DecreaseFactor: 0.5, BreakerFailures: 5, BreakerOpen: 15 * time.Second, BreakerOpenMax: 2 * time.Minute, BreakerJitter: 0.2, BreakerMaxTrips: 3, ProbeTimeout: 30 * time.Second, RetryAfterMax: 5 * time.Minute, HoldMax: 10 * time.Second, FloorStrikes: 3, BlockRun: 20, MaxHosts: 65536, MaxOpenPorts: 64, MaxGaps: 100, MaxChanges: 20, ToolHostErrors: 30, }
TargetRate is the table of target-rate bounds. Why each value:
- Floor 1/s: the slowest pace still useful for a scan in a job's time; slower than that the result is reported as a gap, not stretched.
- DefaultCeiling 100/s: the forwarder's default before this controller.
- StartFraction 0.25: a new host is probed at a quarter of the ceiling; doubling per clean window reaches the ceiling in two windows.
- Window 2s: several round trips to a distant host, and at least two requests at the floor, so a window has evidence before it moves.
- IncreaseFraction 0.10, DecreaseFactor 0.5: additive increase of a tenth of the ceiling per clean window, halving on push-back; one decrease per window so the replies to a burst already in flight do not collapse the rate.
- BreakerFailures 5, BreakerOpen 15s doubling to 2m, ±20 % jitter, 3 trips: a host that stops answering is left alone for a while and probed once; after three trips (about 1.5 minutes of silence) it is a coverage gap for the task.
- RetryAfterMax 5m: the same cap the control client honors.
- HoldMax 10s: within a tool's usual connect timeout; longer waits are refused instead of hanging the tool.
- FloorStrikes 3, BlockRun 20: enough repetitions that one refusal or a handful of forbidden paths do not count as a block.
- ToolHostErrors 30: a tool's own breaker counts every failed request of every concurrent check (not consecutive failures), so it needs a larger budget than BreakerFailures × BreakerMaxTrips (15); 30 lets a few slow checks fail without dropping a live host.
Functions ¶
func Ceiling ¶
Ceiling is the lowest of the positive limits (a scan's rate_limit, a scan window's, a program rule's, the local policy's rate.max_rps, the sensor's own setting): the rate a controller never exceeds. Limits of zero or less are "not set"; with none set it is TargetRate.DefaultCeiling.
func FailureReason ¶
FailureReason classifies a network error toward a target as a failure signal (ReasonTimeout, ReasonRefused, ReasonReset), or "" when it says nothing about the target (the task was canceled, or the error is not a network one).
Types ¶
type Change ¶
type Change struct {
Host string `json:"host"`
From float64 `json:"from"`
To float64 `json:"to"`
Reason string `json:"reason"`
}
Change is one move of a host's rate or breaker.
type Gap ¶
type Gap struct {
Host string `json:"host"`
Reason string `json:"reason"`
Detail string `json:"detail,omitempty"`
}
Gap is one host whose coverage is incomplete, and why.
type HostRate ¶
type HostRate struct {
// OnChange, when set, receives every rate or breaker change (called
// without the controller's lock held).
OnChange func(Change)
// contains filtered or unexported fields
}
HostRate is one task's per-host rate controller: additive increase, multiplicative decrease (AIMD) on each destination host, with a breaker per host. Its state belongs to one task: a forwarder creates it and drops it with the task, so nothing a host taught one task (or one tenant) reaches another.
Signals: a connection or an answer is health; 429 and 503 (with their Retry-After) are congestion; a reset, a timeout, a refused connection on a port that answered before and a failed TLS handshake are failures; a run of 403 answers after other answers is a block. The rate never goes above the ceiling the caller gives (Ceiling of every limit) and never below TargetRate.Floor (or the ceiling, when that is lower).
A host the controller cannot cover (the breaker refused connections to it, it kept pushing back at the floor, or it blocked the scan) is a coverage gap: the task completes and reports it (Gaps), it is not read as "no findings".
func NewHostRate ¶
NewHostRate returns the controller of one task with the given ceiling (Ceiling of every limit that applies; zero or less: the default).
func (*HostRate) Acquire ¶
Acquire waits until a connection (or a request) to host may start, bounded by ctx. It returns ErrHostUnavailable when the host's breaker is open or the host asked to wait longer than TargetRate.HoldMax.
func (*HostRate) Gaps ¶
Gaps are the hosts whose coverage is incomplete (at most TargetRate.MaxGaps) and how many there are in all.
func (*HostRate) ReportDial ¶
ReportDial feeds the outcome of a connection attempt to host:port. A failure counts only on a port that accepted a connection before in this task: a closed or filtered port of a host is not push-back (a port scan meets thousands of them). Cancellation is not a signal.
func (*HostRate) ReportHTTP ¶
ReportHTTP feeds one HTTP answer of host (status and its Retry-After header value).
type Signal ¶
type Signal int
Signal is what one connection or answer said about its host.
const ( // Connected: a connection was established. Connected Signal = iota + 1 // Answered: the host answered (an HTTP response other than 403, 429 // or 503, or a tunnel that ended cleanly after the host sent data). Answered // Congested: 429 or 503 (Event.RetryAfter set when the host sent one). Congested // Forbidden: a 403 answer. Forbidden // Failed: a reset, a timeout, a refused connection on a port that // answered before, or a failed TLS handshake (Event.Reason says which). Failed )
type Summary ¶
type Summary struct {
Ceiling float64 `json:"ceiling"`
Floor float64 `json:"floor"`
Start float64 `json:"start"`
// Min is the lowest rate any host reached; Final the lowest rate a
// host ended at.
Min float64 `json:"min"`
Final float64 `json:"final"`
Hosts int `json:"hosts"`
// Throttled hosts were slowed at least once; Unreachable hosts were
// given up by their breaker.
Throttled int `json:"hosts_throttled"`
Unreachable int `json:"hosts_unreachable"`
// Gaps lists at most TargetRate.MaxGaps hosts; GapsTotal counts all.
Gaps []Gap `json:"gaps,omitempty"`
GapsTotal int `json:"gaps_total,omitempty"`
}
Summary is a task's target-rate outcome, for its result and its log.
type TargetRateBounds ¶
type TargetRateBounds struct {
// Floor is the lowest rate the controller paces a host at. A host that
// still pushes back at the floor is reported as a coverage gap
// (GapRateFloor) instead of being slowed further.
Floor float64
// DefaultCeiling is the ceiling when no limit was given at all (the
// forwarder's historical default; never "unlimited").
DefaultCeiling float64
// StartFraction of the ceiling is where a host starts (slow start).
StartFraction float64
// Window is how often a host's rate is re-evaluated: a window with
// answers and no push-back raises the rate.
Window time.Duration
// IncreaseFraction of the ceiling is added per clean window once slow
// start ended (additive increase).
IncreaseFraction float64
// DecreaseFactor multiplies the rate on push-back (multiplicative
// decrease), at most once per window.
DecreaseFactor float64
// BreakerFailures consecutive failures open a host's breaker.
BreakerFailures int
// BreakerOpen is how long a breaker stays open the first time; it
// doubles on each further trip up to BreakerOpenMax.
BreakerOpen time.Duration
BreakerOpenMax time.Duration
// BreakerJitter spreads the open time by ± this fraction.
BreakerJitter float64
// BreakerMaxTrips trips give a host up for the rest of the task.
BreakerMaxTrips int
// ProbeTimeout is how long a half-open probe may stay unanswered before
// another one is allowed.
ProbeTimeout time.Duration
// RetryAfterMax caps a target's Retry-After.
RetryAfterMax time.Duration
// HoldMax is the longest a connection is held to honor a Retry-After;
// a longer one refuses connections to the host until it passes.
HoldMax time.Duration
// FloorStrikes push-backs while at the floor make a host a coverage gap
// (GapRateFloor).
FloorStrikes int
// BlockRun consecutive 403 answers, from a host that answered otherwise
// before, read as a block.
BlockRun int
// MaxHosts bounds the hosts one task tracks; further hosts share one
// conservative state.
MaxHosts int
// MaxOpenPorts bounds the reachable ports remembered per host.
MaxOpenPorts int
// MaxGaps bounds the coverage gaps one result lists (the total is
// still counted).
MaxGaps int
// MaxChanges bounds the rate changes one task logs.
MaxChanges int
// ToolHostErrors is the per-host error budget handed to a tool that
// keeps its own host breaker (nuclei -max-host-error): errors on one
// host, over all its requests, before the tool skips the host.
ToolHostErrors int
}
TargetRateBounds bound the per-host request rate toward scan targets (HostRate). Rates are requests (connections, and requests on a connection the forwarder reads) per second.