httpsec

package
v0.33.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 14, 2026 License: GPL-3.0 Imports: 16 Imported by: 0

Documentation

Overview

Package httpsec provides SSRF-safe URL validation and HTTP client construction for the OpenCTEM SDK.

This is a lifted copy of api/pkg/httpsec. Originally the SDK lived in its own Go module and could not import from api/, so the 9+ outbound-client sites across sdk-go/pkg/* each carried a bare &http.Client{Timeout: …} with no dialer-level blocklist. That was fine when the SDK only talked to the pinned OpenCTEM API over a trusted operator-provided URL, but third-party users of the SDK (custom scanners, integration shims) can point it at arbitrary hostnames. Any of those callers inherit the same SSRF gap as the API did before the audit.

This package mirrors api/pkg/httpsec verbatim — when you change one, change the other. A follow-up task tracks hoisting the single canonical copy into a top-level shared Go module; until then, the drift is caught by scripts/security-lint.sh which grep-checks both copies for the same CIDR blocklist.

Any outbound HTTP call in sdk-go/pkg/* that reaches a hostname chosen at runtime (API URL from config, KEV/EPSS feed, bootstrap token endpoint, command poller, lease endpoint) MUST use SafeHTTPClient + ValidateURL, not &http.Client{} directly.

Stability: Stable (docs/STABILITY.md).

Index

Constants

View Source 
const (
	// PinElementAnchorSPKI: SHA-256 of the SubjectPublicKeyInfo of the
	// verified chain's trust anchor, recorded at pairing.
	PinElementAnchorSPKI = "anchor_spki"
	// PinElementCACert: SHA-256 of a CA certificate in the presented chain
	// (SENSOR_CA_FINGERPRINT; see SetAPIPinnedCA).
	PinElementCACert = "ca_cert"
)

Platform TLS pin elements, as identity.json records them.

View Source 
const ProxyDirectValue = "direct"

ProxyDirectValue is the setting value that means "no proxy".

Variables

View Source 
var AllowLoopback = os.Getenv(allowLoopbackEnvVar) == "1"

AllowLoopback, when set to true, skips the loopback (127.0.0.0/8, ::1/128) branch of the blocklist. All other CIDRs (RFC1918, link-local IMDS, CGNAT, …) remain enforced. Prefer the env var for external test harnesses; this variable is for in-process overrides.

View Source 
var ErrPinnedCAMismatch = errors.New("httpsec: the platform's certificate chain does not contain the pinned CA")

ErrPinnedCAMismatch is a server whose chain does not lead to the pinned CA.

View Source 
var ErrPinnedCANoServerName = errors.New("httpsec: a pinned platform CA needs a host name in the platform URL, not an IP address")

ErrPinnedCANoServerName is a pinned connection without a server name to check the certificate against. crypto/tls leaves ConnectionState.ServerName empty when the platform URL is an IP address (no SNI), and x509 skips the name check for an empty DNSName, so such a connection is refused: with a pin, the platform URL must use a host name.

View Source 
var ErrPlatformChangedDuringPairing = errors.New("httpsec: the platform's certificate chain changed to another CA during pairing")

ErrPlatformChangedDuringPairing is a pairing whose later connections presented a certificate leading to another anchor than the first.

View Source 
var ErrPlatformPinMismatch = errors.New("httpsec: platform certificate does not match the pin stored at pairing; re-pair or set SENSOR_CA_FINGERPRINT")

ErrPlatformPinMismatch is a platform certificate whose verified chain does not lead to the key pinned at pairing. It is never answered by trusting the system roots instead.

View Source 
var ErrProxiedTargetBlocked = errors.New("httpsec: target refused before the proxy")

ErrProxiedTargetBlocked is returned (wrapped) when a request that would go through a proxy is refused because its target is not allowed.

View Source 
var ProxyEnvVars = []string{
	"HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "ALL_PROXY",
	"http_proxy", "https_proxy", "no_proxy", "all_proxy",
}

ProxyEnvVars are the proxy variables of a process environment, in both spellings. ScannerEnviron passes or removes them as a group.

View Source 
var SensitiveHeaders = []string{
	"Authorization",
	"Proxy-Authorization",
	"Cookie",
	"X-API-Key",
	"X-Api-Token",
	"X-Auth-Token",
	"X-Agent-Key",
	"X-Agent-API-Key",
	"X-Sensor-API-Key",
	"X-ApiKeys",
	"X-Bootstrap-Token",
}

SensitiveHeaders are request headers that carry credentials. They are stripped from a redirected request whenever the redirect changes origin (scheme, host or port). net/http only strips Authorization/Cookie, and only when the host leaves the original domain — it keeps them on an https->http same-host hop and never touches custom API-key headers.

Functions

func APIPinnedCA added in v0.18.0 

func APIPinnedCA() []byte

APIPinnedCA returns the pin set with SetAPIPinnedCA.

func APIPinnedSPKI added in v0.19.0 

func APIPinnedSPKI() []byte

APIPinnedSPKI returns the pin set with SetAPIPinnedSPKI.

func APIRootCAs added in v0.12.0 

func APIRootCAs() *x509.CertPool

APIRootCAs returns the pool set with SetAPIRootCAs (nil: system trust).

func AllowPrivate 

func AllowPrivate() bool

AllowPrivate reports the current runtime posture for startup-logging; do not consult it per-call (that happens inside IsIPBlocked).

func AnchorSPKI added in v0.19.0 

func AnchorSPKI(chains [][]*x509.Certificate) []byte

AnchorSPKI is the pin of a verified connection: the SHA-256 of the SubjectPublicKeyInfo of the trust anchor of its shortest verified chain (nil without a verified chain).

func CheckAPIBaseURL added in v0.7.0 

func CheckAPIBaseURL(raw string) (warning string, err error)

CheckAPIBaseURL validates an operator-supplied OpenCTEM API base URL.

It returns an error for anything that cannot be a sane API endpoint: a non-http(s) scheme, a missing host, or embedded userinfo (credentials in the URL end up in logs and error strings). For a plain-http URL whose host is not loopback it returns a non-empty warning instead of an error: the API key is sent as a bearer token, so clear-text transport outside a local dev setup exposes it, but rejecting it outright would break existing in-cluster deployments that terminate TLS elsewhere.

func ContentRootCAs added in v0.15.0 

func ContentRootCAs() *x509.CertPool

ContentRootCAs returns the pool set with SetContentRootCAs.

func EnvironmentProxySummary added in v0.15.0 

func EnvironmentProxySummary() string

EnvironmentProxySummary describes the proxy variables set in the process environment, credentials removed, e.g. "HTTPS_PROXY=http://proxy:3128". It is empty when none is set.

func FirstWarning added in v0.8.0 

func FirstWarning(baseURL string) bool

FirstWarning reports whether the CheckAPIBaseURL warning for baseURL should be printed now: true the first time it is asked for a given base URL in this process, false afterwards. pkg/client and pkg/platform share it, so a sensor that builds several clients for one platform prints the plain-http warning once, not once per client or per request.

func FormatFingerprint added in v0.19.0 

func FormatFingerprint(fp []byte) string

FormatFingerprint is a SHA-256 fingerprint as "sha256:<64 hex digits>".

func HasAPIPin added in v0.19.0 

func HasAPIPin() bool

HasAPIPin reports whether API clients created now pin the platform's TLS identity (a CA certificate or an anchor key).

func IsIPBlocked 

func IsIPBlocked(ip net.IP) bool

IsIPBlocked reports whether the given IP falls in a blocked CIDR. The hard-blocked set always rejects. The RFC1918 / ULA set is conditional on allowPrivate (set via env).

AllowLoopback (test-only) still overrides loopback for httptest servers; it does NOT touch IMDS or RFC1918 — those remain on their own toggle chains.

func IsIPBlockedWith added in v0.7.0 

func IsIPBlockedWith(ip net.IP, allowPrivateRanges, allowLoopback bool) bool

IsIPBlockedWith is IsIPBlocked with the two toggles supplied by the caller instead of read from the process-wide env-derived defaults. It lets a component (e.g. the scan-target policy in pkg/core) carry its own allow-private posture without mutating package globals. The hard-blocked set (IMDS/link-local, CGNAT, multicast, unspecified, reserved) is enforced regardless of either toggle; allowLoopback only relaxes 127.0.0.0/8 and ::1.

func LoadCAFile added in v0.12.0 

func LoadCAFile(path string) (*x509.CertPool, error)

LoadCAFile returns the system trust store plus the PEM certificates in path, for SetAPIRootCAs: the platform's private CA is trusted and the public CAs keep working (a proxy or a platform that later moves to a public certificate). A file without a certificate is an error.

func NewAPIClient added in v0.7.0 

func NewAPIClient(timeout time.Duration) *http.Client

NewAPIClient returns the client for requests that carry an OpenCTEM API key. Every such request goes to the operator-configured API base URL, so the destination is trusted configuration, not attacker input: on-prem and in-cluster platforms live on loopback, RFC1918, ULA or CGNAT (Tailscale) addresses, and SafeHTTPClient's blocklist would refuse all of them. The client still refuses every redirect (the bearer key never follows one), still refuses link-local destinations such as the cloud metadata service and multicast/reserved/unspecified addresses (so a poisoned DNS answer cannot point the key there), and uses the API path's proxy setting (SetAPIProxy; by default the environment's HTTP(S)_PROXY / NO_PROXY).

func NewAPIClientRecordingPin added in v0.19.0 

func NewAPIClientRecordingPin(timeout time.Duration, r *PinRecorder) *http.Client

NewAPIClientRecordingPin is NewAPIClient that also records the platform's pin in r (pairing).

func ParseCAFingerprint added in v0.18.0 

func ParseCAFingerprint(s string) ([]byte, error)

ParseCAFingerprint reads a SHA-256 fingerprint written as 64 hex digits, optionally with colons and an "SHA256:" prefix.

func RefuseRedirects added in v0.7.0 

func RefuseRedirects(req *http.Request, _ []*http.Request) error

RefuseRedirects is a CheckRedirect policy that never follows a redirect. Use it for clients that only ever talk to the OpenCTEM API: the API does not issue redirects, so a 3xx means a misconfigured base URL or a hostile intermediary, and following it would forward the bearer key.

func SafeCheckRedirect added in v0.7.0 

func SafeCheckRedirect(req *http.Request, via []*http.Request) error

SafeCheckRedirect is the CheckRedirect policy installed on every SafeHTTPClient:

  • at most 10 hops;
  • a redirect may never downgrade https to http (the credential and the response would cross the network in clear text);
  • when a hop changes origin relative to the original request, every header in SensitiveHeaders is removed before the request is sent.

The dialer still applies the IP blocklist to the redirect target, so a redirect into IMDS/loopback fails at connect time.

func SafeHTTPClient 

func SafeHTTPClient(timeout time.Duration) *http.Client

SafeHTTPClient returns an *http.Client whose dialer rejects any connection attempt to a blocked CIDR at the transport layer. This is the belt to ValidateURL's braces: even if a caller forgets to validate up front, the dial fails closed.

Callers should still ValidateURL themselves because the dialer-only check happens AFTER DNS resolution, so a request to an attacker- supplied URL will have burned a DNS lookup (possibly against an attacker-controlled resolver). ValidateURL rejects before the lookup leaves the host process.

Proxy: the content path's setting (SetContentProxy; by default the environment's HTTP_PROXY / HTTPS_PROXY / NO_PROXY). When a request goes through a proxy, its target is checked before the proxy is used (the dialer then only sees the proxy's address): see guardedProxy. The proxy itself is operator configuration and is dialed with the API-destination policy (private addresses allowed; link-local, multicast and reserved refused).

func SetAPIPinnedCA added in v0.18.0 

func SetAPIPinnedCA(fingerprint []byte)

SetAPIPinnedCA pins every API client created afterwards to the CA with this SHA-256 fingerprint (nil removes the pin).

func SetAPIPinnedSPKI added in v0.19.0 

func SetAPIPinnedSPKI(fingerprint []byte)

SetAPIPinnedSPKI pins every API client created afterwards to the trust anchor key with this SHA-256 SubjectPublicKeyInfo fingerprint (nil removes the pin). A CA certificate pin (SetAPIPinnedCA) wins over it.

func SetAPIProxy added in v0.15.0 

func SetAPIProxy(s ProxySetting)

SetAPIProxy sets the proxy of every API client created afterwards (NewAPIClient). The zero ProxySetting goes back to the environment.

func SetAPIRootCAs added in v0.12.0 

func SetAPIRootCAs(pool *x509.CertPool)

SetAPIRootCAs makes every API client created afterwards (NewAPIClient) trust pool instead of the system trust store; nil goes back to the system trust store. Clients created before keep their trust.

func SetContentProxy added in v0.15.0 

func SetContentProxy(s ProxySetting)

SetContentProxy sets the proxy of every SafeHTTPClient created afterwards. The zero ProxySetting goes back to the environment.

func SetContentRootCAs added in v0.15.0 

func SetContentRootCAs(pool *x509.CertPool)

SetContentRootCAs makes every SafeHTTPClient created afterwards trust pool instead of the system trust store (nil: the system trust store). It is for a TLS-inspecting egress proxy, whose CA re-signs every public certificate: build pool with LoadCAFile, which keeps the public CAs.

func TrustUpstreamHosts added in v0.15.0 

func TrustUpstreamHosts(hosts ...string)

TrustUpstreamHosts registers host names that the program itself hard-codes (a public feed or content source, such as the KEV feed's host). When a SafeHTTPClient request goes through a proxy and such a name does not resolve on this host, which is normal on a network whose only way out is the proxy, the request is still sent and the proxy resolves the name. Any other name that does not resolve locally is refused, because its address cannot be checked.

Never register a name that comes from configuration the platform or a job can set: that would let it skip the address check.

Types

type PinRecorder added in v0.19.0 

type PinRecorder struct {
	// contains filtered or unexported fields
}

PinRecorder learns the platform pin of a pairing: the anchor key of the first verified connection, which every later connection of the same client must then lead to as well.

func (*PinRecorder) Pin added in v0.19.0 

func (r *PinRecorder) Pin() []byte

Pin is the recorded anchor key fingerprint (nil: no verified TLS connection yet, a plain http platform, or a connection a CA certificate pin verified, whose pin is that certificate's).

type ProxyFunc added in v0.15.0 

type ProxyFunc = func(*http.Request) (*url.URL, error)

ProxyFunc is the type of http.Transport.Proxy.

type ProxyMode added in v0.15.0 

type ProxyMode int

ProxyMode says where a ProxySetting takes its proxy from. 

const (
	// ProxyEnvironment uses HTTP_PROXY, HTTPS_PROXY and NO_PROXY of the
	// process (http.ProxyFromEnvironment). It is the zero value.
	ProxyEnvironment ProxyMode = iota
	// ProxyDirect never uses a proxy, whatever the environment says.
	ProxyDirect
	// ProxyURL sends every request through one proxy, except the hosts in
	// the setting's NoProxy list.
	ProxyURL
)

type ProxySetting added in v0.15.0 

type ProxySetting struct {
	Mode ProxyMode
	// URL is the proxy for ProxyURL: http, https, socks5 or socks5h, with
	// optional user:password (Basic, or SOCKS5 user/password).
	URL *url.URL
	// NoProxy is the bypass list for ProxyURL, in NO_PROXY syntax (hosts,
	// domains, IPs, CIDRs, ":port", "*"). Loopback is never proxied.
	NoProxy string
}

ProxySetting is the proxy configuration of one outbound path.

func APIProxy added in v0.15.0 

func APIProxy() ProxySetting

APIProxy returns the setting of the API path.

func ContentProxy added in v0.15.0 

func ContentProxy() ProxySetting

ContentProxy returns the setting of the content path.

func ParseProxySetting added in v0.15.0 

func ParseProxySetting(value, noProxy string) (ProxySetting, error)

ParseProxySetting parses a proxy setting as an operator writes it: "" means the environment, "direct" (or "none") means no proxy, anything else is a proxy URL. A URL without a scheme is http, as in Go and curl. noProxy is the bypass list used with a URL (normally the NO_PROXY of the process).

func (ProxySetting) Func added in v0.15.0 

func (s ProxySetting) Func() ProxyFunc

Func returns the http.Transport.Proxy function for the setting. It never returns nil, so it can be installed as is.

func (ProxySetting) String added in v0.15.0 

func (s ProxySetting) String() string

String describes the setting for logs, without credentials: "environment", "direct" or the redacted proxy URL.

func (ProxySetting) URLFor added in v0.15.0 

func (s ProxySetting) URLFor(target *url.URL) (*url.URL, error)

URLFor returns the proxy a request to target would use (nil: direct). It answers "which proxy does this path use" for logs and for building a child process's environment.

type ValidationResult 

type ValidationResult struct {
	URL         *url.URL
	ResolvedIPs []net.IP
}

ValidationResult carries the parsed URL + the DNS-pinned IP set so callers that want to prevent DNS rebinding can dial one of the resolved IPs rather than re-resolve at dial time.

func ValidateURL 

func ValidateURL(rawURL string) (*ValidationResult, error)

ValidateURL parses rawURL, confirms scheme is http/https, blocks common dangerous hostnames, resolves DNS, and rejects if any A/AAAA hits a blocked CIDR. On any failure returns an error — callers MUST NOT proceed to dial.

Fail-closed on DNS lookup failure: if we cannot resolve, we cannot verify the target is safe, so the request is rejected.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.