resilience

type Admission ¶ added in v0.1.15

type Admission struct {
	// contains filtered or unexported fields
}

Admission — degrade instead of tipping (ENG-52, MOTOR-PRODUCCION-S2).

THE FAILURE IT REMOVES, measured in the isolated laboratory (docs/BENCHMARKS.md §4e): the engine accepted UNBOUNDED concurrency. In an open-arrival overload — offered load past the box's ceiling (~1 100 rps on a customer s-2vcpu-2gb, ~1 600 on a dedicated c-2) — in-flight requests accumulate without bound: each one is a goroutine, buffers, and a place in the pgxpool acquire queue (unbounded, FIFO). Every queued request — including the ones that will die at the 5 s query timeout or the client's own patience — pays the WHOLE pre-pool pipeline (parse, route, tenant, JWT, RBAC, query build), so wasted work grows with the backlog and eats exactly the CPU the admitted work needed. That is the positive feedback that makes the collapse METASTABLE: backlog → less useful capacity → more backlog; measured, a run at 1 100 rps tips to a seconds-scale p50 with LESS goodput and never comes back inside the run (6/8 runs; service p50 jumps to ~450–650 ms, all of it queue). The old backstop (queryTimeout = 5 s) fires after five seconds of held resources — far too late, and the work is thrown away after being paid for.

THE MECHANISM: a hard cap on in-flight admission-scoped requests, enforced at the FRONT of the chain (before the tenant limiter, the logger, the cache, JWT — before any per-request work beyond one atomic add and a path check). Over the cap → immediate 429 with Retry-After — the cheapest possible refusal, milliseconds-early instead of seconds-late.

WHY A CONCURRENCY CAP AND NOT THE ALTERNATIVES (argued, not asserted):

• Little's law makes concurrency the SELF-ADAPTIVE quantity: X = N/R. At full capacity with healthy latency the measured N is tiny (≈ 1–5 at 1 000–1 400 rps × 1.6–3 ms); tipped runs measure N in the THOUSANDS. Three orders of magnitude separate the regimes, so a crude cap cleanly splits them with margin on both sides — no estimator needed. The same N cap yields each box's own rps ceiling (a 40 % faster plan just serves more per slot) and each workload's own (a heavy screen with 10× the R admits 10× fewer rps — which is exactly its real capacity). A RATE admission would need the ceiling in rps, which varies 20× per endpoint and 40 % per plan — the ENG-53 trap.
• Latency-gradient adaptive limits (the Netflix concurrency-limits family) buy precision this system does not need (see the 1000× regime gap) and pay for it with an estimator that whipsaws on benign latency spikes — GC, autovacuum, a shared-vCPU neighbour (measured on the $18 box: p99 wandering 10–314 ms between healthy repeats). A false rejection under normal load is the one non-negotiable failure mode.
• A bounded queue with deadline (CoDel-style) still buffers (memory + added latency) and adds a tuning surface; clients already retry, and an immediate 429 + Retry-After is both cheaper and more honest.
• Pool-pressure admission (reject when acquire wait is high) fires LATE — after JWT/RBAC/parse are paid — and misses CPU-bound overload that never queues on the pool (aggregates, hooks).

SCOPE: everything except infra (probes, /metrics, /debug, /admin, /editor), OPTIONS preflight (CORS answers pre-auth), SSE streams (held open for minutes BY DESIGN — the `/events` suffix, the same rule the response cache uses), and byte-serving downloads (client-paced sendfile, not CPU). A long-lived connection inside the cap would consume a slot doing no work.

KNOB: APPXIMO_MAX_INFLIGHT. Unset/"auto" → max(32, 4×(GOMAXPROCS + pool max conns)) — on the reference 2-vCPU/10-conn box that is 48, bounding the admitted queue at ≈ 48 slots ≈ tens of ms of admitted latency at capacity while leaving ≥ 10× headroom over the healthy N of the fastest measured workload. "0" disables. A non-integer refuses to boot (the ENG-47 rule: a safety knob never falls back silently).

func NewAdmissionFromEnv(cores, poolConns int, exempt func(r *http.Request) bool) (*Admission, error)

func (a *Admission) InFlight() int64

func (a *Admission) Limit() int64

func (a *Admission) Middleware(next http.Handler) http.Handler

func (a *Admission) PromCollector() prometheus.Collector

func (a *Admission) Rejected() int64

type MemoryGuard ¶ added in v0.1.12

type MemoryGuard struct {
	// contains filtered or unexported fields
}

MemoryGuard — the minimal, honest write-admission guard (MIGRACION-CONFIANZA-S1, D-ter).

WHAT IT IS NOT: capacity. It does not make the engine "hold" a bulk load on a small box. It makes the engine STOP ACCEPTING NEW WRITES when the HOST is about to run out of memory, answering a 503 that says why, instead of accepting until the kernel's OOM killer takes PostgreSQL — and with it every app that shares that PostgreSQL. Measured in the field (a Symfony migration, 46k rows, 957 MiB box, no swap, five apps on one Postgres): the engine kept accepting writes until `postgresql@14-main.service: Failed with result 'oom-kill'` and all five apps went down. The audit before this guard: the engine had NO notion of host memory pressure anywhere — GOMEMLIMIT bounds its OWN heap (which was never the problem: the memory that grew was Postgres's backends), the pool is a fixed 10 connections, the rate limiter counts requests, not bytes.

WHAT IT MEASURES — and why not MemAvailable alone: on a box that runs PostgreSQL, `shared_buffers` shows up as Cached but is NOT reclaimable, so MemAvailable at rest sits at a few tens of MiB however much RAM the box has. A guard on MemAvailable alone would trip permanently and be switched off on day one. The signal is MemAvailable + SwapFree: what the kernel can still hand out before it has to kill something. (On a box with NO swap the two coincide — and the installer now warns loudly about exactly that box, scripts/install.sh.)

COST: one atomic load per write request; /proc/meminfo is read at most once per second, by the first writer to notice the sample is stale (others keep the previous sample — never a stampede, never a lock on the hot path). Reads never consult it: a saturated host still serves what it can.

KNOBS: APPXIMO_MEMORY_GUARD_MIN_MB — the floor of MemAvailable + SwapFree in MiB under which writes are refused. Default: max(32, 2 % of MemTotal) MiB — deliberately LOW, so it fires only when the kernel is genuinely about to run out, never as a permanent false positive on a busy but healthy box. `0` disables the guard. A non-integer value refuses to boot (a safety knob never falls back silently — the ENG-47 rule).

func NewMemoryGuard(minBytes int64, meminfoPath string, interval time.Duration) *MemoryGuard

func NewMemoryGuardFromEnv() (*MemoryGuard, error)

func (g *MemoryGuard) Allow() (ok bool, availableBytes int64)

func (g *MemoryGuard) Middleware(next http.Handler) http.Handler

func (g *MemoryGuard) MinBytes() int64

func (g *MemoryGuard) Sample() (int64, error)

type RateLimitConfig ¶

type RateLimitConfig struct {
	RPS   float64 // sustained requests per second, per tenant
	Burst int     // bucket capacity for short spikes
}

RateLimitConfig sets an explicit per-tenant token-bucket policy, independent of subscription tier. Used to wire the limiter from environment configuration.

type TenantLimiter ¶

type TenantLimiter struct {
	// contains filtered or unexported fields
}

TenantLimiter provides per-tenant token-bucket rate limiting. Each tenant gets an independent limiter keyed by tenantID. With a nil cfg the burst size equals the tier rate (1-second burst capacity); with cfg set, every tenant uses cfg.RPS / cfg.Burst and the tier is ignored.

Past maxLimiters distinct tenants, additional (unknown) tenants share a single overflow bucket: memory stays bounded and the rate limit still applies.

func NewConfiguredLimiter(cfg RateLimitConfig) *TenantLimiter

func NewTenantLimiter() *TenantLimiter

func (tl *TenantLimiter) Allow(tenantID string, tier Tier) bool

type Tier ¶

type Tier string

Tier is a subscription plan that determines the allowed request rate.

const (
	TierFree Tier = "free" // 100 req/s
	TierPro  Tier = "pro"  // 1000 req/s
)