ratelimiter

ratelimiter

Package ratelimiter provides a distributed, Redis-backed fixed-window rate limiter used by the Aibrix gateway to enforce per-user and per-model request limits across multiple gateway instances.

Interface

type RateLimiter interface {
    Get(ctx context.Context, key string) (int64, error)
    GetLimit(ctx context.Context, key string) (int64, error)
    Incr(ctx context.Context, key string, val int64) (int64, error)
}

Implementations

redisRateLimiter — NewRedisAccountRateLimiter(name, client, windowSize)

A fixed-window counter backed by Redis. Suitable for cluster-wide enforcement when multiple gateway replicas share the same Redis instance.

Key scheme: {name}:{key}:{timebin}

The time bin is computed as:

timebin = (time.Now().Unix() / windowSeconds) % 64

This cycles through 64 buckets, so at most 64 keys exist per logical counter. The TTL on each key is set to windowSize, so stale bins expire automatically.

• Minimum windowSize is 1 second (smaller values are clamped up).
• Incr uses a Redis pipeline (INCRBY + EXPIRE) for atomicity within a single round-trip.
• A missing key (redis.Nil) is treated as 0, not an error.

noopRateLimiter — NewNoopRateLimiter()

A no-op implementation that always allows requests. Used when Redis is unavailable (e.g., local development without a Redis sidecar). GetLimit returns math.MaxInt64.

Usage in the gateway

Two limiter instances are created at server startup:

The separate name prefix ensures the two limiters' keys never collide in Redis.

Per-user limits (RPM / TPM)

Keys follow the pattern aibrix:{username}_{RPM|TPM}_CURRENT:{timebin}. Limits are read from the user record resolved during request header processing.

Per-model RPS

Keys follow the pattern aibrix_model:{modelName}_MODEL_RPS_CURRENT:{timebin}. The RPS limit is configured per model via the requestsPerSecond field in a config profile:

{
  "profiles": {
    "rps-limited": {
      "routingStrategy": "least-request",
      "requestsPerSecond": 1
    }
  }
}

The active profile is selected by passing the config-profile request header. If no profile is active or requestsPerSecond is unset, the RPS check is skipped entirely.

Enforcement flow

Per-model RPS enforcement is handled in HandleRequestBody via enforceModelRPS and a deferred compensation step:

1. Pre-routing gate (enforceModelRPS) — called before routing. It:
   • atomically increments the counter with Incr(..., +1) and receives the new value
   • rejects with HTTP 429 and rolls back with Incr(..., -1) if newVal > limit
   • allows the request through when newVal <= limit
2. Deferred compensation (decrModelRPS) — armed immediately after a successful pre-charge. If routing later fails, the deferred call refunds quota with Incr(..., -1).
3. Success path — after routing succeeds and request accounting is attached, compensation is disabled, so the pre-charge remains counted.

Using incr-then-check (rather than check-then-increment) means the INCRBY is the sole gate. Because Redis processes INCRBY atomically, each concurrent caller receives a unique sequential result — eliminating the TOCTOU race that would otherwise allow over-admission during the check→increment window.