Documentation
¶
Overview ¶
Package quota answers the question pkg/authz does not.
RBAC decides whether an identity *may* act. Nothing decided how *much* it may consume, so on a shared hub one tenant could open unlimited projects, enrol unlimited executors, hold unlimited sessions, and — the expensive one — start unlimited concurrent runs against a token budget the whole organisation shares. pkg/globalbudget exists but is keyed by project, which is the wrong axis for a multi-tenant deployment: a user who can create projects can create budget headroom.
This package is the identity-keyed half. It is deliberately stdlib-only apart from pkg/authz (whose Subject and ClaimKind it reuses so that quota bindings read exactly like role bindings in the same config file). The SQLite adapter lives in pkg/quotastore, the same seam as oidcauth.SessionStore/sessionstore and secretbroker.Store/secretstore.
The two kinds of resource ¶
Gauges (projects, concurrent tasks, executors, sessions) go up on admission and back down on release. They are the ones that can drift: a hub that crashes mid-run leaves a counter claiming a task is still running forever, which would slowly starve the tenant it was meant to protect. Enforcer therefore treats the persisted gauge as a cache and rebuilds it from live state at startup — see Enforcer.Reconcile.
Daily counters (tokens, cost) only ever go up, inside a UTC-day bucket. They are the opposite case: they must survive a restart exactly, because re-deriving "spend so far today" from anything other than the recorded increments would hand a compromised account a fresh budget on every crash.
Why the most restrictive binding wins a tie ¶
Resolution is per-resource and most-specific-wins: a limit set directly on a subject beats one set on their email, which beats one set on a role, which beats one set on a group, which beats the deployment default. Within one tier — the case that actually happens, a user in several groups — the *smallest* limit wins.
That is the opposite of how role bindings resolve (strongest role wins) and the difference is deliberate. A role is a grant, so unioning grants is the safe reading of "member of both". A quota is a ceiling, so unioning ceilings would mean any tenant could lift their own cap by joining one more group — turning group membership into a privilege-escalation primitive. The minimum is also order-independent, which matters because the alternative (first match wins) makes a security property depend on YAML line order.
Index ¶
- Variables
- func SubjectForIdentity(id string) *authz.Subject
- type Binding
- type Config
- type CounterRow
- type Denial
- type Effective
- type Enforcer
- func (e *Enforcer) Admit(subject *authz.Subject, res Resource, n float64) (Effective, error)
- func (e *Enforcer) CheckSpend(subject *authz.Subject) error
- func (e *Enforcer) CheckSpendIdentity(identity string) error
- func (e *Enforcer) ClearOverride(identity string) (bool, error)
- func (e *Enforcer) Denials() map[Resource]uint64
- func (e *Enforcer) Effective(subject *authz.Subject) Effective
- func (e *Enforcer) Enabled() bool
- func (e *Enforcer) HasClaimBindings() bool
- func (e *Enforcer) KnowsSubject(identity string) bool
- func (e *Enforcer) Load() error
- func (e *Enforcer) Reconcile(live LiveState) error
- func (e *Enforcer) Release(identity string, res Resource, n float64)
- func (e *Enforcer) SessionAdmission(subject *authz.Subject, existing int) int
- func (e *Enforcer) SetOverride(identity string, limits Limits, actor string) error
- func (e *Enforcer) Snapshot(extra []string) []View
- func (e *Enforcer) Spend(identity string, tokens, usd float64)
- func (e *Enforcer) Usage(identity string) Usage
- func (e *Enforcer) ViewFor(subject *authz.Subject) View
- type Kind
- type Limits
- type LiveState
- type Option
- type Override
- type Resolver
- type Resource
- type Store
- type Usage
- type View
Constants ¶
This section is empty.
Variables ¶
var AllResources = []Resource{ ResProjects, ResConcurrentTasks, ResConcurrentReproductions, ResExecutors, ResSessions, ResDailyTokens, ResDailyCostUSD, }
AllResources lists every resource in a stable order: config validation, the admin panel's column order, and the /metrics gauge order all read it.
Functions ¶
func SubjectForIdentity ¶
SubjectForIdentity reconstructs the claims an identity key encodes. It is exported for callers that hold an identity string but no live subject — an API token's minter, a project registry owner — and resolves email and sub bindings but not group ones, which the key does not carry.
Types ¶
type Binding ¶
type Binding struct {
// Claim selects what Value is compared against: group, role, email, sub.
Claim authz.ClaimKind
// Value is the claim value to match, normalized like authz's.
Value string
// Limits are the ceilings this binding contributes.
Limits Limits
}
Binding assigns limits to everyone matching one claim. It mirrors authz.Binding minus the scope fields: a quota is a property of an identity across the whole hub, not of an identity within one project. Scoping it per project would defeat the point, since the tenant chooses how many projects they have.
type Config ¶
type Config struct {
// Defaults apply to every identity that no binding covers. Leaving it
// empty means "unlimited by default", which is the correct behaviour
// for the single-tenant hub this feature must not disturb.
Defaults Limits
// Bindings assign narrower (or wider) limits per claim.
Bindings []Binding
}
Config is the deployment's quota policy.
type CounterRow ¶
type CounterRow struct {
Identity string
Resource Resource
// Bucket is the UTC day ("2006-01-02") for daily resources and "" for
// gauges. Keeping it in the key rather than resetting rows at midnight
// means a day rollover needs no scheduled job and no clock the process
// has to stay awake for: yesterday's row simply stops being read.
Bucket string
Value float64
UpdatedAt time.Time
}
CounterRow is one persisted usage counter.
type Denial ¶
type Denial struct {
Identity string
Resource Resource
Limit float64
Used float64
Requested float64
// RetryAfter is non-zero when waiting alone will clear the denial: a
// running task will finish, a UTC day will roll over. It is zero when
// the tenant or an admin must actually do something — delete a project,
// revoke an executor, raise the cap — because a Retry-After that will
// never come true trains clients to hammer a wall.
RetryAfter time.Duration
// Source is where the breached limit came from, for the error detail.
Source string
}
Denial is the typed refusal returned by Admit. It carries everything the HTTP layer needs to build the response and everything an operator needs to understand the refusal without reading the server log.
type Effective ¶
type Effective struct {
// Identity is the key limits are accounted against — authz.Subject's
// Label(), the same string oidcauth.Identity.OwnerKey() produces, so a
// project's Owner and a quota's identity are always the same value.
Identity string
// Limits are the ceilings in force.
Limits Limits
// Sources says where each ceiling came from ("override", "group:eng",
// "default"). The admin panel shows it so "why is this user capped at
// two projects?" is answerable without reading YAML.
Sources map[Resource]string
}
Effective is the resolved ceiling set for one identity, with provenance.
type Enforcer ¶
type Enforcer struct {
// contains filtered or unexported fields
}
Enforcer holds the live counters and decides admissions. Safe for concurrent use; the zero value is not usable, call NewEnforcer.
func NewEnforcer ¶
NewEnforcer builds an Enforcer over resolver and store. A nil store makes the enforcer purely in-memory, which is what `cloop ui` falls back to when the control-plane database cannot be opened: caps still hold for the life of the process rather than silently vanishing.
func (*Enforcer) Admit ¶
Admit reserves n units of res for subject, or returns a *Denial.
The reservation and the limit check happen in one critical section, which is the whole point: concurrent callers cannot both observe headroom that only one of them can have.
func (*Enforcer) CheckSpend ¶
CheckSpend reports whether identity has headroom left in its daily budgets without reserving anything. Used before dispatching a run: the exact cost is unknown up front, so the gate is "has this tenant already blown its budget?" rather than a reservation.
func (*Enforcer) CheckSpendIdentity ¶ added in v0.0.2
CheckSpendIdentity is CheckSpend for a caller that holds an identity string rather than a live subject — the hub's spend drain, which books a run's cost long after the request that started it is gone (Task 20264).
It exists because the obvious spelling, CheckSpend(SubjectForIdentity(id)), is wrong in two independent ways:
The counter key does not round-trip. Spend keys by the identity string; SubjectForIdentity("local") yields Subject{Sub: "local"}, whose Label() is "sub:local". Spend books to one key and the check reads another, so every run attributed to the "local" sentinel — or to any registry owner that is a bare username rather than an email — is billed to a counter nothing ever consults. The budget silently never applies.
The limits come out wrong. A synthesized subject carries no groups and no roles, so every group- and role-derived binding fails to match and resolution falls back to the defaults. A tenant admitted at start against a generous group budget would then be refused mid-run against a tighter default, and killed for exceeding a limit that is not theirs.
Both are avoided by keying the counter on the identity exactly as Spend does, and resolving limits through the subject the enforcer remembers for that identity — the real one, with its claims, when this process has seen it.
func (*Enforcer) ClearOverride ¶
ClearOverride drops an identity's edit, returning it to policy.
func (*Enforcer) Enabled ¶
Enabled reports whether any policy is in force. When false every Admit succeeds, which is what keeps a single-tenant hub untouched.
func (*Enforcer) HasClaimBindings ¶ added in v0.0.2
HasClaimBindings reports whether the policy resolves any limit from a group or role claim. See Resolver.HasClaimBindings.
func (*Enforcer) KnowsSubject ¶ added in v0.0.2
KnowsSubject reports whether this process has seen the real claims for identity, as opposed to only ever synthesizing a subject from the identity string.
Callers that act on a resolved limit — rather than merely displaying one — use it together with Resolver.HasClaimBindings to tell "this tenant is over their ceiling" apart from "this tenant is over the default ceiling and their real one is unknown". The two are indistinguishable from the resolved limit alone, and only the first justifies stopping work.
func (*Enforcer) Load ¶
Load reads overrides and counters from the store. Stale day buckets are dropped on the way in and pruned from storage, so a long-lived hub does not accumulate a row per identity per day forever.
func (*Enforcer) Reconcile ¶
Reconcile replaces every gauge counter from live state.
This runs at startup because the alternative — trusting the persisted counter — fails in the direction that hurts: a hub killed while three runs were in flight comes back believing that tenant is still using three slots, and keeps believing it forever. Nothing decrements a counter for a task whose process no longer exists. Rebuilding from what is actually running, enrolled and signed in makes a crash cost a moment of accuracy instead of a permanently narrowed tenant.
Daily counters are deliberately untouched: they are cumulative, they have no live equivalent to rebuild from, and re-deriving them would hand a compromised account a fresh token budget on every restart.
func (*Enforcer) Release ¶
Release returns n units of a gauge resource. Releasing more than is held clamps at zero rather than going negative: a double release (a run that reports completion twice) must not mint headroom.
func (*Enforcer) SessionAdmission ¶
SessionAdmission computes how many of an identity's existing sessions must be evicted to make room for one more.
Sessions are the one resource enforced by eviction rather than refusal, and the reason is a lockout: refusing the login of a user who is already at their session cap leaves them with no session, and the self-service remedy (POST /api/session/logout-all) requires one. Evicting the least recently used session preserves the invariant the cap exists for — no identity ever holds more than N — without ever locking anyone out of their own account.
func (*Enforcer) SetOverride ¶
SetOverride records an admin's per-identity edit.
func (*Enforcer) Snapshot ¶
Snapshot returns one View per identity the enforcer knows about: everyone with an override, everyone holding a counter, everyone seen this process, plus any extra identities the caller supplies (the project registry knows about owners who have not made a request yet).
func (*Enforcer) Spend ¶
Spend records consumption of a daily resource after the fact. Unlike Admit it never refuses: the tokens are already gone, and refusing to record them would be the one bug that lets a tenant exceed a budget indefinitely.
type Kind ¶
type Kind int
Kind distinguishes the two accounting models described in the package doc.
type Limits ¶
Limits is a sparse set of ceilings. A resource absent from the map is unlimited; a resource present with 0 means zero is allowed, which is a real and useful setting (a tenant who may run but may not enrol hardware).
Sparseness is what makes partial overrides work: an admin can raise one user's max_projects without restating — and accidentally freezing — every other ceiling that user inherits from their group.
func (Limits) Normalize ¶
Normalize returns a copy with unknown resources rejected and negative ceilings treated as "unlimited" rather than "deny everything".
The negative case is a footgun worth disarming explicitly: an operator who types -1 means "no limit" in almost every system that has ever had a quota, and reading it as a ceiling of -1 would deny every request from that tenant with no obvious cause.
type LiveState ¶
type LiveState struct {
Projects map[string]float64
Tasks map[string]float64
Executors map[string]float64
Sessions map[string]float64
}
LiveState is ground truth for the gauge resources, gathered from the things that actually exist rather than from a counter that may have drifted.
Every map is identity → count. An identity absent from a map holds none of that resource; the whole gauge table is replaced, not merged, so a counter left behind by a crash mid-run is erased rather than decremented forever.
type Option ¶
type Option func(*Enforcer)
Option configures an Enforcer.
func WithStoreErrorHandler ¶
WithStoreErrorHandler installs a callback for persistence failures.
type Resolver ¶
type Resolver struct {
// contains filtered or unexported fields
}
Resolver turns a Config into per-identity limits. It is immutable after New and safe for concurrent use.
func New ¶
New validates cfg and returns a Resolver.
Validation is strict for the same reason authz.New's is: a typo in a resource name or a claim kind produces a binding that matches nothing, which presents as "the quota I configured is not being enforced" — the failure mode a quota system can least afford to have silently.
func (*Resolver) Configured ¶
Configured reports whether any policy was set. A hub with no quota config enforces nothing, which keeps single-tenant local use exactly as it was.
func (*Resolver) HasClaimBindings ¶ added in v0.0.2
HasClaimBindings reports whether any binding resolves off a group or role claim, so a caller holding only an identity string can tell whether a synthesized subject would under-resolve that identity's ceiling.
It exists for the hub's spend drain, which runs long after the request that carried the claims. When this is true and the enforcer has not seen the real subject, the resolved limit is the default rather than whatever binding the tenant was actually admitted under — and acting on that would stop work for exceeding a limit that does not apply.
func (*Resolver) Resolve ¶
Resolve computes the ceilings for subject, with override applied on top.
override is the admin's per-identity edit from the store. It is applied last and per-resource, so an admin can raise one ceiling for one person without inheriting responsibility for the rest of their limits.
type Resource ¶
type Resource string
Resource names one thing an identity can consume. The string values are the wire and YAML form and are part of the API contract: never rename one.
const ( // ResProjects caps how many projects an identity may own at once. ResProjects Resource = "max_projects" // ResConcurrentTasks caps how many runs an identity may have executing // at once. This is the cap that actually protects the fleet: every // concurrent run holds an executor slot and spends tokens. ResConcurrentTasks Resource = "max_concurrent_tasks" // ResConcurrentReproductions caps how many hermetic reproductions // (Task 20221) an identity may have executing at once. // // Deliberately its own gauge rather than a share of ResConcurrentTasks. // A reproduction costs about what the run it reproduces cost — a full // sandbox, a full model call, and the project's test suite on two trees — // so an auditor sweeping a quarter of commits would drain the same pool the // tenant's real work draws from. Counting them apart means a reproduction // storm slows reproductions and nothing else, which is the non-destructive // contract extended from the repository to the fleet's capacity. ResConcurrentReproductions Resource = "max_concurrent_reproductions" // ResExecutors caps how many executors an identity may enrol. Enrolling // a device grants it the right to run workloads, so an unbounded tenant // can grow the trusted compute base without an admin ever looking. ResExecutors Resource = "max_executors" // ResSessions caps concurrent signed-in sessions per identity. Enforced // by evicting the oldest rather than by refusing the newest — see // SessionAdmission. ResSessions Resource = "max_sessions" // ResDailyTokens caps input+output tokens spent per UTC day. ResDailyTokens Resource = "daily_token_budget" // ResDailyCostUSD caps estimated USD spend per UTC day. ResDailyCostUSD Resource = "daily_cost_usd" )
type Store ¶
type Store interface {
// LoadOverrides returns every per-identity override.
LoadOverrides() ([]Override, error)
// PutOverride writes (or replaces) one identity's override. Empty
// limits are stored as an explicit "no overrides" rather than deleting
// the row, so the admin panel can distinguish "reset to policy" from
// "never edited".
PutOverride(o Override) error
// DeleteOverride removes one identity's override, reporting whether a
// row existed.
DeleteOverride(identity string) (bool, error)
// LoadCounters returns every persisted counter, including stale day
// buckets; the caller decides which are still current.
LoadCounters() ([]CounterRow, error)
// PutCounter writes one counter's value.
PutCounter(c CounterRow) error
// ReplaceGauges atomically swaps the whole set of gauge counters
// (bucket "") for the supplied rows. Used by Reconcile: a per-row
// update could not delete the rows for identities that no longer hold
// anything, which is exactly the drift reconciliation exists to remove.
ReplaceGauges(rows []CounterRow) error
// PruneCountersBefore deletes daily counters older than bucket.
PruneCountersBefore(bucket string) error
}
Store persists overrides and counters. The SQLite implementation is pkg/quotastore; keeping it behind an interface is what lets this package stay free of a driver and lets tests run without a database.
type View ¶
type View struct {
Identity string `json:"identity"`
Limits Limits `json:"limits,omitempty"`
Usage Usage `json:"usage"`
Sources map[Resource]string `json:"sources,omitempty"`
// Overridden reports whether an admin has edited this identity, so the
// panel can offer "reset to policy" only where there is something to
// reset.
Overridden bool `json:"overridden"`
}
View pairs an identity's ceilings with its consumption. It is what the admin panel renders and what /metrics exports.
Source Files