gateway

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2026 License: Apache-2.0 Imports: 37 Imported by: 0

Documentation

Overview

Package gateway holds the traffic-plane logic shared by the halo-kong plugin and halo-shadow: request inspection, cohort decision and model rewrite. It is pure (no Kong, no network) so the plugin stays a thin adapter.

Index

Constants

View Source
const (
	// HeaderActor carries the verified actor chain ("planner>reviewer") and
	// HeaderActorDepth its length upstream. Gateway-owned like every x-halo-*:
	// a client copy is always discarded.
	HeaderActor      = "x-halo-actor"
	HeaderActorDepth = "x-halo-actor-depth"
)
View Source
const (
	DefaultBudgetTTL   = 30 * time.Second // a verdict is reused this long per principal
	DefaultBudgetGrace = 15 * time.Minute // a cached verdict outlives an unreachable halo-server this long
	DefaultBudgetFlush = 5 * time.Second  // usage report interval

)

Budget client defaults (ADR-0017).

View Source
const (
	HaloPrefix       = "x-halo-"
	HeaderRing       = "x-halo-ring"
	HeaderRelease    = "x-halo-release"
	HeaderExperiment = "x-halo-experiment"
	HeaderVariant    = "x-halo-variant"

	RingUnknown = "unknown"
)

Headers the gateway stamps on upstream requests. Everything under HaloPrefix is owned by the gateway; client-supplied values are always discarded.

View Source
const (
	DefaultDirectoryTTL   = time.Minute     // an entry is reused this long before halo-server is asked again
	DefaultDirectoryGrace = 5 * time.Minute // a cached entry outlives an unreachable halo-server this long

)

Directory defaults.

View Source
const (
	// DirectoryFailOpen (default) keeps the verified subject with no groups:
	// default routing, no experiments, and still the posture gate and per-user
	// budgets. A deactivated user is never demoted to an anonymous caller.
	DirectoryFailOpen = "open"
	// DirectoryFailClosed refuses the request (ErrDirectoryUnavailable, a 503).
	DirectoryFailClosed = "closed"
)

Directory fail modes (directory.failMode): what Resolve does when the directory cannot be asked and no entry younger than Grace is cached.

View Source
const (
	GatePosture = "posture"
	GateVersion = "version"
)

Gates the traffic plane applies per ring (policy Ring.Posture / Ring.VersionGate).

View Source
const (
	DefaultPostureTTL   = time.Minute      // a verdict is reused this long before halo-server is asked again
	DefaultPostureGrace = 15 * time.Minute // a cached verdict outlives an unreachable halo-server this long

)

Posture defaults.

View Source
const (
	ProtoAnthropic = "anthropic-messages"
	ProtoBedrock   = "bedrock-invoke"
	ProtoResponses = "openai-responses"
	ProtoGemini    = "gemini"
)

Wire protocols (match policy.Gateway.Protocols values).

View Source
const (
	PathMessages    = "/v1/messages"
	PathCountTokens = "/v1/messages/count_tokens" //nolint:gosec // URL path, not a credential
	PathResponses   = "/v1/responses"
)

Exact model-call paths. Anything else under a model-bearing prefix (/v1/messages/*, /v1/responses/*, /model/*) is rejected by Prepare.

View Source
const (
	DefaultKillInterval = 10 * time.Second
	DefaultKillMaxAge   = 10 * time.Minute
)

Kill-switch poller defaults.

View Source
const (
	// DefaultOutputCeiling caps the output estimate of a request that sets no
	// maximum output tokens: no supported model generates more in one call.
	DefaultOutputCeiling = 128_000
	// DefaultServerStateInputTokens is the input charged, when final usage is
	// missing, for a request that names server-side state (ServerStateInput):
	// a full long-context window, since its real size is invisible here.
	DefaultServerStateInputTokens = 200_000
)

Estimate defaults.

Variables

View Source
var ErrDirectoryUnavailable = errors.New("gateway: directory unavailable")

ErrDirectoryUnavailable is Resolve's answer under FailClosed when halo-server cannot be asked and no entry younger than Grace is cached. Callers answer RejectDirectoryUnavailable.

View Source
var ErrNotProvisioned = errors.New("gateway: subject is deactivated or not provisioned in the directory")

ErrNotProvisioned is Resolve's answer for a verified subject the directory does not list as active (deactivated, deleted or never provisioned). Callers refuse the request (RejectNotProvisioned): demoting it to an anonymous caller would skip the posture gate and per-user budgets.

Functions

func Background added in v0.3.0

func Background(proto string, body []byte) bool

Background reports an OpenAI Responses background request.

func CheckDirectoryFailMode added in v0.3.0

func CheckDirectoryFailMode(m string) error

CheckDirectoryFailMode validates a directory.failMode value ("" = open).

func EstimateUsage added in v0.3.0

func EstimateUsage(r UsageResult, e Estimate) budget.Usage

EstimateUsage is the pessimistic charge for a metered 2xx response whose final usage was not read (r.Complete false), so hanging up before the last event never makes a call free:

  • input as reported so far (Anthropic's message_start), else request bytes / 3; at least e.ServerStateInput when the request names server-side state, whose size the request bytes do not show;
  • output the larger of the output reported so far and a measure of what was generated: for a decoded stream the delta payload text / 4 (at least one token per delta event), so SSE framing and JSON escaping are not charged; otherwise response bytes / 4. With reasoning on, at least the thinking budget (else the limit); in background mode the limit. Capped at the request's maximum output tokens, else e.OutputCeiling;
  • never zero.

func EventStreamToSSE

func EventStreamToSSE(rc io.ReadCloser) io.ReadCloser

EventStreamToSSE converts an AWS event stream (Bedrock invoke-with-response-stream) carrying Anthropic events into the SSE the Messages API streams: each chunk's {"bytes": base64(event json)} becomes "event: <type>\ndata: <json>\n\n". Exception frames become an SSE error event.

func ForceNonStreaming

func ForceNonStreaming(protocol, path string, body []byte) (string, []byte, error)

ForceNonStreaming makes a request non-streaming (used for shadow copies).

func HarnessFromUA

func HarnessFromUA(ua string) string

HarnessFromUA maps a User-Agent to a harness adapter name.

func IdentityHeaderFor

func IdentityHeaderFor(org *policy.Org, override string) string

IdentityHeaderFor picks the configured header, falling back to the org's.

func Inspect

func Inspect(protocol, path string, body []byte) (model string, firstTurn bool)

Inspect extracts the requested model alias and whether the request is a first turn. Bedrock carries the alias in the path, the others in the body. Unparseable bodies yield ("", false): never shadowed, routed by default.

func IsCountTokens

func IsCountTokens(p string) bool

IsCountTokens reports whether p is a token-counting call (never mirrored, excluded from request metrics).

func MaxOutputTokens added in v0.3.0

func MaxOutputTokens(body []byte) int64

MaxOutputTokens is a model request's output token limit on any supported wire; 0 when it sets none or does not parse.

func ParseUA

func ParseUA(ua string) (harness, version string)

ParseUA returns the harness and CLI version a User-Agent names. The formats (sources in docs/adr/0011-gateway-enforced-posture-and-version.md):

claude-cli/2.1.287 (external, sdk-cli)               Claude Code
codex_cli_rs/0.99.0 (Mac OS 15.1.0; arm64) iTerm.app  Codex (originator/version (os; arch) terminal)
GeminiCLI/0.26.0/gemini-2.5-pro (darwin; x64)        Gemini CLI (GeminiCLI[-client]/version/model (platform; arch[; surface]))
opencode/1.19.3 ai-sdk/provider-utils/4.0.46 runtime/bun/1.3.14   OpenCode (opencode/version, then the AI SDK and runtime)

The version is the token after the first '/', up to a space, '/', '(' or ';'. version is "" when the harness is known but no semver follows; harness is "" for an unrecognised UA (Copilot CLI's is not documented and has none).

func ProtocolForPath

func ProtocolForPath(p string) string

ProtocolForPath classifies a request path by exact match; "" means "not a model call". count_tokens is anthropic-messages (it carries a model alias).

func ReasoningBudget added in v0.3.0

func ReasoningBudget(proto string, body []byte) (bool, int64)

ReasoningBudget reports whether a request enables hidden reasoning, and the token budget it sets (0 = none): OpenAI Responses reasoning (unless effort "none"); Anthropic thinking enabled or adaptive (budget_tokens) or an output_config effort, also under Bedrock Converse additionalModelRequestFields; Gemini generationConfig.thinkingConfig.

func RewriteRequest

func RewriteRequest(protocol, path string, body []byte, model string) (string, []byte, error)

RewriteRequest points a request at upstream model `model`: the JSON "model" field for anthropic-messages / openai-responses, the /model/{id}/ path segment for bedrock-invoke. Only the request is touched; responses stream through untouched. Empty model or unknown protocol is a no-op.

func RingGate

func RingGate(org *policy.Org, ring, gate string) string

RingGate is ring's mode for gate (GatePosture or GateVersion); off when the ring is unknown.

func RingPins

func RingPins(org *policy.Org, ring string) map[string][]string

RingPins returns the CLI versions ring r's devices may run per harness: the ring profile's pin plus the variant profiles of its running client-axis experiment (the gateway cannot tell which variant a device applied).

func RouteOrder

func RouteOrder(alias string, r policy.ModelRoute, key string) []policy.RouteTarget

RouteOrder returns the route's targets in attempt order for key. Targets are grouped into priority tiers (ascending). Within a tier the weighted pick (internal/assign) goes first and the rest follow in policy order; a tier without weights keeps policy order. The first element is the target a request lands on when everything is healthy.

func ServerStateInput added in v0.3.0

func ServerStateInput(proto string, body []byte) bool

ServerStateInput reports whether a request makes the provider bill input tokens the request body does not carry, so request bytes underestimate it:

  • OpenAI Responses: previous_response_id, conversation, a stored prompt id, background mode, input_file by file_id or file_url, input_image by file_id or an http(s) URL, item_reference, and the file_search, web_search*, mcp and code_interpreter tools;
  • Anthropic Messages (and Bedrock invoke): server tools (web_search, web_fetch, code_execution, tool_search), MCP servers, container uploads, and document or image blocks whose source is a file or a URL.

func SessionID

func SessionID(h http.Header) string

SessionID is the harness session id: Claude Code's header, then Codex's.

func StickyKey

func StickyKey(userID, sessionID string) string

StickyKey is what weighted route choice hashes: the verified identity plus the harness session, so a session stays on one target. "" (anonymous, no session) means no stickiness is possible: the first-priority target is used.

func Target

func Target(raw string) (scheme, host string, port int, pathPrefix string, err error)

Target splits an upstream URL for Kong's balancer.

func UsageTap added in v0.3.0

func UsageTap(proto, contentType, contentEncoding string, rc io.ReadCloser, done func(UsageResult)) io.ReadCloser

UsageTap wraps a model response body in the caller's wire protocol and reports its token usage once the body ends (EOF, read error or Close), exactly once. It never changes the bytes the client receives. A body with Content-Encoding gzip or deflate is decompressed for the scan only, by a goroutine that exits when the body ends (the caller must Close it); any other encoding is not scanned and reports Complete false.

The response is scanned as it streams: SSE line by line (Anthropic message_start/message_delta, OpenAI response.completed, Gemini chunks), an AWS event stream frame by frame (Bedrock invoke/converse streams), or, for a plain JSON body, its last 64 KiB. Counts merge by maximum, since every wire reports cumulative totals.

Types

type Breaker

type Breaker struct {
	Threshold int           // consecutive failures that open the circuit (default 5)
	Cooldown  time.Duration // open time before a half-open probe (default 30s)
	// contains filtered or unexported fields
}

Breaker is a per-key circuit breaker. After Threshold consecutive failures a key is open for Cooldown; then exactly one probe request is let through (half-open): success closes it, failure re-opens it for another Cooldown. Safe for concurrent use. Allow must be followed by Done or Cancel.

func (*Breaker) Allow

func (b *Breaker) Allow(key string) bool

Allow reports whether a request may be sent to key now.

func (*Breaker) Cancel

func (b *Breaker) Cancel(key string)

Cancel releases an allowed request that never produced a verdict (the client went away), so a half-open probe slot is not leaked.

func (*Breaker) Done

func (b *Breaker) Done(key string, ok bool)

Done records the outcome of an allowed request.

func (*Breaker) State

func (b *Breaker) State(key string) string

State is "closed", "open" or "half-open" (an expired open circuit).

type BudgetCheck added in v0.3.0

type BudgetCheck struct {
	budget.Principal
}

BudgetCheck is halo-server's POST <url>/check body.

type BudgetClient added in v0.3.0

type BudgetClient struct {
	URL, Token        string // URL: .../api/v1/gateway/budget
	TTL, Grace, Flush time.Duration
	Client            *http.Client
	Now               func() time.Time
	Log               *slog.Logger
	// contains filtered or unexported fields
}

BudgetClient is one halo-proxy replica's side of the budget design: halo-server holds the authoritative counters; the replica caches each principal's verdict for TTL and reports its own usage every Flush. Between refreshes the replica adds the usage it has seen to the cached verdict, so one busy caller cannot outrun the cache on a single replica.

func NewBudgetClient added in v0.3.0

func NewBudgetClient(url, token string, ttl, grace, flush time.Duration, log *slog.Logger, allowInsecureInCluster bool) (*BudgetClient, error)

NewBudgetClient validates the config (https unless loopback or allowInsecureInCluster).

func (*BudgetClient) Check added in v0.3.0

func (c *BudgetClient) Check(ctx context.Context, p budget.Principal) (st []budget.Status, known bool)

Check returns the counters p is charged to, with this replica's unreported usage added. known=false when halo-server could not be asked and no verdict younger than Grace is cached. ponytail: concurrent misses for one principal each call halo-server, as in PostureClient; add singleflight if a burst after TTL expiry shows up.

func (*BudgetClient) Counters added in v0.3.0

func (c *BudgetClient) Counters() (reported, failed, dropped int64)

Counters for /metrics: reports sent, reports failed, records dropped.

func (*BudgetClient) FlushNow added in v0.3.0

func (c *BudgetClient) FlushNow(ctx context.Context) error

FlushNow sends pending usage in batches until none is left or a send fails. A failed batch is kept and resent with the same id.

func (*BudgetClient) Record added in v0.3.0

func (c *BudgetClient) Record(p budget.Principal, s budget.Spend, at time.Time)

Record queues usage for the next report and charges it to p's cached verdict.

func (*BudgetClient) Run added in v0.3.0

func (c *BudgetClient) Run(ctx context.Context)

Run reports usage every Flush until ctx ends. Call FlushNow after it returns to send what the last interval left.

func (*BudgetClient) Time added in v0.3.0

func (c *BudgetClient) Time() time.Time

Time is the client's clock (Now, or the wall clock).

type BudgetUsage added in v0.3.0

type BudgetUsage struct {
	Batch   string          `json:"batch"`
	Records []budget.Record `json:"records"`
}

BudgetUsage is halo-server's POST <url>/usage body. Batch is stable across retries so the server applies a batch once.

type BudgetVerdict added in v0.3.0

type BudgetVerdict struct {
	Budgets []budget.Status `json:"budgets"`
}

BudgetVerdict is halo-server's answer: every counter the principal is charged to.

type Decision

type Decision struct {
	Ring          string
	Release       string
	Experiment    string
	Variant       string
	UpstreamName  string
	Upstream      string // URL; "" = leave Kong's default service
	UpstreamModel string // "" = leave the model untouched
	// Route is the ordered attempt list (failover order) for the alias;
	// UpstreamName/UpstreamModel/Upstream are its first element. halo-kong
	// uses only those; halo-proxy walks Route.
	Route   []policy.RouteTarget
	Headers map[string]string
	Shadow  []ShadowTarget
}

Decision is the routing outcome for one request.

func Decide

func Decide(org *policy.Org, req RequestInfo) Decision

Decide is pure and deterministic for a given (org, req).

type DirectoryClient added in v0.3.0

type DirectoryClient struct {
	URL, Token string
	TTL, Grace time.Duration
	Client     *http.Client
	Now        func() time.Time
	Log        *slog.Logger
	// FailClosed: refuse (ErrDirectoryUnavailable) instead of failing open
	// when the directory cannot be asked (DirectoryFailClosed).
	FailClosed bool
	// contains filtered or unexported fields
}

DirectoryClient makes halo-server's SCIM directory authoritative for a verified subject (ADR-0022): the subject's groups are the directory's, and a subject the directory does not list as active is refused (ErrNotProvisioned). Groups never come from the request.

func NewDirectoryClient added in v0.3.0

func NewDirectoryClient(url, token string, ttl, grace time.Duration, log *slog.Logger, allowInsecureInCluster bool) (*DirectoryClient, error)

NewDirectoryClient validates the config (https unless loopback or allowInsecureInCluster).

func (*DirectoryClient) Resolve added in v0.3.0

func (c *DirectoryClient) Resolve(ctx context.Context, sub *policy.Subject) (*policy.Subject, error)

Resolve returns sub with the directory's groups. It returns nil, nil (unknown caller) only when sub is nil, and ErrNotProvisioned when the directory does not list sub as active, when halo-server answers the lookup with a non-200 status below 500, or when the last cached answer for sub was inactive, however old. When halo-server cannot be asked (transport error or 5xx) and no entry younger than Grace is cached it returns sub with no groups (fail open), or ErrDirectoryUnavailable under FailClosed: never nil, which would let an outage turn a deactivated user into an anonymous caller that skips the posture gate and per-user budgets. Workload subjects (ADR-0023) are not in SCIM: they are returned unchanged without a lookup. Kind and Delegation always come from the verified credential, never from the directory: a delegated agent (ADR-0024) is looked up as its root principal, so an agent acting for a deprovisioned user is refused like the user, and keeps its chain (and the agent policy) on every path, fail open included. ponytail: concurrent misses for one subject each call halo-server, as PostureClient.Check does.

type DirectoryEntry added in v0.3.0

type DirectoryEntry struct {
	Active bool     `json:"active"`
	Groups []string `json:"groups,omitempty"`
}

DirectoryEntry is halo-server's SCIM directory answer for one subject (GET /api/v1/gateway/directory). Inactive covers deactivated, deleted and never provisioned users alike.

type Estimate added in v0.3.0

type Estimate struct {
	RequestBytes int
	// MaxOutput is the request's maximum output tokens (MaxOutputTokens); 0 = none.
	MaxOutput int64
	// OutputCeiling caps the output estimate when MaxOutput is 0 (0 = DefaultOutputCeiling).
	OutputCeiling int64
	// ServerStateInput is the input to charge when the request makes the
	// provider bill input its bytes do not show (ServerStateInput true); 0 = it does not.
	ServerStateInput int64
	// Reasoning: the request enables hidden reasoning (ReasoningBudget), so
	// the output estimate is at least ThinkingBudget, or the output limit when
	// ThinkingBudget is 0: reasoning tokens are billed as output but never
	// streamed, so a stream cut before the final usage event shows none of them.
	Reasoning      bool
	ThinkingBudget int64
	// Background: an OpenAI Responses background request; its response is
	// queued (usage null), so the output is charged at the limit.
	Background bool
}

Estimate is what EstimateUsage knows about the request (RequestEstimate).

func RequestEstimate added in v0.3.0

func RequestEstimate(proto string, body []byte, serverStateInput, outputCeiling int64) Estimate

RequestEstimate is the Estimate for a metered request body: maximum output, server-side state (charged serverStateInput, else DefaultServerStateInputTokens), reasoning and background mode.

type Finding

type Finding struct {
	Gate   string // GatePosture | GateVersion
	Mode   string // policy.GateWarn | policy.GateEnforce
	Reason string
}

Finding is a failed gate check under a ring whose mode is warn or enforce.

func CheckPosture

func CheckPosture(ctx context.Context, c *PostureClient, org *policy.Org, ring string, sub *policy.Subject) (f *Finding, unknown bool)

CheckPosture applies ring's posture gate to a verified subject. unknown reports that the gate is on but no verdict was available (no client, an anonymous caller, or halo-server unreachable past the grace window): the request passes and callers count it.

func CheckVersion

func CheckVersion(org *policy.Org, ring, ua string) *Finding

CheckVersion applies ring's versionGate to a request's User-Agent. nil = pass (or gate off). A ring that pins no version for the detected harness passes: there is nothing to compare against.

func (*Finding) Message

func (f *Finding) Message() string

Message is what the developer's CLI shows on a refusal: what failed and what to run.

func (*Finding) Reject

func (f *Finding) Reject(proto string) *Rejection

Reject is the 403 to answer in enforce mode, in the caller's wire format; nil in warn mode.

type KillEnvelope

type KillEnvelope struct {
	Payload   []byte `json:"payload"`   // base64 in JSON
	Signature []byte `json:"signature"` // base64 in JSON
}

KillEnvelope is GET /api/v1/gateway/killswitch: Payload is the JSON KillList, Signature is ed25519 over killDomain+Payload.

func SignKillList

func SignKillList(key ed25519.PrivateKey, l KillList) (KillEnvelope, error)

SignKillList seals l for gateways.

type KillList

type KillList struct {
	Version     uint64   `json:"version"` // grows with every kill/unkill; informational
	Experiments []string `json:"experiments"`
	// Toggles are killed feature-toggle names: treated as off everywhere, whatever
	// their rules say. Part of the signed payload, so covered by the same
	// signature, freshness and monotonicity checks as Experiments.
	Toggles  []string  `json:"toggles,omitempty"`
	IssuedAt time.Time `json:"issuedAt"`
}

KillList is the set of experiments halo-server has killed: gateways treat them as not running (control routing, no shadow) without waiting for a policy PR to merge and a snapshot to ship.

func VerifyKillList

func VerifyKillList(pub ed25519.PublicKey, env KillEnvelope, now time.Time, maxAge time.Duration) (KillList, error)

VerifyKillList checks the signature and freshness (IssuedAt within maxAge of now, at most a minute in the future) and returns the list.

type KillSwitch

type KillSwitch struct {
	URL      string
	Token    string
	PubKey   ed25519.PublicKey
	Interval time.Duration // default DefaultKillInterval
	MaxAge   time.Duration // default DefaultKillMaxAge
	Client   *http.Client  // default: 10s timeout, no redirects
	Log      *slog.Logger
	Now      func() time.Time
	// contains filtered or unexported fields
}

KillSwitch polls halo-server for the signed kill list.

Failure policy: a fetch or verification failure keeps the last accepted list (a kill never silently lapses because the control plane is down); if no list was ever accepted the set is empty and the policy snapshot's statuses alone decide. A list is accepted only if it is strictly newer (IssuedAt) than the current one, so a replayed older envelope cannot undo a kill. A nil *KillSwitch is valid and kills nothing.

func NewKillSwitch

func NewKillSwitch(url, token string, pub ed25519.PublicKey, interval time.Duration, log *slog.Logger, allowInsecureInCluster bool) (*KillSwitch, error)

NewKillSwitch validates the config (see bundle.ParseEd25519Verifier for PEM keys). allowInsecureInCluster permits plain http to a non-loopback host.

func (*KillSwitch) Apply

func (k *KillSwitch) Apply(org *policy.Org) *policy.Org

Apply returns org with every killed experiment's status forced to "paused", so Decide routes its users to default (control) and mirrors nothing. org itself is never mutated; it is returned as-is when nothing applies. ponytail: copies per request while a kill is active; cache by (org, version) if it shows in profiles.

func (*KillSwitch) Killed

func (k *KillSwitch) Killed() KillList

Killed returns the current list (a copy).

func (*KillSwitch) Refresh

func (k *KillSwitch) Refresh(ctx context.Context) error

Refresh fetches and verifies the list once; on error the previous list stays.

func (*KillSwitch) Restore

func (k *KillSwitch) Restore(l KillList)

Restore seeds a last-known list persisted by the caller (halod keeps it in its state file) so kills survive a restart and envelopes issued before it are refused as replays. It is a no-op unless l is newer than the current list. The list is trusted as given: pass only one that was verified.

func (*KillSwitch) Run

func (k *KillSwitch) Run(ctx context.Context)

Run refreshes immediately and then every Interval until ctx ends.

type Outbound

type Outbound struct {
	Path     string
	RawQuery string
	SetQuery bool // replace the client's query string with RawQuery (else keep it)
	Body     []byte
	// SSE: the response is a Bedrock event stream the anthropic-messages client
	// cannot read; convert it with EventStreamToSSE.
	SSE bool
}

Outbound is the request an upstream expects, derived from the client's request. Path is escaped and relative to the upstream's base URL.

func BuildOutbound

func BuildOutbound(proto string, up policy.Upstream, model, path string, body []byte) (Outbound, error)

BuildOutbound rewrites the client's request (protocol proto, escaped path, body) for upstream up serving model. Same-wire upstreams only get the model swapped (RewriteRequest). Translations:

anthropic-messages -> bedrock   POST /model/{id}/invoke[-with-response-stream], body without model/stream + anthropic_version
anthropic-messages -> vertex    POST .../publishers/anthropic/models/{id}:[stream]rawPredict, body without model + anthropic_version
openai-responses   -> azure     POST /openai/v1/responses (or /openai/responses?api-version=), model = deployment

Responses stay in the provider's format except Bedrock streams, which are converted to SSE (see Outbound.SSE).

type PostureClient

type PostureClient struct {
	URL, Token string
	TTL, Grace time.Duration
	Client     *http.Client
	Now        func() time.Time
	Log        *slog.Logger
	// contains filtered or unexported fields
}

PostureClient asks halo-server for a subject's device posture and caches the verdict. It fails closed only on a verdict halo-server returned: when the server cannot be reached, the last verdict serves for Grace, and after that the posture is unknown (callers let the request through and count it).

func NewPostureClient

func NewPostureClient(url, token string, ttl, grace time.Duration, log *slog.Logger, allowInsecureInCluster bool) (*PostureClient, error)

NewPostureClient validates the config (https unless loopback or allowInsecureInCluster).

func (*PostureClient) Check

func (c *PostureClient) Check(ctx context.Context, subject string) (v PostureVerdict, known bool)

Check returns subject's verdict; known=false when halo-server could not be asked and no verdict younger than Grace is cached. ponytail: concurrent misses for one subject each call halo-server; add singleflight if a burst after TTL expiry shows up in the server's load.

type PostureVerdict

type PostureVerdict struct {
	Compliant bool   `json:"compliant"`
	Reason    string `json:"reason,omitempty"`
}

PostureVerdict is halo-server's answer for one subject (GET /api/v1/gateway/posture).

type Rejection

type Rejection struct {
	Status   int
	Protocol string // selects the error body shape
	Message  string
	Code     string // OpenAI Responses error code; "" = derived from Status
	// RetryAfter, when > 0, is sent as Retry-After (whole seconds, rounded up).
	RetryAfter time.Duration
}

Rejection is a request the gateway answers itself. Fail closed: unknown model endpoints (404), unreadable/oversize bodies (413), models that are not a configured alias (400: a permanent client error, which the CLIs do not retry, unlike 403/5xx/429).

func RejectDirectoryUnavailable added in v0.3.0

func RejectDirectoryUnavailable(path string) *Rejection

RejectDirectoryUnavailable is the 503, in the caller's protocol, for ErrDirectoryUnavailable.

func RejectNotProvisioned added in v0.3.0

func RejectNotProvisioned(path string) *Rejection

RejectNotProvisioned is the 403, in the caller's protocol, for ErrNotProvisioned.

func RejectTooLarge

func RejectTooLarge(path string) *Rejection

RejectTooLarge is the 413 for a model call whose body could not be read in full (Kong spooled it to disk, or it exceeds the proxy limit).

func (*Rejection) Error

func (r *Rejection) Error() string

func (*Rejection) JSON

func (r *Rejection) JSON() []byte

JSON renders the error in the caller's wire format so the CLI shows Message.

func (*Rejection) SetHeaders added in v0.3.0

func (r *Rejection) SetHeaders(h http.Header)

SetHeaders adds the response headers that go with the body: Retry-After, and for Bedrock the x-amzn-ErrorType its SDKs classify errors by.

type RequestInfo

type RequestInfo struct {
	UserID     string // from the auth gateway's identity header only
	Groups     []string
	ModelAlias string
	SessionID  string
	Harness    string
	Protocol   string
	FirstTurn  bool
	// AnyTurn: a parsed, replayable model call at any step of a session
	// (eligible for shadow experiments with turns: all).
	AnyTurn bool
	// Workload: the verified caller is a CI / cloud workload (ADR-0023). It
	// gets no experiment or shadow unless policy workloads.experiments is true.
	Workload bool
	// Agent: the verified caller is a delegated agent acting for UserID
	// (ADR-0024). It gets no experiment or shadow unless policy
	// agents.experiments is true (and, for a workload root, workloads.experiments).
	Agent bool
}

RequestInfo is everything Decide may look at.

func NewRequestInfo

func NewRequestInfo(h http.Header, path string, body []byte) RequestInfo

NewRequestInfo builds RequestInfo from the request alone. UserID/Groups are left empty: cohort comes only from a verified identity (PrepareVerified).

type Result

type Result struct {
	Decision Decision
	Protocol string
	// ClearHeaders are client-supplied x-halo-* names to drop (all of them; the
	// ones Decision sets are then re-set with gateway-owned values).
	ClearHeaders []string
	// Path/Body are the rewritten forms; Rewritten is false when unchanged.
	Path      string
	Body      []byte
	Rewritten bool
	// RewriteErr is non-nil when a rewrite was needed but failed (Reject is
	// then set too: an un-rewritten alias must never reach the upstream).
	RewriteErr error
	// Reject, when non-nil, must be answered by the adapter itself; the
	// request must NOT be forwarded.
	Reject *Rejection
}

Result is everything the Kong plugin must apply to the upstream request.

func PrepareVerified

func PrepareVerified(org *policy.Org, sub *policy.Subject, h http.Header, path string, body []byte) Result

PrepareVerified inspects a request and computes the full set of mutations, or a Rejection the adapter must answer instead of forwarding. Cohort comes only from sub, verified by internal/identity (JWT, or trusted-proxy headers); sub == nil means anonymous: default routing, no experiments. Client-supplied headers (x-halo-* or identity) never influence the outcome.

type RouteRow

type RouteRow struct {
	Alias      string      `json:"alias"`
	Ring       string      `json:"ring"`
	Experiment string      `json:"experiment,omitempty"` // traffic-axis experiment that supplied the route
	Variant    string      `json:"variant,omitempty"`
	Targets    []TargetRow `json:"targets"`
}

RouteRow is the effective route of one alias for a caller.

func RouteTable

func RouteTable(org *policy.Org, req RequestInfo) []RouteRow

RouteTable returns every alias's effective routes for req (UserID, Groups and SessionID decide the ring, experiment variant and weighted pick; ModelAlias is ignored). Anonymous callers (no user, no session) see policy order.

type ShadowTarget

type ShadowTarget struct {
	Experiment string
	Variant    string
	Upstream   string // URL
	Model      string
}

ShadowTarget is one candidate a request should be mirrored to.

type Snapshot

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot serves a compiled org policy (policy.Compile output, or a bare json.Marshal of *policy.Org) from a file and reloads it when the mtime/size changes. A failed reload keeps serving the last good policy.

func NewSnapshot

func NewSnapshot(path string) *Snapshot

func (*Snapshot) Get

func (s *Snapshot) Get() (*policy.Org, error)

Get returns the current org. err is non-nil if the latest load/reload failed; org may still be the previous good snapshot (nil if none ever loaded).

func (*Snapshot) Refused added in v0.3.0

func (s *Snapshot) Refused() uint64

Refused counts snapshots refused for security validation errors (policy.SecurityIssues): a compiled snapshot that skipped `halo validate` never activates.

type TargetRow

type TargetRow struct {
	Order          int      `json:"order"`
	Selected       bool     `json:"selected"`
	Upstream       string   `json:"upstream"`
	Kind           string   `json:"kind"`
	Model          string   `json:"model"`
	Weight         float64  `json:"weight,omitempty"`
	Priority       int      `json:"priority"`
	TimeoutSeconds int      `json:"timeoutSeconds,omitempty"`
	SkippedBy      []string `json:"skippedByHarnesses,omitempty"` // harnesses whose wire protocol this target cannot serve
}

TargetRow is one attempt-order entry; Selected marks where a request lands when every target is healthy.

type Tool added in v0.3.0

type Tool struct {
	Name   string
	Hosted bool
}

Tool is one tool a model call declares. Hosted marks a tool the provider runs or builds in (web search, code execution, a remote MCP server, a versioned Anthropic tool): Name is then its type, its Gemini or Bedrock tool key, or mcp:<server>, and policy matches it as a prefix (see policy.Agents.DeniesTool).

func ToolNames added in v0.3.0

func ToolNames(protocol, path string, body []byte) ([]Tool, error)

ToolNames returns every tool a model call declares, or an error when the body or its tool declarations cannot be read: a body nested past maxJSONDepth, trailing data, a tools member of the wrong shape, a tool without a name, or a (protocol, path) without an extractor. The inspectable shapes are an explicit list:

  • Anthropic Messages and count_tokens: tools[].name and tools[].type (hosted unless "custom"), mcp_servers[].name as mcp:<name>.
  • OpenAI Responses: tools[].name of function and custom tools; any other tools[].type is hosted, mcp as mcp:<server_label>.
  • Bedrock invoke and converse: the Anthropic shape, toolConfig.tools[].toolSpec.name, toolConfig.tools[].systemTool.name (hosted), any other toolConfig.tools[] member (hosted) and additionalModelRequestFields.tools.
  • Gemini generateContent, streamGenerateContent and countTokens: tools[].functionDeclarations[].name, every other tools[] member key (codeExecution, googleSearch, urlContext, ...) as hosted, and the same under generateContentRequest.

Keys match ignoring case and underscores (function_declarations is functionDeclarations) and duplicate keys all count, so a body cannot show the gateway one tool list and the upstream parser another.

type UsageResult added in v0.3.0

type UsageResult struct {
	Usage budget.Usage // field-wise maximum over every usage object read
	Found bool         // at least one usage object was read
	// Complete: the upstream body reached its clean end (EOF) and a usage
	// object was read, so Usage is the provider's final count. A client that
	// hangs up early, an upstream cut, a Content-Encoding the tap cannot decode
	// or a body without usage leave it false: charge EstimateUsage instead.
	Complete bool
	Bytes    int64 // response bytes seen (decoded bytes when the tap decoded them)
	// Streamed: the body was an SSE or AWS event stream the tap decoded, so
	// OutputText and OutputEvents measure the generated output.
	Streamed bool
	// OutputText is the UTF-8 bytes of generated output (text, thinking, tool
	// arguments) in the stream's delta events, without framing or JSON
	// escaping; OutputEvents counts those events.
	OutputText, OutputEvents int64
}

UsageResult is what a UsageTap saw by the time its body ended.

Directories

Path Synopsis
Package adapters renders the "put halo-proxy in front of / behind stack X" config snippets shipped under deploy/integrations.
Package adapters renders the "put halo-proxy in front of / behind stack X" config snippets shipped under deploy/integrations.
Package kong generates the decK declarative config for Kong OSS (db-less) that fronts an org's model traffic and attaches the halo-kong plugin.
Package kong generates the decK declarative config for Kong OSS (db-less) that fronts an org's model traffic and attaches the halo-kong plugin.
Package upstreamauth signs requests to provider upstreams that need a gateway-held credential instead of the caller's, so halo-proxy can talk to Amazon Bedrock directly (no orchestrator, no SigV4 sidecar).
Package upstreamauth signs requests to provider upstreams that need a gateway-held credential instead of the caller's, so halo-proxy can talk to Amazon Bedrock directly (no orchestrator, no SigV4 sidecar).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL