testlive

package
v0.8.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 30, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package testlive is the opt-in live-local harness: tests that reach the owner's real identity provider, Azure DevOps organisation and AWS account. The live suites build only under `-tags live`, skip unless their own WARDYN_LIVE_* gate is set, and never run in CI. docs/LIVE-TESTS.md is the owner's setup and run guide.

Everything the live suites print goes through Redact first, and they take secrets only as paths to files kept outside the repository.

Index

Constants

View Source 
const (
	// MaxTokens is the ceiling on one call's max_tokens.
	MaxTokens = 32
	// DefaultMaxCalls and HardMaxCalls bound WARDYN_LIVE_BEDROCK_MAX_CALLS.
	DefaultMaxCalls = 5
	HardMaxCalls    = 20
	// DefaultModel is Claude Haiku 4.5 through the US cross-region profile.
	DefaultModel = "us.anthropic.claude-haiku-4-5-20251001-v1:0"
)
View Source 
const (
	EnvEntra    = "WARDYN_LIVE_ENTRA"
	EnvADO      = "WARDYN_LIVE_ADO"
	EnvADOWrite = "WARDYN_LIVE_ADO_WRITE" // LL2b: pushes and deletes one scratch branch
	EnvBedrock  = "WARDYN_LIVE_BEDROCK"
	EnvAWSSSO   = "WARDYN_LIVE_AWS_SSO"
	// EnvBedrockWardyn (LL3w) is Bedrock reached THROUGH a Wardyn run — a
	// governed sandbox's own model call — as opposed to EnvBedrock (LL3),
	// which drives the Bedrock data plane directly from the test process.
	EnvBedrockWardyn = "WARDYN_LIVE_BEDROCK_WARDYN"
	// EnvAutonomy (LL5) gates TestLive_AutonomyL0Enforced (#705).
	EnvAutonomy   = "WARDYN_LIVE_AUTONOMY"
	EnvBaseURL    = "WARDYN_LIVE_BASE_URL"
	EnvIdentities = "WARDYN_LIVE_IDENTITIES_FILE"

	EnvADOOrg     = "WARDYN_LIVE_ADO_ORG"
	EnvADOProject = "WARDYN_LIVE_ADO_PROJECT"
	EnvADORepo    = "WARDYN_LIVE_ADO_REPO"
	// The LL2 project and repository whose names carry a space (#485);
	// optional, defaulting to the fixture names docs/LIVE-TESTS.md lists.
	EnvADOSpacedProject = "WARDYN_LIVE_ADO_SPACED_PROJECT"
	EnvADOSpacedRepo    = "WARDYN_LIVE_ADO_SPACED_REPO"

	// LL2c, the personal-access-token mint probe, on its own throwaway app.
	EnvADOPATProbe       = "WARDYN_LIVE_ADO_PAT_PROBE"
	EnvADOPATProbeTenant = "WARDYN_LIVE_ADO_PAT_PROBE_TENANT_ID"
	EnvADOPATProbeClient = "WARDYN_LIVE_ADO_PAT_PROBE_CLIENT_ID"
	EnvADOPATProbeScope  = "WARDYN_LIVE_ADO_PAT_PROBE_SCOPE"

	EnvSSORegion    = "WARDYN_LIVE_AWS_SSO_REGION"
	EnvSSOTokenFile = "WARDYN_LIVE_AWS_SSO_TOKEN_FILE"

	EnvBedrockAccount  = "WARDYN_LIVE_BEDROCK_ACCOUNT_ID"
	EnvBedrockRole     = "WARDYN_LIVE_BEDROCK_ROLE_NAME"
	EnvBedrockRegion   = "WARDYN_LIVE_BEDROCK_REGION"
	EnvBedrockModel    = "WARDYN_LIVE_BEDROCK_MODEL"
	EnvBedrockMaxCalls = "WARDYN_LIVE_BEDROCK_MAX_CALLS"

	// EnvBedrockWardynModelProvider names the model provider (GET
	// /model-providers) LL3w runs on (CreateRunRequest.ModelProvider) — this
	// install's Amazon Bedrock SSO provider, which the member has signed in
	// to. Install-specific, so it is never defaulted.
	EnvBedrockWardynModelProvider = "WARDYN_LIVE_BEDROCK_WARDYN_MODEL_PROVIDER"
	// EnvBedrockWardynDeniedModelProvider is OPTIONAL: a second Bedrock
	// provider this install has pointed at a model the capped account's
	// service control policy denies, for the forced-AccessDenied half of LL3w.
	// Unset skips that half by name, never the whole suite.
	EnvBedrockWardynDeniedModelProvider = "WARDYN_LIVE_BEDROCK_WARDYN_DENIED_MODEL_PROVIDER"

	// EnvAutonomyModelProvider names a WORKING model provider for the autonomy
	// gate's own member — required so the run's model credential resolves
	// cleanly and the request reaches the autonomy gate itself, rather than
	// failing earlier on an unrelated missing-credential refusal.
	EnvAutonomyModelProvider = "WARDYN_LIVE_AUTONOMY_MODEL_PROVIDER"
)

The live suites' environment. Values that are secrets are never read from the environment: the *_FILE names hold a path to a file outside the repo.

Variables

View Source 
var ErrTokenExpired = errors.New("the Identity Center sign-in has expired")

ErrTokenExpired means the owner has to sign in again; the suite skips on it.

Functions

func AgentPolicyDeliveredOK added in v0.8.0 

func AgentPolicyDeliveredOK(events []types.AuditEvent, wantLevel string) error

AgentPolicyDeliveredOK reports a descriptive error unless events (an interactive run's own audit trail) carries a run.agent_policy.write row for wantLevel with delivered=true — proof the managed-settings file that closes claude-code's own bypass route actually reached the sandbox, on whichever runner substrate this deployment uses.

It does NOT also check for an absent run.exec row: an interactive run records run.interactive and returns before any Exec call exists, at EVERY autonomy level, so an absent row here proves nothing L0-specific. The real "no run.exec at L0" proof is AutonomyL0RefusalOK's refusal, which happens before any run exists at all.

On an exec-less (krun) runner substrate, interactive dispatch never writes this row at all, so this check reports "never even attempted" rather than a real delivery failure there.

func AutonomyL0RefusalOK added in v0.8.0 

func AutonomyL0RefusalOK(body string) error

AutonomyL0RefusalOK reports a descriptive error unless body is the refusal autonomyLadder writes when a governance profile caps a member at AutonomyL0 and the request is an unattended run.

It is PURE — no HTTP, no client — so a live test and a hermetic fixture prove the SAME check: a 403 alone isn't proof of L0 enforcement, since a profile could refuse for an unrelated reason or resolve to a different rung and still return 403.

It grades body text only, never the machine "reason" field, which the server always leaves empty on this refusal — checking APIError.Reason would fail red against a server enforcing L0 correctly.

func BedrockWardynForcedFaultOK added in v0.8.0 

func BedrockWardynForcedFaultOK(hint, class string) error

BedrockWardynForcedFaultOK reports a descriptive error unless hint is the sentence bedrockFaultHints writes to a run's failure_hint when AWS refuses the model call with the given error class ("AccessDeniedException" or "ThrottlingException") — Wardyn's own sentence, keyed by the AWS class the proxy's bedrockUpstreamFault classified, never AWS's raw response verbatim. class must be one of those two exact strings.

Observable ONLY on a lane the proxy actually MITMs bedrock-runtime on — the bearer (API-key) lane, or a provider's plain-HTTP bedrock.base_url test hatch. The per-user AWS SSO lane's bedrock-runtime traffic is an opaque, un-MITM'd tunnel, so a denial on that lane never produces this hint at all — see the live test's own mode check before it calls this.

func BedrockWardynRunProvesPerUserSSO added in v0.8.0 

func BedrockWardynRunProvesPerUserSSO(events []types.AuditEvent, memberPrincipal string) error

BedrockWardynRunProvesPerUserSSO reports a descriptive error unless events (a completed run's own audit trail) prove the run's Bedrock call went out on THIS MEMBER's own per-user AWS SSO capture, on an allow-listed model.

"per-user" is proven on the credential.mint row itself, not on run.bedrock.configure's mode: mode=="sso-inject-proxy" is chosen from b.ssoInject && b.ssoProxyInject regardless of whether the roster row behind it is `per_user` or `shared` — an operator's shared captured session produces the exact same mode. What actually distinguishes them is the mint's own scope snapshot (authorBedrockSSOInjection): its secret_name is the AWS-SSO sentinel, its credential_source is CredentialSourcePerUser only for a per-user row, and its owner_subject is the subject the roster resolved the capture for. A mint whose snapshot says "shared", or whose owner is anyone but memberPrincipal, must be graded a failure here, or the proof is false.

owner_subject is runIdentitySubject(ctx, run.CreatedBy) at dispatch, evaluated on the dispatcher's own detached context where localPrincipalFromContext is always empty outside local mode — so for the OIDC-backed deployment this suite targets, owner_subject is run.CreatedBy verbatim, the same "sub" GET /me's "principal" reports for the same token. Compared for exact equality here; an install where the two are not spelled the same is disclosed in docs/LIVE-TESTS.md rather than silently worked around.

func Errorf 

func Errorf(t testing.TB, format string, args ...any)

func Fatalf 

func Fatalf(t testing.TB, format string, args ...any)

func LoadIdentities 

func LoadIdentities() (map[string]Identity, error)

LoadIdentities parses WARDYN_LIVE_IDENTITIES_FILE: an object keyed by role ("admin", "member", "norole").

func LoadSSOToken 

func LoadSSOToken(now time.Time) (string, error)

LoadSSOToken reads WARDYN_LIVE_AWS_SSO_TOKEN_FILE.

func Logf 

func Logf(t testing.TB, format string, args ...any)

Logf, Errorf, Fatalf and Skipf are the only ways a live suite writes output.

func ModelAllowed 

func ModelAllowed(id string) bool

ModelAllowed reports whether id is on the allow-list.

func ReadSecretFile 

func ReadSecretFile(name string) ([]byte, error)

ReadSecretFile reads the file whose path is in env var name. It refuses a file inside the checkout, where a stray `git add` would publish it, and its errors name the variable, never the path or the contents.

func Redact 

func Redact(s string) string

Redact masks every credential- or identity-shaped substring of s.

func Require 

func Require(t testing.TB, gate string, names ...string)

Require skips t unless gate is "1"; that is the only condition a skip may mean "prove nothing, on purpose". Once the operator has opted in by setting gate, a missing name is a broken invocation, not an absence of intent, so it is Fatalf, not Skipf — a skip and a pass must never share an exit code on a run the operator explicitly asked to prove something (#463). The message names the variables and never prints a value.

func RequireAccount 

func RequireAccount(want, got string) error

RequireAccount refuses unless the credential's account is the capped one.

func RunCreatedByIsMember added in v0.8.0 

func RunCreatedByIsMember(createdBy, memberPrincipal string) error

RunCreatedByIsMember reports a descriptive error unless createdBy (the run's own AgentRun.CreatedBy — the human principal, token `sub`) is exactly memberPrincipal. An empty createdBy, or a match against anyone else (including an admin), fails it.

func SignV4 

func SignV4(req *http.Request, body []byte, c Creds, region, service string, now time.Time)

SignV4 signs req with AWS Signature Version 4, over host, x-amz-date and, when present, content-type and the session token.

func Skipf 

func Skipf(t testing.TB, format string, args ...any)

func TranscriptContainsReply added in v0.8.0 

func TranscriptContainsReply(transcript []byte, want string) error

TranscriptContainsReply reports a descriptive error unless transcript (a run's session recording, raw asciicast bytes — the JSON escaping of plain ASCII text leaves a literal substring intact) contains want, case insensitive.

Types

type Bedrock 

type Bedrock struct {
	// contains filtered or unexported fields
}

Bedrock is a client that has already proven it holds member-account credentials. The only way to get one is NewMemberBedrock.

func NewMemberBedrock 

func NewMemberBedrock(ctx context.Context, cfg BedrockConfig, ssoToken string, ep Endpoints, hc *http.Client) (*Bedrock, error)

NewMemberBedrock exchanges the Identity Center token for role credentials on cfg.AccountID and then refuses unless STS says those credentials belong to exactly that account.

func (*Bedrock) Converse 

func (b *Bedrock) Converse(ctx context.Context, prompt string, maxTokens int) (string, error)

Converse makes one model call. It refuses a max_tokens above MaxTokens and any call past the process's budget, before anything is sent.

type BedrockConfig 

type BedrockConfig struct {
	AccountID, RoleName, SSORegion, Region, Model string
	MaxCalls                                      int
}

BedrockConfig is the validated WARDYN_LIVE_BEDROCK_* environment.

func LoadBedrockConfig 

func LoadBedrockConfig(getenv func(string) string) (BedrockConfig, error)

LoadBedrockConfig validates the suite's environment. Anything outside the limits is an error, never clamped into range.

type Creds 

type Creds struct{ AccessKeyID, SecretAccessKey, SessionToken string }

Creds are temporary role credentials.

type Endpoints 

type Endpoints struct{ Portal, STS, Bedrock string }

Endpoints are the three AWS base URLs the suite talks to; tests point them at local fakes.

func DefaultEndpoints 

func DefaultEndpoints(c BedrockConfig) Endpoints

DefaultEndpoints returns the real regional endpoints.

type Identity 

type Identity struct {
	StorageState string `json:"storage_state"`
	APIToken     string `json:"api_token"`
}

Identity is one test identity's entry in WARDYN_LIVE_IDENTITIES_FILE. The Playwright suites read storage_state; the Go suites read api_token.

type SSOToken 

type SSOToken struct {
	AccessToken string    `json:"accessToken"`
	ExpiresAt   time.Time `json:"expiresAt"`
}

SSOToken is the part of an AWS CLI `sso login` cache file the suite uses.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.