ai

package
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 7, 2026 License: Apache-2.0 Imports: 40 Imported by: 0

Documentation

Overview

Package ai generates Stampede scenarios with a language model, before any load is run. It is optional and bring-your-own-key: the model is called only here, never during a load test.

The pipeline has six stages:

  1. Understand: build a dependency map and traffic mix from a description, an OpenAPI spec, a HAR recording and/or an access log.
  2. Draft: the model writes a scenario constrained to the JSON Schema.
  3. Static check: the scenario must parse and compile (variable flow), use only known endpoints and never call blocked third parties.
  4. Dry run: each journey runs once with one user against the target, recording every request, response, extracted value and check.
  5. Repair: problems and redacted evidence go back to the model, up to three rounds; journeys that still fail are flagged for a human.
  6. Approve: the result is a proposal (YAML, traces, diff). Nothing is saved until a person approves it.

Index

Constants

View Source
const (
	StageUnderstand  = "understand"
	StageDraft       = "draft"
	StageStaticCheck = "static-check"
	StageDryRun      = "dry-run"
	StageRepair      = "repair"
	StageDone        = "done"
)

Pipeline stages, reported through Options.Progress.

View Source
const (
	JourneyPassed  = "passed"
	JourneyFlagged = "flagged"
	JourneyNotRun  = "not-run"
)

Journey statuses.

View Source
const DefaultMaxRepairs = 3

DefaultMaxRepairs is how many times failures go back to the model.

View Source
const IntrospectionQuery = `` /* 284-byte string literal not displayed */

IntrospectionQuery asks a GraphQL server for its schema.

Variables

View Source
var ErrBudget = errors.New("the AI token budget is exhausted")

ErrBudget means the token budget ran out before the first draft.

Functions

func GuardList

func GuardList() (blocked map[string]string, testMode []string)

GuardList returns the blocked and test-mode hosts, for documentation and the CLI's help output.

func Narrate

Narrate asks a model to summarise a finished report. Every claim it keeps cites facts from r.Facts(); claims citing unknown facts, measured claims citing none, and measured claims with figures not found in their citations are dropped. The model never sees request or response data, only the report's aggregate figures, with error texts redacted.

func NormalizePath

func NormalizePath(p string) string

NormalizePath turns a concrete path into a template: numeric, UUID and long hex segments become {id}.

func RenderTrace

func RenderTrace(t Trace) string

RenderTrace is renderTrace for callers outside the package.

func SensitiveName

func SensitiveName(name string) bool

SensitiveName reports whether a header, field or parameter name usually holds a credential or payment data.

func ThirdPartyCategory

func ThirdPartyCategory(host string) string

ThirdPartyCategory returns the category of a host on the guard list ("payment", "sms", "email" or "captcha"), or "" when the host is not blocked. Test-mode hosts are never blocked.

func Truncate

func Truncate(s string, limit int) string

Truncate shortens s to at most limit bytes on a rune boundary, noting how much was cut. limit <= 0 leaves s alone.

func UnifiedDiff

func UnifiedDiff(a, b, nameA, nameB string) string

UnifiedDiff compares two texts line by line and returns a unified diff with three lines of context, or "" when they are equal.

Types

type CheckResult

type CheckResult struct {
	Name   string `json:"name"`
	OK     bool   `json:"ok"`
	Detail string `json:"detail,omitempty"`
}

CheckResult is one assertion on a response.

type Coverage

type Coverage struct {
	Endpoints []EndpointCoverage `json:"endpoints"`
	// Unmatched are requests that use no endpoint of the API.
	Unmatched []RequestRef `json:"unmatched,omitempty"`
	// Templated counts requests whose whole URL is an expression, which
	// cannot be matched statically.
	Templated int `json:"templated,omitempty"`
	Covered   int `json:"covered"`
	Total     int `json:"total"`
}

Coverage compares the requests of a scenario with the endpoints of an API: which endpoints some journey exercises, which none does, and which requests match no endpoint at all. It needs no model.

func CoverageOf

func CoverageOf(s *scenario.Scenario, und *Understanding) *Coverage

CoverageOf maps a scenario's requests onto the endpoints of und.

func (*Coverage) Percent

func (c *Coverage) Percent() float64

Percent is the share of endpoints covered.

type Dependency

type Dependency struct {
	Producer string `json:"producer"` // "POST /api/login"
	Value    string `json:"value"`    // "$.token", "cookie session", "header Location"
	Consumer string `json:"consumer"` // "GET /api/me"
	Via      string `json:"via"`      // "Authorization: Bearer", "path {id}", "json productId"
	Kind     string `json:"kind"`     // token, id, cookie
}

Dependency says one call produces a value another call needs.

type DryRunner

type DryRunner struct {
	Program *scenario.Program
	// BaseURL is the target the relative paths join.
	BaseURL string
	Env     map[string]string
	Secrets map[string]string
	// Allow vets every request URL; it returns a reason when the request
	// must not be sent.
	Allow func(*url.URL) string
	// DataDir resolves relative feeder files. With ConfineData, files must
	// stay inside DataDir (server mode).
	DataDir     string
	ConfineData bool
	Redactor    *Redactor
	// Transport overrides the HTTP transport (tests).
	Transport http.RoundTripper
	// BodyLimit bounds recorded bodies (default 2 KiB).
	BodyLimit int
	// MaxRequests bounds requests per pass (default 100).
	MaxRequests int
	// GRPCFiles are descriptors for grpc steps (from the generator's .proto
	// inputs). Steps whose method is not there load their own proto or
	// protoset files, or ask the server's reflection service.
	GRPCFiles *protoregistry.Files
	// contains filtered or unexported fields
}

DryRunner runs journeys once each.

func (*DryRunner) Close

func (d *DryRunner) Close()

Close releases connections the dry run opened for gRPC steps.

func (*DryRunner) RunJourney

func (d *DryRunner) RunJourney(ctx context.Context, j *scenario.CJourney) []Trace

RunJourney executes one journey once per branch alternative (up to 4).

type Endpoint

type Endpoint struct {
	Method string `json:"method"`
	// Path uses {name} for parameters, as in OpenAPI.
	Path    string `json:"path"`
	Summary string `json:"summary,omitempty"`
	// Auth is set when the endpoint needs credentials.
	Auth bool `json:"auth,omitempty"`
	// Source is openapi, har or log.
	Source string `json:"source"`
	// Count is how often the endpoint appeared in recorded traffic.
	Count int `json:"count,omitempty"`
}

Endpoint is one operation of the system under test.

func (Endpoint) Key

func (e Endpoint) Key() string

Key is "METHOD /path".

type EndpointCoverage

type EndpointCoverage struct {
	Endpoint
	Journeys []string `json:"journeys,omitempty"`
}

EndpointCoverage is one endpoint and the journeys that call it.

type GRPCMethod

type GRPCMethod struct {
	// Name is "package.Service/Method", as grpc steps write it.
	Name            string `json:"name"`
	Input           string `json:"input"`
	Output          string `json:"output"`
	ServerStreaming bool   `json:"serverStreaming,omitempty"`
	// Example is the request message as JSON with every field set to an
	// example value of its type.
	Example string `json:"example"`
	Comment string `json:"comment,omitempty"`
}

GRPCMethod is a callable method of a gRPC service from .proto files.

type Inputs

type Inputs struct {
	// Description is plain language: who the users are and what they do.
	Description string
	// OpenAPI is an OpenAPI 3.x document (YAML or JSON).
	OpenAPI []byte
	// HAR is a browser or proxy recording (HTTP Archive 1.2).
	HAR []byte
	// GraphQL is a GraphQL schema: an introspection result (JSON) or SDL.
	GraphQL []byte
	// GraphQLPath is where the GraphQL API is served (default /graphql).
	GraphQLPath string
	// AccessLog is a web server access log (common, combined or any format
	// with a quoted "METHOD /path" request line).
	AccessLog []byte
	// Existing is a scenario to compare the proposal with.
	Existing []byte
	// Proto holds .proto sources by file name (the name imports use). The
	// generator writes grpc steps for their services' methods.
	Proto map[string][]byte
	// ProtoPaths, when set, are the paths grpc steps name in proto: so a
	// run loads the same descriptors (the CLI passes the files it read).
	// Without them steps rely on the server's reflection service.
	ProtoPaths []string
	// ProtoImportPaths are directories where imports of the .proto files
	// that are not in Proto are found (CLI only; steps name them in
	// importPaths).
	ProtoImportPaths []string
}

Inputs are what a user gives the generator. At least one is required.

type JourneyCheck

type JourneyCheck struct {
	Journey string  `json:"journey"`
	OK      bool    `json:"ok"`
	Traces  []Trace `json:"traces"`
}

JourneyCheck is the dry-run result of one journey.

func DryRunScenario

func DryRunScenario(ctx context.Context, s *scenario.Scenario, target string, allowHosts []string, env, secrets map[string]string, dataDir string) ([]JourneyCheck, error)

DryRunScenario runs every journey of s once against target with one user, under the same host policy and third-party guard as generation.

func (JourneyCheck) Problem

func (c JourneyCheck) Problem() string

Problem describes the first failure of a journey's dry run in one line, or returns "" when it passed.

type JourneyResult

type JourneyResult struct {
	Name   string `json:"name"`
	Status string `json:"status"`
	// Attempts counts the dry runs of this journey across repair rounds.
	Attempts int      `json:"attempts"`
	Traces   []Trace  `json:"traces"`
	Problems []string `json:"problems,omitempty"`
}

JourneyResult is the outcome for one journey.

type MixEntry

type MixEntry struct {
	Endpoint string  `json:"endpoint"`
	Count    int     `json:"count"`
	Share    float64 `json:"share"`
}

MixEntry is an endpoint's share of logged traffic.

type NarrateOptions

type NarrateOptions struct {
	Provider  provider.Provider
	MaxTokens int // per reply; default 4000
	// Repairs is how many times a reply with uncited or unsupported claims
	// is sent back; default 1, negative disables.
	Repairs int
}

NarrateOptions configures Narrate.

type Options

type Options struct {
	Provider provider.Provider
	// Target is the base URL of the system under test. It is required for
	// the dry run.
	Target string
	// AllowHosts are extra hosts requests may reach besides the target and
	// private addresses.
	AllowHosts []string
	// Env and Secrets back ${env.X} and ${secret.X} in the dry run. Secret
	// values are redacted everywhere.
	Env     map[string]string
	Secrets map[string]string
	// DryRun runs each journey once against Target.
	DryRun bool
	// MaxRepairs bounds repair rounds (0 uses DefaultMaxRepairs; negative
	// disables repair).
	MaxRepairs int
	// OmitBaseURL leaves target.baseURL out of the YAML (the server
	// supplies it from the run's target). Otherwise it is set to Target.
	OmitBaseURL bool
	// DataDir and ConfineData resolve file feeders in the dry run.
	DataDir     string
	ConfineData bool
	// TokenBudget stops calling the model once this many tokens were used
	// (0 means no limit).
	TokenBudget int64
	// MaxTokens bounds each reply (default 16000).
	MaxTokens int
	// Transport overrides the dry run's HTTP transport (tests).
	Transport http.RoundTripper
	// Progress receives stage changes. It must not block.
	Progress func(Progress)
}

Options configure a generation.

type Problem

type Problem struct {
	// Journey is the journey the problem belongs to; empty means the whole
	// scenario.
	Journey string `json:"journey,omitempty"`
	Message string `json:"message"`
	// Fatal problems make the scenario unusable: it does not validate or
	// it would send traffic to a blocked third party. Other problems flag
	// a journey for review.
	Fatal bool `json:"fatal,omitempty"`
}

Problem is something wrong with a proposed scenario.

func (Problem) String

func (p Problem) String() string

type Progress

type Progress struct {
	Stage   string
	Round   int
	Message string
	Usage   provider.Usage
}

Progress reports the pipeline's state.

type Redactor

type Redactor struct {
	// contains filtered or unexported fields
}

Redactor removes secrets and personal data from anything that is sent to a model provider. Recorded traffic (HAR files, access logs, dry-run traces) gets full redaction: credentials, tokens, cookies, emails, phone numbers and card numbers. Documents the user wrote on purpose (the OpenAPI spec and the description) only lose credentials, since they often name test accounts the model needs.

func NewRedactor

func NewRedactor(known ...string) *Redactor

NewRedactor redacts the given exact values (secrets, extracted tokens) wherever they appear, plus everything the patterns catch.

func (*Redactor) AddSecret

func (r *Redactor) AddSecret(v string)

AddSecret registers an exact value to redact. Values shorter than 4 characters are ignored to avoid erasing ordinary text.

func (*Redactor) Body

func (r *Redactor) Body(b []byte, limit int) string

Body redacts a request or response body and truncates it to limit bytes (0 means no limit). JSON bodies are redacted field by field so values under sensitive keys disappear even when they look harmless.

func (*Redactor) Header

func (r *Redactor) Header(name, value string) string

Header redacts one header value.

func (*Redactor) Headers

func (r *Redactor) Headers(h http.Header) map[string]string

Headers redacts a header set into a flat map, one value per name.

func (*Redactor) Secrets

func (r *Redactor) Secrets(s string) string

Secrets removes credentials only: registered values, JWTs, bearer and basic credentials, well-known API key formats and key=value pairs whose key names a secret.

func (*Redactor) Text

func (r *Redactor) Text(s string) string

Text applies full redaction to recorded traffic.

func (*Redactor) URL

func (r *Redactor) URL(raw string) string

URL redacts query parameters with sensitive names and personal data in the rest of the URL.

type RequestRef

type RequestRef struct {
	Journey string `json:"journey"`
	Method  string `json:"method"`
	URL     string `json:"url"`
}

RequestRef is a request step in a journey.

type Result

type Result struct {
	// YAML is the proposed scenario, with flagged journeys marked by a
	// comment. Empty when the model never produced a usable scenario.
	YAML     string             `json:"yaml"`
	Scenario *scenario.Scenario `json:"-"`
	Journeys []JourneyResult    `json:"journeys"`
	// Problems are static problems in the proposal (fatal ones mean the
	// YAML must not be used).
	Problems []Problem `json:"problems"`
	// Diff compares Inputs.Existing with the proposal.
	Diff          string         `json:"diff,omitempty"`
	Usage         provider.Usage `json:"usage"`
	Rounds        int            `json:"rounds"`
	Provider      string         `json:"provider"`
	Model         string         `json:"model"`
	DryRun        bool           `json:"dryRun"`
	Understanding *Understanding `json:"understanding,omitempty"`
}

Result is a proposal awaiting approval.

func Generate

func Generate(ctx context.Context, in Inputs, opts Options) (*Result, error)

Generate runs the pipeline. It returns a result even with an error when some work was done, so token usage can always be recorded.

func (*Result) Fatal

func (r *Result) Fatal() bool

Fatal reports whether the proposal is unusable.

func (*Result) Flagged

func (r *Result) Flagged() []string

Flagged lists journeys that need a human.

func (*Result) Validated

func (r *Result) Validated() bool

Validated reports whether every journey passed (or, without a dry run, the static check found nothing).

type SpecDiff

type SpecDiff struct {
	Added   []Endpoint            `json:"added,omitempty"`
	Removed []Endpoint            `json:"removed,omitempty"`
	Broken  map[string][]Endpoint `json:"broken,omitempty"` // journey -> removed endpoints it calls
}

SpecDiff lists endpoints added and removed between two versions of an API, and the journeys that call a removed one.

func DiffSpecs

func DiffSpecs(s *scenario.Scenario, old, cur *Understanding) *SpecDiff

DiffSpecs compares two understandings by method and path, and finds the journeys of s that call an endpoint the new one no longer has.

type StepTrace

type StepTrace struct {
	Step            string            `json:"step"`
	Method          string            `json:"method,omitempty"`
	URL             string            `json:"url,omitempty"`
	RequestHeaders  map[string]string `json:"requestHeaders,omitempty"`
	RequestBody     string            `json:"requestBody,omitempty"`
	Status          int               `json:"status,omitempty"`
	ResponseHeaders map[string]string `json:"responseHeaders,omitempty"`
	ResponseBody    string            `json:"responseBody,omitempty"`
	DurationMs      float64           `json:"durationMs"`
	Extracted       map[string]string `json:"extracted,omitempty"`
	Checks          []CheckResult     `json:"checks,omitempty"`
	OK              bool              `json:"ok"`
	Error           string            `json:"error,omitempty"`
	// Note explains a step that was not executed as a request.
	Note string `json:"note,omitempty"`
}

StepTrace records one request of a dry run. Everything is redacted.

type ThirdPartyCall

type ThirdPartyCall struct {
	// Journey and Step locate the call; both are empty for the target's
	// base URL.
	Journey  string `json:"journey,omitempty"`
	Step     string `json:"step,omitempty"`
	Host     string `json:"host"`
	Category string `json:"category"`
}

ThirdPartyCall is a place in a scenario that sends traffic to a host on the guard list.

func ThirdPartyCalls

func ThirdPartyCalls(s *scenario.Scenario) []ThirdPartyCall

ThirdPartyCalls lists the steps of s (and its base URL) that send traffic to a payment, SMS, email or CAPTCHA provider on the guard list. Hosts written as templates cannot be known before a run and are not reported; the run's host policy still applies to them.

func (ThirdPartyCall) String

func (c ThirdPartyCall) String() string

type Trace

type Trace struct {
	// Pass numbers passes from 1. A journey with branches gets one pass
	// per alternative so every branch is exercised.
	Pass     int         `json:"pass"`
	Branches []string    `json:"branches,omitempty"`
	OK       bool        `json:"ok"`
	Error    string      `json:"error,omitempty"`
	Steps    []StepTrace `json:"steps"`
}

Trace is one pass through a journey.

type Understanding

type Understanding struct {
	Endpoints    []Endpoint     `json:"endpoints"`
	Dependencies []Dependency   `json:"dependencies"`
	Mix          []MixEntry     `json:"mix,omitempty"`
	Visits       []VisitPattern `json:"visits,omitempty"`
	// HasSpec is set when endpoints come from OpenAPI or a HAR, so the
	// static check can require every request to use a known endpoint.
	HasSpec bool `json:"hasSpec"`
	// BasePath is the path of the spec's first server URL ("/api/v1"),
	// which prefixes every spec path on the wire.
	BasePath string `json:"basePath,omitempty"`
	// GRPCMethods are the methods of the .proto inputs; ProtoFiles their
	// compiled descriptors, which the dry run uses.
	GRPCMethods []GRPCMethod         `json:"grpcMethods,omitempty"`
	ProtoFiles  *protoregistry.Files `json:"-"`
	// Context is the redacted text the model sees.
	Context string `json:"-"`
}

Understanding is the result of the first pipeline stage.

func Understand

func Understand(in Inputs, red *Redactor) (*Understanding, error)

Understand builds the dependency map and the model's context from the inputs. Traffic is redacted with red; documents lose credentials only.

type VisitPattern

type VisitPattern struct {
	Steps []string `json:"steps"`
	Count int      `json:"count"`
	Share float64  `json:"share"`
}

VisitPattern is a sequence of endpoints seen in one client visit.

Directories

Path Synopsis
Package provider talks to large language model APIs for Stampede's optional AI journey generation.
Package provider talks to large language model APIs for Stampede's optional AI journey generation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL