Documentation
¶
Overview ¶
Package ai generates Stampede scenarios with a language model, before any load is run. It is optional and bring-your-own-key: the model is called only here, never during a load test.
The pipeline has six stages:
- Understand: build a dependency map and traffic mix from a description, an OpenAPI spec, a HAR recording and/or an access log.
- Draft: the model writes a scenario constrained to the JSON Schema.
- Static check: the scenario must parse and compile (variable flow), use only known endpoints and never call blocked third parties.
- Dry run: each journey runs once with one user against the target, recording every request, response, extracted value and check.
- Repair: problems and redacted evidence go back to the model, up to three rounds; journeys that still fail are flagged for a human.
- Approve: the result is a proposal (YAML, traces, diff). Nothing is saved until a person approves it.
Index ¶
- Constants
- Variables
- func GuardList() (blocked map[string]string, testMode []string)
- func Narrate(ctx context.Context, r *report.Report, opts NarrateOptions) (*report.Narrative, provider.Usage, error)
- func NormalizePath(p string) string
- func RenderTrace(t Trace) string
- func SensitiveName(name string) bool
- func ThirdPartyCategory(host string) string
- func Truncate(s string, limit int) string
- func UnifiedDiff(a, b, nameA, nameB string) string
- type CheckResult
- type Coverage
- type Dependency
- type DryRunner
- type Endpoint
- type EndpointCoverage
- type GRPCMethod
- type Inputs
- type JourneyCheck
- type JourneyResult
- type MixEntry
- type NarrateOptions
- type Options
- type Problem
- type Progress
- type Redactor
- func (r *Redactor) AddSecret(v string)
- func (r *Redactor) Body(b []byte, limit int) string
- func (r *Redactor) Header(name, value string) string
- func (r *Redactor) Headers(h http.Header) map[string]string
- func (r *Redactor) Secrets(s string) string
- func (r *Redactor) Text(s string) string
- func (r *Redactor) URL(raw string) string
- type RequestRef
- type Result
- type SpecDiff
- type StepTrace
- type ThirdPartyCall
- type Trace
- type Understanding
- type VisitPattern
Constants ¶
const ( StageUnderstand = "understand" StageDraft = "draft" StageStaticCheck = "static-check" StageDryRun = "dry-run" StageRepair = "repair" StageDone = "done" )
Pipeline stages, reported through Options.Progress.
const ( JourneyPassed = "passed" JourneyFlagged = "flagged" JourneyNotRun = "not-run" )
Journey statuses.
const DefaultMaxRepairs = 3
DefaultMaxRepairs is how many times failures go back to the model.
const IntrospectionQuery = `` /* 284-byte string literal not displayed */
IntrospectionQuery asks a GraphQL server for its schema.
Variables ¶
var ErrBudget = errors.New("the AI token budget is exhausted")
ErrBudget means the token budget ran out before the first draft.
Functions ¶
func GuardList ¶
GuardList returns the blocked and test-mode hosts, for documentation and the CLI's help output.
func Narrate ¶
func Narrate(ctx context.Context, r *report.Report, opts NarrateOptions) (*report.Narrative, provider.Usage, error)
Narrate asks a model to summarise a finished report. Every claim it keeps cites facts from r.Facts(); claims citing unknown facts, measured claims citing none, and measured claims with figures not found in their citations are dropped. The model never sees request or response data, only the report's aggregate figures, with error texts redacted.
func NormalizePath ¶
NormalizePath turns a concrete path into a template: numeric, UUID and long hex segments become {id}.
func RenderTrace ¶
RenderTrace is renderTrace for callers outside the package.
func SensitiveName ¶
SensitiveName reports whether a header, field or parameter name usually holds a credential or payment data.
func ThirdPartyCategory ¶
ThirdPartyCategory returns the category of a host on the guard list ("payment", "sms", "email" or "captcha"), or "" when the host is not blocked. Test-mode hosts are never blocked.
func Truncate ¶
Truncate shortens s to at most limit bytes on a rune boundary, noting how much was cut. limit <= 0 leaves s alone.
func UnifiedDiff ¶
UnifiedDiff compares two texts line by line and returns a unified diff with three lines of context, or "" when they are equal.
Types ¶
type CheckResult ¶
type CheckResult struct {
Name string `json:"name"`
OK bool `json:"ok"`
Detail string `json:"detail,omitempty"`
}
CheckResult is one assertion on a response.
type Coverage ¶
type Coverage struct {
Endpoints []EndpointCoverage `json:"endpoints"`
// Unmatched are requests that use no endpoint of the API.
Unmatched []RequestRef `json:"unmatched,omitempty"`
// Templated counts requests whose whole URL is an expression, which
// cannot be matched statically.
Templated int `json:"templated,omitempty"`
Covered int `json:"covered"`
Total int `json:"total"`
}
Coverage compares the requests of a scenario with the endpoints of an API: which endpoints some journey exercises, which none does, and which requests match no endpoint at all. It needs no model.
func CoverageOf ¶
func CoverageOf(s *scenario.Scenario, und *Understanding) *Coverage
CoverageOf maps a scenario's requests onto the endpoints of und.
type Dependency ¶
type Dependency struct {
Producer string `json:"producer"` // "POST /api/login"
Value string `json:"value"` // "$.token", "cookie session", "header Location"
Consumer string `json:"consumer"` // "GET /api/me"
Via string `json:"via"` // "Authorization: Bearer", "path {id}", "json productId"
Kind string `json:"kind"` // token, id, cookie
}
Dependency says one call produces a value another call needs.
type DryRunner ¶
type DryRunner struct {
Program *scenario.Program
// BaseURL is the target the relative paths join.
BaseURL string
Env map[string]string
Secrets map[string]string
// Allow vets every request URL; it returns a reason when the request
// must not be sent.
Allow func(*url.URL) string
// DataDir resolves relative feeder files. With ConfineData, files must
// stay inside DataDir (server mode).
DataDir string
ConfineData bool
Redactor *Redactor
// Transport overrides the HTTP transport (tests).
Transport http.RoundTripper
// BodyLimit bounds recorded bodies (default 2 KiB).
BodyLimit int
// MaxRequests bounds requests per pass (default 100).
MaxRequests int
// GRPCFiles are descriptors for grpc steps (from the generator's .proto
// inputs). Steps whose method is not there load their own proto or
// protoset files, or ask the server's reflection service.
GRPCFiles *protoregistry.Files
// contains filtered or unexported fields
}
DryRunner runs journeys once each.
type Endpoint ¶
type Endpoint struct {
Method string `json:"method"`
// Path uses {name} for parameters, as in OpenAPI.
Path string `json:"path"`
Summary string `json:"summary,omitempty"`
// Auth is set when the endpoint needs credentials.
Auth bool `json:"auth,omitempty"`
// Source is openapi, har or log.
Source string `json:"source"`
// Count is how often the endpoint appeared in recorded traffic.
Count int `json:"count,omitempty"`
}
Endpoint is one operation of the system under test.
type EndpointCoverage ¶
EndpointCoverage is one endpoint and the journeys that call it.
type GRPCMethod ¶
type GRPCMethod struct {
// Name is "package.Service/Method", as grpc steps write it.
Name string `json:"name"`
Input string `json:"input"`
Output string `json:"output"`
ServerStreaming bool `json:"serverStreaming,omitempty"`
// Example is the request message as JSON with every field set to an
// example value of its type.
Example string `json:"example"`
Comment string `json:"comment,omitempty"`
}
GRPCMethod is a callable method of a gRPC service from .proto files.
type Inputs ¶
type Inputs struct {
// Description is plain language: who the users are and what they do.
Description string
// OpenAPI is an OpenAPI 3.x document (YAML or JSON).
OpenAPI []byte
// HAR is a browser or proxy recording (HTTP Archive 1.2).
HAR []byte
// GraphQL is a GraphQL schema: an introspection result (JSON) or SDL.
GraphQL []byte
// GraphQLPath is where the GraphQL API is served (default /graphql).
GraphQLPath string
// AccessLog is a web server access log (common, combined or any format
// with a quoted "METHOD /path" request line).
AccessLog []byte
// Existing is a scenario to compare the proposal with.
Existing []byte
// Proto holds .proto sources by file name (the name imports use). The
// generator writes grpc steps for their services' methods.
Proto map[string][]byte
// ProtoPaths, when set, are the paths grpc steps name in proto: so a
// run loads the same descriptors (the CLI passes the files it read).
// Without them steps rely on the server's reflection service.
ProtoPaths []string
// ProtoImportPaths are directories where imports of the .proto files
// that are not in Proto are found (CLI only; steps name them in
// importPaths).
ProtoImportPaths []string
}
Inputs are what a user gives the generator. At least one is required.
type JourneyCheck ¶
type JourneyCheck struct {
Journey string `json:"journey"`
OK bool `json:"ok"`
Traces []Trace `json:"traces"`
}
JourneyCheck is the dry-run result of one journey.
func DryRunScenario ¶
func DryRunScenario(ctx context.Context, s *scenario.Scenario, target string, allowHosts []string, env, secrets map[string]string, dataDir string) ([]JourneyCheck, error)
DryRunScenario runs every journey of s once against target with one user, under the same host policy and third-party guard as generation.
func (JourneyCheck) Problem ¶
func (c JourneyCheck) Problem() string
Problem describes the first failure of a journey's dry run in one line, or returns "" when it passed.
type JourneyResult ¶
type JourneyResult struct {
Name string `json:"name"`
Status string `json:"status"`
// Attempts counts the dry runs of this journey across repair rounds.
Attempts int `json:"attempts"`
Traces []Trace `json:"traces"`
Problems []string `json:"problems,omitempty"`
}
JourneyResult is the outcome for one journey.
type NarrateOptions ¶
type NarrateOptions struct {
Provider provider.Provider
MaxTokens int // per reply; default 4000
// Repairs is how many times a reply with uncited or unsupported claims
// is sent back; default 1, negative disables.
Repairs int
}
NarrateOptions configures Narrate.
type Options ¶
type Options struct {
Provider provider.Provider
// Target is the base URL of the system under test. It is required for
// the dry run.
Target string
// AllowHosts are extra hosts requests may reach besides the target and
// private addresses.
AllowHosts []string
// Env and Secrets back ${env.X} and ${secret.X} in the dry run. Secret
// values are redacted everywhere.
Env map[string]string
Secrets map[string]string
// DryRun runs each journey once against Target.
DryRun bool
// MaxRepairs bounds repair rounds (0 uses DefaultMaxRepairs; negative
// disables repair).
MaxRepairs int
// OmitBaseURL leaves target.baseURL out of the YAML (the server
// supplies it from the run's target). Otherwise it is set to Target.
OmitBaseURL bool
// DataDir and ConfineData resolve file feeders in the dry run.
DataDir string
ConfineData bool
// TokenBudget stops calling the model once this many tokens were used
// (0 means no limit).
TokenBudget int64
// MaxTokens bounds each reply (default 16000).
MaxTokens int
// Transport overrides the dry run's HTTP transport (tests).
Transport http.RoundTripper
// Progress receives stage changes. It must not block.
Progress func(Progress)
}
Options configure a generation.
type Problem ¶
type Problem struct {
// Journey is the journey the problem belongs to; empty means the whole
// scenario.
Journey string `json:"journey,omitempty"`
Message string `json:"message"`
// Fatal problems make the scenario unusable: it does not validate or
// it would send traffic to a blocked third party. Other problems flag
// a journey for review.
Fatal bool `json:"fatal,omitempty"`
}
Problem is something wrong with a proposed scenario.
type Redactor ¶
type Redactor struct {
// contains filtered or unexported fields
}
Redactor removes secrets and personal data from anything that is sent to a model provider. Recorded traffic (HAR files, access logs, dry-run traces) gets full redaction: credentials, tokens, cookies, emails, phone numbers and card numbers. Documents the user wrote on purpose (the OpenAPI spec and the description) only lose credentials, since they often name test accounts the model needs.
func NewRedactor ¶
NewRedactor redacts the given exact values (secrets, extracted tokens) wherever they appear, plus everything the patterns catch.
func (*Redactor) AddSecret ¶
AddSecret registers an exact value to redact. Values shorter than 4 characters are ignored to avoid erasing ordinary text.
func (*Redactor) Body ¶
Body redacts a request or response body and truncates it to limit bytes (0 means no limit). JSON bodies are redacted field by field so values under sensitive keys disappear even when they look harmless.
func (*Redactor) Secrets ¶
Secrets removes credentials only: registered values, JWTs, bearer and basic credentials, well-known API key formats and key=value pairs whose key names a secret.
type RequestRef ¶
type RequestRef struct {
Journey string `json:"journey"`
Method string `json:"method"`
URL string `json:"url"`
}
RequestRef is a request step in a journey.
type Result ¶
type Result struct {
// YAML is the proposed scenario, with flagged journeys marked by a
// comment. Empty when the model never produced a usable scenario.
YAML string `json:"yaml"`
Scenario *scenario.Scenario `json:"-"`
Journeys []JourneyResult `json:"journeys"`
// Problems are static problems in the proposal (fatal ones mean the
// YAML must not be used).
Problems []Problem `json:"problems"`
// Diff compares Inputs.Existing with the proposal.
Diff string `json:"diff,omitempty"`
Usage provider.Usage `json:"usage"`
Rounds int `json:"rounds"`
Provider string `json:"provider"`
Model string `json:"model"`
DryRun bool `json:"dryRun"`
Understanding *Understanding `json:"understanding,omitempty"`
}
Result is a proposal awaiting approval.
func Generate ¶
Generate runs the pipeline. It returns a result even with an error when some work was done, so token usage can always be recorded.
type SpecDiff ¶
type SpecDiff struct {
Added []Endpoint `json:"added,omitempty"`
Removed []Endpoint `json:"removed,omitempty"`
Broken map[string][]Endpoint `json:"broken,omitempty"` // journey -> removed endpoints it calls
}
SpecDiff lists endpoints added and removed between two versions of an API, and the journeys that call a removed one.
type StepTrace ¶
type StepTrace struct {
Step string `json:"step"`
Method string `json:"method,omitempty"`
URL string `json:"url,omitempty"`
RequestHeaders map[string]string `json:"requestHeaders,omitempty"`
RequestBody string `json:"requestBody,omitempty"`
Status int `json:"status,omitempty"`
ResponseHeaders map[string]string `json:"responseHeaders,omitempty"`
ResponseBody string `json:"responseBody,omitempty"`
DurationMs float64 `json:"durationMs"`
Extracted map[string]string `json:"extracted,omitempty"`
Checks []CheckResult `json:"checks,omitempty"`
OK bool `json:"ok"`
Error string `json:"error,omitempty"`
// Note explains a step that was not executed as a request.
Note string `json:"note,omitempty"`
}
StepTrace records one request of a dry run. Everything is redacted.
type ThirdPartyCall ¶
type ThirdPartyCall struct {
// Journey and Step locate the call; both are empty for the target's
// base URL.
Journey string `json:"journey,omitempty"`
Step string `json:"step,omitempty"`
Host string `json:"host"`
Category string `json:"category"`
}
ThirdPartyCall is a place in a scenario that sends traffic to a host on the guard list.
func ThirdPartyCalls ¶
func ThirdPartyCalls(s *scenario.Scenario) []ThirdPartyCall
ThirdPartyCalls lists the steps of s (and its base URL) that send traffic to a payment, SMS, email or CAPTCHA provider on the guard list. Hosts written as templates cannot be known before a run and are not reported; the run's host policy still applies to them.
func (ThirdPartyCall) String ¶
func (c ThirdPartyCall) String() string
type Trace ¶
type Trace struct {
// Pass numbers passes from 1. A journey with branches gets one pass
// per alternative so every branch is exercised.
Pass int `json:"pass"`
Branches []string `json:"branches,omitempty"`
OK bool `json:"ok"`
Error string `json:"error,omitempty"`
Steps []StepTrace `json:"steps"`
}
Trace is one pass through a journey.
type Understanding ¶
type Understanding struct {
Endpoints []Endpoint `json:"endpoints"`
Dependencies []Dependency `json:"dependencies"`
Mix []MixEntry `json:"mix,omitempty"`
Visits []VisitPattern `json:"visits,omitempty"`
// HasSpec is set when endpoints come from OpenAPI or a HAR, so the
// static check can require every request to use a known endpoint.
HasSpec bool `json:"hasSpec"`
// BasePath is the path of the spec's first server URL ("/api/v1"),
// which prefixes every spec path on the wire.
BasePath string `json:"basePath,omitempty"`
// GRPCMethods are the methods of the .proto inputs; ProtoFiles their
// compiled descriptors, which the dry run uses.
GRPCMethods []GRPCMethod `json:"grpcMethods,omitempty"`
ProtoFiles *protoregistry.Files `json:"-"`
// Context is the redacted text the model sees.
Context string `json:"-"`
}
Understanding is the result of the first pipeline stage.
func Understand ¶
func Understand(in Inputs, red *Redactor) (*Understanding, error)
Understand builds the dependency map and the model's context from the inputs. Traffic is redacted with red; documents lose credentials only.
type VisitPattern ¶
VisitPattern is a sequence of endpoints seen in one client visit.