Documentation
¶
Overview ¶
Package registry — acceptance criteria runner.
A skill's `acceptance_criteria` frontmatter is a list of Given/When/Then prose strings. The runner evaluates each criterion against the skill's body and frontmatter, returning a TestReport that callers (CLI, API, Web UI) render directly.
Two evaluators ship:
- LLMEvaluator: hands the skill + criterion to an agent.ChatModel and asks for a structured pass/fail verdict. Production path; the prose contract stays free-form.
- DeterministicEvaluator: a no-LLM adapter for CI and unit tests. It looks for "PASS:" or "FAIL:" markers inside each criterion so fixture skills can encode expected outcomes without standing up an LLM provider.
The runner is read-only — it never mutates the skill or its files.
Index ¶
- Variables
- func RenderSkillMD(skill *AgentSkill) ([]byte, error)
- func ValidateSkill(s *AgentSkill) error
- func ValidateSkillName(name string) error
- type AgentSkill
- type DeterministicEvaluator
- type Evaluator
- type ItemState
- type LLMEvaluator
- type RegistryStatus
- type RunOptions
- type Server
- func (s *Server) CallTool(ctx context.Context, name string, arguments map[string]any) (*mcp.ToolCallResult, error)
- func (s *Server) GetPromptData(name string) (*mcp.PromptData, error)
- func (s *Server) HasContent() bool
- func (s *Server) Initialize(ctx context.Context) error
- func (s *Server) IsInitialized() bool
- func (s *Server) ListPromptData() []mcp.PromptData
- func (s *Server) Name() string
- func (s *Server) RefreshTools(ctx context.Context) error
- func (s *Server) ServerInfo() mcp.ServerInfo
- func (s *Server) SetSkillRegistry(r SkillRegistry)
- func (s *Server) SetTSDispatcher(d TSDispatcher)
- func (s *Server) Store() *Store
- func (s *Server) Tools() []mcp.Tool
- type SkillFile
- type SkillRegistry
- type Store
- func (s *Store) ActiveSkills() []*AgentSkill
- func (s *Store) DeleteFile(skillName, filePath string) error
- func (s *Store) DeleteSkill(name string) error
- func (s *Store) Dir() string
- func (s *Store) GetSkill(name string) (*AgentSkill, error)
- func (s *Store) HandlerPath(name string) (string, bool)
- func (s *Store) HasContent() bool
- func (s *Store) ListFiles(skillName string) ([]SkillFile, error)
- func (s *Store) ListSkills() []*AgentSkill
- func (s *Store) Load() error
- func (s *Store) ReadFile(skillName, filePath string) ([]byte, error)
- func (s *Store) RenameSkill(oldName, newName string) error
- func (s *Store) SaveSkill(sk *AgentSkill) error
- func (s *Store) Status() RegistryStatus
- func (s *Store) WriteFile(skillName, filePath string, data []byte) error
- type TSDispatcher
- type TestReport
- type TestResult
- type TestSeverity
- type ValidationResult
Constants ¶
This section is empty.
Variables ¶
var ErrCriterionOutOfRange = errors.New("criterion index out of range")
ErrCriterionOutOfRange is returned when RunOptions.CriterionIndex names a criterion that does not exist on the skill. The CLI maps this to exit code 2.
var ErrNoCriteria = errors.New("skill has no acceptance_criteria")
ErrNoCriteria signals that the skill has no acceptance_criteria frontmatter. The runner returns this distinct error so callers can emit a clear "nothing to test" message and exit cleanly rather than reporting zero failures (which is ambiguous: did the skill pass, or was there nothing to check?).
var ErrNotFound = errors.New("not found")
ErrNotFound is returned when a skill does not exist in the store.
Functions ¶
func RenderSkillMD ¶
func RenderSkillMD(skill *AgentSkill) ([]byte, error)
RenderSkillMD serializes an AgentSkill back to SKILL.md format.
func ValidateSkill ¶
func ValidateSkill(s *AgentSkill) error
ValidateSkill validates an AgentSkill and returns just the error (convenience wrapper).
func ValidateSkillName ¶
ValidateSkillName validates a skill name against the agentskills.io spec.
Types ¶
type AgentSkill ¶
type AgentSkill struct {
// --- Frontmatter fields (from YAML between --- delimiters) ---
Name string `yaml:"name" json:"name"`
Description string `yaml:"description" json:"description"`
License string `yaml:"license,omitempty" json:"license,omitempty"`
Compatibility string `yaml:"compatibility,omitempty" json:"compatibility,omitempty"`
Metadata map[string]string `yaml:"metadata,omitempty" json:"metadata,omitempty"`
AllowedTools string `yaml:"allowed-tools,omitempty" json:"allowedTools,omitempty"`
// AcceptanceCriteria documents expected skill behavior as human-readable
// Given/When/Then scenarios. Gridctl extension; not part of agentskills.io spec.
// See https://agentskills.io/specification
AcceptanceCriteria []string `yaml:"acceptance_criteria,omitempty" json:"acceptanceCriteria,omitempty"`
// --- Gridctl extensions (not in agentskills.io spec) ---
State ItemState `yaml:"state,omitempty" json:"state"`
// --- Parsed from file content (not in frontmatter YAML) ---
Body string `yaml:"-" json:"body"` // Markdown content after frontmatter
// --- Computed fields (not serialized to YAML) ---
FileCount int `yaml:"-" json:"fileCount"` // Number of supporting files (scripts/, references/, assets/)
Dir string `yaml:"-" json:"dir,omitempty"` // Relative path from skills/ root (e.g., "git-workflow/branch-fork")
// --- Typed-skill fields (Phase C) ---
// Set when a skill.go or skill.ts sibling is present alongside
// SKILL.md. The walker discovers the handler at load time but
// does not load or compile it; runtime dispatch happens through
// the registry server's typed-skill path.
HandlerLanguage string `yaml:"-" json:"handlerLanguage,omitempty"` // "go", "ts", or empty
HandlerPath string `yaml:"-" json:"handlerPath,omitempty"` // Relative path from skill dir (e.g., "skill.ts")
}
AgentSkill represents an Agent Skills standard SKILL.md file. See https://agentskills.io/specification for the full spec.
func ParseSkillMD ¶
func ParseSkillMD(data []byte) (*AgentSkill, error)
ParseSkillMD parses a SKILL.md file into an AgentSkill. The file format is YAML frontmatter between --- delimiters followed by a markdown body.
func (*AgentSkill) Validate ¶
func (s *AgentSkill) Validate() error
Validate checks the skill against the agentskills.io specification.
type DeterministicEvaluator ¶
type DeterministicEvaluator struct{}
DeterministicEvaluator is the zero-LLM evaluator used by CI and unit tests. It inspects each criterion for the case-sensitive prefix markers "PASS:" or "FAIL:" and produces the corresponding verdict; criteria without a marker return TestSeverityError so test authors know the fixture needs to be explicit.
This adapter exists so the acceptance contract can be exercised end to end (CLI → API → evaluator) without standing up an LLM provider.
func (DeterministicEvaluator) Evaluate ¶
func (DeterministicEvaluator) Evaluate(_ context.Context, _ *AgentSkill, idx int, criterion string) TestResult
Evaluate parses the marker convention. Markers are checked case-sensitively to avoid false positives in natural-language criteria like "should pass validation" — only explicit "PASS:" or "FAIL:" at the start of the trimmed criterion fires.
func (DeterministicEvaluator) Name ¶
func (DeterministicEvaluator) Name() string
Name returns "deterministic".
type Evaluator ¶
type Evaluator interface {
// Name identifies the evaluator backend for the TestReport.
Name() string
// Evaluate produces a verdict for criterion idx against skill.
// Implementations MUST NOT return both a TestResult and an error;
// transient infrastructure failures should be returned via a
// TestResult with Severity = TestSeverityError so the report
// remains complete for the user.
Evaluate(ctx context.Context, skill *AgentSkill, idx int, criterion string) TestResult
}
Evaluator runs a single acceptance criterion against a skill and returns the verdict. Implementations MUST honor ctx cancellation.
type ItemState ¶
type ItemState string
ItemState represents the lifecycle state of a skill. Note: state is a gridctl extension, not part of the agentskills.io spec.
type LLMEvaluator ¶
type LLMEvaluator struct {
// Model is the canonical model ID handed to ChatRequest.Model.
// Falls back to defaultJudgeModel when empty.
Model string
// Provider is the gridctl LLM provider. Required; constructing
// the evaluator without one and calling Evaluate produces
// TestSeverityError on every criterion.
Provider agent.ChatModel
// MaxTokens caps the judge's output. Defaults to 256 — verdicts
// fit comfortably in a few sentences.
MaxTokens int
}
LLMEvaluator is the production evaluator. It asks an agent.ChatModel to render a verdict on each criterion against the skill's body, then parses a strict JSON envelope from the model's response.
The prompt instructs the judge to emit exactly one JSON object with `verdict` (pass|fail) and `rationale` (one sentence). Anything else is treated as TestSeverityError so the user sees the model misbehaved rather than guessing whether absence-of-marker meant pass or fail.
func (LLMEvaluator) Evaluate ¶
func (e LLMEvaluator) Evaluate(ctx context.Context, skill *AgentSkill, idx int, criterion string) TestResult
Evaluate prompts the judge and parses its verdict. Errors at any stage become TestSeverityError so the report stays complete.
type RegistryStatus ¶
type RegistryStatus struct {
TotalSkills int `json:"totalSkills"`
ActiveSkills int `json:"activeSkills"`
}
RegistryStatus contains summary statistics.
type RunOptions ¶
type RunOptions struct {
// CriterionIndex, when non-negative, scopes the run to a single
// criterion. -1 (the zero value via NewRunOptions) means "all".
CriterionIndex int
// DryRun lists criteria without evaluating them. Severity on each
// result is empty.
DryRun bool
// Now overrides the wall-clock used for EvaluatedAt and
// GeneratedAt. Tests inject a fixed time; production callers
// leave this zero so the runner defaults to time.Now().
Now time.Time
}
RunOptions tunes a single Run pass.
func NewRunOptions ¶
func NewRunOptions() RunOptions
NewRunOptions returns RunOptions with CriterionIndex set to -1, the "run every criterion" sentinel. Use this instead of a literal zero value so future-added fields stay backward-compatible.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is an in-process MCP server that serves Agent Skills as prompts. It implements mcp.AgentClient so it can be registered with the gateway router, and mcp.PromptProvider so the gateway can serve skills via MCP prompts and resources.
func (*Server) CallTool ¶
func (s *Server) CallTool(ctx context.Context, name string, arguments map[string]any) (*mcp.ToolCallResult, error)
CallTool dispatches a registered typed skill. The lookup order is:
- The typed-skill registry (programmatic registrations — Go skills etc.). This is the path Phase D's orchestrator and Phase G's CLI will most often hit.
- The TS dispatcher (skill.ts handlers discovered by the walker).
Tools that are not registered surface the same error every other AgentClient does, so the gateway router's "no such tool" path works unchanged.
func (*Server) GetPromptData ¶
func (s *Server) GetPromptData(name string) (*mcp.PromptData, error)
GetPromptData returns a specific active skill's content as MCP PromptData.
func (*Server) HasContent ¶
HasContent returns true if the registry has any skills.
func (*Server) Initialize ¶
Initialize loads the store.
func (*Server) IsInitialized ¶
IsInitialized returns whether the server has been initialized.
func (*Server) ListPromptData ¶
func (s *Server) ListPromptData() []mcp.PromptData
ListPromptData returns active Agent Skills as MCP PromptData. Each skill gets a single optional "context" argument for clients to pass additional context when requesting the skill via prompts/get.
func (*Server) RefreshTools ¶
RefreshTools reloads the store from disk.
func (*Server) ServerInfo ¶
func (s *Server) ServerInfo() mcp.ServerInfo
ServerInfo returns server information.
func (*Server) SetSkillRegistry ¶
func (s *Server) SetSkillRegistry(r SkillRegistry)
SetSkillRegistry installs the typed-skill registry the server consults for programmatically registered skills (Go skills, plus any other in-process Definitions). Pass nil to detach.
func (*Server) SetTSDispatcher ¶
func (s *Server) SetTSDispatcher(d TSDispatcher)
SetTSDispatcher installs the TypeScript dispatcher the server uses to execute skill.ts handlers discovered by the walker. Pass nil to detach.
func (*Server) Tools ¶
Tools returns the registered typed skills as MCP tool entries — the programmatically registered skills first, then any TS-handler skills the walker found on disk. The returned slice is sorted by name so the wire output is deterministic across reloads.
Phase C: Markdown-only skills (no skill.go / skill.ts sibling) remain prompts and do not appear here. Go-handler skills are not exposed by the walker either — they require an explicit registration through SetSkillRegistry, which Phase G will do as part of `gridctl agent build`.
type SkillFile ¶
type SkillFile struct {
Path string `json:"path"` // Relative path within the skill dir (e.g., "scripts/lint.sh")
Size int64 `json:"size"` // File size in bytes
IsDir bool `json:"isDir"` // True for directories
}
SkillFile represents a file within a skill directory.
type SkillRegistry ¶
type SkillRegistry interface {
Tools() []mcp.Tool
CallTool(ctx context.Context, name string, arguments map[string]any) (*mcp.ToolCallResult, error)
}
SkillRegistry is the typed-skill surface the registry server consults to expose typed skills as MCP tools. The registry server holds a concrete *skill.Registry (from pkg/agent/skill) wrapped in this interface so the registry package does not have to import pkg/agent directly — that would push pkg/agent into the dependency closure of every consumer of pkg/registry.
Implementations MUST be safe for concurrent reads.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store manages skill directories on disk. Each skill is a directory containing a required SKILL.md and optional supporting files (scripts/, references/, assets/).
func (*Store) ActiveSkills ¶
func (s *Store) ActiveSkills() []*AgentSkill
ActiveSkills returns only skills with State == "active". Returned pointers are copies.
func (*Store) DeleteFile ¶
DeleteFile removes a file from a skill directory.
func (*Store) DeleteSkill ¶
DeleteSkill removes a skill directory and cache entry.
func (*Store) GetSkill ¶
func (s *Store) GetSkill(name string) (*AgentSkill, error)
GetSkill returns a skill by name.
func (*Store) HandlerPath ¶
HandlerPath returns the absolute path to a typed skill's handler file ("skill.go" / "skill.ts") on disk, or ("", false) if the skill does not exist or has no typed handler. The store walker populates HandlerLanguage/HandlerPath at Load time; this helper resolves the pair against baseDir so dispatchers do not have to know the on-disk layout.
func (*Store) HasContent ¶
HasContent returns true if there is at least one skill.
func (*Store) ListSkills ¶
func (s *Store) ListSkills() []*AgentSkill
ListSkills returns all skills (all states). Returned pointers are copies.
func (*Store) Load ¶
Load scans the skills/ subdirectory for SKILL.md files and checks for legacy YAML registry files.
func (*Store) RenameSkill ¶
RenameSkill renames a skill directory and updates its frontmatter.
func (*Store) SaveSkill ¶
func (s *Store) SaveSkill(sk *AgentSkill) error
SaveSkill creates or updates a skill (validates, writes SKILL.md, updates cache).
func (*Store) Status ¶
func (s *Store) Status() RegistryStatus
Status returns registry summary counts.
type TSDispatcher ¶
type TSDispatcher interface {
Dispatch(ctx context.Context, name, sourcePath string, arguments map[string]any) (*mcp.ToolCallResult, error)
}
TSDispatcher is the runtime hook for executing TypeScript-handler skills the walker discovered on disk. The registry server hands a dispatch each call by handler-path; the dispatcher reads the file, runs it through the agent sandbox, and returns the typed-skill result. Set via Server.SetTSDispatcher; nil is a valid configuration — TS skills then surface as a "no dispatcher wired" error at call time rather than failing registry load.
type TestReport ¶
type TestReport struct {
// SkillName echoes the skill the runner ran against.
SkillName string `json:"skill_name"`
// Results is the per-criterion verdicts in criterion order.
Results []TestResult `json:"results"`
// PassCount, FailCount, ErrorCount summarize Results so callers
// don't re-tally.
PassCount int `json:"pass_count"`
FailCount int `json:"fail_count"`
ErrorCount int `json:"error_count"`
// Evaluator names the backend that ran the criteria (e.g. "llm",
// "deterministic"). Surfaced in JSON output so CI logs make the
// provenance explicit.
Evaluator string `json:"evaluator"`
// DryRun is true when Results was populated by listing criteria
// without executing them. Each TestResult in that case carries
// Severity = "" so renderers can show "—" instead of a verdict.
DryRun bool `json:"dry_run,omitempty"`
// GeneratedAt is the wall-clock time the report was assembled.
GeneratedAt time.Time `json:"generated_at"`
}
TestReport is the runner's full output for one skill.
func RunAcceptance ¶
func RunAcceptance(ctx context.Context, skill *AgentSkill, ev Evaluator, opts RunOptions) (TestReport, error)
RunAcceptance evaluates the skill's acceptance_criteria using the supplied evaluator and returns a populated TestReport.
Returns:
- ErrNoCriteria when the skill has no acceptance_criteria.
- ErrCriterionOutOfRange when opts.CriterionIndex is set and points outside the skill's criteria slice.
Per-criterion infrastructure failures are reported as TestResults with Severity = TestSeverityError; only the two errors above stop the whole run.
func (TestReport) HasErrors ¶
func (r TestReport) HasErrors() bool
HasErrors reports whether any criterion errored during evaluation. The CLI uses this to map to exit code 2.
func (TestReport) HasFailures ¶
func (r TestReport) HasFailures() bool
HasFailures reports whether any criterion failed. The CLI uses this to map to exit code 1. Error-severity results are separate and map to exit code 2 in the CLI; here we only flag fail.
type TestResult ¶
type TestResult struct {
// Index is the criterion's position in the skill's
// AcceptanceCriteria slice (zero-based). Stable across runs.
Index int `json:"index"`
// Criterion is the verbatim Given/When/Then string from the
// skill's frontmatter.
Criterion string `json:"criterion"`
// Severity classifies the outcome.
Severity TestSeverity `json:"severity"`
// Message is a short human-readable rationale. Required for fail
// and error; optional for pass.
Message string `json:"message,omitempty"`
// EvaluatedAt is the wall-clock time the verdict was rendered.
EvaluatedAt time.Time `json:"evaluated_at"`
}
TestResult is the verdict for one criterion. Mirrors optimize.Finding's shape so the JSON envelope feels familiar.
type TestSeverity ¶
type TestSeverity string
TestSeverity classifies a single criterion's outcome. The vocabulary mirrors pkg/optimize.Severity so renderers can share badge styling.
const ( // TestSeverityPass means the evaluator verified the criterion is // satisfied by the skill as written. TestSeverityPass TestSeverity = "pass" // TestSeverityFail means the evaluator believes the criterion is // NOT satisfied. This is the case the runner reports back to CI. TestSeverityFail TestSeverity = "fail" // TestSeverityError means evaluation could not complete (LLM // unavailable, malformed criterion, etc.). Mapped to exit code 2 // by the CLI so infrastructure failures never look like a real // fail verdict. TestSeverityError TestSeverity = "error" )
type ValidationResult ¶
type ValidationResult struct {
Errors []string // Fatal validation failures
Warnings []string // Non-fatal advisories (e.g., body too long)
}
ValidationResult contains errors and warnings from skill validation.
func ValidateSkillFull ¶
func ValidateSkillFull(s *AgentSkill) *ValidationResult
ValidateSkillFull validates an AgentSkill and returns both errors and warnings.
func (*ValidationResult) Error ¶
func (v *ValidationResult) Error() error
Error returns the first error as a Go error, or nil if valid.
func (*ValidationResult) Valid ¶
func (v *ValidationResult) Valid() bool
Valid returns true if there are no errors (warnings are OK).