Documentation
¶
Overview ¶
Package registry — acceptance criteria runner.
A skill's `acceptance_criteria` frontmatter is a list of Given/When/Then prose strings. The runner evaluates each criterion against the skill's body and frontmatter, returning a TestReport that callers can render.
The DeterministicEvaluator ships in-tree: it looks for "PASS:" or "FAIL:" markers inside each criterion so fixture skills can encode expected outcomes without standing up an LLM provider. Production evaluation against a live model was previously provided by the LLMEvaluator; that path moved out when the agent runtime was retired.
The runner is read-only — it never mutates the skill or its files.
Index ¶
- Constants
- Variables
- func AgentHonorMatrix() map[string]HonorStatus
- func IsKnownModelValue(v string) bool
- func KnownModelAliases() []string
- func NormalizeModelValue(v string) string
- func RenderSkillMD(skill *AgentSkill) ([]byte, error)
- func RenderWithModelPreference(sk *AgentSkill, value string) ([]byte, error)
- func RewriteTopLevelModelLine(raw []byte, value string) ([]byte, bool)
- func SkillHonorMatrix() map[string]HonorStatus
- func ValidateSkill(s *AgentSkill) error
- func ValidateSkillName(name string) error
- type AgentSkill
- type DeterministicEvaluator
- type Evaluator
- type HonorStatus
- type ItemState
- type ModelPolicy
- type ModelPreference
- type RegistryStatus
- type RunOptions
- type Server
- func (s *Server) CallTool(_ context.Context, _ string, _ map[string]any) (*mcp.ToolCallResult, error)
- func (s *Server) GetPromptData(name string) (*mcp.PromptData, error)
- func (s *Server) HasContent() bool
- func (s *Server) Initialize(ctx context.Context) error
- func (s *Server) IsInitialized() bool
- func (s *Server) ListPromptData() []mcp.PromptData
- func (s *Server) Name() string
- func (s *Server) RefreshTools(ctx context.Context) error
- func (s *Server) ServerInfo() mcp.ServerInfo
- func (s *Server) Store() *Store
- func (s *Server) Tools() []mcp.Tool
- type SkillFile
- type SkillMetadata
- type Store
- func (s *Store) ActiveSkills() []*AgentSkill
- func (s *Store) DeleteFile(skillName, filePath string) error
- func (s *Store) DeleteSkill(name string) error
- func (s *Store) Dir() string
- func (s *Store) GetSkill(name string) (*AgentSkill, error)
- func (s *Store) HasContent() bool
- func (s *Store) ListFiles(skillName string) ([]SkillFile, error)
- func (s *Store) ListSkills() []*AgentSkill
- func (s *Store) Load() error
- func (s *Store) ReadFile(skillName, filePath string) ([]byte, error)
- func (s *Store) RefreshFileCount(name string) error
- func (s *Store) RenameSkill(oldName, newName string) error
- func (s *Store) SaveSkill(sk *AgentSkill) error
- func (s *Store) Status() RegistryStatus
- func (s *Store) WriteFile(skillName, filePath string, data []byte) error
- type TestReport
- type TestResult
- type TestSeverity
- type ValidationResult
Constants ¶
const ( ResolutionAuthor = "author" ResolutionDefault = "default" ResolutionOverride = "override" )
Resolution provenance values, shared with the REST surface.
const ( ModelSourceTopLevel = "model" ModelSourceMetaPreferred = "metadata.preferred-model" ModelSourceMetaModel = "metadata.model" )
Model preference source keys, in recognition precedence order.
Variables ¶
var ErrCriterionOutOfRange = errors.New("criterion index out of range")
ErrCriterionOutOfRange is returned when RunOptions.CriterionIndex names a criterion that does not exist on the skill. The CLI maps this to exit code 2.
var ErrNoCriteria = errors.New("skill has no acceptance_criteria")
ErrNoCriteria signals that the skill has no acceptance_criteria frontmatter. The runner returns this distinct error so callers can emit a clear "nothing to test" message and exit cleanly rather than reporting zero failures (which is ambiguous: did the skill pass, or was there nothing to check?).
var ErrNotFound = errors.New("not found")
ErrNotFound is returned when a skill does not exist in the store.
Functions ¶
func AgentHonorMatrix ¶
func AgentHonorMatrix() map[string]HonorStatus
AgentHonorMatrix returns a copy of the full agent-target matrix, keyed by target slug.
func IsKnownModelValue ¶
IsKnownModelValue reports whether a value is a documented alias, a suffixed alias form, or shaped like a full model ID. Callers treat a false as advisory-warn material, never an error.
func KnownModelAliases ¶
func KnownModelAliases() []string
KnownModelAliases returns the documented alias vocabulary, for lint messages and docs.
func NormalizeModelValue ¶
NormalizeModelValue canonicalizes a model value for comparison: trimmed and lowercased. Two declarations that normalize equal are the same preference; a rewrite is never forced over case or whitespace.
func RenderSkillMD ¶
func RenderSkillMD(skill *AgentSkill) ([]byte, error)
RenderSkillMD serializes an AgentSkill back to SKILL.md format.
func RenderWithModelPreference ¶
func RenderWithModelPreference(sk *AgentSkill, value string) ([]byte, error)
RenderWithModelPreference renders SKILL.md bytes with the resolved model preference applied to a copy of the skill. It backs the projection-time policy rewrite: the registry canonical is NEVER mutated; callers write the returned bytes into projected copies only.
The honored key is always set: Claude Code reads top-level `model:` on projected skill files, so the rewrite guarantees it even for authors who declared only a metadata key. Any metadata declaration the author wrote is updated in place as well, so a projected file never carries two disagreeing declarations. All other frontmatter rides through the existing parse/render round trip untouched.
func RewriteTopLevelModelLine ¶
RewriteTopLevelModelLine returns frontmatter-bearing bytes with the top-level `model:` key set to value (or removed when value is empty), preserving every other byte verbatim. It is deliberate line surgery, not a parse/re-render, for files whose bytes are the contract (identity-projected agents, and adopt's reversal of a projected rewrite). ok is false when the file has no recognizable frontmatter block or an existing model value is not a single-line scalar (never valid per the client docs, and not safely replaceable line-wise); callers fall back to pass-through. Callers that can parse the result should verify it, since text surgery cannot see every YAML form.
func SkillHonorMatrix ¶
func SkillHonorMatrix() map[string]HonorStatus
SkillHonorMatrix returns a copy of the full skill-target matrix, keyed by target slug (REST and lint consumers).
func ValidateSkill ¶
func ValidateSkill(s *AgentSkill) error
ValidateSkill validates an AgentSkill and returns just the error (convenience wrapper).
func ValidateSkillName ¶
ValidateSkillName validates a skill name against the agentskills.io spec.
Types ¶
type AgentSkill ¶
type AgentSkill struct {
// --- Frontmatter fields (from YAML between --- delimiters) ---
Name string `yaml:"name" json:"name"`
Description string `yaml:"description" json:"description"`
License string `yaml:"license,omitempty" json:"license,omitempty"`
Compatibility string `yaml:"compatibility,omitempty" json:"compatibility,omitempty"`
Metadata SkillMetadata `yaml:"metadata,omitempty" json:"metadata,omitempty"`
AllowedTools string `yaml:"allowed-tools,omitempty" json:"allowedTools,omitempty"`
// AcceptanceCriteria documents expected skill behavior as human-readable
// Given/When/Then scenarios. Gridctl extension; not part of agentskills.io spec.
// See https://agentskills.io/specification
AcceptanceCriteria []string `yaml:"acceptance_criteria,omitempty" json:"acceptanceCriteria,omitempty"`
// --- Gridctl extensions (not in agentskills.io spec) ---
State ItemState `yaml:"state,omitempty" json:"state"`
// Extra holds frontmatter keys this struct does not model, preserved
// through parse, render, and the JSON API so imports and edits never
// strip them (clients read keys like argument-hint and
// disable-model-invocation from projected skills). Values are the
// decoded YAML; gridctl never interprets them. The yaml tag is unused:
// UnmarshalYAML populates it and RenderSkillMD emits it explicitly.
Extra map[string]any `yaml:"-" json:"extra,omitempty"`
// --- Parsed from file content (not in frontmatter YAML) ---
Body string `yaml:"-" json:"body"` // Markdown content after frontmatter
// --- Computed fields (not serialized to YAML) ---
FileCount int `yaml:"-" json:"fileCount"` // Number of supporting files (scripts/, references/, assets/)
Dir string `yaml:"-" json:"dir,omitempty"` // Relative path from skills/ root (e.g., "git-workflow/branch-fork")
}
AgentSkill represents an Agent Skills standard SKILL.md file. See https://agentskills.io/specification for the full spec.
func ParseSkillMD ¶
func ParseSkillMD(data []byte) (*AgentSkill, error)
ParseSkillMD parses a SKILL.md file into an AgentSkill. The file format is YAML frontmatter between --- delimiters followed by a markdown body.
func (*AgentSkill) UnmarshalYAML ¶
func (s *AgentSkill) UnmarshalYAML(value *yaml.Node) error
UnmarshalYAML decodes frontmatter into the typed fields and captures every unmodeled top-level key into Extra, so unknown keys survive the registry's parse/render round trip instead of being silently dropped. The mapping node is walked by hand for the same reason SkillMetadata's unmarshaler does it: yaml.v3's map decoding would reject a duplicate key outright, and a plain struct decode has no way to see unmatched keys at all. Known fields keep struct-decode strictness (a type mismatch fails the parse, as before); unknown values that fail to decode are skipped rather than fatal.
func (*AgentSkill) Validate ¶
func (s *AgentSkill) Validate() error
Validate checks the skill against the agentskills.io specification.
type DeterministicEvaluator ¶
type DeterministicEvaluator struct{}
DeterministicEvaluator is the zero-LLM evaluator used by CI and unit tests. It inspects each criterion for the case-sensitive prefix markers "PASS:" or "FAIL:" and produces the corresponding verdict; criteria without a marker return TestSeverityError so test authors know the fixture needs to be explicit.
This adapter exists so the acceptance contract can be exercised end to end (CLI → API → evaluator) without standing up an LLM provider.
func (DeterministicEvaluator) Evaluate ¶
func (DeterministicEvaluator) Evaluate(_ context.Context, _ *AgentSkill, idx int, criterion string) TestResult
Evaluate parses the marker convention. Markers are checked case-sensitively to avoid false positives in natural-language criteria like "should pass validation" — only explicit "PASS:" or "FAIL:" at the start of the trimmed criterion fires.
func (DeterministicEvaluator) Name ¶
func (DeterministicEvaluator) Name() string
Name returns "deterministic".
type Evaluator ¶
type Evaluator interface {
// Name identifies the evaluator backend for the TestReport.
Name() string
// Evaluate produces a verdict for criterion idx against skill.
// Implementations MUST NOT return both a TestResult and an error;
// transient infrastructure failures should be returned via a
// TestResult with Severity = TestSeverityError so the report
// remains complete for the user.
Evaluate(ctx context.Context, skill *AgentSkill, idx int, criterion string) TestResult
}
Evaluator runs a single acceptance criterion against a skill and returns the verdict. Implementations MUST honor ctx cancellation.
type HonorStatus ¶
type HonorStatus string
HonorStatus is what one projection target does with a declared model preference. The matrix is static, maintained truth about client behavior; it will go stale on client-doc timescales, so every entry cites its source in the tables below and the sync packages carry exhaustiveness tests that fail when a projection target gains no row.
const ( // HonorHonored: the client reads the key and uses it as a model // resolution input (below its own env var and per-invocation // overrides). HonorHonored HonorStatus = "honored" // HonorIgnored: the client documents no model key for this surface. HonorIgnored HonorStatus = "ignored" // HonorUnknown: consumer-dependent or unverified; surfaced as such // rather than guessed. HonorUnknown HonorStatus = "unknown" // HonorDropped: the projection render deliberately drops the key // (client model vocabularies are not Claude's); reported in // Rendered.Dropped at sync time too. HonorDropped HonorStatus = "dropped-on-render" )
func AgentHonor ¶
func AgentHonor(slug string) HonorStatus
AgentHonor returns the honor status for one agent projection target slug; unknown slugs report HonorUnknown.
func SkillHonor ¶
func SkillHonor(slug string) HonorStatus
SkillHonor returns the honor status for one skill projection target slug; unknown slugs report HonorUnknown rather than guessing.
type ItemState ¶
type ItemState string
ItemState represents the lifecycle state of a skill. Note: state is a gridctl extension, not part of the agentskills.io spec.
type ModelPolicy ¶
type ModelPolicy struct {
// Rewrite opts the scope into projection rewrite. False means
// surfacing-only: Resolve still answers (for status and REST), but
// nothing on disk changes.
Rewrite bool
// Default applies where the author declared nothing.
Default string
// Overrides maps exact registry names to the preference applied
// regardless of the author's declaration, in either direction.
Overrides map[string]string
}
ModelPolicy is one scope's compiled model preference policy (the stack.yaml `model_preferences:` block for either the skills or the agents scope), consumed by the projection engines. The controller compiles it from the loaded stack; CLI call sites without stack context run with a nil policy, which is pure pass-through and, for projections a policy previously rewrote, preserve-not-revert; the sync packages own that rule.
func (*ModelPolicy) NeedsRewrite ¶
func (p *ModelPolicy) NeedsRewrite(name, declared string) (string, bool)
NeedsRewrite reports whether projection must write a resolved value that differs from the author's declaration. Equality is judged after normalization, so case or whitespace never forces a rewrite (and never forces a skill off its symlink channel).
func (*ModelPolicy) Resolve ¶
func (p *ModelPolicy) Resolve(name, declared string) (value, resolution string)
Resolve answers the effective preference for one name: override beats the author's declaration beats the default. The returned resolution names which source won ("override", "author", "default"), or "" when nothing applies.
type ModelPreference ¶
type ModelPreference struct {
// Values holds the declared preference(s), most preferred first.
Values []string `json:"values"`
// SourceKey names where the author declared it: "model",
// "metadata.preferred-model", or "metadata.model".
SourceKey string `json:"sourceKey"`
}
ModelPreference is the typed read-side view of a skill or agent author's declared model preference. It is a preference, never an enforcement point: clients resolve their own model (env vars and per-invocation parameters outrank projected frontmatter), and gridctl only surfaces, defaults, and overrides what the projected files declare.
Values is a list so a future agentskills.io ordered-preference shape is a consumption change, not a migration; today it always holds one element. The author's raw key is untouched wherever it was written (top-level Extra or metadata); this view is derived, never stored.
func ExtractModelPreference ¶
func ExtractModelPreference(sk *AgentSkill) *ModelPreference
ExtractModelPreference derives the typed preference from a skill's frontmatter. Recognition precedence: top-level `model:` beats `metadata.preferred-model` beats `metadata.model`. Non-string or empty declarations yield nil. Extraction never mutates the skill.
func ModelPreferenceFromKeys ¶
func ModelPreferenceFromKeys(topLevel string, metadata map[string]string) *ModelPreference
ModelPreferenceFromKeys derives the typed preference from raw key values, for callers whose documents are not AgentSkill-shaped (agent definitions keep frontmatter as raw YAML nodes). topLevel is the top-level `model:` scalar; metadata the `metadata:` map (nil allowed).
func (*ModelPreference) Value ¶
func (p *ModelPreference) Value() string
Value returns the single effective preference (the first value), or "" for a nil preference.
type RegistryStatus ¶
type RegistryStatus struct {
TotalSkills int `json:"totalSkills"`
ActiveSkills int `json:"activeSkills"`
}
RegistryStatus contains summary statistics.
type RunOptions ¶
type RunOptions struct {
// CriterionIndex, when non-negative, scopes the run to a single
// criterion. -1 (the zero value via NewRunOptions) means "all".
CriterionIndex int
// DryRun lists criteria without evaluating them. Severity on each
// result is empty.
DryRun bool
// Now overrides the wall-clock used for EvaluatedAt and
// GeneratedAt. Tests inject a fixed time; production callers
// leave this zero so the runner defaults to time.Now().
Now time.Time
}
RunOptions tunes a single Run pass.
func NewRunOptions ¶
func NewRunOptions() RunOptions
NewRunOptions returns RunOptions with CriterionIndex set to -1, the "run every criterion" sentinel. Use this instead of a literal zero value so future-added fields stay backward-compatible.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is an in-process MCP server that serves Agent Skills as prompts. It implements mcp.AgentClient so it can be registered with the gateway router, and mcp.PromptProvider so the gateway can serve skills via MCP prompts and resources.
func (*Server) CallTool ¶
func (s *Server) CallTool(_ context.Context, _ string, _ map[string]any) (*mcp.ToolCallResult, error)
CallTool always returns an error. See Tools().
func (*Server) GetPromptData ¶
func (s *Server) GetPromptData(name string) (*mcp.PromptData, error)
GetPromptData returns a specific active skill's content as MCP PromptData.
func (*Server) HasContent ¶
HasContent returns true if the registry has any skills.
func (*Server) Initialize ¶
Initialize loads the store.
func (*Server) IsInitialized ¶
IsInitialized returns whether the server has been initialized.
func (*Server) ListPromptData ¶
func (s *Server) ListPromptData() []mcp.PromptData
ListPromptData returns active Agent Skills as MCP PromptData. Each skill gets a single optional "context" argument for clients to pass additional context when requesting the skill via prompts/get.
func (*Server) RefreshTools ¶
RefreshTools reloads the store from disk.
func (*Server) ServerInfo ¶
func (s *Server) ServerInfo() mcp.ServerInfo
ServerInfo returns server information.
type SkillFile ¶
type SkillFile struct {
Path string `json:"path"` // Relative path within the skill dir (e.g., "scripts/lint.sh")
Size int64 `json:"size"` // File size in bytes
IsDir bool `json:"isDir"` // True for directories
}
SkillFile represents a file within a skill directory.
type SkillMetadata ¶
SkillMetadata holds the frontmatter metadata mapping. The agentskills.io spec defines it as string-to-string, but ecosystems like openclaw/ClawHub publish nested values there, so decoding is lenient: non-string values are coerced to strings (see UnmarshalYAML in frontmatter.go).
func (*SkillMetadata) UnmarshalYAML ¶
func (m *SkillMetadata) UnmarshalYAML(value *yaml.Node) error
UnmarshalYAML decodes metadata leniently. Values that are not strings (nested mappings, sequences, booleans, numbers) are coerced to their string form instead of failing the whole SKILL.md parse. Non-mapping metadata (e.g. a bare string) is ignored rather than treated as an error. The mapping node is walked by hand so one bad entry (or a duplicate key, which yaml.v3's map decoding rejects) cannot discard the valid keys.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store manages skill directories on disk. Each skill is a directory containing a required SKILL.md and optional supporting files (scripts/, references/, assets/).
func (*Store) ActiveSkills ¶
func (s *Store) ActiveSkills() []*AgentSkill
ActiveSkills returns only skills with State == "active". Returned pointers are copies.
func (*Store) DeleteFile ¶
DeleteFile removes a file from a skill directory.
func (*Store) DeleteSkill ¶
DeleteSkill removes a skill directory and cache entry.
func (*Store) GetSkill ¶
func (s *Store) GetSkill(name string) (*AgentSkill, error)
GetSkill returns a skill by name.
func (*Store) HasContent ¶
HasContent returns true if there is at least one skill.
func (*Store) ListSkills ¶
func (s *Store) ListSkills() []*AgentSkill
ListSkills returns all skills (all states). Returned pointers are copies.
func (*Store) Load ¶
Load scans the skills/ subdirectory for SKILL.md files and checks for legacy YAML registry files.
func (*Store) RefreshFileCount ¶
RefreshFileCount recomputes a skill's supporting-file count from disk. Callers that write files into a skill directory after SaveSkill (the git importer installs scripts/ and references/ once the skill validates) use this so the cached count matches what actually landed.
func (*Store) RenameSkill ¶
RenameSkill renames a skill directory and updates its frontmatter.
func (*Store) SaveSkill ¶
func (s *Store) SaveSkill(sk *AgentSkill) error
SaveSkill creates or updates a skill (validates, writes SKILL.md, updates cache).
func (*Store) Status ¶
func (s *Store) Status() RegistryStatus
Status returns registry summary counts.
type TestReport ¶
type TestReport struct {
// SkillName echoes the skill the runner ran against.
SkillName string `json:"skill_name"`
// Results is the per-criterion verdicts in criterion order.
Results []TestResult `json:"results"`
// PassCount, FailCount, ErrorCount summarize Results so callers
// don't re-tally.
PassCount int `json:"pass_count"`
FailCount int `json:"fail_count"`
ErrorCount int `json:"error_count"`
// Evaluator names the backend that ran the criteria (e.g. "llm",
// "deterministic"). Surfaced in JSON output so CI logs make the
// provenance explicit.
Evaluator string `json:"evaluator"`
// DryRun is true when Results was populated by listing criteria
// without executing them. Each TestResult in that case carries
// Severity = "" so renderers can show "—" instead of a verdict.
DryRun bool `json:"dry_run,omitempty"`
// GeneratedAt is the wall-clock time the report was assembled.
GeneratedAt time.Time `json:"generated_at"`
}
TestReport is the runner's full output for one skill.
func RunAcceptance ¶
func RunAcceptance(ctx context.Context, skill *AgentSkill, ev Evaluator, opts RunOptions) (TestReport, error)
RunAcceptance evaluates the skill's acceptance_criteria using the supplied evaluator and returns a populated TestReport.
Returns:
- ErrNoCriteria when the skill has no acceptance_criteria.
- ErrCriterionOutOfRange when opts.CriterionIndex is set and points outside the skill's criteria slice.
Per-criterion infrastructure failures are reported as TestResults with Severity = TestSeverityError; only the two errors above stop the whole run.
func (TestReport) HasErrors ¶
func (r TestReport) HasErrors() bool
HasErrors reports whether any criterion errored during evaluation. The CLI uses this to map to exit code 2.
func (TestReport) HasFailures ¶
func (r TestReport) HasFailures() bool
HasFailures reports whether any criterion failed. The CLI uses this to map to exit code 1. Error-severity results are separate and map to exit code 2 in the CLI; here we only flag fail.
type TestResult ¶
type TestResult struct {
// Index is the criterion's position in the skill's
// AcceptanceCriteria slice (zero-based). Stable across runs.
Index int `json:"index"`
// Criterion is the verbatim Given/When/Then string from the
// skill's frontmatter.
Criterion string `json:"criterion"`
// Severity classifies the outcome.
Severity TestSeverity `json:"severity"`
// Message is a short human-readable rationale. Required for fail
// and error; optional for pass.
Message string `json:"message,omitempty"`
// EvaluatedAt is the wall-clock time the verdict was rendered.
EvaluatedAt time.Time `json:"evaluated_at"`
}
TestResult is the verdict for one criterion. Mirrors optimize.Finding's shape so the JSON envelope feels familiar.
type TestSeverity ¶
type TestSeverity string
TestSeverity classifies a single criterion's outcome. The vocabulary mirrors pkg/optimize.Severity so renderers can share badge styling.
const ( // TestSeverityPass means the evaluator verified the criterion is // satisfied by the skill as written. TestSeverityPass TestSeverity = "pass" // TestSeverityFail means the evaluator believes the criterion is // NOT satisfied. This is the case the runner reports back to CI. TestSeverityFail TestSeverity = "fail" // TestSeverityError means evaluation could not complete (LLM // unavailable, malformed criterion, etc.). Mapped to exit code 2 // by the CLI so infrastructure failures never look like a real // fail verdict. TestSeverityError TestSeverity = "error" )
type ValidationResult ¶
type ValidationResult struct {
Errors []string // Fatal validation failures
Warnings []string // Non-fatal advisories (e.g., body too long)
}
ValidationResult contains errors and warnings from skill validation.
func ValidateSkillFull ¶
func ValidateSkillFull(s *AgentSkill) *ValidationResult
ValidateSkillFull validates an AgentSkill and returns both errors and warnings.
func (*ValidationResult) Error ¶
func (v *ValidationResult) Error() error
Error returns the first error as a Go error, or nil if valid.
func (*ValidationResult) Valid ¶
func (v *ValidationResult) Valid() bool
Valid returns true if there are no errors (warnings are OK).