registry

package
v0.1.0-beta.11 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 23, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package registry — acceptance criteria runner.

A skill's `acceptance_criteria` frontmatter is a list of Given/When/Then prose strings. The runner evaluates each criterion against the skill's body and frontmatter, returning a TestReport that callers can render.

The DeterministicEvaluator ships in-tree: it looks for "PASS:" or "FAIL:" markers inside each criterion so fixture skills can encode expected outcomes without standing up an LLM provider. Production evaluation against a live model was previously provided by the LLMEvaluator; that path moved out when the agent runtime was retired.

The runner is read-only — it never mutates the skill or its files.

Index

Constants

This section is empty.

Variables

View Source
var ErrCriterionOutOfRange = errors.New("criterion index out of range")

ErrCriterionOutOfRange is returned when RunOptions.CriterionIndex names a criterion that does not exist on the skill. The CLI maps this to exit code 2.

View Source
var ErrNoCriteria = errors.New("skill has no acceptance_criteria")

ErrNoCriteria signals that the skill has no acceptance_criteria frontmatter. The runner returns this distinct error so callers can emit a clear "nothing to test" message and exit cleanly rather than reporting zero failures (which is ambiguous: did the skill pass, or was there nothing to check?).

View Source
var ErrNotFound = errors.New("not found")

ErrNotFound is returned when a skill does not exist in the store.

Functions

func RenderSkillMD

func RenderSkillMD(skill *AgentSkill) ([]byte, error)

RenderSkillMD serializes an AgentSkill back to SKILL.md format.

func ValidateSkill

func ValidateSkill(s *AgentSkill) error

ValidateSkill validates an AgentSkill and returns just the error (convenience wrapper).

func ValidateSkillName

func ValidateSkillName(name string) error

ValidateSkillName validates a skill name against the agentskills.io spec.

Types

type AgentSkill

type AgentSkill struct {
	// --- Frontmatter fields (from YAML between --- delimiters) ---
	Name          string            `yaml:"name" json:"name"`
	Description   string            `yaml:"description" json:"description"`
	License       string            `yaml:"license,omitempty" json:"license,omitempty"`
	Compatibility string            `yaml:"compatibility,omitempty" json:"compatibility,omitempty"`
	Metadata      map[string]string `yaml:"metadata,omitempty" json:"metadata,omitempty"`
	AllowedTools  string            `yaml:"allowed-tools,omitempty" json:"allowedTools,omitempty"`
	// AcceptanceCriteria documents expected skill behavior as human-readable
	// Given/When/Then scenarios. Gridctl extension; not part of agentskills.io spec.
	// See https://agentskills.io/specification
	AcceptanceCriteria []string `yaml:"acceptance_criteria,omitempty" json:"acceptanceCriteria,omitempty"`

	// --- Gridctl extensions (not in agentskills.io spec) ---
	State ItemState `yaml:"state,omitempty" json:"state"`

	// --- Parsed from file content (not in frontmatter YAML) ---
	Body string `yaml:"-" json:"body"` // Markdown content after frontmatter

	// --- Computed fields (not serialized to YAML) ---
	FileCount int    `yaml:"-" json:"fileCount"`     // Number of supporting files (scripts/, references/, assets/)
	Dir       string `yaml:"-" json:"dir,omitempty"` // Relative path from skills/ root (e.g., "git-workflow/branch-fork")
}

AgentSkill represents an Agent Skills standard SKILL.md file. See https://agentskills.io/specification for the full spec.

func ParseSkillMD

func ParseSkillMD(data []byte) (*AgentSkill, error)

ParseSkillMD parses a SKILL.md file into an AgentSkill. The file format is YAML frontmatter between --- delimiters followed by a markdown body.

func (*AgentSkill) Validate

func (s *AgentSkill) Validate() error

Validate checks the skill against the agentskills.io specification.

type DeterministicEvaluator

type DeterministicEvaluator struct{}

DeterministicEvaluator is the zero-LLM evaluator used by CI and unit tests. It inspects each criterion for the case-sensitive prefix markers "PASS:" or "FAIL:" and produces the corresponding verdict; criteria without a marker return TestSeverityError so test authors know the fixture needs to be explicit.

This adapter exists so the acceptance contract can be exercised end to end (CLI → API → evaluator) without standing up an LLM provider.

func (DeterministicEvaluator) Evaluate

func (DeterministicEvaluator) Evaluate(_ context.Context, _ *AgentSkill, idx int, criterion string) TestResult

Evaluate parses the marker convention. Markers are checked case-sensitively to avoid false positives in natural-language criteria like "should pass validation" — only explicit "PASS:" or "FAIL:" at the start of the trimmed criterion fires.

func (DeterministicEvaluator) Name

Name returns "deterministic".

type Evaluator

type Evaluator interface {
	// Name identifies the evaluator backend for the TestReport.
	Name() string

	// Evaluate produces a verdict for criterion idx against skill.
	// Implementations MUST NOT return both a TestResult and an error;
	// transient infrastructure failures should be returned via a
	// TestResult with Severity = TestSeverityError so the report
	// remains complete for the user.
	Evaluate(ctx context.Context, skill *AgentSkill, idx int, criterion string) TestResult
}

Evaluator runs a single acceptance criterion against a skill and returns the verdict. Implementations MUST honor ctx cancellation.

type ItemState

type ItemState string

ItemState represents the lifecycle state of a skill. Note: state is a gridctl extension, not part of the agentskills.io spec.

const (
	StateDraft    ItemState = "draft"
	StateActive   ItemState = "active"
	StateDisabled ItemState = "disabled"
)

type RegistryStatus

type RegistryStatus struct {
	TotalSkills  int `json:"totalSkills"`
	ActiveSkills int `json:"activeSkills"`
}

RegistryStatus contains summary statistics.

type RunOptions

type RunOptions struct {
	// CriterionIndex, when non-negative, scopes the run to a single
	// criterion. -1 (the zero value via NewRunOptions) means "all".
	CriterionIndex int

	// DryRun lists criteria without evaluating them. Severity on each
	// result is empty.
	DryRun bool

	// Now overrides the wall-clock used for EvaluatedAt and
	// GeneratedAt. Tests inject a fixed time; production callers
	// leave this zero so the runner defaults to time.Now().
	Now time.Time
}

RunOptions tunes a single Run pass.

func NewRunOptions

func NewRunOptions() RunOptions

NewRunOptions returns RunOptions with CriterionIndex set to -1, the "run every criterion" sentinel. Use this instead of a literal zero value so future-added fields stay backward-compatible.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is an in-process MCP server that serves Agent Skills as prompts. It implements mcp.AgentClient so it can be registered with the gateway router, and mcp.PromptProvider so the gateway can serve skills via MCP prompts and resources.

func New

func New(store *Store) *Server

New creates a registry server that serves skills as MCP prompts.

func (*Server) CallTool

func (s *Server) CallTool(_ context.Context, _ string, _ map[string]any) (*mcp.ToolCallResult, error)

CallTool always returns an error. See Tools().

func (*Server) GetPromptData

func (s *Server) GetPromptData(name string) (*mcp.PromptData, error)

GetPromptData returns a specific active skill's content as MCP PromptData.

func (*Server) HasContent

func (s *Server) HasContent() bool

HasContent returns true if the registry has any skills.

func (*Server) Initialize

func (s *Server) Initialize(ctx context.Context) error

Initialize loads the store.

func (*Server) IsInitialized

func (s *Server) IsInitialized() bool

IsInitialized returns whether the server has been initialized.

func (*Server) ListPromptData

func (s *Server) ListPromptData() []mcp.PromptData

ListPromptData returns active Agent Skills as MCP PromptData. Each skill gets a single optional "context" argument for clients to pass additional context when requesting the skill via prompts/get.

func (*Server) Name

func (s *Server) Name() string

Name returns "registry".

func (*Server) RefreshTools

func (s *Server) RefreshTools(ctx context.Context) error

RefreshTools reloads the store from disk.

func (*Server) ServerInfo

func (s *Server) ServerInfo() mcp.ServerInfo

ServerInfo returns server information.

func (*Server) Store

func (s *Server) Store() *Store

Store returns the underlying store for REST API access.

func (*Server) Tools

func (s *Server) Tools() []mcp.Tool

Tools always returns an empty slice. Skills are served as MCP prompts via the PromptProvider surface, not as executable tools. The method survives to satisfy mcp.AgentClient so the gateway router can keep the registry in its client set without a special case.

type SkillFile

type SkillFile struct {
	Path  string `json:"path"`  // Relative path within the skill dir (e.g., "scripts/lint.sh")
	Size  int64  `json:"size"`  // File size in bytes
	IsDir bool   `json:"isDir"` // True for directories
}

SkillFile represents a file within a skill directory.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store manages skill directories on disk. Each skill is a directory containing a required SKILL.md and optional supporting files (scripts/, references/, assets/).

func NewStore

func NewStore(baseDir string) *Store

NewStore creates a store rooted at the given directory.

func (*Store) ActiveSkills

func (s *Store) ActiveSkills() []*AgentSkill

ActiveSkills returns only skills with State == "active". Returned pointers are copies.

func (*Store) DeleteFile

func (s *Store) DeleteFile(skillName, filePath string) error

DeleteFile removes a file from a skill directory.

func (*Store) DeleteSkill

func (s *Store) DeleteSkill(name string) error

DeleteSkill removes a skill directory and cache entry.

func (*Store) Dir

func (s *Store) Dir() string

Dir returns the store's base directory path.

func (*Store) GetSkill

func (s *Store) GetSkill(name string) (*AgentSkill, error)

GetSkill returns a skill by name.

func (*Store) HasContent

func (s *Store) HasContent() bool

HasContent returns true if there is at least one skill.

func (*Store) ListFiles

func (s *Store) ListFiles(skillName string) ([]SkillFile, error)

ListFiles returns all files in a skill directory (excluding SKILL.md).

func (*Store) ListSkills

func (s *Store) ListSkills() []*AgentSkill

ListSkills returns all skills (all states). Returned pointers are copies.

func (*Store) Load

func (s *Store) Load() error

Load scans the skills/ subdirectory for SKILL.md files and checks for legacy YAML registry files.

func (*Store) ReadFile

func (s *Store) ReadFile(skillName, filePath string) ([]byte, error)

ReadFile reads a specific file from a skill directory.

func (*Store) RenameSkill

func (s *Store) RenameSkill(oldName, newName string) error

RenameSkill renames a skill directory and updates its frontmatter.

func (*Store) SaveSkill

func (s *Store) SaveSkill(sk *AgentSkill) error

SaveSkill creates or updates a skill (validates, writes SKILL.md, updates cache).

func (*Store) Status

func (s *Store) Status() RegistryStatus

Status returns registry summary counts.

func (*Store) WriteFile

func (s *Store) WriteFile(skillName, filePath string, data []byte) error

WriteFile writes a file to a skill directory, creating parent directories as needed.

type TestReport

type TestReport struct {
	// SkillName echoes the skill the runner ran against.
	SkillName string `json:"skill_name"`

	// Results is the per-criterion verdicts in criterion order.
	Results []TestResult `json:"results"`

	// PassCount, FailCount, ErrorCount summarize Results so callers
	// don't re-tally.
	PassCount  int `json:"pass_count"`
	FailCount  int `json:"fail_count"`
	ErrorCount int `json:"error_count"`

	// Evaluator names the backend that ran the criteria (e.g. "llm",
	// "deterministic"). Surfaced in JSON output so CI logs make the
	// provenance explicit.
	Evaluator string `json:"evaluator"`

	// DryRun is true when Results was populated by listing criteria
	// without executing them. Each TestResult in that case carries
	// Severity = "" so renderers can show "—" instead of a verdict.
	DryRun bool `json:"dry_run,omitempty"`

	// GeneratedAt is the wall-clock time the report was assembled.
	GeneratedAt time.Time `json:"generated_at"`
}

TestReport is the runner's full output for one skill.

func RunAcceptance

func RunAcceptance(ctx context.Context, skill *AgentSkill, ev Evaluator, opts RunOptions) (TestReport, error)

RunAcceptance evaluates the skill's acceptance_criteria using the supplied evaluator and returns a populated TestReport.

Returns:

  • ErrNoCriteria when the skill has no acceptance_criteria.
  • ErrCriterionOutOfRange when opts.CriterionIndex is set and points outside the skill's criteria slice.

Per-criterion infrastructure failures are reported as TestResults with Severity = TestSeverityError; only the two errors above stop the whole run.

func (TestReport) HasErrors

func (r TestReport) HasErrors() bool

HasErrors reports whether any criterion errored during evaluation. The CLI uses this to map to exit code 2.

func (TestReport) HasFailures

func (r TestReport) HasFailures() bool

HasFailures reports whether any criterion failed. The CLI uses this to map to exit code 1. Error-severity results are separate and map to exit code 2 in the CLI; here we only flag fail.

type TestResult

type TestResult struct {
	// Index is the criterion's position in the skill's
	// AcceptanceCriteria slice (zero-based). Stable across runs.
	Index int `json:"index"`

	// Criterion is the verbatim Given/When/Then string from the
	// skill's frontmatter.
	Criterion string `json:"criterion"`

	// Severity classifies the outcome.
	Severity TestSeverity `json:"severity"`

	// Message is a short human-readable rationale. Required for fail
	// and error; optional for pass.
	Message string `json:"message,omitempty"`

	// EvaluatedAt is the wall-clock time the verdict was rendered.
	EvaluatedAt time.Time `json:"evaluated_at"`
}

TestResult is the verdict for one criterion. Mirrors optimize.Finding's shape so the JSON envelope feels familiar.

type TestSeverity

type TestSeverity string

TestSeverity classifies a single criterion's outcome. The vocabulary mirrors pkg/optimize.Severity so renderers can share badge styling.

const (
	// TestSeverityPass means the evaluator verified the criterion is
	// satisfied by the skill as written.
	TestSeverityPass TestSeverity = "pass"

	// TestSeverityFail means the evaluator believes the criterion is
	// NOT satisfied. This is the case the runner reports back to CI.
	TestSeverityFail TestSeverity = "fail"

	// TestSeverityError means evaluation could not complete (LLM
	// unavailable, malformed criterion, etc.). Mapped to exit code 2
	// by the CLI so infrastructure failures never look like a real
	// fail verdict.
	TestSeverityError TestSeverity = "error"
)

type ValidationResult

type ValidationResult struct {
	Errors   []string // Fatal validation failures
	Warnings []string // Non-fatal advisories (e.g., body too long)
}

ValidationResult contains errors and warnings from skill validation.

func ValidateSkillFull

func ValidateSkillFull(s *AgentSkill) *ValidationResult

ValidateSkillFull validates an AgentSkill and returns both errors and warnings.

func (*ValidationResult) Error

func (v *ValidationResult) Error() error

Error returns the first error as a Go error, or nil if valid.

func (*ValidationResult) Valid

func (v *ValidationResult) Valid() bool

Valid returns true if there are no errors (warnings are OK).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL