agentcoretesting

package
v0.54.10 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 7 Imported by: 0

README

agentcore/testing

agentcore/testing 提供面向外部实现者的 package-local conformance helpers,用来验证自定义 Storage、ToolRegistry、Provider 是否满足 agentcore 的公共 contract。

它的定位不是:

  • 再造一套复杂测试框架
  • 暴露 agentcore 仓库内部测试细节
  • 替代 API 文档或 examples

它的定位是:

  • 给外部实现一个可复用的 contract test 入口
  • 让独立仓库里的适配器可以自证兼容
  • 把“文档说明的预期”变成“可以直接跑的验证”

当前提供的 suites

  • RunStorageSuite 验证 run snapshot、append-only transcript、resume 后 authority transcript 可恢复
  • RunStorageFixtureSuite 在保留 happy path 的同时,验证 SaveRun / LoadRun / AppendMessages / LoadMessages 的错误传播
  • RunToolRegistrySuite 验证 ListTools schema 与 ExecuteTool 的基本结果 contract
  • RunProviderSuite 验证 GenerateResult 的最小合法性,以及可选 streaming provider 的基本行为

Contract Checklist

如果你不确定“应该先跑哪组 suite,或者某个 surface 目前属于哪一层证据”,可以先看 package-local checklist:

  • ContractChecklist()

它会返回一组结构化条目,描述:

  • 哪些 surface 当前已经属于 runtime contract
  • 哪些 surface 已经到 Runtime + Host 证据层
  • 哪些 suite 适合优先拿来验证外部实现

这不是新的测试框架,而是一层更接近文档矩阵的公开元数据入口。

Onboarding Checklist

如果你是外部适配器作者,想从“我应该先看什么、跑什么、参考什么”这个角度进入,也可以先看:

  • OnboardingChecklist()

它会按实现类别给出一组最小导航:

  • 推荐先跑的 suites
  • 推荐先看的 docs
  • 推荐参考的 package-local examples

当前覆盖:

  • Storage
  • ToolRegistry
  • Provider

一个很实用的使用顺序是:

  1. 先读 OnboardingChecklist(),确定你属于哪一类实现者
  2. 再读 ContractChecklist(),确认当前这个 surface 的证据层与推荐 suites
  3. 最后按对应 suite 和 example test 开始落你的自定义实现验证

最小使用方式

Storage
package mystorage_test

import (
    "testing"

    "github.com/inhere/suga/pkg/agentcore"
    agentcoretesting "github.com/inhere/suga/pkg/agentcore/testing"
)

func TestStorageContract(t *testing.T) {
    agentcoretesting.RunStorageSuite(t, agentcoretesting.NewStorageFactoryFromPath(
        func(t *testing.T, root string) agentcore.Storage {
            t.Helper()
            return mystorage.New(root)
        },
    ))
}

如果你还想验证 storage 的错误传播 contract,可以使用 RunStorageFixtureSuite(...):

fixture := agentcoretesting.NewStorageFixture(erroringStorage).
    WithSaveRunError(errSaveBoom).
    WithLoadRunError(errLoadBoom).
    WithAppendMessagesError(errAppendBoom).
    WithLoadMessagesError(errLoadMessagesBoom)

agentcoretesting.RunStorageFixtureSuite(t, func(t *testing.T) agentcoretesting.StorageFixture {
    t.Helper()
    return fixture
})

当你既想保留 round-trip / resume 这类 happy path,又想单独验证错误路径时,可以给 fixture 分开提供:

  • RoundTripStorage
  • MessagesStorage
  • ResumeStorage

这样 happy-path 子测可以继续跑在正常 storage 上,错误传播子测则跑在错误 stub 上。

ToolRegistry
package mytools_test

import (
    "testing"

    "github.com/inhere/suga/pkg/agentcore"
    agentcoretesting "github.com/inhere/suga/pkg/agentcore/testing"
)

func TestRegistryContract(t *testing.T) {
    agentcoretesting.RunToolRegistrySuite(t, func(t *testing.T) agentcoretesting.ToolRegistryFixture {
        t.Helper()

        registry := mytools.NewRegistry()
        registry.Register(...)

        return agentcoretesting.NewToolRegistryFixture(
            registry,
            agentcore.ToolSchema{
                Name: "echo",
            },
            agentcore.ToolCall{
                CallID: "call-1",
                Name:   "echo",
            },
            &agentcore.ToolResult{
                CallID: "call-1",
                Name:   "echo",
            },
        ).WithExecuteError(
            "tool_failure",
            agentcore.ToolCall{
                CallID: "call-2",
                Name:   "fail",
            },
            errToolBoom,
        )
    })
}
Provider
package myprovider_test

import (
    "testing"

    "github.com/inhere/suga/pkg/agentcore"
    agentcoretesting "github.com/inhere/suga/pkg/agentcore/testing"
)

func TestProviderContract(t *testing.T) {
    agentcoretesting.RunProviderSuite(t, func(t *testing.T) agentcoretesting.ProviderFixture {
        t.Helper()

        provider := myprovider.New(...)
        streamProvider := myprovider.NewStreaming(...)

        return agentcoretesting.NewProviderFixture(
            provider,
            agentcoretesting.NewGenerateCase(
                "final",
                &agentcore.GenerateRequest{},
                &agentcore.GenerateResult{
                    Content:      "done",
                    FinishReason: agentcore.FinishFinal,
                },
            ),
        ).WithStreamingProvider(streamProvider).WithStreaming(agentcoretesting.StreamingCase{
            Name:    "stream_success",
            Request: &agentcore.GenerateRequest{},
            WantEvents: []agentcore.StreamEvent{
                {Kind: agentcore.StreamEventTextDelta, Delta: "hel"},
                {Kind: agentcore.StreamEventTextDelta, Delta: "lo"},
                {Kind: agentcore.StreamEventMessageStop},
            },
            WantResult: &agentcore.GenerateResult{
                Content:      "hello",
                FinishReason: agentcore.FinishFinal,
            },
        })
    })
}

Streaming cases

RunProviderSuite 现在不仅能验证 streaming success path,也能验证两类常见错误路径:

  • stream provider 自己返回错误
  • host 侧 stream callback 返回错误

这几个字段分别表示:

  • WithStreamingProvider(...) 当你的 Generate(...) 和 Stream(...) 不适合共用同一个脚本化 provider,或者你想把 generate/stream contract 分开验证时使用
  • StreamingCase.WantErr 表示这次 Stream(...) 调用最终应该返回的错误
  • StreamingCase.EmitErr 表示 host 侧 stream callback 主动返回错误,用来验证 provider 是否正确把 callback 失败向上传播

一个最小 streaming error 示例:

fixture := agentcoretesting.NewProviderFixture(
    generateProvider,
    agentcoretesting.NewGenerateCase(
        "final",
        &agentcore.GenerateRequest{},
        &agentcore.GenerateResult{
            Content:      "done",
            FinishReason: agentcore.FinishFinal,
        },
    ),
).WithStreamingProvider(streamProvider).WithStreamingError(agentcoretesting.StreamingCase{
    Name:    "stream_provider_error",
    Request: &agentcore.GenerateRequest{},
    WantErr: errStreamBoom,
}).WithStreamingHandlerError(agentcoretesting.StreamingCase{
    Name:    "stream_handler_error",
    Request: &agentcore.GenerateRequest{},
    WantErr: errHandlerBoom,
    EmitErr: errHandlerBoom,
})

Storage error cases

RunStorageFixtureSuite 当前主要用于验证 storage 的错误传播,而不替代最简单的 RunStorageSuite。

推荐把它理解成两层:

  • RunStorageSuite(...) 只关心最小 happy path
  • RunStorageFixtureSuite(...) 在 happy path 之外,额外验证错误路径 contract

当前支持的最小错误路径包括:

  • SaveRun 返回错误
  • LoadRun 返回错误
  • AppendMessages 返回错误
  • LoadMessages 返回错误
  • SaveRun 在缺失 RunID 时返回错误

如果你的错误 stub 会破坏 happy path,推荐像下面这样把正常 storage 和错误 storage 分开:

agentcoretesting.RunStorageFixtureSuite(t, func(t *testing.T) agentcoretesting.StorageFixture {
    t.Helper()

    return agentcoretesting.NewStorageFixture(erroringStorage).
        WithSaveRunError(errSaveBoom).
        WithLoadRunError(errLoadBoom).
        WithAppendMessagesError(errAppendBoom).
        WithLoadMessagesError(errLoadMessagesBoom)
})

如果还需要同时保留 happy path:

agentcoretesting.RunStorageFixtureSuite(t, func(t *testing.T) agentcoretesting.StorageFixture {
    t.Helper()

    return agentcoretesting.StorageFixture{
        Storage:          erroringStorage,
        RoundTripStorage: healthyStorageA,
        MessagesStorage:  healthyStorageB,
        ResumeStorage:    healthyStorageC,
    }.WithSaveRunError(errSaveBoom)
})

设计原则

  • 只依赖 public agentcore contract
  • 失败信息尽量指向 contract 级问题,而不是内部实现细节
  • 优先补最容易被独立实现的接口
  • 保持 additive 演进,不轻易把第一版 suite 做成重量级框架

什么时候用它

推荐在下面这些场景里使用:

  • 你在独立仓库里实现了自己的 Storage
  • 你在桥接已有工具系统到 ToolRegistry
  • 你在封装自己的 provider / streaming provider
  • 你准备把适配器对外发布,希望先验证自己没有偏离 agentcore contract

如果你想先理解 contract 本身,再来使用这些 suites,推荐先看:

如果你想直接看“外部适配器作者”的最小示例测试,继续看:

其中也包含了如何读取:

  • ContractChecklist()
  • OnboardingChecklist()

Documentation

Overview

Package agentcoretesting provides package-local conformance helpers for validating public agentcore implementations against the expected contracts.

The helpers in this package are intended for external implementers of agentcore.Provider, agentcore.Storage, and agentcore.ToolRegistry. They are deliberately small and only rely on the public agentcore contracts so they can be reused outside this repository.

Beyond runnable suites such as RunStorageSuite, RunToolRegistrySuite, and RunProviderSuite, the package also exposes lightweight navigation metadata:

  • ContractChecklist reports the current contract/evidence matrix
  • OnboardingChecklist reports recommended docs, suites, and examples for common external implementer roles

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RunProviderSuite

func RunProviderSuite(t *stdtesting.T, factory ProviderFactory)

RunProviderSuite runs the minimal public provider conformance checks.

func RunStorageFixtureSuite

func RunStorageFixtureSuite(t *stdtesting.T, factory StorageCaseFactory)

RunStorageFixtureSuite runs the storage conformance checks against one fixture.

func RunStorageSuite

func RunStorageSuite(t *stdtesting.T, factory StorageFactory)

RunStorageSuite runs the minimal public storage conformance checks.

func RunToolRegistrySuite

func RunToolRegistrySuite(t *stdtesting.T, factory ToolRegistryFactory)

RunToolRegistrySuite runs the minimal public tool registry conformance checks.

Types

type ChecklistItem

type ChecklistItem struct {
	Surface           string
	Status            ContractStatus
	Evidence          EvidenceLevel
	RecommendedSuites []string
	Notes             string
}

ChecklistItem summarizes one contract surface together with the recommended verification entry points currently available in agentcore/testing.

func ContractChecklist

func ContractChecklist() []ChecklistItem

ContractChecklist returns the current package-local contract matrix in a machine-readable form so docs and tooling can point at the same source of truth.

type ContractStatus

type ContractStatus string

ContractStatus describes the current stability posture of one public surface.

const (
	StatusHighConfidence       ContractStatus = "High confidence in v0"
	StatusStableDirection      ContractStatus = "Stable direction, careful evolution"
	StatusMustStayConservative ContractStatus = "Must stay conservative"
	StatusShouldKeepTightening ContractStatus = "Should keep tightening"
	StatusUsefulButEvolving    ContractStatus = "Useful but still evolving"
	StatusNeededForPosture     ContractStatus = "Needed for independent package posture"
	StatusHostReadyCandidate   ContractStatus = "Kernel-stable candidate"
	StatusImportantRelease     ContractStatus = "Important release surface"
)

type EvidenceLevel

type EvidenceLevel string

EvidenceLevel describes what kind of verification currently supports a surface.

const (
	EvidenceRuntime     EvidenceLevel = "Runtime"
	EvidenceRuntimeHost EvidenceLevel = "Runtime + Host"
	EvidenceDefaultPath EvidenceLevel = "Default Path"
)

type GenerateCase

type GenerateCase struct {
	Name       string
	Request    *agentcore.GenerateRequest
	WantResult *agentcore.GenerateResult
	WantErr    error
}

GenerateCase describes one expected Generate call against a provider.

func NewGenerateCase

func NewGenerateCase(name string, req *agentcore.GenerateRequest, want *agentcore.GenerateResult) GenerateCase

NewGenerateCase creates a named provider generate contract case.

type OnboardingItem

type OnboardingItem struct {
	Kind              string
	RecommendedSuites []string
	Docs              []string
	Examples          []string
	Notes             string
}

OnboardingItem describes the recommended starting points for one external implementation category.

func OnboardingChecklist

func OnboardingChecklist() []OnboardingItem

OnboardingChecklist returns a small package-local navigation guide for external implementers who want to validate a custom agentcore integration.

type ProviderFactory

type ProviderFactory func(t *stdtesting.T) ProviderFixture

ProviderFactory returns a fresh provider fixture for one test case.

type ProviderFixture

type ProviderFixture struct {
	Provider       agentcore.Provider
	StreamProvider agentcore.StreamingProvider
	GenerateCases  []GenerateCase
	Streaming      *StreamingCase
	StreamingCases []StreamingCase
}

ProviderFixture configures provider conformance checks.

func NewProviderFixture

func NewProviderFixture(provider agentcore.Provider, cases ...GenerateCase) ProviderFixture

NewProviderFixture creates a minimal provider conformance fixture.

func (ProviderFixture) WithGenerateError

func (f ProviderFixture) WithGenerateError(name string, req *agentcore.GenerateRequest, wantErr error) ProviderFixture

WithGenerateError appends one expected provider error case.

func (ProviderFixture) WithStreaming

func (f ProviderFixture) WithStreaming(tc StreamingCase) ProviderFixture

WithStreaming returns a copy of the fixture with one streaming case attached.

func (ProviderFixture) WithStreamingError

func (f ProviderFixture) WithStreamingError(tc StreamingCase) ProviderFixture

WithStreamingError appends one expected streaming provider error case.

func (ProviderFixture) WithStreamingHandlerError

func (f ProviderFixture) WithStreamingHandlerError(tc StreamingCase) ProviderFixture

WithStreamingHandlerError appends one expected stream-handler failure case.

func (ProviderFixture) WithStreamingProvider

func (f ProviderFixture) WithStreamingProvider(provider agentcore.StreamingProvider) ProviderFixture

WithStreamingProvider attaches a dedicated streaming-capable provider for stream cases.

type StorageCaseFactory

type StorageCaseFactory func(t *stdtesting.T) StorageFixture

StorageCaseFactory returns a fresh storage-conformance fixture for one test case.

type StorageFactory

type StorageFactory func(t *stdtesting.T) agentcore.Storage

StorageFactory creates an isolated storage implementation for one test case.

func NewStorageFactoryFromPath

func NewStorageFactoryFromPath(factory StoragePathFactory) StorageFactory

NewStorageFactoryFromPath adapts a path-based storage constructor into a StorageFactory backed by t.TempDir().

type StorageFixture

type StorageFixture struct {
	Storage             agentcore.Storage
	RoundTripStorage    agentcore.Storage
	MessagesStorage     agentcore.Storage
	ResumeStorage       agentcore.Storage
	SaveRunErr          error
	LoadRunErr          error
	AppendMessagesErr   error
	LoadMessagesErr     error
	SaveRunMissingIDErr error
}

StorageFixture configures storage conformance checks.

func NewStorageFixture

func NewStorageFixture(storage agentcore.Storage) StorageFixture

NewStorageFixture creates a minimal storage conformance fixture.

func (StorageFixture) WithAppendMessagesError

func (f StorageFixture) WithAppendMessagesError(err error) StorageFixture

WithAppendMessagesError appends one expected AppendMessages failure case.

func (StorageFixture) WithLoadMessagesError

func (f StorageFixture) WithLoadMessagesError(err error) StorageFixture

WithLoadMessagesError appends one expected LoadMessages failure case.

func (StorageFixture) WithLoadRunError

func (f StorageFixture) WithLoadRunError(err error) StorageFixture

WithLoadRunError appends one expected LoadRun failure case.

func (StorageFixture) WithSaveRunError

func (f StorageFixture) WithSaveRunError(err error) StorageFixture

WithSaveRunError appends one expected SaveRun failure case.

func (StorageFixture) WithSaveRunMissingIDError

func (f StorageFixture) WithSaveRunMissingIDError(err error) StorageFixture

WithSaveRunMissingIDError appends one expected SaveRun invalid-input case.

type StoragePathFactory

type StoragePathFactory func(t *stdtesting.T, root string) agentcore.Storage

StoragePathFactory creates a storage rooted at an isolated directory path.

type StreamingCase

type StreamingCase struct {
	Name       string
	Request    *agentcore.GenerateRequest
	WantEvents []agentcore.StreamEvent
	WantResult *agentcore.GenerateResult
	WantErr    error
	EmitErr    error
}

StreamingCase describes one optional streaming provider scenario.

type ToolExecuteCase

type ToolExecuteCase struct {
	Name       string
	Call       agentcore.ToolCall
	WantResult *agentcore.ToolResult
	WantErr    error
}

ToolExecuteCase describes one expected ExecuteTool call against a registry.

func NewToolExecuteCase

func NewToolExecuteCase(name string, call agentcore.ToolCall, want *agentcore.ToolResult) ToolExecuteCase

NewToolExecuteCase creates a named tool execution case.

type ToolRegistryFactory

type ToolRegistryFactory func(t *stdtesting.T) ToolRegistryFixture

ToolRegistryFactory returns a preconfigured registry fixture for one test case.

type ToolRegistryFixture

type ToolRegistryFixture struct {
	Registry        agentcore.ToolRegistry
	ExpectedSchema  agentcore.ToolSchema
	ExpectedSchemas []agentcore.ToolSchema
	ExecuteCall     agentcore.ToolCall
	WantResult      *agentcore.ToolResult
	WantErr         error
	ExecuteCases    []ToolExecuteCase
}

ToolRegistryFixture configures a tool registry conformance scenario.

func NewToolRegistryFixture

func NewToolRegistryFixture(
	registry agentcore.ToolRegistry,
	schema agentcore.ToolSchema,
	call agentcore.ToolCall,
	want *agentcore.ToolResult,
) ToolRegistryFixture

NewToolRegistryFixture creates a minimal tool-registry conformance fixture.

func (ToolRegistryFixture) WithExecuteCase

WithExecuteCase appends one additional execute success case.

func (ToolRegistryFixture) WithExecuteError

func (f ToolRegistryFixture) WithExecuteError(name string, call agentcore.ToolCall, wantErr error) ToolRegistryFixture

WithExecuteError appends one expected execute error case.

func (ToolRegistryFixture) WithExpectedSchema

func (f ToolRegistryFixture) WithExpectedSchema(schema agentcore.ToolSchema) ToolRegistryFixture

WithExpectedSchema appends one additional listed-schema expectation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL