eval_mcp_surfaces

eval_mcp_surfaces

cmd/eval_mcp_surfaces evaluates whether a model can use the model-facing MCP catalog correctly. It can run schema-only validation, model-backed schema evaluation, and Docker-backed live MCP execution.

Inputs

Supported model providers are anthropic, google, openai, and qwen. The evaluator reads ANTHROPIC_API_KEY, GOOGLE_API_KEY, OPENAI_API_KEY, and QWEN_API_KEY only for the providers selected by --model, --models, or EVAL_MODELS. When no model source is configured, the default is anthropic:claude-haiku-4-5-20251001. Qwen uses QWEN_API_KEY directly, defaults to the international DashScope OpenAI-compatible endpoint, disables thinking for tool calls, and supports QWEN_BASE_URL or QWEN_CHAT_COMPLETIONS_URL for regional endpoints.

By default, model-backed runs expose the same full MCP capability surface as a normal client for the selected tool surface. The evaluator adds bridge tools for resources/list, resources/read, prompts/list, prompts/get, completion/complete, and initialize capability metadata, then records which bridge tools and targets each model uses.

Implementation Layout

The command root is intentionally thin:

Keep model-facing evaluation behavior in internal/evaluator. Extract a helper into a sibling internal package only when it is independent of cases, tool schemas, provider prompts, and report semantics.

MCP-First Failure Triage

Full Docker runs use real model calls against the selected MCP surface and, when --execute-tools is enabled, execute validated tool calls against GitLab. Use traces to decide where fixes belong:

1. Discovery or route-selection failures point first to MCP action descriptions, aliases, dynamic ranking, or gitlab://tools metadata.
2. Correct route with wrong parameters points first to canonical ActionSpec schemas, JSON schema tags, parameter guidance, and output next_steps.
3. Wrong follow-up after a successful MCP call points first to tool output, Markdown hints, and response formatting.
4. Only change evaluator prompts, compact workflow plans, fixtures, or assertions when the MCP metadata is already precise and the failure is harness-specific.

--check-report-clean is a gate, not the whole audit. Also inspect repaired first-pass diagnostics and trace JSON when tuning model-facing MCP behavior.

Common Commands

Recommended Makefile shortcuts for full Docker-backed evaluator runs:

# CE presets (read + mutating-safe + destructive-safe + capability-discovery)
make eval-surfaces-docker SURFACE=dynamic
make eval-surfaces-docker SURFACE=meta

# Enterprise-only presets on GitLab EE runtime
make eval-surfaces-docker-enterprise SURFACE=dynamic
make eval-surfaces-docker-enterprise SURFACE=meta

# CE + Enterprise presets together on GitLab EE runtime
make eval-surfaces-docker-enterprise-all SURFACE=dynamic
make eval-surfaces-docker-enterprise-all SURFACE=meta

Notes:

• eval-surfaces-docker runs CE-only Docker presets.
• eval-surfaces-docker-enterprise runs Enterprise-only Docker presets (docker-enterprise-*).
• eval-surfaces-docker-enterprise-all runs both CE and Enterprise Docker presets in one run (EVAL_SURFACE_CASE_SET=all) on GitLab EE.
• Add PRESET=... to any command to run a single preset instead of the full set.

Dry-run the current catalog without model calls:

GITLAB_ENTERPRISE=false timeout 180s go run ./cmd/eval_mcp_surfaces \
  --dry-run \
  --repeat=1 \
  --out /tmp/eval-dry.md

Dry-run a saved snapshot partition and emit uncovered high-risk routes:

timeout 180s go run ./cmd/eval_mcp_surfaces \
  --tools-file dist/evaluation/mcp-surfaces/snapshots/current/tools.json \
  --dry-run \
  --partition base-read \
  --coverage-report dist/evaluation/mcp-surfaces/snapshots/current/coverage-base-read.md \
  --out dist/evaluation/mcp-surfaces/snapshots/current/schema-base-read.md

Run a targeted model-backed schema sample:

timeout 900s go run ./cmd/eval_mcp_surfaces \
  --model anthropic:claude-sonnet-4-6 \
  --task MS-001,MF-001 \
  --repeat=1 \
  --pause=250ms \
  --retries=8 \
  --retry-wait=65s \
  --out dist/evaluation/mcp-surfaces/schema-sample.md

Run the same small local analysis batch across all models configured in .env:

timeout 1800s go run ./cmd/eval_mcp_surfaces \
  --models "$EVAL_MODELS" \
  --task MT-001,MS-001,MF-001 \
  --repeat=1 \
  --pause=250ms \
  --retries=4 \
  --retry-wait=30s \
  --out dist/evaluation/mcp-surfaces/current-multi-model-smoke.md

When neither --model nor --models is provided, .env EVAL_MODELS is used if present; otherwise the evaluator uses the source-defined default model.

Run the full fixture corpus against Haiku using the current low-token dynamic surface:

timeout 7200s go run ./cmd/eval_mcp_surfaces \
  --tool-surface=dynamic \
  --model anthropic:claude-haiku-4-5-20251001 \
  --repeat=1 \
  --pause=250ms \
  --retries=4 \
  --retry-wait=65s \
  --out dist/evaluation/mcp-surfaces/dynamic-haiku-all.md

Run the deterministic dynamic search corpus after ranker changes:

go test ./internal/tools/dynamic/ -run TestDynamicSearchCorpus -count=1

Prepare Docker fixtures:

GITLAB_ENTERPRISE=false timeout 600s go run ./cmd/eval_mcp_surfaces \
  --backend=gitlab \
  --gitlab-env-file test/e2e/.env.docker \
  --prepare-fixtures \
  --fixtures-only \
  --fixtures dist/evaluation/mcp-surfaces/e2e-fixtures.json

Execute validated model calls against Docker GitLab CE:

GITLAB_ENTERPRISE=false timeout 900s go run ./cmd/eval_mcp_surfaces \
  --preset docker-read \
  --fixtures dist/evaluation/mcp-surfaces/e2e-fixtures.json \
  --task MS-014,MS-017,MS-020 \
  --out dist/evaluation/mcp-surfaces/live-smoke.md

Exercise MCP resource, prompt, completion, and capability bridge access against Docker GitLab CE:

GITLAB_ENTERPRISE=false timeout 900s go run ./cmd/eval_mcp_surfaces \
  --preset docker-capability-discovery \
  --tool-surface=dynamic \
  --fixtures dist/evaluation/mcp-surfaces/e2e-fixtures.json \
  --out dist/evaluation/mcp-surfaces/dynamic-capability-discovery.md

Run the Enterprise schema-only batch:

GITLAB_ENTERPRISE=true timeout 180s go run ./cmd/eval_mcp_surfaces \
  --preset schema-enterprise \
  --out dist/evaluation/mcp-surfaces/schema-enterprise.md

Compare token and evaluation reports:

timeout 180s go run ./cmd/eval_mcp_surfaces \
  --compare dist/evaluation/mcp-surfaces/snapshots/release-2.0.0/tokens.md \
  --compare dist/evaluation/mcp-surfaces/snapshots/release-2.0.0/schema-base-read.md \
  --compare dist/evaluation/mcp-surfaces/snapshots/current/schema-base-read.md \
  --out dist/evaluation/mcp-surfaces/comparison/version-summary.md

Check dynamic call-efficiency gates from trace JSONL without parsing Markdown reports:

timeout 180s go run ./cmd/eval_mcp_surfaces \
  --check-efficiency dist/evaluation/mcp-surfaces/dynamic-full-live.traces/traces.jsonl \
  --out dist/evaluation/mcp-surfaces/dynamic-efficiency-check.md

Compare dynamic and meta-tool traces on identical task/model rows only:

timeout 180s go run ./cmd/eval_mcp_surfaces \
  --compare-traces dist/evaluation/mcp-surfaces/dynamic-full-live.traces/traces.jsonl \
  --compare-traces dist/evaluation/mcp-surfaces/meta-default-opaque-full-plus-reactivated-2026-05-13.traces/traces.jsonl \
  --out dist/evaluation/mcp-surfaces/meta-vs-dynamic-trace-comparison.md

Publish reviewed Docker reports into managed documentation blocks:

timeout 180s go run ./cmd/eval_mcp_surfaces \
  --publish-docs \
  --publish-from dist/evaluation/mcp-surfaces/docker-read-all-models.md \
  --publish-from dist/evaluation/mcp-surfaces/docker-mutating-safe-all-models.md \
  --publish-from dist/evaluation/mcp-surfaces/docker-destructive-safe-all-models.md \
  --publish-label "2026-05-05 Docker economy models"

Check the committed docs against the same reviewed reports without writing:

timeout 180s go run ./cmd/eval_mcp_surfaces \
  --check-docs \
  --publish-from dist/evaluation/mcp-surfaces/docker-read-all-models.md \
  --publish-from dist/evaluation/mcp-surfaces/docker-mutating-safe-all-models.md \
  --publish-from dist/evaluation/mcp-surfaces/docker-destructive-safe-all-models.md \
  --publish-label "2026-05-05 Docker economy models"

Run validated calls through an older or separately built stdio MCP server:

E2E_MODE=docker timeout 900s go run ./cmd/eval_mcp_surfaces \
  --tools-file dist/evaluation/mcp-surfaces/snapshots/release-2.0.0/tools.json \
  --mcp-command dist/evaluation/mcp-surfaces/snapshots/release-2.0.0/gitlab-mcp-server-release-2.0.0 \
  --mcp-env-file test/e2e/.env.docker \
  --execute-tools \
  --use-fixtures \
  --fixtures dist/evaluation/mcp-surfaces/e2e-fixtures.json \
  --task MS-028 \
  --skip-unavailable \
  --out dist/evaluation/mcp-surfaces/snapshots/release-2.0.0/live-ms-028.md

The Docker presets apply safe defaults for --backend=gitlab, --gitlab-env-file test/e2e/.env.docker, --execute-tools, --use-fixtures, --skip-unavailable, the matching partition, and the expected edition. docker-capability-discovery selects the capability-fallback partition and keeps mutating/destructive GitLab operations out of the CE batch. Override any preset flag explicitly when debugging a narrower case.

Long model-backed runs create the selected --out Markdown file at startup with a Status: running placeholder. The placeholder is replaced by the final metrics report when the run completes, or by a failure report if the evaluator stops before final metrics are available. For long local runs, always set an explicit --out path and redirect stdout/stderr to a sibling .log file so the terminal does not become the report artifact.

Use scripts/eval-compare-version.sh to orchestrate the standard snapshot, token audit, schema dry-run, optional model-backed run, and optional comparison report for one target label.

Safety

--execute-tools can mutate GitLab resources. It requires --backend=gitlab or --mcp-command plus E2E_MODE=docker unless --allow-live-mutations is explicitly set.

Keep reports, traces, snapshots, and fixture state under dist/evaluation/mcp-surfaces/; that directory is ignored by git.

Model-backed trace JSON records the normalized prompt flow plus provider HTTP request/response bodies and MCP CallTool request/response payloads. Provider authentication headers are not serialized; raw trace artifacts remain local and should not be published or committed.

--publish-docs is intentionally separate from normal runs. It consumes reviewed Markdown reports selected with --publish-from. Full GitLab-backed MCP reports without an explicit preset also read their local Trace artifacts JSONL. That lets the publisher split the table by preset and special partitions. The publisher routes CE/base and Enterprise/Premium rows to separate dynamic and meta-tool managed blocks. It refuses to publish Docker metrics from GitLab-backed reports that did not use MCP tool execution. Partial Docker preset reports must use a --publish-label containing targeted so they are not mistaken for full preset results.

Docker live reports include a failure-triage section. It separates MCP implementation bugs, GitLab CE limitations, model route-selection misses, model parameter-shape misses, fixture setup failures, transient GitLab 5xx responses, timeout/resource exhaustion, destructive safety failures, and not-found results. Dynamic-surface reports also separate ranker_miss diagnostics from model discovery and execution failures when ranker-specific notes are available.

Related Documentation

• AI Model Evaluation
• AI Model Evaluation Developer Guide
• AI Model Evaluation Results