Documentation
¶
Overview ¶
Package modelcorpus is the model evaluation corpus: for each case, the stimulus a model is given and the key that stimulus is scored against, held apart so that no code which produces a stimulus can read a key.
The boundary, and what it does and does not close ¶
A Case carries both halves and publishes them through two accessors. Stimuli returns what a run may see: the identifier, the prompt template, the recipe that builds the world, the environment the case needs and the surfaces it may run on. Keys returns what only a scorer may see: the ordered steps and, per argument, where its right value comes from.
The key is an unexported field, so a builder, a repair message, a simulated result or a mock fixture cannot be derived from it: the runner imports this package and calls Stimuli, and a test in this package fails when any package other than the scorer, the two generators that publish about the key and the fake provider so much as names Keys.
That closes the machine half of the leak. It does nothing about the author, who writes both halves in one file, and calling it the structural fix for that would be a claim the mechanism does not support. The authorial half is held by two other things: the prompt audit in this package's tests, which renders every stimulus on every surface and refuses one that spells its own answer, and review, under the rule that changing a prompt to make a case pass is the defect rather than the fix.
What is data here and what is read from the catalog ¶
A step names one canonical catalog action, never a per-surface tool name. How that action is spelled on each surface, which arguments its schema admits, whether it destroys and which tier it needs are all read from the catalog at run time, by the harness that calls it and by the tests that gate this corpus. Written by hand, and audited, are only the prompt, the key and the three surface contracts in contract.go.
Why it is untagged and imports nothing ¶
Two programs share it: the runner, which is built with the e2e tag because it starts the real server binary against a real GitLab, and the scoring command, which is an ordinary binary. A shared type can only be shared from a package both may import, so this one is untagged and stdlib only. Its sibling github.com/jmrplens/gitlab-mcp-server/v3/internal/testutil/modelrecord is the same arrangement for what a run writes down.
Being untagged means an ordinary build compiles it, so it is held to the same bar as production code, and cmd/server's TestDependencies_TestSupport_NeverReachesTheServerBinary names it so that it can never reach the binary a user downloads.
Identifiers ¶
A case identifier is allocated once and retired rather than reused, which is why the numbers are not contiguous: MT-199 and above were renumbered out of a collision, and every identifier a retirement takes out of the corpus stays spent. access_test.go is what holds that.
What a retirement is ¶
Retired answers what became of an identifier the corpus no longer holds, so a person reading a report from before the move is not left with a number and nothing else. The categories are the survivor rule read backwards: a case survives when every step of its key is a tools/call this server registers, its result comes from the server or from GitLab, and a recipe can build its world. retired.go carries the two classes that could be decided by reading the old corpus; the third needs a builder to have been tried, so its category is declared there and used by the step that tries.
Index ¶
- Constants
- func Contract(surface Surface) (string, bool)
- func Digest() string
- func Domains(id string) ([]string, bool)
- func FactPlaceholder(key string) string
- func Facts(recipe Recipe) ([]string, bool)
- func IDs() []string
- func InterpolatedFacts(prompt string) ([]string, error)
- func Keys() map[string]Key
- func Render(prompt string, facts map[string]string) (string, error)
- func StepCount(id string) (int, bool)
- type Action
- type Arg
- type Case
- type Key
- type Needs
- type Recipe
- type Ref
- type Restrict
- type Retirement
- type Step
- type Stimulus
- type Surface
- type Tier
- type Truth
Constants ¶
const ( // FactProjectPath is the project a case works in, rendered as its full // path. A recipe accepts its numeric ID as a second spelling, so an // argument that takes either is satisfied by either. FactProjectPath = "project_path" // FactProjectID is that same project rendered as its numeric ID, and is // a fact of its own for the reason [FactGroupID] is: two actions type // project_id as an integer (project_alias.create and // storage_move.schedule_project), and an integer argument refuses a // path whatever spellings a scorer would accept. FactProjectID = "project_id" // FactDefaultBranch is that project's default branch. FactDefaultBranch = "default_branch" // FactRemoteURL is that project's git remote URL, which the project // discovery tool resolves. FactRemoteURL = "remote_url" // FactGroupPath is the group a case works in, rendered as its path and // accepting its numeric ID. FactGroupPath = "group_path" // FactGroupID is that same group rendered as its numeric ID. // // It is a fact of its own rather than a second spelling of // [FactGroupPath], because what a prompt hands the model is the value the // fact renders as, not the set of spellings the scorer would accept: an // argument typed as an integer refuses a path, so a case binding one has // to give the model the number. FactGroupID = "group_id" FactArtifactPath = "artifact_path" FactBranchName = "branch_name" FactCIVariableKey = "ci_variable_key" FactDeploymentID = "deployment_id" FactDiscussionID = "discussion_id" FactEnvironmentID = "environment_id" FactEnvironmentName = "environment_name" FactInstanceCIVariableKey = "instance_ci_variable_key" FactIssueIID = "issue_iid" FactJobID = "job_id" FactMergeRequestIID = "merge_request_iid" FactMergeRequestSource = "mr_source_branch" FactPackageDir = "package_dir" FactPackageFilesDisplay = "package_files_display" FactPackageName = "package_name" FactPackageTag = "package_tag" FactPackageVersion = "package_version" // FactPipelineID is a pipeline as the instance numbers it, and // FactPipelineIID is the same pipeline as the project numbers it. They // are two facts rather than two spellings of one, because an action takes // one or the other and nothing in a number says which it is: a world that // offered both spellings under one key rendered the project's number to a // model whose action takes the instance's, and GitLab then answered about // whichever pipeline on the instance carries that number. FactPipelineID = "pipeline_id" FactPipelineIID = "pipeline_iid" FactReleaseName = "release_name" FactReleaseTagName = "release_tag_name" FactRunnerID = "runner_id" FactSnippetID = "snippet_id" )
The fact keys the recipes promise. They are spelled as the evaluator this replaces spelled them, so a reader comparing the two corpora is comparing cases rather than vocabularies.
const ( FactAttestationIID = "attestation_iid" FactAuditEventID = "audit_event_id" FactAwardID = "award_id" FactBadgeID = "badge_id" FactBroadcastMessageID = "broadcast_message_id" FactCommitDiscussionID = "commit_discussion_id" FactCommitNoteID = "commit_note_id" FactCommitSHA = "commit_sha" FactCustomEmojiID = "custom_emoji_id" FactDatabaseMigration = "database_migration_version" FactDependencyExportID = "dependency_export_id" FactDeployKeyID = "deploy_key_id" FactDeployTokenID = "deploy_token_id" FactEpicDiscussionID = "epic_discussion_id" FactEpicIID = "epic_iid" FactEpicNoteID = "epic_note_id" FactExternalStatusCheckID = "external_status_check_id" FactFeatureFlagName = "feature_flag_name" FactFilePath = "file_path" FactGeoSiteID = "geo_site_id" FactGeoSiteName = "geo_site_name" FactGeoSiteURL = "geo_site_url" FactGroupAccessTokenID = "group_access_token_id" FactHookID = "hook_id" FactJobTokenTargetProjectID = "job_token_target_project_id" FactLDAPProvider = "ldap_provider" FactMemberRoleID = "member_role_id" FactMilestoneIID = "milestone_iid" FactMirrorID = "mirror_id" FactModelFileName = "model_file_name" FactModelFilePath = "model_file_path" FactModelVersionID = "model_version_id" FactPackageID = "package_id" FactPipelineScheduleID = "pipeline_schedule_id" FactPipelineTriggerID = "pipeline_trigger_id" FactProjectAccessTokenID = "project_access_token_id" FactProjectAliasName = "project_alias_name" FactProtectedBranchName = "protected_branch_name" FactProtectedEnvironment = "protected_environment_name" FactSAMLGroupName = "saml_group_name" FactScimUID = "scim_uid" FactServiceAccountID = "service_account_id" FactServiceAccountTokenID = "service_account_token_id" FactServiceAccountUsername = "service_account_username" FactSSHCertificateID = "ssh_certificate_id" FactStorageMoveID = "storage_move_id" FactTagName = "tag_name" FactTerraformStateName = "terraform_state_name" FactUserID = "user_id" FactVulnerabilityID = "vulnerability_id" FactWikiSlug = "wiki_slug" )
The fact keys the destructive and licensed worlds add.
Every one of them names a thing a builder has to have made, which is what separates this block from the one above: a read case can be satisfied by a collection that happens to be empty, and a case that deletes something cannot.
const ( // RetiredBridgeTool is a case whose step called a tool the evaluator // invented to ask the MCP **client** for something: a resource, a // prompt, a completion, a capability listing. No client exposes those to // a model under those names, each client that exposes them at all does // it its own way, and whether this server answers resources/read // correctly is a deterministic question the end-to-end suite already // asks on every push. RetiredBridgeTool = "bridge-tool" // RetiredSimulatedResult is a case whose step had its result fabricated // by the harness rather than returned by the server, so the model // reacted to something nothing under test produced. RetiredSimulatedResult = "simulated-result" // RetiredNoRecipe is a case whose world no builder can raise on a Docker // instance. It is declared here and used where the evidence is: the // fixture step finds out by trying, and a retirement written before // anybody tried would be a decision with no grounds. // // **Nothing carries it, and that is the answer rather than an omission.** // The step that built the worlds tried every recipe this corpus names and // raised one for each, so no case was left without a world. A handful of // the licensed worlds do name an object a Docker instance cannot hold: an // attestation is published by a CI job that attests, a SCIM identity by an // identity provider, an enterprise user by a verified domain, a storage // move by a second Gitaly storage. None of those retires its case. The // world is raised around the object and the identifier it would have had // is reserved, so the case still dispatches the action it is about and // GitLab answers the not-found, which is the "dispatched, GitLab refused" // class the scoring keeps apart from a model failure, and which the // end-to-end suite's own scenarios already assert for the same actions. // The category stays declared because that judgement can change: a // fixture stack that grows an identity provider, or a corpus that grows a // case whose world needs one, would use it. RetiredNoRecipe = "no-recipe" )
The three categories a retirement can have.
Variables ¶
This section is empty.
Functions ¶
func Contract ¶
Contract returns the text one surface is introduced with, and whether the surface is one of the three.
func Digest ¶
func Digest() string
Digest returns a fingerprint of the whole corpus, key included.
A run writes it on its record, and the report is scored from the corpus at HEAD rather than from anything the run carried: a key that moved between the two is a record being scored against an answer it was never put to, and the digest is what lets the publisher refuse that instead of publishing it.
It therefore has to cover the key, which is why it is here and not in the runner. A run may not read a key, and a digest of the stimuli alone would be unmoved by exactly the edit it exists to catch.
Sixteen hex characters, the repository's own length for a comparison fingerprint a person reads off two rows of a table.
func Domains ¶
Domains returns the catalog domains one case's key touches, sorted, and whether the corpus has that case.
It is the widest of the three narrow doors, so what it is for has to be stated rather than assumed. The individual surface publishes one tool per action and its whole list is 682,878 tokens at Ultimate, over the context window of at least one provider outright, so a model there is shown a slice: every tool of these domains, plus distractors from others, filled to a budget seeded by the case ID. A slice chosen without them would be a list the case's own tools are missing from, and an attempt against that measures nothing but the shuffle.
What it hands over and what it does not. It narrows the field from a thousand tools to a few dozen, which is a hint no dynamic or meta attempt gets, and that is exactly why an individual row is published as a comparison class of its own rather than beside them: choice within a slice is a different question from choice across a catalog. It says nothing about which tool of a domain a step wants, in what order, or with what arguments, so a case that touches issues is shown every issue tool with the right one among them.
A step naming a standalone tool contributes no domain, because such a tool is registered outside the catalog under one name on every surface. The slice keeps every tool it cannot place in a domain, so the standalone ones are shown whether or not a case names one, and this accessor stays a statement about the catalog rather than about a particular key's tools.
func FactPlaceholder ¶
FactPlaceholder is how a prompt interpolates one fact. It is spelled once here so that the corpus gate looks for what Render resolves.
func Facts ¶
Facts returns the fact keys a recipe promises, and whether the recipe is one this corpus knows at all.
func IDs ¶
func IDs() []string
IDs returns every case identifier, sorted, which is what a report iterates and what a run filters against.
func InterpolatedFacts ¶
InterpolatedFacts returns the fact keys a prompt interpolates, in the order they first appear. It reads the parsed template rather than the text, so a spelling Render would resolve and a spelling a search would find are the same set.
func Keys ¶
Keys returns the answer for every case, by case ID.
Four packages outside this one may use it, and access_test.go fails on a fifth: the scorer, which is the one thing a key is for; the two generators, each of which publishes about a key and produces no stimulus; and the fake provider, which replays a key to prove the pipe and whose every row the publisher refuses. The list that test reads names this package too, since the corpus's own gate is written against the key, which is why it holds five entries and this sentence names four. It is one of three such lists, one per accessor answering about a key: StepCount and Digest have their own, and are allowed where this is not.
func Render ¶
Render renders one case's prompt over the facts its recipe produced.
It is here rather than in the runner because the corpus gate and the run must render the same text: a template the gate accepted and the runner then rendered differently would be audited in one spelling and sent in another, which is the drift the two-text case shape had and this shape exists to end.
A template naming a fact the recipe did not produce is an error rather than an empty string, because a stimulus with a hole in it is a paid request that measures nothing.
func StepCount ¶
StepCount returns how many steps one case's key declares, and whether the corpus has that case.
It is one of the two things a run is told about a key, Domains being the other, and it is told for one reason: the turn cap an attempt is bounded by is a multiple of the steps plus a margin, so a case of one step cannot spend a conversation's worth of tokens going nowhere. A cap is an ending and never a message, so nothing a model is shown can be derived from it; what a run learns is how long it may go on, never what it should say.
It is deliberately not a field of Stimulus. That type is the enumeration of what a run may see, and growing it is how the boundary erodes; a function asked for by name is a use a reader can find, and access_test.go holds this name to a list of callers of its own, as it holds Keys to one. The runner is on that list; the file that composes a prompt is refused by a second lock of its own.
Types ¶
type Action ¶
type Action string
Action is a canonical catalog action ID as data, such as "issue.list".
It is deliberately not harness.ActionID. The harness is built behind the e2e tag, and the end-to-end coverage gate credits every harness.ActionID constant it finds to the e2e ratchet and holds it to a runtime placement; a model case is neither a scenario nor placed, so borrowing that type would make every case look like an end-to-end scenario to a gate that has no way to tell otherwise. The runner converts at the call site, and types_test.go checks every Action here against the catalog at Ultimate.
type Case ¶
type Case struct {
// ID is the case identifier, allocated once and never reused.
ID string
// Prompt is the stimulus, a template over {{ .Facts.<key> }}.
Prompt string
// Recipe names the world this case runs in.
Recipe Recipe
// Needs is what the instance must offer for the case to run at all.
Needs Needs
// Surfaces restricts the case to some of the three, with the reason.
Surfaces Restrict
// contains filtered or unexported fields
}
Case is one stimulus and its key, held apart by the two accessors.
Prompt is a text/template rendered over the recipe's facts and over nothing else: a value a case needs is a fact the recipe produces, never a constant written into the text, because a text carrying the value one run of one fixture happened to have is a case that silently stops being about anything the next time the fixture changes.
type Key ¶
type Key struct {
Steps []Step
}
Key is the answer: the steps in the order they must dispatch, each with the arguments whose values are compared.
type Needs ¶
type Needs struct {
// Tier is the lowest licensing tier that can run this case. Empty means
// Free; [Needs.MinimumTier] is the reader.
Tier Tier
// Runner requires a CI runner able to pick a job up.
Runner bool
// Admin requires a token whose user administers the instance.
Admin bool
// FixtureService requires the fixture HTTP service the Docker stack
// runs, which answers what GitLab cannot be made to produce
// deterministically.
FixtureService bool
}
Needs is what a case asks of the instance, never of the catalog: what the server serves is decided by the surface under test, not by a need.
func (Needs) MinimumTier ¶
MinimumTier returns the tier this case needs, reading the empty tier as Free so that the great majority of cases, which need no license, can leave the field out.
type Recipe ¶
type Recipe string
Recipe names the world a case runs in. The builders live in the end-to-end fixture library, behind the e2e tag; what lives here is the name a case refers to and the fact keys the builder promises to produce.
The facts are the join between the two halves of a case. A prompt may interpolate a promised key and nothing else, and an argument may bind its truth to a promised key and nothing else, so a case cannot ask about a value its world does not build, and a value the world builds is written into the stimulus by the run rather than into the corpus by an author.
const ( // RecipeWorld is the shared read-only project and its group. RecipeWorld Recipe = "world" // RecipeProject is a project of this attempt's own, with a README and a // CI configuration, so a case may create a pipeline in it. RecipeProject Recipe = "project" // RecipeGroup is a group of this attempt's own, for the cases that // create something directly under a group. RecipeGroup Recipe = "group" // RecipeBranch is a project with a branch beside the default one. RecipeBranch Recipe = "branch" // RecipeIssue is a project with an issue in it. RecipeIssue Recipe = "issue" // RecipeMergeRequest is a project with an open merge request. RecipeMergeRequest Recipe = "merge_request" // RecipeMergeRequestDiscussion is that merge request with an unresolved // discussion on it. RecipeMergeRequestDiscussion Recipe = "merge_request_discussion" // RecipeMergeRequestSource is a project with a branch ahead of the // default one, ready for a merge request that does not exist yet. RecipeMergeRequestSource Recipe = "merge_request_source" // RecipePipelineJob is a project with a pipeline and its jobs. A case // that needs the job to have finished says so with Needs.Runner; a case // that needs it unfinished, such as the cancel one, does not. RecipePipelineJob Recipe = "pipeline_job" // RecipeFailedJob is a project with a pipeline whose job failed and left // an artifact behind. RecipeFailedJob Recipe = "failed_job_artifact" // RecipeEnvironment is a project with an environment and a deployment // recorded against it. RecipeEnvironment Recipe = "environment_deployment" // RecipeRelease is a project with a release. RecipeRelease Recipe = "release" // RecipeSnippet is a personal snippet. RecipeSnippet Recipe = "snippet" // RecipeMember is a project with another user added to it. RecipeMember Recipe = "member" // RecipeRunner is a project with a runner registered for it and a job // that runner has run. RecipeRunner Recipe = "runner" // RecipeCIVariable is a project with a CI variable already in it, for // the cases that change one rather than create one. RecipeCIVariable Recipe = "ci_variable" // RecipeInstanceVariable is a CI variable name no other attempt uses. // Instance scope is shared by every attempt on the instance, so unlike a // project-scoped name this one cannot be a literal. RecipeInstanceVariable Recipe = "instance_ci_variable" // RecipePackageFiles is a project and local files to publish to it. RecipePackageFiles Recipe = "package_files" )
The worlds the Free corpus runs in.
RecipeWorld is the one shared, read-only world: a project with a README and a default branch, inside a group, which any number of attempts may read at once. Every other recipe builds state of the attempt's own, which is what lets a mutating case name a literal ("Evaluation Sprint", "EVAL_TOKEN") without two attempts colliding over it.
const ( // RecipeMergeableMergeRequest is a merge request with nothing standing // between it and a merge: the approval requirements a project may carry // are cleared, so a refusal is about what the case did rather than about // the project it did it in. It runs no pipeline, so a merge asked for as // soon as one passes is refused by GitLab for want of one. RecipeMergeableMergeRequest Recipe = "mergeable_merge_request" // RecipeFile is a project with a file already committed on a branch // beside the default one. RecipeFile Recipe = "repository_file" // RecipeMilestone is a project with a milestone in it. RecipeMilestone Recipe = "milestone" // RecipeProjectAccessToken is a project with an access token of its own. RecipeProjectAccessToken Recipe = "project_access_token" // RecipePackage is a project with a generic package published to it. RecipePackage Recipe = "package" // RecipeBroadcastMessage is an instance broadcast message. RecipeBroadcastMessage Recipe = "broadcast_message" // RecipeProjectHook is a project with a webhook on it. RecipeProjectHook Recipe = "project_hook" // RecipeProjectBadge is a project with a badge on it. RecipeProjectBadge Recipe = "project_badge" // RecipeDraftNote is a merge request with an unpublished draft note. RecipeDraftNote Recipe = "draft_note" // RecipeJobTokenScope is a project whose CI job token allowlist already // names a second project. RecipeJobTokenScope Recipe = "job_token_scope" // RecipeInstanceVariableSeeded is an instance CI variable that already // exists, which is the opposite of [RecipeInstanceVariable]: that one // reserves a name nothing has used, this one has used it. RecipeInstanceVariableSeeded Recipe = "instance_ci_variable_seeded" // RecipeTag is a project with a tag on it. RecipeTag Recipe = "tag" // RecipePipelineTrigger is a project with a pipeline trigger token. RecipePipelineTrigger Recipe = "pipeline_trigger" // RecipePipelineSchedule is a project with a pipeline schedule. RecipePipelineSchedule Recipe = "pipeline_schedule" // RecipeUser is a user of this attempt's own, so blocking one or turning // its two-factor authentication off touches nobody else. It is what the // old evaluator's skip reason for MT-105 was about: that case ran // against the shared evaluator user and had to be skipped. RecipeUser Recipe = "user" // RecipeMemberCandidate is a project and a user who is not a member of // it yet. RecipeMemberCandidate Recipe = "project_member_candidate" // RecipeFeatureFlag is a project with a feature flag. RecipeFeatureFlag Recipe = "feature_flag" // RecipeCustomEmoji is a group with a custom emoji. RecipeCustomEmoji Recipe = "custom_emoji" // RecipeWikiPage is a project with a wiki page. RecipeWikiPage Recipe = "wiki_page" // RecipeMergeRequestAward is a merge request with an award emoji on it. RecipeMergeRequestAward Recipe = "merge_request_award" // RecipeIssueAward is an issue with an award emoji on it. RecipeIssueAward Recipe = "issue_award" // RecipeDeployKey is a project with a deploy key. RecipeDeployKey Recipe = "deploy_key" // RecipeDeployToken is a project with a deploy token. RecipeDeployToken Recipe = "deploy_token" // RecipeCommitDiscussion is a project with a discussion on a commit and // a note in that discussion. RecipeCommitDiscussion Recipe = "commit_discussion" // RecipeTerraformState is a project with a locked Terraform state. RecipeTerraformState Recipe = "terraform_state" // RecipeDatabaseMigration is a database migration version the instance // has not marked applied. RecipeDatabaseMigration Recipe = "database_migration" // RecipeProjectMirror is a project with a remote mirror configured. RecipeProjectMirror Recipe = "project_mirror" )
The worlds the destructive partition runs in.
Almost every one of them is "a project of this attempt's own with one thing already in it", because a case that deletes something needs that something to exist and needs it to belong to nobody else. They are named after what they seed rather than after the case that destroys it, so a second case asking about the same object reuses the world instead of declaring a near-copy of it.
const ( // RecipeEpic is a group with an epic, a note on it and a discussion. RecipeEpic Recipe = "epic" // RecipeEpicIssue is a group with an epic and a project whose issue is // assigned to it. RecipeEpicIssue Recipe = "epic_issue" // RecipePushRule is a project with a push rule already on it. RecipePushRule Recipe = "push_rule" // RecipeProjectServiceAccount is a project with a service account and a // token for it. RecipeProjectServiceAccount Recipe = "project_service_account" // RecipeGroupServiceAccount is a group with a service account and a // token for it. RecipeGroupServiceAccount Recipe = "group_service_account" // RecipeServiceAccountName is a service account username nobody on the // instance signs in as, for the case that creates an instance service // account. A username is unique instance-wide, so it cannot be a // literal. RecipeServiceAccountName Recipe = "service_account_name" // RecipeGroupProtectedBranch is a group with a protected branch rule. RecipeGroupProtectedBranch Recipe = "group_protected_branch" // RecipeGroupProtectedEnvironment is a group with a protected // environment. RecipeGroupProtectedEnvironment Recipe = "group_protected_environment" // RecipeGeoSite is a registered Geo site. RecipeGeoSite Recipe = "geo_site" // RecipeGeoSiteName is a Geo site name and URL no site on the instance // holds, which is the opposite of [RecipeGeoSite]: that one registers a // site, this one reserves what a case may register one as. GitLab holds // both a site's name and its URL unique instance-wide, so neither can // be a literal: one case runs three times against one instance, and // every attempt after the first would be refused. RecipeGeoSiteName Recipe = "geo_site_name" // RecipeScimIdentity is a group with a SCIM identity. RecipeScimIdentity Recipe = "scim_identity" // RecipeLDAPLink is a group with an LDAP link. RecipeLDAPLink Recipe = "ldap_link" // RecipeSAMLLink is a group with a SAML group link. RecipeSAMLLink Recipe = "saml_link" // RecipeGroupSSHCertificate is a group with an SSH certificate. RecipeGroupSSHCertificate Recipe = "group_ssh_certificate" // RecipeGroupWikiPage is a group with a wiki page. RecipeGroupWikiPage Recipe = "group_wiki_page" // RecipeMemberRole is an instance custom member role. RecipeMemberRole Recipe = "member_role" // RecipeProjectAlias is a project with an alias. RecipeProjectAlias Recipe = "project_alias" // RecipeProjectAliasName is a project and an alias name nothing on the // instance has claimed, for the case that creates one. An alias names a // project instance-wide, so it cannot be a literal for the reason // [RecipeInstanceVariable]'s key cannot. RecipeProjectAliasName Recipe = "project_alias_name" // RecipeExternalStatusCheck is a project with an external status check // and a merge request the check applies to. RecipeExternalStatusCheck Recipe = "external_status_check" // RecipeMergeTrainEntry is a project with a merge request on its merge // train. RecipeMergeTrainEntry Recipe = "merge_train_entry" // RecipeStorageMove is a group with a repository storage move recorded // against it. RecipeStorageMove Recipe = "storage_move" // RecipeDependencyExport is a pipeline with a dependency list export. RecipeDependencyExport Recipe = "dependency_export" // RecipeInstanceAuditEvent is an instance audit event. RecipeInstanceAuditEvent Recipe = "instance_audit_event" // RecipeAttestation is a project with a build attestation. RecipeAttestation Recipe = "attestation" // RecipeVulnerability is a project with a vulnerability record. RecipeVulnerability Recipe = "vulnerability" // RecipeEnterpriseUser is a group with an enterprise user in it. RecipeEnterpriseUser Recipe = "enterprise_user" // RecipeGroupAccessToken is a group with an access token of its own. RecipeGroupAccessToken Recipe = "group_access_token" // RecipeModelVersion is a project with a model registry version and a // file in it. RecipeModelVersion Recipe = "model_version" // RecipeDeploymentApproval is a project with a protected environment and // a deployment waiting for approval on it. RecipeDeploymentApproval Recipe = "deployment_approval" )
The worlds the licensed partitions run in.
A licensed case says what it needs of the instance through Needs.Tier; a recipe here says what it needs of the world, and the two are separate questions. A group holding an epic is a world any tier could describe; only GitLab's license decides whether the epic can exist.
type Ref ¶
type Ref struct {
// Step is the 1-based position of the earlier step in the key.
Step int
// Field is the path into that step's structured result, segments
// separated by dots.
Field string
}
Ref names a field of an earlier step's result.
func (Ref) IsZero ¶
IsZero reports whether this reference names nothing, which is what every truth that is not a Truth.Produced carries.
type Restrict ¶
Restrict is a case that cannot run on every surface, with the reason a report publishes beside it.
It exists because a case that runs on one surface is a declaration, and a runner that special-cases an identifier is a declaration hidden in code. types_test.go refuses Only without a Reason, which is the rule the harness's own OnSurfaces already states, and the runner passes both to it.
type Retirement ¶
type Retirement struct {
// ID is the case identifier, which stays allocated for ever.
ID string
// Category is one of the three above.
Category string
// Reason says why, in the words a reader of an old report needs.
Reason string
}
Retirement is one identifier the corpus no longer holds.
func Retired ¶
func Retired() []Retirement
Retired returns every retired identifier with its category and reason, in declaration order. It is what a reader of an old report looks a case up in, and it answers about the corpus rather than about any run, so it is exported beside Stimuli and needs none of Keys's sanction.
type Step ¶
type Step struct {
// Action is the canonical catalog action this step must dispatch.
Action Action
// Standalone is the registered name of a tool outside the catalog.
Standalone string
// Args are the arguments whose values are compared, with where each
// one's right value comes from. An argument the case does not pin is
// simply absent: comparing a value the model was never given is the
// defect this shape exists to end.
Args []Arg
// Optional marks a step a completed attempt need not have reached.
Optional bool
// Produces names the fields of this step's result that a later step's
// argument may bind to with [Truth.Produced].
Produces []string
}
Step is one tools/call the key expects, named canonically.
Exactly one of Action and Standalone is set. A catalog action is projected to each surface's own spelling by the harness; a standalone tool (gitlab_discover_project, the gitlab_interactive_* flows) is registered outside the catalog under one name on every surface, so it is named here as that name.
type Stimulus ¶
Stimulus is everything about a case a run may see: what the model is asked, which world it is asked in, what the instance must offer and which surfaces the case may run on.
It is a type of its own rather than a Case with the key blanked out, because a blanked field is a field a later edit can fill in by accident.
type Surface ¶
type Surface string
Surface names a tool surface as data, for the same reason Action is not the harness's own type: this package is untagged and cannot import the harness. The runner converts to harness.Surface at the call site.
const ( // SurfaceDynamic is gitlab_find_action plus gitlab_execute_action, the // server's default surface. SurfaceDynamic Surface = "dynamic" // SurfaceMeta is one dispatcher tool per catalog group, the operation // traveling in an action argument. SurfaceMeta Surface = "meta" // SurfaceIndividual is one registered tool per action. SurfaceIndividual Surface = "individual" )
The three surfaces a case can run on.
type Tier ¶
type Tier string
Tier is the licensing tier an instance must have for a case to run, as data rather than as internal/edition.Tier, which this package may not import.
type Truth ¶
type Truth struct {
// Fact names a recipe fact. Every spelling the recipe accepts for it
// matches, which is what lets an argument that takes a project's
// numeric ID be satisfied by its path.
Fact string
// Literal is a value the prompt states verbatim, compared exactly.
Literal string
// Produced binds to a field of an earlier step's result, read from the
// record the run wrote.
Produced Ref
// Authored marks a value the model writes, such as a title or a note
// body derived from the request. It is reported as authored and never
// compared.
Authored bool
}
Truth is where an argument's right value comes from. It is a closed sum of four, so a scorer can be exhaustive and the corpus gate can reject a case that binds a name no recipe produces.
Exactly one is set. A value expressed in the prompt as a verb rather than as a value ("pause the runner", "resolve the discussion") is deliberately not one of them: it is checked by the recipe's own verification against GitLab after the run, because a Literal the prompt never spells is a value the model was never given, and the corpus gate refuses one.