Documentation
¶
Overview ¶
Package module executes detached module processes on behalf of the host.
Every rule the protocol states is enforced here rather than assumed, because a module is a separate program -- possibly buggy, possibly hostile, and in this architecture deliberately written against an independent implementation of the contract.
Three donor defects are fixed by construction, all observed in the upstream prototype this host replaces:
- stdout and stderr are captured SEPARATELY. The donor used CombinedOutput, so any module diagnostic corrupted the JSON stream it was parsing.
- output is BOUNDED. The donor buffered unbounded child output into memory and echoed malformed payloads back into model context.
- the whole PROCESS TREE is killed. The donor's context cancellation killed only the direct child, orphaning grandchildren such as ffmpeg.
Index ¶
- Constants
- func ApplyGrants(d *modproto.Descriptor, req *modproto.Request, policy GrantPolicy)
- func BinaryName(id string) string
- func BinaryPath(home, id string) (string, error)
- func BoundText(s string, max int) string
- func CheckID(id string) error
- func DescribeInstalled(ctx context.Context, home, id string) (*modproto.Descriptor, error)
- func FileDigest(path string) (string, error)
- func GrantBinaries(d *modproto.Descriptor, req *modproto.Request) (missing []string)
- func Install(ctx context.Context, home, binaryPath string) (string, error)
- func InstallDir(home, id string) (string, error)
- func InvalidateDescribeCache()
- func ModulesDir(home string) string
- func NewRequestID() (string, error)
- func Remove(home, id string) error
- func ResolveArtifact(req *modproto.Request, a modproto.Artifact) (string, error)
- func ResolveBinaries(declared []string) (resolved map[string]string, missing []string)
- func ScratchDir(home, id string) string
- func StateDir(home, id string) string
- func ValidID(id string) bool
- func ValidRootName(name string) bool
- func WithinRoot(root, path string) bool
- type GrantPolicy
- type Manifest
- type PartialInstall
- type Result
- type Runner
Constants ¶
const ( // DefaultDeadline is the Runner's own fallback for a caller that sets no // deadline at all. It is DELIBERATELY NOT the invocation default -- see // DefaultInvokeDeadlineMS, which is what every production caller passes and // what the contract documents. // // Two numbers, and the difference is not an oversight. This one predates // the measurement: it was chosen when a module invocation was a small // bounded thing, before a real Remotion render was observed at 1m55s // against a 120s budget. Every production path -- CLI, agent, cockpit -- // sets DeadlineMS explicitly, so this fallback is reached only by a caller // inside this repo that constructed a Request by hand, which today means // tests. // // Kept at 60s rather than raised to match, because the two answer // different questions. A hand-built Request with no deadline is a caller // who has not thought about bounds; giving it the render budget would make // a forgotten field cost three minutes before anyone notices. The // invocation default is measured against real work; this is a guard // against absent-mindedness. // // If a production path ever reaches this, that is the bug -- not the // value. TestEveryProductionInvocationSetsItsOwnDeadline pins that. DefaultDeadline = 60 * time.Second DefaultMaxOutputBytes = 1 << 20 // 1 MiB DefaultMaxStderrBytes = 64 << 10 // DefaultInvokeDeadlineMS is how long a capability may run when the caller // does not say. // // One constant, because three callers each had their own: the CLI allowed // 120s while the agent and the cockpit allowed 180s, so a render that // worked through the browser failed through the CLI with // "command_timeout: node was cancelled or timed out" -- which reads as a // broken module rather than a shorter leash. // // Video rendering is the case that sets the floor. It is a wall-clock // budget rather than a promise: a module that needs longer should return a // job handle, not hold the host. // // The floor is measured, not chosen: the same render takes ~17s on an idle // machine and was observed at 2m32s while a frontend build, a test run and // the gateway competed for the same cores. A budget set from the idle case // would kill real work whenever the machine is busy, which is exactly when // a person is most likely to be waiting on it. DefaultInvokeDeadlineMS = 180000 // MaxErrorText bounds module-supplied text -- an error message, a warning // -- before it reaches a tool result, the chat or a log. A module's error // message could otherwise be most of its 1 MiB envelope. MaxErrorText = 2 << 10 )
Defaults applied when a caller does not specify bounds. They are deliberately modest: a module needing more must say so, so that "unbounded" is never the accidental default.
const DisabledMarker = ".disabled"
DisabledMarker is the file that turns an installed module off. It lives here rather than with the code that reads it because an upgrade carries it over.
const EnvSubprocessAllow = "COMPA_SUBPROCESS_ALLOW"
EnvSubprocessAllow names the executables the host may authorize, overriding the default of "everything the module declared".
It exists because a module choosing among several declared binaries picks the first the host GRANTS, and the host cannot tell a signed-in CLI from a merely installed one -- an AI CLI's `--version` exits 0 on a machine where every real request fails to authenticate. Narrowing the grant is the one lever the host legitimately has: it is a statement about what this machine authorizes, not a guess about which CLI works.
Unset means unchanged: every declared name is authorized, as before. Setting it is the user saying which of their CLIs they are actually signed in to.
const ManifestName = ".compa-install.json"
ManifestName is the file Install writes into a module's directory to record what it installed. Discovery skips dotfiles, so it is never run.
Variables ¶
This section is empty.
Functions ¶
func ApplyGrants ¶
func ApplyGrants(d *modproto.Descriptor, req *modproto.Request, policy GrantPolicy)
ApplyGrants fills req.Grants from what the module DECLARED, intersected with what the host policy authorizes.
This is the per-invocation half of "installing a module grants nothing". Authority is conferred here, on this call, and only for names the module declared up front -- the same rule GrantBinaries applies to executables.
Two properties are load-bearing:
- A grant is not consent. Authorizing a paid provider says the host will let the module reach it; whether to spend money on this particular call is a separate human decision the module still has to ask for. Collapsing the two is how an unpriced call gets made because someone once ticked a box.
- An empty grant list means "nothing authorized", and that is different from a module being invoked outside any host mediation. A module that receives an explicit empty list must fail closed rather than assume it was called directly.
Subprocess IS set here, and the comment that used to say otherwise was wrong in a way that cost a journey. The reasoning -- that req.Binaries carries the resolved paths, so naming them again would be a second source of truth -- sounded right and left Grants.Subprocess permanently empty. A module reads the GRANT to decide whether it may shell out and the PATH to know what to run; supplying only the path handed it an executable with no permission to use it, and a real module refused with "no AI CLI was granted" one line after the host printed "binaries: 3/3 resolved".
The two are not duplicates: one is authority, the other is location.
func BinaryName ¶
BinaryName is the file name of an installed module's executable: the module ID, with .exe on Windows. Discovery runs nothing else.
func BinaryPath ¶
BinaryPath is where an installed module's executable lives.
func BoundText ¶
BoundText cuts s to at most max bytes, at a character boundary, and says how much was left out. Module-supplied text passes through it before it reaches a tool result, the chat or a log.
func DescribeInstalled ¶
DescribeInstalled returns the descriptor of one installed module.
func FileDigest ¶
FileDigest returns the protocol digest of a file's contents, re-reading it only when its size or modification time changed. Discovery checks every installed binary and declared document on each page load, and hashing a large binary each time is what made that slow.
func GrantBinaries ¶
func GrantBinaries(d *modproto.Descriptor, req *modproto.Request) (missing []string)
GrantBinaries fills req.Binaries from the module's declared subprocess names, intersected with what the host could actually resolve.
This is the per-invocation half of "installing a module grants nothing": authority to execute a binary is conferred here, on this call, and only for names the module declared up front.
func Install ¶
Install registers a module binary with this host.
Registration is a HOST action, deliberately. A module never writes into the host's state directory and never needs to know its layout: it hands the host a path and the host decides everything else. That keeps the confinement rule intact -- a module that could write into host state could grant itself authority -- and it means a module installer only has to find `compa-kernel` on PATH.
The order matters:
- describe and VALIDATE before copying, so a binary that does not speak the protocol is refused rather than installed and discovered broken later;
- derive the module ID from the DESCRIPTOR, never from the filename or the caller, so a binary cannot be installed under a name it does not claim;
- copy everything into a staging directory beside the install;
- re-describe the STAGED copy, so what was verified is what will run, and record its digest in an install manifest;
- swap the staged directory into place, keeping the previous install until the swap has succeeded.
Nothing an earlier install left is touched before step 5, so a failed upgrade leaves the working module exactly as it was. Installs of the same module are serialised, across processes too, so the CLI and the dashboard cannot interleave.
It is idempotent: installing over an existing module replaces it, which is how upgrade works.
func InstallDir ¶
InstallDir is the directory module id is installed in, after checking the ID and that the result is a direct child of the modules directory.
func InvalidateDescribeCache ¶
func InvalidateDescribeCache()
InvalidateDescribeCache forgets every remembered descriptor. Install, Remove and enabling or disabling a module call it, so the next discovery describes what is actually there.
func ModulesDir ¶
ModulesDir is where installed modules live under the host state root.
func NewRequestID ¶
NewRequestID generates the host-side correlation ID a module must echo verbatim. It is host-generated so correlation never depends on trusting a module to be unique.
func Remove ¶
Remove uninstalls a module and deletes its binary.
A module's own state under <home>/state/<id>/ is deliberately left alone: uninstalling should not destroy a user's work, and reinstalling should find it again. Removing state is a separate, explicit action.
The ID is checked against the one ID grammar and the target must be a direct child of the modules directory. The check this replaces accepted "..", and Remove("..") deleted the whole Compa home.
func ResolveArtifact ¶
ResolveArtifact turns a module-reported artifact into an absolute host path, refusing anything that escapes its declared root.
Both the root and the path are resolved through every kind of link, junctions included, so a link pointing outside the root cannot smuggle a path past a purely lexical check.
func ResolveBinaries ¶
ResolveBinaries turns a module's DECLARED subprocess names into absolute paths for one invocation.
Modules run with no inherited environment, so there is no PATH for them to search. The host resolves on their behalf, which is deliberately stronger than handing over a minimal PATH: a search can resolve to something that was never declared -- a shadowing entry, or a second binary with the same name in a supplied directory -- whereas an absolute path is an identity rather than a query. It also makes Permissions.Subprocess load-bearing: a binary the module did not declare is never supplied, so it cannot be run.
Only names in `declared` are ever looked up. A name absent from the descriptor is not resolvable by asking, which is what keeps the capability list and the execution authority in agreement.
A declared binary that cannot be resolved is OMITTED rather than mapped to an empty string, so "not supplied" and "supplied as nothing" stay distinguishable -- the same rule that keeps unknown cost from becoming zero. The unresolved names are returned so the host can tell the user which requirement is unmet instead of letting the module fail opaquely.
func ScratchDir ¶
ScratchDir is the working directory a module runs in.
A module used to run in its own install directory, beside the host's state, so a relative path it resolved landed among installed files it could then change. The name starts with a dot, which no declared root name can, so it never collides with a root under the same state directory.
func StateDir ¶
StateDir is a module's own state directory. Removing a module leaves it in place, so reinstalling finds the user's work again.
func ValidID ¶
ValidID reports whether id is a usable module ID.
The check that preceded this compared id with filepath.Base(id), which accepts "." and "..": Remove("..") deleted the whole Compa home. Every place that turns an ID into a path -- install, remove, enable, disable, the API and the CLI -- uses this one test.
A trailing dot is refused because Windows drops it, so "fake." would name the directory of the module "fake".
func ValidRootName ¶
ValidRootName reports whether a root name a module declares can be used. A writable root becomes <home>/state/<module>/<name>, so it is held to the same grammar as a module ID.
func WithinRoot ¶
WithinRoot reports whether path is inside root, treating both as cleaned absolute-ish paths.
Exported because three other places were doing this with strings.HasPrefix, which is wrong in a way that matters: "/srv/modules/acme-evil" has the prefix "/srv/modules/acme" and is a different directory. A module declaring "../acme-evil/overlay.md" escaped its own tree and the check accepted it.
One implementation, so a fix here cannot leave a copy behind.
Types ¶
type GrantPolicy ¶
type GrantPolicy struct {
// Network, Credentials and PaidProviders are the names the host will
// authorize when a module declares them.
Network []string
Credentials []string
PaidProviders []string
// Subprocess is the set of executable NAMES the host will authorize.
//
// It is separate from Request.Binaries, which carries the resolved absolute
// PATHS. A module reads the grant to decide whether it MAY shell out, and
// the path to know what to run -- so filling only Binaries left a module
// with a path it was not authorized to use. One module's evidence.extract
// refused with "no AI CLI was granted for this invocation" one line after
// the host printed "binaries: 3/3 resolved"; both were true, about
// different fields.
Subprocess []string
// AllowDeclared authorizes every name the module declared, without the
// host listing them. It is the trusted-desktop shortcut GrantAll uses.
//
// It used to be Publish that meant this, so a policy could not allow
// publishing without also allowing every network, credential and paid
// provider a module named.
AllowDeclared bool
// Publish is separate because it is a single capability rather than a set,
// and because publishing is the one effect a user almost always wants to
// approve per act rather than per session.
Publish bool
}
GrantPolicy is what the HOST is willing to authorize for an invocation.
It is the host's half of the grant: a module declares what it may need, the policy says what the host will actually allow, and the grant is the intersection. Neither side alone decides -- a module cannot widen its own authority by declaring more, and the host cannot confer authority for something the module never declared and therefore cannot be expecting.
The zero value authorizes nothing, which is the correct default for a host that has not made a decision yet.
func GrantAll ¶
func GrantAll() GrantPolicy
GrantAll authorizes everything a module declared, for a call the operator approved: by a rule of the approval policy (tools.approval) that allows it, or by answering an ask.
This is for the trusted single-user desktop case this host was built for: the person running it installed the module deliberately, and approved this call against the effects it declares. Consent and cost gates still apply -- a grant says "you may reach this provider", never "you may spend this money".
Credentials are granted as NAMES. The host holds no secret for a module and passes none: a module that needs a key reads it from its own configuration.
func GrantDeclared ¶
func GrantDeclared() GrantPolicy
GrantDeclared is GrantAll without publishing: every name the module declared, for a call the operator did not approve. Publishing is approved per act, so only an approved call gets GrantAll.
func GrantNothing ¶
func GrantNothing() GrantPolicy
GrantNothing authorizes nothing while still being an explicit decision.
func GrantsFor ¶
func GrantsFor(approved bool) GrantPolicy
GrantsFor is the authority one call carries: GrantAll when the operator approved it (approval.Approved), GrantDeclared otherwise.
type Manifest ¶
type Manifest struct {
ID string `json:"id"`
Dir string `json:"dir"`
SHA256 string `json:"sha256"`
}
Manifest records one install: which module, in which directory, and the digest of the binary that was verified. A binary that no longer matches it was changed after the host checked it, and is not run until reinstalled.
func ReadManifest ¶
ReadManifest reads the manifest in a module directory. A directory without one returns nil and no error: a binary the owner put there by hand is theirs to run.
type PartialInstall ¶
PartialInstall reports a module that installed and runs, but whose declared knowledge could not be copied.
It is an error type rather than a silent warning because the consequence is invisible at runtime: the agent behaves as though the module documented nothing, with no signal that anything is missing.
func (*PartialInstall) Error ¶
func (e *PartialInstall) Error() string
type Result ¶
type Result struct {
Envelope *modproto.Envelope
// Stderr is advisory diagnostics only: bounded, surfaced to humans, and
// never parsed for control flow or protocol data.
Stderr string
// Truncated reports that output hit its bound. The envelope is then
// unusable, since a truncated JSON document cannot be trusted even if it
// happens to parse.
Truncated bool
Duration time.Duration
ExitCode int
// Warnings are host-side advisory findings about this invocation, distinct
// from the module's own Envelope.Warnings.
Warnings []string
}
Result is one completed module invocation, including the evidence needed to explain what happened when it went wrong.
type Runner ¶
type Runner struct {
// Binary is the absolute path to the module executable.
Binary string
// ModuleID is the expected module ID; a descriptor claiming a different
// one is rejected, so a binary cannot impersonate another module.
ModuleID string
// WorkDir is the working directory the module runs in, created when
// needed. Empty means a fresh temporary directory per run. Installed
// modules get ScratchDir: never the install directory, never the host's.
WorkDir string
// Digest is the binary's digest recorded at install time. When set, a
// binary that no longer matches it is refused before it runs.
Digest string
MaxOutputBytes int
MaxStderrBytes int
}
Runner executes one installed module binary.
func NewRunner ¶
NewRunner returns a runner for the module installed as id: its binary, its identity, its scratch directory and the digest recorded when it was installed, if any.
func (*Runner) Describe ¶
Describe runs `<binary> module describe --json` and returns a validated descriptor.
func (*Runner) DescribeCached ¶
DescribeCached returns the module's descriptor, running the binary only when it changed since it was last described.
Discovery described every module, one after another with a 10s budget each, on every Modules or Tools page load, every invocation from the cockpit and every artifact request. A describe is a process start; an unchanged binary answers the same way, so the answer is kept.
Failures are not kept: a module that timed out once, while a virus scanner read it, should be asked again.
func (*Runner) Invoke ¶
func (r *Runner) Invoke(ctx context.Context, d *modproto.Descriptor, req *modproto.Request) (*Result, error)
Invoke runs one capability and returns a fully validated envelope.
The descriptor is required rather than optional: without it the host cannot check reported execution against declared effects, which is the check that keeps an unpriced call from rendering as free and a writing capability from routing around approval.
Source Files