plugins/

Generators

A generator turns the catalog into something else. The built-in generators produce Markdown and LLM indexes, standalone Mermaid diagrams, and a Backstage Software Catalog bundle; each lives in its own directory beside the plugin.

The contract is one JSON message in and one JSON message out:

→ { "portolanVersion": "0.1.0", "catalog": { ... }, "options": { ... } }
← { "files": [{ "name": "shop/oms/README.md", "contents": "..." }] }

There is a second question, asked with "kind": "describe", and the answer is the plugin itself rather than its work:

→ { "portolanVersion": "0.1.0", "kind": "describe" }
← { "files": [],
    "describe": { "name": "extract-go", "summary": "...",
                  "phases": ["extract"],
                  "options": { "type": "object", "additionalProperties": false,
                               "properties": { "context": { ... } } } } }

The options a plugin takes are facts the source does not carry, so only the plugin knows what it can be told. npm run schema asks all of them and composes schema/portolan.schema.json, which an editor reads while the manifest is being written and gen checks before it runs anything. additionalProperties: false is what makes that worth having: encoding/json drops a field it does not recognise, so before this a misspelled option was no option at all and nothing said so.

A descriptor may also carry needs: what the host must put in the request beyond the tree, because a sandboxed module cannot reach it. The one need so far is history (portolan.0007) - when each file under the root was first committed and last changed, read by the host from one git log and handed over as input.history, keyed by the path the plugin would open. extract-adr asks for it; a plugin that does not ask is handed nothing.

A generator names files; it never writes them. scripts/gen.mjs writes what comes back, refuses a name that points outside the output directory, and deletes pages that stopped being generated. That is what lets a generator run as a wasm module with no directory preopened at all — the sandbox is not a restriction worked around, it is the reason the protocol has this shape.

Three obligations, and they are the whole of it:

1. Valid output. Every file a generator names is committed and read.
2. Determinism. The same catalog produces byte-identical output, so reviewing generated documentation is reviewing a diff. Sort anything that comes out of a map; never read a clock.
3. One decisive response. A malformed request, incompatible protocol, unsafe filename or invalid response fails the run. Non-fatal extraction notes go to stderr; there is no advisory response property a caller may accidentally ignore. A note that opens with warning: is kept beside the step in .portolan/build-report.json and listed on the Settings page.

The repository enforces those obligations with schema/field coverage tests, byte-for-byte permutation tests, generated-link and anchor checks, Mermaid parser checks, and Backstage relationship validation. A new catalog field must either be rendered or be explicitly acknowledged by the relevant exporter.

Adding one

1. Write it. In Go, a new package here whose run hands its options type to plugin.Serve, which reads the request, answers a describe and calls the work; catalog.Catalog from github.com/shortlink-org/portolan/catalog is the mirror of the schema, and internal/goscan is the tree as syntax - the files parsed once, the import path of each package, the string constants followed to their literals - which River and Watermill share and the next Go extractor should not copy. In Python, the same three things live in pyplugin/ - protocol.py, source.py for the tree as syntax, and catalog.py for the fragment shapes - and extract-django and extract-celery are what using them looks like. In any other language, anything that speaks the protocol above.
2. Describe it. An options.schema.json beside the source, embedded with go:embed and returned in the descriptor. schematest.Check in a test keeps it from drifting from the options struct: a field renamed on one side and not the other fails, and so does an option with no description.
3. Build it. A built-in Go plugin is a library package with Serve(io.Reader, io.Writer) error; add it to the map in plugins/cmd/portolan-go/main.go, and plugins:build in package.json puts it in plugins/portolan-go.wasm with the rest (a test keeps the map and portolan.json in step). A plugin of your own is its own module: GOOS=wasip1 GOARCH=wasm go build.
4. Declare it in portolan.json, under plugins (how to run it) and generate (what to run it on), then run npm run schema so the manifest schema learns its options.

{
  "plugins": [
    { "name": "markdown", "wasm": { "url": "file://plugins/portolan-go.wasm" } }
  ],
  "generate": [
    { "plugin": "markdown", "out": "docs", "options": { "title": "Example estate" } }
  ]
}

Lifecycles

An aggregate whose root has a status gets a lifecycle on the catalog: the states, the first being where a new root starts, and one move per edge - the method that makes it and the event it hands back. All three extractors read it off a table the code keeps, never off the branches of the methods, because the table is the claim and the methods are held to it: an edge in the table no method makes, a move into a state the table lacks, and a status changed outside the one way through the table are each reported.

In Go the table is a go-sdk/fsm rule set, var Rules = fsm.TransitionRuleSet{ StateLive: {EventRevoke: StateRevoked}, …}, with the states and events as string constants; the method whose body calls TriggerEvent is the mover, and every exported method that hands it a constant makes the edges that constant names. The event type a method returns is what its last move publishes - a method lapsing a lock and then locking again hands AccountLocked back for the lock. In TypeScript it is export const TRANSITIONS = { open: ["checked-out"], … } and the method that assigns this.status; see extract-ts/README.md. In Rust it is pub const TRANSITIONS: &[(&str, &[&str])] = &[("placed", &["confirmed"]), …] and the method that assigns self.status, handed a string or a variant of the status enum; see extract-rust/README.md. In Python it is that mapping on the Django model, beside the TextChoices that names the states, and the method assigning self.status - or django-fsm's @transition(field=status, source=…, target=…), which is the same table written one edge at a time; see extract-django/README.md. In Java it is TRANSITIONS beside the status enum and the method assigning this.status; Java is also the one language here with a vocabulary for the model, so the rest of extract-java reads what jMolecules declares rather than what the layout implies. Terminal states are derived on the page - nothing leads out - and never written down. A move the clock makes, a session expiring, a lock running out, is not a move: nothing runs when it happens, so it is not in the table.

Enums

An aggregate's fields take some of their values from closed sets - a reason, a status, a code - and a consumer of its events switches on them. Those sets are the aggregate's enums: one entry per set, id <aggregate>.<slug>, the values in declaration order, each with its doc and a deprecated mark when the source carries one. A status enum is read here as well as by the lifecycle: the lifecycle keeps the moves, the enum keeps the doc on each value, and a page may draw both.

What counts as one is a convention per language, and each extractor's README says which. In Go, which has no enum, it is a named type over a basic one - type Reason string - and a const block whose constants are of that type, looked for in the aggregate's package, under vo/, and under event/; a value's name is the constant's literal, because that is what the wire carries, and the constant's own name only for an iota. A Deprecated: paragraph in the doc marks the value. In Rust it is a pub enum whose every variant is a bare name, the literal an as_str answers standing in for the variant. In Java it is a top-level enum in the aggregate's package. In proto, the enums the messages reach through their fields sit on the interface as enums, with the numbers the wire uses.

A repository without a domain model

extract-project is the baseline extractor for any repository. It reads only README and build/deployment manifests, emits a neutral group and component, and leaves aggregates empty. This is intentional: a package called domain is not evidence that the project models aggregates. OpenAPI, AsyncAPI, GraphQL, proto and SQL extractors merge their facts into the same component.

The language-specific domain extractors are optional enrichments. The local setup wizard offers one only when it finds the structure that extractor requires; for Go this means an aggregate package in either internal/domain/<aggregate> or internal/<aggregate>/domain, containing the root struct named after that aggregate. The extractor follows the same choice for application use cases, transport adapters, policies, integration-event DTOs, and assembly bindings, so horizontal layers and feature slices can coexist while a service is being migrated.

The SQL extractor follows the same migration path. With no repositories or projectors option it discovers both internal/infrastructure/repository/<aggregate>/migrations and internal/<aggregate>/infrastructure/repository/migrations (and the matching projector forms), merging every package into one store. An explicit root keeps the original collection layout for TypeScript, Rust, Java, or custom trees and may also point directly at one feature repository.

extract-river is another independent enrichment for Go repositories. It joins a job argument's Kind() to Client.Insert/InsertTx, the selected queue, Worker[Args].Work, and river.AddWorker. The result is a work-queue channel plus a two-hop enqueue/dispatch flow, with payload fields and source lines. It does not need aggregates and does not treat a job as a domain event.

extract-redis finds runtime construction of go-redis, rueidis and redigo clients in non-test Go source. That source evidence adds a service-owned Redis store to the catalog even when the repository has no SQL migrations or domain model. It follows literals, constants, concatenation, fmt.Sprintf, helper functions and conditional suffixes into common Redis operations, producing key patterns with their read/write/delete use, TTL, value type and source. The patterns remain Redis keyspaces rather than being presented as SQL tables.

extract-watermill reads Watermill Router.AddHandler and AddNoPublisherHandler registrations. It resolves literal and constant topics, plus defaults on env-config structs, follows direct Publisher.Publish calls and one-hop publishing helpers, and traces JSON marshal/unmarshal values back to their Go payload structs. It carries enclosing if/else conditions and early returns through the handler's control flow: proven alternatives become one catalog alt; publications whose relationship cannot be proven remain separate possible routes. The channels merge normally with AsyncAPI declarations by address. Generic NewEventHandler[T] and NewCommandHandler[T] registrations on Watermill CQRS processors are also extracted; fixed topic generators and the standard event/command-name generator form are resolved from source.

extract-go-nats reads nats.go and JetStream calls into the subjects a service listens on and publishes to. A call is known by the type it is made on - *nats.Conn, nats.JetStreamContext, jetstream.JetStream, jetstream.Stream - and not by its name, because a service's own bus port has a Subscribe too. The subject is followed to a literal, a constant, a config default, or a parameter; a parameter is followed up to two hops through the callers, including calls through an interface the adapter satisfies, which is how a port Subscribe(ctx, topic, name, handler) reads as the assembly's Subscribe(ctx, cart.Topic, cart.BasketCheckedOut{}.Name(), …). When the port takes exactly one other string beside the subject, that string is the message's name; a direct call names no message, and its direction is in the channel's doc. A subject read off a database row is a warning at the call, not a channel. Consumer configs give the filter subject and the durable name; streams, wildcard subjects and work-queue retention are not read yet.

extract-python-kafka is the framework-independent Kafka enrichment for Python. It recognizes confluent-kafka, kafka-python and aiokafka by their imported client types, follows literal topics through constants, environment defaults, settings and local factories, and emits generic message streams with producer and consumer flows. Only client-side configuration proven in source is retained; authentication values are omitted, while broker-side partitions, replication and retention remain explicitly unknown. See extract-python-kafka/README.md for the supported call shapes.

extract-wsdl reads WSDL 1.1 contracts as structured SOAP APIs. It follows local WSDL imports and XSD imports/includes without network access, keeps distinct services, ports and SOAP 1.1/1.2 bindings, and records operation actions, request/response messages, faults, headers and reachable XSD shapes. It can describe a contract implemented by the component or a vendored copy for an external system (mode: external). Remote imports are reported as missing evidence rather than fetched during generation.

extract-http-clients is the outbound counterpart and does not require a domain layout. It reads net/http request construction, calls through an oapi-codegen client, and SOAP Call/CallContext sites. A generated client is joined to the OpenAPI document beside it, so the call uses the document's operation id and lands on an external with the contract the document declares. SOAP actions are joined to WSDL bindings when the action matches. Generated and hand-written wrapper signatures are learned from their call into the SOAP transport or their SOAPAction/SOAP 1.2 content-type header, so the action, request and response positions are taken from code rather than assumed. A raw request whose peer or contract cannot be proved is still useful evidence: it is emitted as unresolved, with its method, path and source line, rather than being assigned to a guessed system. Conditions guarding a call and the opposite path after an early return are carried into the flow note. Calls through local wrappers retain their argument values, including closure arguments, so a path and HTTP method declared by a business operation survive the trip into the transport. URL-shaped configuration is followed through a constructor and client field to the request, and the resulting evidence chain is included in the flow instead of presenting a receiver field as an endpoint. Inbound composition also follows fixed factories, string-keyed constructor maps, capability type assertions, and interface fields wired by composite literals, direct assignments, or setters. A standalone flow says whether no source caller exists or callers exist but no inbound/asynchronous root was proved, so the UI exposes the missing evidence instead of implying a complete business path. When a provider branch still ends before its transport, the extractor loads the module with go/packages, builds SSA, and uses x/tools VTA to resolve calls through interface parameters, function values, return values, and interface-typed struct fields. Typed edges are followed only after a concrete factory branch is selected: applying context-insensitive VTA to a shared dispatcher would otherwise attach every request implementation to every endpoint. Factory conditions and HTTP/SOAP meaning continue to come from the source extractor. Module loading is read-only and bounded; unavailable private dependencies, type errors, or a timeout produce a warning and retain the syntax-only result rather than failing generation. Routes without a provider factory are also joined to their outbound calls, including handlers invoked from closures and methods on locally constructed values. A Swagger @Router annotation is medium-confidence root evidence for handler factories behind custom registries. Direct calls reached through a main → Run/Start/Bootstrap assembly path become high-confidence startup flows. AddFunc, AfterFunc, and Schedule registrations create scheduled roots when their handler reaches an outbound call. Extracted HTTP, callback, startup, scheduled, River-job and Watermill-event flows publish that trigger provenance; unmatched transport fragments are explicitly unproven with low confidence.

After fragments merge, Portolan composes them into root-oriented, cross-protocol flows. The seam is machine evidence rather than a display-name match: an exact source function reached by another extractor, a River job's queue plus Kind(), or a Watermill transport plus topic address. Composition is recursive, so one request can continue through provider HTTP/SOAP calls, enqueue a River job, enter its worker, and make further outbound calls. Ambiguous handoffs remain separate. Each composed flow records the source fragment slugs it includes, and the UI exposes that provenance as a cross-protocol badge. Trigger-bearing job and event flows stay available as standalone entry views; transport-only fragments consumed by a proven root are removed from the top-level flow list.

Flows written by hand

Some flows will always be written by people: the design doc for something not built yet, the reconstruction after an incident, the path no test pins. The catalog's JSON is the wrong place to write one - a tree of nodes, a unique id per step, every lane declared twice - so extract-flows reads a text form that reads like the sequence diagram it becomes: one file per flow, one line per hop, frames closed by end. The estate here declares no such step: every flow it shows is read out of code that runs, and one written by hand over files nobody wrote would be the single page in the catalog nothing holds to account. A repository that has a real one points an extract-flows step at the directory the files live in.

# Order accepted
owner: shop
source: services/oms/test/integration/order_accepted_test.go

The narrow slice one integration test pins end to end.

## Participants
- oms-db: store in shop "oms-db (postgres)"
- psp-gateway: external "psp-gateway (external)"

## Steps
shop.oms -> oms-db: insertOrderAndOutboxRow [verified] @internal/oms/adapter/postgres/order_repo.go:141 #a1
  > The order row and the outbox row commit in one transaction.
shop.oms -> bus: event shop.oms.order.OrderPlaced [verified]
shop.oms -> shop.pricing: rpc shop.v1.Pricing/GetQuote as "GetQuote (250 ms)"

alt score below 40 #alt-risk
  bus -> payments.ledger: event shop.oms.order.OrderPlaced
else score at or above 40
  shop.oms -> bus: event shop.oms.order.OrderCancelled
  stop
else
end

par OrderPlaced fan-out
  bus -> payments.ledger: event shop.oms.order.OrderPlaced
and
  bus -> delivery.core: event shop.oms.order.OrderPlaced
end

loop outbox relay, every 200 ms until the batch is empty
  shop.oms -> oms-db: SELECT ... FOR UPDATE SKIP LOCKED
end

The head is the name, owner: (the context the flow belongs to), an optional source: (where it was read from; the file itself when left out) and an optional slug: (the file's name when left out), then the summary. A hop is from -> to: [call|rpc|event] label-or-ref - call when no kind is written - followed in any order by as "label", [status], @where and #id. An event names its ref, shop.oms.order.OrderPlaced; an rpc names its call id, shop.v1.Pricing/GetQuote, or, for a call no interface declares - a webhook arriving on a route - just a label. The label of an event or rpc is the last segment of its ref unless as says otherwise; the status is declared unless written; @ is a file and line, or wherever the hop was seen. A note is the > lines under the hop. Ids are numbered unless given, and giving them is what keeps deep links to a step stable across edits.

alt <when> … else <when> … end is a choice; else alone is "otherwise", a branch with nothing in it is allowed, and stop as the last line of a branch says the flow ends there rather than rejoining. par [title] … and … end runs its branches side by side; loop <until> … end repeats. Frames nest.

Services are known by their context.service id, bus and client by name; any other lane is declared under Participants, in the order the lanes should be drawn, as - <id>: <kind> [in <context>] ["label"]. Lines starting with // are comments. A mistake fails the run with its file and line, the way a compiler would: a flow silently left out is the kind of missing nobody notices. A ref that resolves to nothing is caught later, by the validator, because only the merged catalog can say.

Decisions written by hand

Every decision worth keeping is already written down, in a file beside the code it constrains, in the MADR shape a person reads. Typing it a second time into the catalog's JSON - an id, a slug, a number, a scope and a body with every newline escaped - makes the JSON the source and the markdown a copy, and the copy is the one that goes stale: the three records that lived in data/catalog.json before extract-adr existed all named files that were never in the repository at all.

So the markdown is the source and the fragment is the output. extract-adr reads docs/adr/*.md under a service's root - a README.md among them is the directory's index, not a decision, and is skipped - and answers with one fragment holding the records.

# auth.0003 — Session expiry publishes no event

- **Status:** accepted
- **Date:** 2026-08-22
- **Scope:** auth.auth
- **Superseded by:** auth.0007
- **Supersedes:** auth.0001, auth.0002
- **Relates:** auth.auth.session.SessionEnded, shop.cart, checkout
- **Note:** how a revocation is kept out of the cache was decided again in
  auth.0010; the drop described below turned out not to be enough.

## Context and Problem Statement
…
## Decision Outcome
…

The title carries the record's id, an em dash, and the title. The id is a prefix and four padded digits, and the prefix is whatever the record is about - a service (auth, cart, oms), a context (payments) or the organisation (org). The file is named NNNN-kebab-slug.md with the same number, because the slug the catalog keeps is built from both: the id with its dots opened out, then the file's kebab, as in auth-0003-expiry-publishes-nothing. A file renamed away from its record would silently change the address of a decision somebody linked to, so the two are held against each other.

Status and Date are required, and so is Scope unless the step's scope option names it for the whole tree; the rest are written when there is something to write, and a bullet the format does not have fails the record rather than being dropped. A bullet may wrap onto the next line, indented under itself - the break is the author's line width and closes up into a space. Scope says what the record is about by how many segments it has: none, or org, for the organisation, one for a context, two for <context>.<service>. Relates names events, services and flows in one list and they are told apart by their shape - a flow by its slug, which has no dots, a service by <context>.<service>, an event by the aggregate and Name after that - because an author should not have to remember which of three lists a name belongs in. Note is prose no other field holds, most often that part of a record was decided again somewhere else without the whole of it being superseded; it sits in the page's header, above the frozen body.

Everything from the first ## onward is the record. It goes into the catalog exactly as written and comes back out onto the page the same way, headings and all: an ADR is frozen history, and nothing on its page is redrawn from the model as it stands now. Prose above that first ## is a mistake - a paragraph that drifted up there would be read by a person and dropped by the extractor.

The adr-tools shape is read too: number and title on the title line, Date: above the record, status as the first section with its Superseded by / Supersedes link. It has no prefix or scope, so both come from the step's scope option.

src/catalog.ts fails the whole app on load if a record breaks any of its rules, so the extractor checks first, where the file that caused it can be named: ids and slugs unique, an id ending in its own zero-padded number, a date that parses, a status from the five, and both halves of a supersession. A file that breaks one of them is left out with a warning naming the file and line, and the rest of the tree is read; only a supersession with one half recorded refuses the whole run. That is why Superseded by and Supersedes are two bullets rather than one derived from the other - supersession is a two-way fact, and half of it recorded is a bug. The halves that live in one step's tree are held against each other there; a record superseded by one in another service's tree is a claim only the merged catalog can check, and the validator checks it. The same goes for Scope and Relates: an extractor sees one root, so it validates the shape of a name and leaves whether the thing exists to the far side.

The demo estate's org-wide and context-wide records live in data/adr, and are read by a step that points at that directory with a glob of its own. Root docs/ is where gen-markdown writes, so nothing hand-written can live there. in is the directory of records rather than data itself: a step's fragment is only left out of its own stamp when the output is inside the input root, and in: data with out: data would be stamped from the file it writes.

{
  "plugins": [{ "name": "adr", "wasm": { "url": "file://plugins/portolan-go.wasm" } }],
  "extract": [
    {
      "plugin": "adr",
      "in": "data/adr",
      "out": "data",
      "options": { "files": ["*.md"], "out": "adr.json" }
    },
    {
      "plugin": "adr",
      "in": "examples/auth",
      "out": "examples/auth/portolan",
      "options": { "out": "adr.json" }
    }
  ]
}

Vocabulary written by hand

A context's glossary is the one file in a service written for a person and read by everyone: what a word means inside the boundary it is spoken in. Nothing generates it and nothing should - a definition is a decision about language, not a fact about a type - so extract-glossary only reads. It takes GLOSSARY.md at a service's root and answers with a fragment of terms.

# Glossary — auth

One meaning per word inside this context.

**Session.** Proof that a user logged in, how long that proof is good for, and
whether it has been taken away.

A title, an optional line or two saying what the vocabulary covers, then one paragraph per term in alphabetical order. The paragraph opens with the term in bold and the full stop inside the bold, so **Email address.** names a two-word term and nothing has to guess where the name ends. Everything after it is the definition, carried through as written. Hard wrapping is the author's business: a soft break inside a paragraph is a space, here as in every markdown renderer.

Nothing reads the definition for structure. A glossary is a person explaining a word to another person, and a parser that went looking for shapes inside the explanation would be a parser telling an estate how to phrase itself.

The shapes a glossary is otherwise written in are refused by name: a table, a bullet list, a heading per term. Also refused: a file that does not open with # Glossary, an entry that defines nothing, and one word defined twice - inside a file or across the files of one step - because a word with two meanings in one context is the failure the glossary exists to prevent.

What is merely untidy comes back as a warning and the fragment is still written: a file that has drifted out of alphabetical order, a root with no glossary at all.

The term's id is <context>.<slug> - auth.session, shop.order - so the context has to be told to the step rather than derived from the directory: a glossary sits beside a SERVICE, and examples/shop/oms/GLOSSARY.md holds words that belong to shop. The same word in two contexts is two terms, which is the point of the id; the same word twice in one context is an error.

{
  "plugins": [{ "name": "glossary", "wasm": { "url": "file://plugins/portolan-go.wasm" } }],
  "extract": [
    {
      "plugin": "glossary",
      "in": "examples/auth",
      "out": "examples/auth/portolan",
      "options": { "context": "auth", "out": "glossary.json" }
    }
  ]
}

Commands: what to type

The first thing a reader new to a checkout wants is not the aggregate list but how do I build this, test it, run it. The README answers that sometimes; the runner files answer it always, and they are already in the tree. extract-commands reads them and puts the answer on the service as a list of commands - the line to type, what the file says it is for, what the runner would execute, and where it was read.

Left out, because the runner leaves them out too: _-prefixed and [private] entries, internal tasks, make's special and pattern targets (.PHONY, %.o), targets spelled through a variable, and pre/post hooks of a script that is itself listed. [project.scripts] in a pyproject is not read: those are programs the package installs, not tasks a developer runs. uv has no task section and Go has no runner of its own, so a repository with only those declares no commands.

Maven and Gradle are the other way round: the build is the runner, and what a project declares is which plugins extend it. So a pom lists the two lifecycle phases every pom answers to, test and package, and then the goals a person types that its <build><plugins> add - spring-boot:run for the Spring Boot plugin, flyway:migrate for Flyway - from a short table of the plugins an estate meets. A plugin bound to a phase, like protobuf generation, is run by the phase and is not listed; neither is one under <pluginManagement>, which pins a version and runs nothing. A Gradle script lists build and test, run when it applies the application plugin, bootRun for Spring Boot, and every task it registers itself, with the description it sets. The script is read as text, not run: a task registered in a loop or by an unnamed plugin is not here.

Cargo declares commands in one place, [alias] in .cargo/config.toml, and the xtask convention is an alias that runs a package: xtask = "run --package xtask --". Each alias is a command with its expansion as the body, and when the expansion runs a package and leaves the subcommand to the caller, that package's main.rs is read for the subcommands - the variants of a clap enum deriving Subcommand, in kebab-case, with their /// comment as the doc, and the string literals of a match on the first argument, with a // comment over the arm. An alias that names its subcommand already, like gen = "run -p xtask -- gen", is listed as the one command it is.

Nothing is evaluated. A target inside an ifeq is listed; a name that depends on a variable's value is not, because the name it would have is not in the file. The pyproject reader is a line scanner that follows the table headers and the shapes both runners document, not a TOML parser.

The fragment names the service and lists its commands, and claims nothing else: merging puts the list on the service a domain extractor described. The runner files sit beside the service, so the step is told which one it is.

{
  "plugins": [{ "name": "commands", "wasm": { "url": "file://plugins/portolan-go.wasm" } }],
  "extract": [
    {
      "plugin": "commands",
      "in": "examples/shop/pricing",
      "out": "examples/shop/pricing/portolan",
      "options": { "context": "shop", "service": "pricing", "out": "commands.json" }
    }
  ]
}

extract-project reads the same files for the component it describes, so a repository read by that plugin does not need this one.

Downstream, gen-markdown draws the list as a Commands table on the service page, and gen-backstage puts it on the Component twice, because Backstage has no field for it: as the portolan.io/commands annotation, one command a line with its description, and as entity links of type command, each leading to the line of the runner file it was read from, when sourceBaseUrl says where the repository is.

Outside the estate: an external with a contract

A service calls things nobody here builds - a card network, a tax API, a carrier. The first answer the catalog gave was unknown: the call was recorded and left unresolved, the lane drawn dashed and red, and the Problems page listed it beside real defects. True, and unhelpful, once the far end is Stripe and Stripe publishes a document.

An external is what the catalog may claim about such a system, and no more: what it answers on, read from the copy of its document vendored beside the adapter that calls it, and what the manifest says it is called and is for. It sits at the root beside the contexts (catalog.externals) with a bare id, no context, no aggregates and no repository, and the estate's picture draws it outside, muted, exactly where it is.

Two steps describe one, and neither knows the other exists:

{ "plugin": "openapi", "in": "examples/payments/ledger",
  "out": "examples/payments/ledger/portolan",
  "options": { "external": "stripe", "externalName": "Stripe",
               "externalUrl": "https://docs.stripe.com/api",
               "spec": "src/main/java/.../infrastructure/stripe/openapi/openapi.yaml",
               "out": "stripe.json" } },
{ "plugin": "java-domain", "in": "examples/payments/ledger", "...": "...",
  "options": { "externals": { "stripe.v1": "stripe" } } }

The first reads the copy and says what stripe answers on. The second reads the adapter, finds the verb and the route in each call, looks the operation up in the same copy, and - told by externals that the copy's api id belongs to stripe - records the call as stripe.v1/PostPaymentIntents, declared, on a lane of kind external. The merge joins the two by the id; a call to an operation the copy declares resolves, one it does not is reported by the extractor and left out.

Neither line is needed when the tree can say it itself. An openapi step with no spec walks the tree for every document and reads what sits beside each one: a server generated from it - oapi-codegen's ServerInterface, swag's docs package, a handler or controller written against it - means the service implements it, and the document is what the service provides; a client generated from it - ClientInterface, a client.gen.go, an adapter named after a client - means the service calls it, and the document names a system this tree does not implement. That system is an external, with the id, the name and the summary the document gives itself: a copy titled "Gordian Flights & Ancillaries API" becomes gordian-flights-ancillaries, the trailing "API" dropped because the system is the thing and not its interface. peers on the same step says which called documents are ours - {"auth.v1": "auth.auth"} - and those are skipped, because the service that implements them describes them; externals names a system when the title would not, {"stripe.v1": "stripe"}. A document with neither a server nor a client beside it is reported and left alone, since reading it as either would be a guess.

The domain extractor keeps the same rule from its side: a generated HTTP client whose api no peers line claims is read as calling the system the document beside it is titled after, and the call lands on an external lane, declared, under the id both sides derive from that title. externals on the domain step overrides the name; a proto client, whose contract names no system, stays unresolved until peers says who answers.

The copy is narrow - the operations the service calls and the schemas they answer with, every field verbatim - for the reason org.0001 gives for a proto: what is vendored has to be reviewable, and Stripe's whole document is not. It carries one line the original does not, x-portolan-api: stripe.v1 in info: the copy is already the consumer's translation boundary, so it is the one place the estate's name for the document is written, and every reader of the copy takes the id from there rather than from Stripe's title and version

• or from two manifests that would have to agree.

Verifiers: the third phase

An extractor reads source and runs before there is a catalog; a generator reads the catalog and writes pages. A verifier sits between them. It reads something observed - traces today, a test's record tomorrow - and answers with a fragment like an extractor's, but one that only makes sense against the merged catalog: "this hop was seen running" names a hop somebody else declared. So a verify step is handed both input and catalog, and the catalog it is handed leaves out the step's own last output. Without that, what it wrote last time would count as evidence this time, and the fragment could never be checked against a clean run.

{
  "plugins": [{ "name": "otel", "process": { "command": "go", "args": ["run", "./plugins/verify-otel"] } }],
  "verify": [
    {
      "plugin": "otel",
      "in": "examples/auth",
      "out": "examples/auth/portolan",
      "options": { "traces": ["telemetry/traces.jsonl"], "out": "observed.json" }
    }
  ]
}

What comes back is merged like any other source, under two rules that exist for it. A flow declared twice is accepted when the second declaration differs only in status: declared steps become verified, and anything else that differs - a lane, a hop, a branch - is the conflict it always was. A consumer or a call declared twice keeps the first note and takes verified if either side has it.

verify-otel

Reads OTLP JSON - one batch per file or one per line, as a collector's file exporter writes it - and turns each trace into hops between lanes:

A trace whose root opens a declared flow raises the steps it shows: the call in, the events out, the rpcs with a ref. A call step is never raised - a SELECT ran, which is not the same claim as "the repository's ByEmail was called" - and unresolved is never raised, because a trace does not put the far end in the catalog. A consumer span inside a trace opens a flow of its own and is matched the same way, so one password change verifies both the request's flow and the policy's. A root no flow opens is written down as observed-<service>-<route>, once per shape, with a summary saying how many traces showed it.

service.name is matched to the one service whose slug it is, event.name to the one event whose wire.name it is, or failing that to the one event of the publisher's with that last segment; services and events in the options say otherwise where an estate's names differ. A publish span whose messaging.destination.name is not the event's wire.channel is a warning: the event went out, but not where the code says it does.

verify-codeowners

Reads the CODEOWNERS a repository already keeps and says who to ask about each service.

"Who do I ask about shop.oms" was the question the estate answered worst. owner on a flow, a store or a module means the bounded context that holds it

• a grouping, not a team - and there was nothing on any page a reader could act on. The answer was already written down, in the one file the forge itself enforces: a team that owns a directory is a team that gets the pull request.

{
  "plugins": [{ "name": "codeowners", "process": { "command": "go", "args": ["run", "./plugins/verify-codeowners"] } }],
  "verify": [
    { "plugin": "codeowners", "in": ".github", "out": "data", "options": { "out": "owners.json" } }
  ]
}

Point in at the directory the file is in, not at the repository root: the host dates a fragment from the last commit to touch the step's input, and the subject of this one is the CODEOWNERS file. Rooted at the repository, it would be restamped by every commit ever made. Left with no file, the three places a forge looks are tried in order - CODEOWNERS, .github/CODEOWNERS, docs/CODEOWNERS - and a file that names something absent fails the run, because answering "nobody owns anything" to a typo is only noticed a month later.

It is a verifier and not an extractor because a rule is a path and only the merged catalog knows where each service is; an extractor would have to be told, service by service, in the manifest, what the catalog already says. It earns the name twice over. A service no rule matches is reported, and so is a rule that matches no service - a team believing it owns something the estate does not have, which is the one failure a CODEOWNERS file can never report about itself. A rule that matches and never wins is reported differently, because the fix is different: nothing is wrong with the path, everything it covers is just claimed by a rule below it.

What lands on the catalog is owners on the service: handles exactly as the file spells them, @acme/oms-team, @someone, dev@acme.io. Deliberately nothing more - resolving a handle to the people currently in it is a call to a forge's API, which needs a credential this does not have and answers differently tomorrow, and a handle is what a reviewer types anyway. Two sources naming owners are unioned, because two rules that both matched are two facts and not two answers.

The grammar is gitignore's, minus the parts CODEOWNERS does not have. A pattern owns a directory when it names the directory or anything above it, and does not when it names only something inside it: services/oms/internal is a rule about part of a service, and reading it as ownership of the whole would hand a team a page it never asked for. Later rules win. A pattern with no owners after it wins too - taking ownership back is the only reason anybody writes one. GitLab's sections change which rule wins and are read the flatter way GitHub means, with a warning saying so, because the difference only ever shows up as an owner quietly missing from a page.

The demo estate's file is data/codeowners/CODEOWNERS, which is deliberately not one of the three places a forge reads: this repository is both the tool and the estate it describes, and a real CODEOWNERS here would ask GitHub to request reviews from teams that do not exist. A real repository puts it where the forge looks.

wasm or process

wasm is the default and should stay that way. The module gets no network, no environment and no way to start a process. A generator gets no filesystem either. An extract or verify step gets the workspace preopened as / (portolan.0006), which is how the built-in Go extractors read a tree without a Go toolchain on the machine: every one of them, and the three generators, is the single module plugins/portolan-go.wasm, which answers to the plugin name the host passes as argv[0]. WASI preopens read-write, so an extractor is trusted not to write the tree it reads, the same trust a process plugin has today; a sha256 pins that trust to a build.

process is the escape hatch for a plugin that needs a toolchain: the Rust, Java, Python and TypeScript extractors run in their own runtimes, and fetch-bsr still talks to its registry from Go. It gets the same protocol and none of the sandbox, which is the trade being made and the reason it is not the default. It declares command and an args array; the host never feeds a command string through a shell. A built-in Go plugin that still runs as a process is the same code reached as go run ./plugins/cmd/portolan-go <name>.

host is for Portolan's own code that needs what only the host has - a git binary, a socket - and so runs inside the host process (portolan.0008): { "name": "git", "host": "fetch-git" }. The name is resolved against the modules shipped in scripts/host-plugins/ and nothing else, so a manifest cannot point the host at arbitrary code; the contract is the same as any plugin's, files named and never written.

A plugin fetched over https:// must declare its sha256; the host verifies it and caches by digest. A file:// plugin may declare one, but a checksum protects a download, not a module built from the source next to it. Downloads do not follow redirects and are bounded in time and size. Every run has a deadline and bounded stdout/stderr; wasm runs in a worker so even a module stuck in a loop can be terminated. Responses reject unknown properties, duplicate or unsafe filenames, and non-string contents before anything is written.

Services in other repositories: fetch-git

fetch-git is fetch-bsr for a repository rather than a registry, and it lives by the same four rules. A pin is a repository, a commit and the paths actually read; the step fetches exactly those directories at exactly that commit and hands them back as files, so the host writes them into the tree beside a git.lock.json naming the commit and the digest of every file. The paths inside the copy are the repository's own, which is the point: the extract step that follows points its in at the vendored service and reads it exactly as it would read that service's checkout.

It runs inside the host (scripts/host-plugins/fetch-git.mjs, portolan.0008) rather than as a module, because it needs a git binary and a socket, and a manifest names it with host rather than wasm or process. The contract is the same: it names files, the host writes them.

{
  "sources": ["data/*.json", "vendor/repos/*/*/git.repo.json"],
  "plugins": [{ "name": "git", "host": "fetch-git" }],
  "extract": [
    {
      "plugin": "git",
      "in": "vendor",
      "out": "vendor/repos",
      "options": {
        "cache": "vendor/repos",
        "repos": [
          { "repo": "github.com/acme/shop", "commit": "c1d2e3f4…", "paths": ["services/oms", "proto"] }
        ]
      }
    },
    {
      "plugin": "go-domain",
      "in": "vendor/repos/acme/shop/services/oms",
      "out": "data/shop",
      "options": { "context": "shop", "service": "oms", "store": "pg" }
    }
  ]
}

It runs the git the host already needs for stamps: a fetch of the one commit into a directory that exists for one call, and an archive of the paths wanted, read straight into memory. Whatever git is configured to do about credentials and hosts - a helper, a netrc entry, an ssh agent - it does here too, and the plugin reads none of it. PORTOLAN_OFFLINE (or any truthy CI) replays the committed copies against their locks; a commit the manifest does not pin is resolved online with a warning and refused offline; a fetch that fails falls back to the committed copy when there is one, and is a red build when there is not; a vendored file edited by hand is reported by path.

What the copy says about itself

Two files land beside every copy, and the difference between them is who reads them. git.lock.json is for the next run of this step: the commit, and the digest of every file, which is what makes replaying the copy equivalent to fetching it again. git.repo.json is for the estate - a catalog fragment holding one line, the repository and the commit it is a copy of - which is why it is in sources above.

Nothing else can say it. A service says which repository it lives in, and an extractor reads a directory as a pure function of what is on disk; neither has any idea which commit somebody fetched. Without that line, every source path of every vendored service is dead text on the page - the file and the line are known, and there is nowhere to send a reader - and every fragment read out of the copy is stamped with the commit that VENDORED it, so the service looks fresh whenever the fetch is re-run and unchanged when its own repository moves. With it, sourceHref links the line at the commit it was read at and stampFor dates the fragment from the code rather than the vendoring.

One more line is needed for the app itself: SOURCE_GLOBS in src/data.ts, where the same patterns are written out a second time because import.meta.glob resolves at build time and needs literals.

Schema modules: fetch and parse, kept apart

fetch-bsr and extract-proto are two plugins on purpose, and the split is the whole design.

Extraction stays a pure function of the tree. Fetching is the step that can fail, need a credential, or come back with something different than it did yesterday, and confining that to its own step is what lets everything after it be replayed byte-for-byte from a checkout.

The fetched protos are the plugin's Response.Files, not a side effect. The host writes them like any other generated file, so they get a manifest entry, are compared by gen:check, and are removed when the step stops naming them. The cache is not a second copy of anything — it is the tree. Refreshing a pin produces one pull request holding the pin bump, the proto diff, the lock diff and the fragment diff, which is the review worth having.

Declare the fetch step before the extract step: steps run in list order, so its protos and locks are on disk by the time the parser reads them.

{
  "plugins": [
    { "name": "bsr",   "host": "fetch-bsr" },
    { "name": "proto", "wasm": { "url": "file://plugins/portolan-go.wasm" } }
  ],
  "extract": [
    {
      "plugin": "bsr",
      "in": "examples/shop",
      "out": "examples/shop/vendor/proto",
      "options": {
        "cache": "examples/shop/vendor/proto",
        "modules": [
          { "module": "buf.build/acme/shop", "commit": "c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6" }
        ]
      }
    },
    {
      "plugin": "proto",
      "in": "examples/shop",
      "out": "examples/shop/portolan",
      "options": {
        "context": "shop",
        "service": "oms",
        "paths": ["vendor/proto/acme/shop"],
        "vendored": ["internal/infrastructure/pricing"],
        "peers": { "pricing.v1": "shop.pricing" },
        "out": "proto.json"
      }
    }
  ]
}

cache repeats the step's out because a plugin is never told where its output goes. That is the shape of the protocol — a plugin returns files and the host decides what to do with them — and not an oversight to work around.

Why fetch-bsr can never be wasm

It needs a socket and a credential, which is why it runs inside the host (scripts/host-plugins/fetch-bsr.mjs, portolan.0008) and is declared with host. The protocol's "no ambient state" rule is about facts: nothing about the estate may come from anywhere but the request. A credential is not a fact about the estate — it decides whether the fetch succeeds, never what the fetch says — and a test asserts the output is byte-identical with and without a token. The token comes from BUF_TOKEN or the netrc buf registry login wrote, never from the manifest.

Pinning, and the offline rule

Pin every module to a commit. A BSR commit is immutable, so a pinned download is byte-reproducible, which is the only reason replaying from disk is equivalent to fetching again. An unpinned module is resolved and warned about when online, and refused when offline — there is nothing to replay against.

PORTOLAN_OFFLINE=1 (or any truthy CI) turns the fetch off. The step then re-emits the committed copies, checked against their locks. Set it in CI; the workflow already does.

Four rules govern what happens when a fetch does not:

1. Fetch succeeded → the fetched files and a regenerated lock.
2. Skipped or failed, cache complete and matching its digests → the cached files byte-identically, plus a warning. Output unchanged, so --check stays clean.
3. Failed and no usable cache → a non-zero exit, never a short file list. The host deletes files a step stops naming, and dropping a repository's vendored protos because a laptop went offline is worse than a red build.
4. A cached file whose digest no longer matches is reported by path — someone edited a vendored copy, which is the drift docs/adr/org.0001.md wants seen.

Why the proto parser is hand-written

docs/adr/org.0001.md has consumers keeping narrowed vendored copies, and a narrowed copy routinely imports a file nobody vendored beside it. A compiler — protocompile, protoc — refuses to produce anything for that input. The whole point of reading vendored copies is to describe files that do not build standalone, so the parser is tolerant: every construct it declines to model (extend, a proto2 group, an aggregate option body) is named in a diagnostic rather than dropped, and only a file that cannot be tokenised is fatal.

It also keeps types as written — []LineItem, map[string]Money, the optional keyword, the author's declaration order — the same stance the catalog already takes for schemas, and one a descriptor has already thrown away.

What extract-proto will not claim

status is only ever declared or unresolved, never verified. Reading a .proto proves a call was written down. verified in the shipped catalog means a test exercises it end to end, which is a property of the merged catalog — and every extractor runs before one exists.

A message named OrderPlaced in an events.proto stays an RpcMessage and does not become a catalog Event. Event.id is <service>.<aggregate>.<Name> and this extractor knows the package, not the aggregate; a guess would collide with the event extract-go already emits, or invent a ghost aggregate that would sit beside the real one forever.

Schema registry subjects: the same split, one topic further

fetch-csr and extract-csr are the Confluent Schema Registry half of the same argument, and they are shaped like fetch-bsr and extract-proto on purpose: one owns the socket and the credential, the other owns the reading, and CI runs the reading over a tree it can verify without a registry existing at all.

Declared as { "name": "csr", "host": "fetch-csr" } and { "name": "csr-schemas", "wasm": { "url": "file://plugins/portolan-go.wasm" } }: the fetcher runs inside the host (scripts/host-plugins/fetch-csr.mjs, portolan.0008), the reader in the shared module.

A registered version is immutable: subject orders-value at version 3 is the same bytes today and next year, and re-registering a changed schema makes version 4. That is the promise a BSR commit makes, so the same four rules govern a fetch that does not happen, and PORTOLAN_OFFLINE=1 replays the committed copies against their locks exactly as it does there.

A subject a fetched schema references is fetched too, pinned by the reference rather than by the manifest, and needs no entry of its own — the version is part of the bytes we already have, so following one adds no lottery. Each lands in its own directory, and the referring subject's lock is what an offline run follows to find them.

Avro and JSON schemas arrive minified onto one line. They are written out indented — re-spaced token by token, never re-parsed, so the file still says what the registry said in the order it said it — because a version bump that is one unreadable line is a review nobody can do. The digest is over the bytes as written, so verifying needs no reformatting of anything.

The strategy the registry does not record

A subject name is whatever the producer's serializer decided to call the registration, and the rule it used — the SubjectNameStrategy — is nowhere in the registry's answer. shop.oms.order-value is a topic plus a suffix under TopicNameStrategy and a record's full name under RecordNameStrategy, and nothing but the manifest can say which. So strategy is an option, and everything extract-csr does with a name follows from being told it.

Under topic-record the separator is a hyphen and both halves may contain one, so the split is made by matching the schema's own full name as the suffix rather than by searching for a delimiter.

A -key subject is kept as a shape and put on no channel: a key is part of every message on the topic, not a message on it. Under record there is no topic at all, and a fragment with no channels is the right answer rather than a gap — that strategy exists so a record can be reused across many.

What extract-csr will not claim

It emits no events. An Event in the catalog belongs to an aggregate, and a registry holds schemas, not domains — it has no idea which aggregate raises what. So the shapes land in defs, where a shared shape belongs, and the topics land in the service's channels beside the ones an AsyncAPI document declares. The domain extractor says an aggregate raises OrderPlaced and calls it shop.oms.OrderPlaced on the wire; this says a schema by that name is registered against topic shop.oms.order and has these fields. Neither knows the other exists, and the pages hold the two against each other.

It does not say who produces. A registry records no producer and no consumer. direction is told, per step and per subject, or it would be invented.

It does not parse protobuf. That is extract-proto's whole job, and a second, worse parser here would be a second answer to one question. A PROTOBUF subject still names its topic — which is the one thing a .proto file cannot say — and a diagnostic points at extract-proto for the fields.

A field referencing a shape nothing in the estate vendored keeps its name and loses its ref. The catalog validates that every ref resolves, and failing a run over a reference that is genuinely true — the shape really does live in another estate — would be the wrong end of the trade.

The bus: a channel is a claim, not an event

extract-asyncapi reads an AsyncAPI document and answers with the channels a service declares — the address the broker knows, and each message on it with the direction it travels. What it does not answer with is events.

That looks like a gap and is a boundary. Event.id is <service>.<aggregate>.<Name>, and an AsyncAPI document knows the message on the wire, not the aggregate that raised it. An extractor that guessed would either collide with the event extract-go and extract-ts already emit or invent a ghost aggregate that would sit beside the real one forever — the same rule extract-proto keeps about a message called OrderPlaced.

So the two sources meet in the merge instead, and the pages hold them against each other. The domain says an aggregate raises BasketCreated and how it leaves, in wire; the document says the service sends cart.BasketCreated on shop.cart.basket. Where they agree the catalog says the same thing twice, which is worth nothing. Where they disagree it is worth a row on the Problems page, because one of the two is stale:

• an event whose channel the document does not declare — a subscriber reading the document does not know the message exists;
• a channel the document declares and no event names — a promise nothing keeps;
• a message the document listens for that nothing in the estate publishes.

That last one is the only edge in the catalog that runs from the subscriber outwards. Everywhere else a publisher names its consumers; here the subscriber names a message and the estate is searched for whoever puts it on the wire. A subscription that resolves is how two repositories that never mention each other are found to be joined — and a channel that two services both declare a send on is a second publisher, which is an error for the reason a second writer in a database is.

2.x says publish and subscribe backwards

In AsyncAPI 3.x an operation carries action: send or action: receive, from the application's side, and there is nothing to get wrong. In 2.x a channel has publish and subscribe, and both are written from the client's side: publish is what somebody else publishes to the application, so the application receives it, and subscribe is what the application produces for somebody else to subscribe to.

Reading 2.x the obvious way puts every arrow in the estate the wrong way round. The extractor reads both versions and answers in 3.x's vocabulary, which is the one the catalog keeps.

A work queue is a channel of kind job

A task queue is a channel too - an address the broker knows, messages that travel on it - with one difference the catalog has to be told: many callers put the same job on it by design, and no domain event stands behind a job. kind: "job" on the channel says so. The merge does not call two senders on a job queue rival publishers, and the Problems page does not look for an event with the job's wire name. extract-celery answers with these for a Python tree: one channel per queue, a send per task the tree enqueues and a receive per task it declares, and one flow per task that is both - the call that enqueues it, then the worker that runs it. It reads the queue the way Celery decides it, the call before the decorator before task_routes before the default, and transaction.on_commit(...) around an enqueue is a note on the step, which is the one fact about when a message leaves that the code states plainly.