Documentation
¶
Overview ¶
Package ipmtokens runs the ipmt semantic tokenizer used by both the LSP server (cmd/ipm-rpc) and the md-html exporter (pkg/mdhtml).
Token coordinates are 0-based UTF-16 columns, line-relative to the surrounding document — caller supplies `lineOffset` to position a block inside its containing markdown file.
The token kind is a uint32 INDEX into SemanticTokenTypes; the LSP protocol uses the same indexing so the encoder in cmd/ipm-rpc can pass values through unmodified. Callers that need the wire-format string (e.g. the JSON embed-buffer response, or pkg/mdhtml which keys off "ipmEvent" / "ipmThing" / …) use TypeName.
Index ¶
Constants ¶
const ( ModAlias uint32 = 1 << 0 ModLeadsTo uint32 = 1 << 1 ModPartOf uint32 = 1 << 2 ModExpresses uint32 = 1 << 3 ModNearTo uint32 = 1 << 4 ModHasAlias uint32 = 1 << 5 ModMarker uint32 = 1 << 6 )
Modifier bitmask helpers — bit positions mirror SemanticTokenModifiers.
const ( TypEvent uint32 = 0 TypThing uint32 = 1 TypConcept uint32 = 2 TypRelation uint32 = 3 TypArrow uint32 = 4 TypTypeMarker uint32 = 5 TypComment uint32 = 6 TypString uint32 = 7 TypTooltip uint32 = 8 TypUnresolved uint32 = 9 )
Token type indices — mirror SemanticTokenTypes.
Variables ¶
var Palette []PaletteRule
Palette is the ordered rule list parsed from palette.json.
var SemanticTokenModifiers = []string{
"alias",
"leadsTo",
"partOf",
"expresses",
"nearTo",
"hasAlias",
"marker",
}
SemanticTokenModifiers is the LSP token-modifier legend. Index N here corresponds to bit (1<<N) in the Token.Mods field.
var SemanticTokenTypes = []string{
"ipmEvent",
"ipmThing",
"ipmConcept",
"ipmRelation",
"ipmArrow",
"ipmTypeMarker",
"ipmComment",
"ipmString",
"ipmTooltip",
"ipmUnresolved",
}
SemanticTokenTypes is the LSP token-type legend. Index order is part of the LSP wire contract — appending is OK, reordering is not.
Functions ¶
func ClassSuffix ¶
ClassSuffix returns the `ipm-<suffix>` class suffix for an LSP semantic token (type + modifier bitmask), or "" if the type is unknown. The modifier bits a token actually carries are mutually exclusive within a type, so the first matching modifier rule wins; the base rule is the fallback.
func PaletteCSSRules ¶
func PaletteCSSRules() string
PaletteCSSRules returns the generated `.ipm-<suffix> { ... }` rules (one per palette entry, in palette order) for embedding in HTML/preview CSS.
func TokenForClass ¶
TokenForClass is the inverse of ClassSuffix: the (token-type index, modifier bits) that an `as-token:NAME` paints over its text. ok=false for an unknown name.
Types ¶
type PaletteRule ¶
type PaletteRule struct {
Type string `json:"type"`
Mod string `json:"mod,omitempty"`
Class string `json:"class"`
FG string `json:"fg"`
Bold bool `json:"bold,omitempty"`
Italic bool `json:"italic,omitempty"`
}
PaletteRule is one (type, optional modifier) styling entry.
type Token ¶
type Token struct {
Line uint32
StartCol uint32
Length uint32
Type uint32 // index into SemanticTokenTypes
Mods uint32 // bitmask matching SemanticTokenModifiers
}
Token is one un-encoded semantic token. Positions are 0-based UTF-16 columns; Collect adds the caller-supplied `lineOffset` so a block inside a markdown document carries whole-file coordinates.
func Collect ¶
Collect parses the given ipmt source and emits semantic tokens for node names, alias spans, arrow spans, and quoted-tooltip spans. Positions are returned in whole-file LSP coordinates given the line offset of the block within its containing markdown document (0 for pure .ipmt files).
`wholeText` is currently unused: every span is sourced from the parser's graph.Src.Lex structure and positioned via a mapper built from blockText (plus lineOffset). The parameter is retained for call-site symmetry and a possible future cross-block mapper. Callers may pass blockText for it.
func Flatten ¶
Flatten resolves the intentionally-overlapping tokens Collect emits into a sorted, NON-OVERLAPPING stream — the canonical output every consumer (LSP wire encoder, md-html, preview) can use directly.
Resolution is per-character LAST-WINS BY EMISSION ORDER: for every UTF-16 column a later token in the input overwrites an earlier one on the columns they share. This is byte-for-byte the same rule pkg/mdhtml.HighlightTokens and previewHighlight.ts:renderFromLspTokens apply when rendering, so flattening here changes no rendered output — it only moves the resolution upstream so the LSP can ship a spec-clean (disjoint, sorted) token set instead of relying on client leniency.
Tokens that map to no CSS class (ClassSuffix == "") are skipped, never painting over a colored token — matching the `continue` both renderers do. Collect never emits such tokens (all its types are valid), but the guard keeps Flatten equivalent to the renderers for any input.
All Collect tokens are single-line (it clamps a range that crosses a newline to one character on the start line), so painting per line is exact.
func TokensForClass ¶
TokensForClass supports the inline-ipmt `as-token:NAME` marker: it resolves the palette class suffix `name` (e.g. `e-marker`, `L`, `type-marker`) to a single synthetic token covering the whole visible text — `[0, utf16Len(visible))` carrying the (type, mods) that map to `ipm-<name>`. The visible text is NOT parsed as ipmt; only the named style is applied. Returns (nil, false) when `visible` is empty or `name` is not a known class suffix.
`TokenForClass` (the name → type+mods reverse map) is defined in palette.go and derived from the embedded palette.json — the single source of truth. Code generation flows the other way: vscode-ipm/scripts/gen-palette.ts consumes the same palette.json to produce the editor's src/palette.gen.ts. Guarded by the e2e drift test.
Source Files