codegen

package
v0.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 6, 2026 License: BSD-3-Clause Imports: 3 Imported by: 0

README

ZAP v2 schema declarations

This directory holds JSON schema declarations consumed by the zapgen-all bulk emitter (see ../cmd/zapgen-all/). One file per LP family.

Format

Each .json file is a single bundle:

{
  "lp": "lp-201",
  "description": "ZAP P2P transport ...",
  "schemas": [
    {
      "wire_name": "ConsensusVote",
      "go_name":   "ConsensusVoteSchema",
      "kind":      "0xD0",
      "size":      53,
      "package":   "p2pwire",
      "out":       "lp-201/consensus_vote_zap.go",
      "fields": [
        { "name": "Round",   "type": "uint64",  "offset": 1 },
        { "name": "BlockID", "type": "bytes32", "offset": 9 }
      ]
    }
  ]
}
Field types

Two field-type families are supported:

  1. Scalar fields. Type names: bool, int8/int16/int32/int64, uint8/uint16/uint32/uint64, float32/float64. Read/write via zapv2.Read / zapv2.Write with the Field[S, T] handle declared in <GoName>Fields.

  2. Fixed-width byte-array fields. Type names: bytes<N> where N is a positive integer (e.g., bytes20 for NodeID/Address, bytes32 for hashes, bytes16 for session IDs, bytes96 for PQ-signature digests). Emitted as standalone typed accessor functions <WireName><FieldName>(view) [N]byte; NOT in the <GoName>Fields struct because [N]byte is not a zapv2.FieldKind member.

Variable-length tails (lists, bytes, sub-objects) are NOT declared here. They live outside the fixed payload. Convention in this dir: a field named *Off is a uint32 offset pointer into the variable- length tail; the actual tail read/write code is hand-written using zapv2.ListAt, Object.Bytes, or similar helpers.

Bundles in this directory

File LP Schemas Schema-ID range
lp-182-consensus.json LP-182 8 (QuasarCert, QBlock, WitnessProof, MagnetarAggregateCert, PolarisLegs, QuasarSig, TxAuthEnvelope, PQPermit) 0x01..0x0B
lp-186-vms.json LP-186 38 across 12 VMs (aivm:3 / bridgevm:3 / dexvm:3 / evm:2 / graphvm:3 / identityvm:4 / keyvm:3 / oraclevm:2 / quantumvm:3 / relayvm:3 / thresholdvm:4 / zkvm:5) per-VM kind bytes 0x01..0x05 within each VM's <vm>wire package namespace
lp-201-p2p.json LP-201 16 (ConsensusVote ... DHTStore) 0xD0..0xDF
lp-208-dag.json LP-208 2 (DAGHeader, DAGBody) 0xE0..0xE1
lp-211-cross-shard.json LP-211 6 (CrossShardTxGroup ... CrossShardAbortNotify) 0xE2..0xE7
lp-214-light-client.json LP-214 3 (LightClient*) 0xF0..0xF2
lp-218-rollup.json LP-218 4 (Rollup*Tx) kind bytes 0x40..0x43; schema-ID range tracked under the LP-218 / LP-208 / LP-211 collision note in lp-218-rollup.json

Total: 77 schemas declared and ready for bulk emission.

Variable-length tail convention (LP-182 + LP-186)

Each variable-length byte tail (bytes field per LP-182 §"Wire schemas" / LP-186 §"Variable-length fields") occupies an 8-byte slot in the fixed section, matching what zap.ObjectBuilder.SetBytes actually writes: {relOff uint32, length uint32} little-endian. In the JSON schema this is modeled as two uint32 fields: <Name>Off and <Name>Len. The underlying ZAP wire is unchanged — this is purely how the codegen declares the fixed-section byte layout.

LP-201 / LP-214 schemas (declared earlier) use a single *Off uint32 convention without an explicit *Len companion; their generated code remains wire-compatible because the codegen does not synthesise SetBytes calls for Off fields (callers invoke SetBytes themselves when filling tails). The LP-182 + LP-186 Off+Len convention is the preferred form going forward — it faithfully models the 8-byte slot.

Schemas out of scope (deferred)

The following schemas / kinds are declared in their LP but DO NOT yet have a fixed-section byte layout in the canonical source-of-truth file. They are flagged as "deferred pending spec amendment" rather than guessed.

LP-182 — Consensus wire (5 deferred)
Kind Schema Reason for deferral
0x07 EpochBundle No EpochBundle struct in ~/work/lux/consensus/protocol/quasar/epoch.go and no ~/work/lux/consensus/pkg/wire/zap/epoch_bundle.go. LP-182 §Implementation lists the file as new; layout pending.
0x08 PrismCut No PrismCut struct in ~/work/lux/consensus/protocol/prism/cut.go matching the LP-182 schema. Layout pending.
0x09 StakeWeightedCut The in-tree protocol/prism/stake_weighted_cut.go is an unexported runtime struct (validators []Validator, totalWeight uint64). The Validator wire shape is upstream; the LP-182 wire layout depends on that and is undeclared.
0x0C DAGVertex LP-182 §Implementation re-points DAG vertex to schema 0x0C; the corresponding file (pkg/wire/zap/dag_vertex.go) is not yet in tree. Note: LP-208 declares an unrelated DAGHeader/DAGBody pair at 0xE0/0xE1 — different schema family.
0x0D PolicyCert LP-182 §Implementation re-points pkg/wire/policies.go at schema 0x0D; layout file not yet in tree.

Path forward for deferred LP-182 schemas: once ~/work/lux/consensus/pkg/wire/zap/*.go lands the byte-by-byte layout for each deferred kind, append the JSON schema entry to lp-182-consensus.json with the offsets matched to the source file.

LP-186 — Chains-VM wire (~12 kinds deferred)

The LP-186 §Schema table enumerates kinds that DO NOT yet have a matching wire-layout file in ~/work/lux/chains/<vm>/wire/. The kind constant exists in kind.go, but no per-kind .go file declares its fixed-section byte layout:

VM Deferred kinds Source
aivm KindTaskResult (0x04) aivm/wire/kind.go declares it; no task_result.go in tree
dexvm KindCancelTx (0x03), KindLiquidityTx (0x05), KindMatchTx (0x06) dexvm/wire/kind.go declares them; only block.go/order_tx.go/swap_tx.go in tree
graphvm KindIndexUpdateTx (0x03), KindQueryResultTx (0x05) graphvm/wire/kind.go declares them; not in tree
oraclevm KindBlock (0x01), KindOracleQueryTx (0x03) oraclevm/wire/kind.go declares them; not in tree
quantumvm KindQuasarWitness (0x04) quantumvm/wire/kind.go declares it; not in tree
relayvm KindCloseChannelTx (0x02), KindRelayReceiptTx (0x05) relayvm/wire/kind.go declares them; not in tree

Path forward for deferred LP-186 schemas: once each VM's wire/ directory ships the missing per-kind .go file with a fixed-section layout comment, append the JSON schema entry to lp-186-vms.json and re-run zapgen-all.

Running the emitter

cd ~/work/lux/zap
GOWORK=off go run ./v2/codegen/cmd/zapgen-all \
  -schemas ./v2/codegen/schemas \
  -out /tmp/zap-emit

The output /tmp/zap-emit/ will contain one *_zap.go per declared schema under the out path each schema specifies. The bulk emitter prints emit <lp>/<schema> -> <path> per file plus a final summary line.

Schema-ID allocation map (canonical)

Per /tmp/SESSION-HANDOFF.md §"Schema ID Allocation":

Range Owner Source LP
0xC0..0xCB Chains-VM (12 VMs) LP-186
0xD0..0xDF P2P transport (16 streams) LP-201
0xE0..0xE1 DAG header/body LP-208 (override from LP-208 file's 0xF0..0xF1)
0xE2..0xE7 Cross-shard atomic 2PC LP-211
0xE8..0xEF Reserved (future LP-208 cluster extensions)
0xF0..0xF2 Light client LP-214
0xF3..0xFF Reserved (future framing)

The LP-218 file lists 0xE0..0xEF; this collides with LP-208 + LP-211. The session handoff resolves this in favour of LP-208 + LP-211; LP-218 rollups should track at their kind bytes (0x40..0x43) within a yet- to-allocate schema-ID range — likely 0xC0-bucket extensions or a dedicated 0xC bucket once chains-VM finalises. Until that resolves, the lp-218-rollup.json bundle emits at kind bytes 0x40..0x43 and is parked in its own rollupwire package; the schema-ID byte is not load-bearing for these schemas because they live in a unique package namespace.

Documentation

Overview

Package codegen emits per-schema ZAP v2 accessors that match v1's hand-rolled inline-everything performance.

Why codegen

Go's generic dispatch and inlining cost budget combine to put a hard ceiling on how fast a generic API can be: any function whose body is larger than the inliner's 80-cost budget cannot inline into its caller, so the call site pays a function-call cost (~5-7 ns on modern hardware). The work performed by a "Wrap" function (parse the ZAP wire frame, validate the kind discriminator, build a typed view) exceeds that budget. The generic [zapv2.Wrap[S]] function therefore costs one function call per Wrap, even when every other primitive in its body would inline.

Hand-written per-schema [WrapX] shims (e.g. [examples.WrapAdvanceTime]) have the same problem — even though they pin S{}.Kind() and S{}.Size() as constants, the body is still too large to inline.

Codegen solves this by emitting Wrap/Build/Read/Write functions that DON'T live inside a function at all — they expand inline at the call site through Go templates instantiated at build time. The user writes a single schema declaration; the codegen tool produces a *_zap.go file whose functions match v1's hand-rolled pattern byte-for-byte.

Output shape

For a schema declared as:

type AdvanceTimeSchema struct{}
func (AdvanceTimeSchema) Kind() zapv2.KindByte { return 1 }
func (AdvanceTimeSchema) Size() int            { return 9 }
func (AdvanceTimeSchema) Name() string         { return "AdvanceTimeTx" }

//zap:field Time uint64 @1

the codegen tool emits a sibling file (advance_time_zap.go) with:

const sizeAdvanceTimeTx = 9
const kindAdvanceTimeTx uint8 = 1
const offsetAdvanceTimeTx_Time = 1

func WrapAdvanceTime(b []byte) (zapv2.View[AdvanceTimeSchema], error) {
    msg, err := zap.Parse(b)
    ...
}

The emitted code uses v1 primitives directly (zap.Parse, msg.Root, root.Uint8) so that the inliner folds them into the caller's frame, matching v1 hand-rolled performance.

When to use codegen

  • Hot-path schemas where the function-call overhead matters (per-tx assembly, per-block validation).
  • When you have a schema description in a declarative form (a YAML file, a Cap'n-Proto-style schema file, struct tags) and want ZAP v2 accessors emitted automatically.

For cold paths and ad-hoc schemas, the generic [zapv2.Wrap[S]] / [zapv2.Build[S]] are equally correct and one extra function call slower — which is rarely the bottleneck.

Status

This package is the entry point. The current implementation emits the canonical AdvanceTimeTx fast-path file (used as the canary in the bench suite); adding new schemas means feeding the codegen tool a schema declaration in one of the supported forms.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Emit

func Emit(w io.Writer, s Schema) error

Emit writes the per-schema *.go file to w. Returns an error if the schema is invalid (unknown field type, offset overlap, fields past declared Size, etc.).

The emitted code follows the v1 hand-rolled pattern: every step (Parse, Root, Uint reads, kind check, View compose) uses v1 primitives that inline into the caller's frame. The result matches v1 hand-rolled performance to within compiler noise — verified by the bench suite.

Scalar fields go through the [zapv2.Field][S, T] generic handle (declared in the per-schema <GoName>Fields var). Fixed-width byte-array fields ("bytes<N>") are emitted as standalone typed accessor functions that call v1's SetBytesFixed / BytesFixedSlice — they do NOT appear in the Fields struct because [N]byte is not a [zapv2.FieldKind] member. Both forms produce the same wire layout.

Types

type Field

type Field struct {
	// Name is the Go-visible field name (e.g. "Time"). Emitted into
	// the schema's Fields struct as `<SchemaGoName>Fields.Name`.
	Name string
	// Type is the Go type as a string. See Field doc for the supported
	// set: scalar FieldKind types or "bytes<N>" for fixed byte arrays.
	Type string
	// Offset is the byte position within the fixed-size payload.
	Offset uint32
}

Field describes one fixed-size field in a schema.

Two field kinds are supported:

  1. Scalar fields. Type is one of the [zapv2.FieldKind] members (bool, int8/16/32/64, uint8/16/32/64, float32/64). The emitted code uses a [zapv2.Field][S, T] handle and the standard zapv2.Read/Write generic functions.

  2. Fixed-width byte-array fields. Type is "bytes<N>" where N is a positive integer (e.g., "bytes20" for NodeID, "bytes32" for hashes, "bytes16" for session IDs). The emitted code uses the v1 ObjectBuilder.SetBytesFixed / Object.BytesFixedSlice accessors and returns the value as a [N]byte. Byte-array fields do NOT use the zapv2.Field generic handle because [N]byte is not a [zapv2.FieldKind] member; instead they get a typed accessor function emitted alongside the schema.

Variable-length tail fields (lists, bytes, sub-objects) are NOT declared here. They use the generic [zapv2.ListAt] / out-of-line pointer machinery and require a hand-written accessor.

func (Field) IsBytes

func (f Field) IsBytes() (int, bool)

IsBytes reports whether the field is a fixed-width byte-array field (type "bytes<N>"). Returns (n, true) on a match; (0, false) for scalar fields.

type Schema

type Schema struct {
	// GoName is the Go type name for the schema marker struct (e.g.
	// "AdvanceTimeSchema"). Used as the View[S] type parameter.
	GoName string
	// WireName is the human-readable name of the schema (e.g.
	// "AdvanceTimeTx"). Used in error messages and Registry lookup.
	// Stable across versions — part of the schema's identity.
	WireName string
	// Kind is the discriminator byte at object offset 0.
	Kind uint8
	// Size is the fixed object payload in bytes (excluding header).
	Size int
	// Package is the Go package the emitted file should live in.
	Package string
	// Fields is the ordered list of fixed-size fields. Variable-length
	// tails (lists, bytes) are not declared here; they use the
	// generic [zapv2.ListAt] / [zapv2.WriteList] machinery.
	Fields []Field
	// SkipRegistry suppresses the emit of `init() { zapv2.Register[S]
	// (zapv2.DefaultRegistry) }`. Use for schema families that have a
	// PRIVATE Kind namespace (the discriminator byte is unique only
	// within a per-package registry, not globally). Examples:
	//
	//  - LP-186 chains-VM wire: each <vm>wire package has its own
	//    KindBlock=0x01 / KindTx=0x02 / etc. Registering all twelve
	//    KindBlock=0x01 implementations into a single global registry
	//    would panic on duplicate kind byte at init time.
	//  - LP-182 consensus wire: the 0x01..0x0D kind bytes are local to
	//    the consensus-wire registry (`pkg/wire/zap/schemas.go`), not
	//    the global zapv2.DefaultRegistry shared with P2P/light-client
	//    schemas at 0xD0+/0xF0+.
	//
	// Default is false — schemas with globally-unique Kind bytes
	// (LP-201, LP-208, LP-211, LP-214, LP-218) DO register at init.
	SkipRegistry bool
}

Schema is the declarative description of a ZAP v2 schema. The codegen tool consumes a Schema and emits a per-schema *.go file with the Wrap/Build/Read/Write functions hand-rolled to inline.

One schema description, one emitted file, one and only one way to access the wire format from Go code — Hickey-style.

Directories

Path Synopsis
cmd
zapgen command
Command zapgen emits per-schema ZAP v2 accessors from a declarative schema description.
Command zapgen emits per-schema ZAP v2 accessors from a declarative schema description.
zapgen-all command
Command zapgen-all bulk-emits per-schema ZAP v2 accessor files from a directory of JSON schema declarations.
Command zapgen-all bulk-emits per-schema ZAP v2 accessor files from a directory of JSON schema declarations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL