gen

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: MIT Imports: 13 Imported by: 0

README

Engine generator

The generator turns the C engine in engine/ into the Go assembly in internal/cengine. It handles the calling convention, the stack rules and the shadow stack on amd64 and arm64.

It is a maintainer tool: it shells out to LLVM 22.1.8 (clang, opt, llc) and runs its own checks and lowering in Go (debug/elf, golang.org/x/arch/x86/x86asm). Applications never run it.

Pipeline

  1. clang -O2 to IR with a kept frame pointer, no red zone, signed wrap and no tail calls. On amd64 the stack is declared 8-byte aligned, as Go's is, and the target feature slow-two-mem-ops keeps llc from folding the load of a call target into the call, so every indirect call is through a register. On arm64 the target is aarch64-linux-gnu with x18 and x26 to x28 fixed, x29 reserved and LR kept for return addresses. The portable flags (freestanding, no system headers, wrapping signed overflow, unsigned char and the others) are not written here: init runs engine/test/flags.sh, which the scripts of the C tree read too, and appends what it gives (TestFlagsComeFromFlagsSh keeps the Go lists and cycles.sh from repeating them). On amd64 quickjs.c, the one file that holds JSValues, also gets -fno-slp-vectorize and the target feature slow-unaligned-mem-16 (EngineValueFlags), so that LLVM copies a JSValue (16 bytes) as two 8-byte moves and not one SSE move: the interpreter writes the two halves one at a time, and Zen 3 cannot forward a 16-byte load from them (see c-engine.md). The small libc is left out, since its memcpy moves 16 bytes at a time on purpose. CheckIR rejects variadics, byval, sret and long double.
  2. The shadow-stack pass (pass/shadow.cpp) moves every address-taken local and dynamic alloca onto the VM's shadow stack: argument 0 of every engine function is the first free shadow byte, each function hands its callees the byte above everything it has live, and __shadow_remaining() gives C the space left for its RangeError checks. No alloca may remain. A function that needs more integer argument registers than the target allows becomes f(sp, blk): every call site stores the arguments in a block on its shadow frame. With -shadow-cc=preserve_none every unified entry, host callback and call to them uses preserve_none. Register-entry leaves stay C and get a reserved frame pointer: no frame, and llc never allocates RBP, which Go owns. Every unified function and every call to one takes and returns an i1, i8 or i16 as an i32: the caller extends it (sign or zero, as the C says, zero for a bool), the callee truncates it. preserve_none puts only 32 and 64-bit integers in its own registers (amd64 R12 on, arm64 X20 on) and leaves narrow ones to the C rules (EDI or W0 on), and Go writes whole words. The signatures the pass reports then have no narrow integer, and the generator refuses one (and a float, which Go would move as a whole float64). With -shadow-icall-sigs every indirect call gets a "kcfi" operand bundle whose type id encodes its signature (icall.go). Without the "kcfi" module flag llc emits no check code for it, but it keeps the id on the machine call. -shadow-probes writes a module with one probe per unified definition, of its type, convention and attributes, that reads every argument and returns a loaded result. No trap instruction is left: every llvm.trap (the __builtin_trap of abort, and the pass's own check on a dynamic alloca that would cross the end of the shadow stack) becomes a call to host_trap(sp), a callback of the host that panics, since SIGILL and SIGTRAP end a Go process whatever it does. LLVM 22.1.8 writes no trap for an unreachable on the two linux triples, so llvm.trap is the only source. debugtrap and ubsantrap stop the pass, and so does a trap in a leaf, which has no shadow pointer to pass.
  3. llc -O2 with shrink-wrapping off writes assembly and an object, every function in .text (-profile-guided-section-prefix=false: otherwise a cold function such as abort goes in .text.unlikely.). The whole engine loads pass/icall.cpp and pass/facts.cpp (built into the same shared object as the pass) with -load: after the machine passes this run has (the outliner and function splitting are off, and the insertion point has to be checked again if that changes) icall.cpp names a symbol at every indirect call, goasm.icall.<function>.<n>.<id>, a label right before the call in the text and a local symbol at its address in the object. An indirect call without an id stops llc. facts.cpp (-goasm-facts) writes down the registers llc assigned: each function's argument registers, the ones its code reads at the entry, and for each call and return the argument, result and live registers (facts.go). llc runs over the probe module too, for the registers of every argument and result. CheckObject rejects writable or thread-local data and any external symbol outside the engine and the host callbacks.
  4. LowerUnified / LowerRegister (amd64) and LowerARM64 rewrite each function as text: strip the prologue and epilogues, load the Go stack arguments into the native argument registers after Go's prologue, marshal each call inline (keep the registers the mode asks for, move the arguments to the outgoing area, call, read the result, restore), where a call to a unified function the driver lowers takes the result from the registers the callee stored it from and reads nothing back from the Go stack (LowerEnv.RegResult), move LLVM's locals onto the Go frame, keep constant pools inside the function, and turn every return into a jump to one Go RET. A function starts in the lowered object where its body will sit in the binary (FuncPlace: on 16 bytes, or at its -align64 place in a line of 64), so that the .p2align of each pool and table holds where the code runs. Any shape it does not know fails: unknown call forms, calls through memory, tail jumps, SP writes, escaping stack addresses, aligned or folded packed stack operands (the Go stack is only 8-byte aligned), read-only symbols other than constant pools, forbidden instructions. A leaf that uses RBP fails too.
  5. clang -c (llvm-mc on arm64) re-assembles the lowered text so every branch offset is fixed.
  6. Emit writes Go assembly: one TEXT per function, the body as bytes, every call a Go-visible CALL or BL.
  7. VerifyLinked checks the linked binary: body bytes unchanged, a stack-check prologue on every entry off the NOSPLIT list and on host_poll, none on the entries on it, the function table points only at unified entries, no hidden calls or tail jumps.

Call modes

-mode picks how a unified call keeps the C callee-saved registers, which Go code and morestack clobber:

  • body (default): the call site saves the ones the calling function's body uses (amd64 RBX, R12 to R15, arm64 R19 to R25 and F8 to F15).
  • all: the call site saves all of them (for comparison).
  • stub (arm64): the call goes through a stub that saves all 15 (the first arm64 backend, for comparison).
  • pn: the code is compiled with preserve_none, so LLVM keeps across a call only what is live, in its own frame, and no call site saves anything. Arguments then use amd64 R12 to R15, RDI, RSI, RDX, RCX, R8, R9 and arm64 X20 to X25. A seventh arm64 argument would land in X26, which the scan rejects, so the pass moves larger argument lists into a block.

What a call costs in each mode, from a loop that calls a unified entry (the least of 5 runs of 300 million calls, in ns a call). On amd64 (EPYC 7B13) body takes 2.59 for an int and 2.88 for a JSValue, all 0.29 more, and pn 0.3 to 0.6 less. On arm64 (Apple Silicon) stub takes 4.9 and all 4.6, since saving all 15 registers costs 2.6 to 2.7 more than body (1.9). There pn takes the same as body for an int, 0.16 more for a JSValue and 0.5 to 0.6 more for a double, where LLVM stores and reloads the accumulator around the call instead of keeping it in d8. The 48 unified call sites of the test unit (arm64) save 720 registers in all and stub, 233 in body and none in pn, and the lowered code of the unit is 3% smaller on amd64 and 8% on arm64 with pn than with body. pn is the mode of the engine: as fast as body or faster, and the call site saves nothing, which keeps the lowering simplest. The code position moves such a loop by about 0.3 ns, so only the numbers of one build compare.

GEN_NO_SAVES=1 with body and GEN_NO_PACK=1 with pn are negative builds that must fail the tests and the register scan.

Running

proto.sh (from the repository root, linux/amd64) builds the pass, generates internal/proto, builds its tests and runs the post-link verify. ARCH=arm64 generates the arm64 package and cross-builds its tests: copy the package to an arm64 machine to run them. MODE is the call mode. Set CALL_OVERHEAD=1 to run the call-overhead timing. The proto tests are testdata/*.txt, kept as text so this package does not compile them. go test here runs the rule tests. With GEN_LLVM (an LLVM bin dir) and GEN_SHADOW_PASS (the built pass) set, it also runs every sample in testdata/reject through the real pipeline, and with GEN_LLD (ld.lld) as well, TestNarrowArgs runs testdata/narrow natively and generated, and TestSampleRuns does the same for each sample of testdata/run: each defines long run(long *r, int one), is run natively (also under UBSan) and from Go in body, pn and pn poison builds, and must print the same lines, and the post-link check must pass on it. internal/enginecmp measures the current engine under the same GC stop setup as the proto tests.

The whole engine

cmd/engine runs the same pipeline over all of engine/ and engine/libc as one module and writes a Go package (-out): the functions in .s files bounded in size, the read-only data as engine_rodata, and the Go declarations. The host provides host_now, malloc, realloc, free, malloc_usable_size and goasmMissing, which each stub calls with its index in missingNames. A rejection stops the run and nothing is written (a rejection by the pass is one LLVM ERROR: line and exit status 1, not an abort with opt's stack), and the default has no stubs. -keep-going is for surveys: it lists every function the C must change, by class, and stubs it instead of stopping. -release is the formal build: it fails on any stub, needs -out, and refuses every switch made for surveys and tests (-keep-going, -loose-varargs, -poison, GEN_KEEP_NARROW, GEN_EDIT_LOWERED, GEN_FORCE_NOSPLIT, a changed -non-format-variadic, a -postlink-opt other than auto, changed -postlink-opt-args) before it does any work. -mode is body or pn. -varargs-block passes variable arguments in a shadow-stack block instead of rejecting them. -shadow-size-log2 is the size of the shadow-stack region the engine is made for (20 by default, 25 for the host's whole 32 MiB, and from 16 to 40: the driver and the pass refuse any other before they use it): engine_decl_<arch>.go records it as shadowSizeLog2, and the host refuses a smaller stack.

The files are compiled one at a time (step 1 above, with the format checks on the IR as written) and linked with llvm-link. On amd64 and arm64 the optimizer then runs over the linked module (-postlink-opt, default auto: PostLinkOptFor) before the shadow-stack pass, with PostLinkOptArgs: -O2, the inline threshold of 400 that engine/test/flags.sh gives clang, and the SLP vectorizer off. clang had optimized each file alone, so the calls from one file to another (the small libc, the allocator and the heap into the interpreter and the collector) were never inlined. The second run does it. It saves 1.5% to 1.7% of the instructions of a page in native C, 0.6% to 1.2% in the generated engine of amd64 and 0.6% to 1.0% in that of arm64, and makes the generated assembly 13% larger (the code 19% on amd64). A threshold of 1000 for inlining makes the interpreter bigger and its frame 32 bytes taller: not used.

The flags clang took from the command line travel in the IR as attributes of the functions (+slow-unaligned-mem-16 of quickjs.c), and stay with their functions. -fno-slp-vectorize is the exception, since it is an option of the pipeline and not an attribute: a second -O2 would join the two 8-byte moves of a JSValue into one of 16 bytes again (52 lines of vector code in the IR of JS_CallInternal). The SLP vectorizer is turned off with a threshold no tree reaches (TestPostLinkOptKeepsValueFlags).

The optimizer changes names, and each place of the generator that goes by a name reads the module as it stands after it:

  • A static function inlined into all its callers is gone (the pass reports, and the Go declarations and engine_keep list, what is in the module). An external function that is inlined stays defined and declared, since the Go side may call it.
  • It clones a function for the constant arguments of a call (mark_children.specialized.2). The clone is a unified function like any other, with the Go name mark_children_specialized_2 (GoNames), and its code goes with the file of the function it was made from (cloneGroups, cloneOf, which also take the numbers llvm-link adds). The register-entry leaves (-leaves, -shadow-leaf) are the functions of those names and nothing else: a clone of memcpy would be a unified function.
  • -align64 (JS_CallInternal, lre_exec_backtrack), DispatchCopies, the host callbacks, host_poll of the post-link check and js_host_layout name external functions and data, which the optimizer keeps under their names.

-postlink-opt=on or off forces the run and -postlink-opt-args changes its options, for tests and surveys (-release refuses both). TestPostLinkOpt (amd64 and arm64, with the post-link check over a module cross-built for each) and the postlink.c sample of TestSampleRuns (which runs the code, on linux/amd64 only) hold all of this.

-arch is amd64 (the default) or arm64, and every file the driver writes ends in the architecture (engine_arm64.s, engine_decl_arm64.go, engine_keep_arm64.go), so one folder holds both and Go picks one. A build removes only the files of its own architecture. -build-tag T puts the tag T in front of the architecture in the //go:build line of every file (T && amd64 for the .s files, T for engine_decl_<arch>.go). scripts/gen-engine.sh passes no tag: the checked-in files carry the architecture alone (amd64, arm64). The arm64 engine runs on darwin/arm64 and linux/arm64. arm64 takes -mode pn only: preserve_none passes integers in X20 to X25 there, and its next registers, X26 to X28, are Go's. The other modes stay with the prototype (proto.sh). Measured on the prototype on an Apple M-series core, a call through a stub or with all callee-saved registers saved took about 3.9 ns, body about 1 ns and pn about 1.1 ns, and pn has no saves at any call site. arm64 has no -poison or -loose-varargs build and refuses both. It lowers each function in LowerARM64EngineFunc (arm64_engine.go) and holds every entry, call and return to the registers llc's facts and probes name, as the amd64 post-link check does: a narrow integer in another register, a result read from another, or a call llc's facts do not have stops the build. llc writes arm64 in the tiny code model (-code-model=tiny), so each reference to data is one instruction that names the whole address, and the lowering rewrites each on its own to the same address: an adr of engine_rodata or of a function becomes Go's MOVD $sym(SB), an adr of the function's own constant pool or jump table stays PC-relative, and a load of a label reads the data in the function's body (the plan copies an object there when a function loads it that way, as amd64 does for a load through the PC). The page forms of the small code model, adrp and :lo12:, stop the build: their second instruction is right only while the register still holds the page of the first, which no single instruction shows. llc writes its object for the checks with a section per function, since an adr from one function to another may be further than the 1 MiB it reaches. It refuses svc, system registers, X18, X26 to X29 as values, LR outside a save, tail jumps, a frame that grows after the prologue, a stack address that escapes and fused multiply-adds (fmadd, fmla and the rest), with a test for each (TestLowerARM64EngineRejects, TestLowerARM64EngineFacts, TestLowerARM64EngineTiny).

A function whose arguments need more integer registers than a unified entry has (10 on amd64, 6 on arm64) takes the shadow pointer and the address of a block of them instead (f(sp, blk)). Its generated entry is named <name>_block, and the driver writes a Go function by its C name and signature into engine_decl_<arch>.go (PackedGo in packed.go): it stores each argument in its 8-byte slot (16 for a JSValue) of a block at the shadow pointer, as the pass lays the block out, and calls the entry with the shadow pointer above the block. So a host calls the API by the C signature on every architecture, while which functions are packed differs (20 on amd64, 205 on arm64).

The same C gives the same package, byte for byte, whatever the number of compile jobs (-j) and the order the compiles finish in, and no file holds the path of the work folder (TestOutputReproducible).

The engine polls (pass/poll.cpp, see c-engine.md): every cycle of the code counts down a counter at -poll-offset bytes from the base of the shadow-stack region (gen.PollCounterOffset, 384, which engine_decl_<arch>.go records as pollCounterOffset), and calls host_poll when it runs out. -poll=false builds an engine without polls, to measure what they cost, and -release refuses it. The interpreter counts its bytecode itself: the C marks where (poll_count in engine/cutils.h, at the entry of JS_CallInternal and where pc does not go forward), the pass puts a count at each mark, and the edges into its dispatch get none. So the dispatch stays short, and llc copies it into the end of each handler after register allocation, which it does for a computed goto of up to 10 instructions. Without the copies the interpreter is much slower. The driver counts the indirect jumps of each function with a computed goto table in llc's text and stops below gen.DispatchCopies (175 for JS_CallInternal, which has 204). The pass checks its own output: with the blocks that count or poll taken out, the back edges of the loops it proves short, and the edges into the dispatch of the interpreter, no function has a cycle left. The interpreter must count at its entry, and each of its marks must come right after a count of its weight. Each count takes at least 1, and polls whenever it runs out: its branch tests that it is 0 or less and calls host_poll on the side taken. Each count takes from the count as it stands, each store to the counter stores the count as it stands, and each call that may poll and each return find the count in memory: code that keeps it in a register stores it before such a call and loads it after, and stores it where it is left. The hidden option -shadow-poll-fault breaks one of these on purpose, so that a test can see the check stop the pass.

Functions without a stack check: every unified entry starts with Go's stack check unless its TEXT says NOSPLIT, and the driver chooses which ones do (ChooseNoSplit, nosplit.go). The list holds the leaves the Go assembler marks NOSPLIT by itself (no call and a frame under 128 bytes), the register-entry leaves, and the functions it chooses on top of those: a function whose address is not taken, that makes no call through a pointer, that is on no cycle of direct calls and whose chain of NOSPLIT calls fits the room the linker gives one (792 bytes on amd64 and on arm64, counted by AMD64Stack and ARM64Stack, which differ in the frame each call adds). Every recursion then goes through a stack check, which is where the stack grows and where the runtime can stop the goroutine, and every loop still calls host_poll, whose own stack check is the safe point. The room is counted as the linker counts it, with 64 bytes more below the deepest frame of each chain (FaultRoom): a fault there becomes a call of runtime.sigpanic pushed onto the goroutine's stack, and sigpanic's stack check may call morestack, which needs 16 bytes above the bottom of the stack and ends the process with "split stack overflow" when they are not there. A function also leaves room above it for the ABI wrapper of a Go func value of it (its arguments and 16 bytes). The linker checks the chains again on every build and fails it when one does not fit. The driver prints the size of the list, writes each function on it with its frame and height to nosplit.txt in the work folder, and engine_decl_<arch>.go names them in noSplitList, for the tests. The post-link check holds each entry's stack check to the list.

Where a body starts: after Go's prologue each function has a PCALIGN $16, so the body starts on 16 bytes and moves within its lines of 64 bytes when code before it grows. -align64 name:offset,... puts the body of each named function offset bytes past a multiple of 64 instead (a PCALIGN $64, which also puts the function on 64 bytes, then offset bytes of 8-byte no-ops, which run once per call). The default of the amd64 engine is JS_CallInternal:40,lre_exec_backtrack:16, the fastest places measured (see performance.md). arm64 and a -src build have none unless the flag is given. The post-link check holds each body to its place and the padding before it to no-ops.

Three negative builds test the tests. -poison writes garbage into every register at each entry before the arguments are loaded, after each unified call before the result is read and at each return: the engine's tests must still pass, so no code reads a register the lowering did not set. A poison build reads every result back from the Go stack, since its returns write over the result registers. GEN_KEEP_NARROW=1 leaves narrow integers as the generator once took them: the post-link check must reject the result (TestNarrowArgs). Only the value 1 turns it on, and any other value but an empty one is an error. The package of such a build is marked: engine_negative_<arch>.go does not compile without -tags goasmnegative, and engine_decl_<arch>.go says so. A later build into the same folder removes the file. GEN_FORCE_NOSPLIT=f,g puts those functions on the NOSPLIT list whatever the rules say, and marks the package the same way: the linker must refuse a recursion or a chain too deep for the room (TestSampleRuns/nosplit_fault).

go build -o /tmp/engine-gen ./internal/goasm/gen/cmd/engine
L=/usr/lib/llvm-22/bin P=internal/goasm/gen/pass
$L/clang++ -shared -fPIC -O1 $($L/llvm-config --cxxflags) $P/shadow.cpp $P/poll.cpp $P/icall.cpp $P/facts.cpp \
    -o internal/goasm/gen/internal/_shadow.so
/tmp/engine-gen -engine engine -pass internal/goasm/gen/internal/_shadow.so \
    -keep-going -report /tmp/rej.json -out /tmp/cengine/cengine

Before the pass, the C is also compiled without optimization and every call that reaches engine/libc's formatter (snprintf, vsnprintf and any variadic function but code_match and unicode_prop_ops) is held to the formats stdio.c takes (formats.go, CheckFormats). A format must be a string constant FormatError accepts, or a parameter that a wrapper forwards (a variadic function's last fixed parameter, or JS_ThrowError2). Anything else stops the build, so a %f cannot reach abort() at run time. The accept table mirrors stdio.c and testdata/formats/cases.txt lists the formats both must agree on. engine/test/libc-test.c feeds that list to vsnprintf, so that stdio.c and the table cannot drift apart.

The post-link check is the gate for the output. Build a binary of the package with -tags goasmkeep (engine_keep_<arch>.go then keeps every function) and pass it as -verify-bin, with the package's import path as -verify-pkg. The check reads llc's own object file and the binary, decodes both with x/arch, and matches every generated function instruction by instruction. It refuses code that would not run as decoded: a prefix the processor rejects or ignores, an unknown or cut-short instruction, a no-op other than the assemblers' padding. The only differences it allows are the rewrites the lowering wrote down for that function, and it works out what each must be from llc's object: the prologue and epilogue it replaced (llc's pushes and reserved locals), where each frame offset moves and that the access stays inside the Go frame, each call's saves, argument moves, target and result, and where each data reference points. The registers in those moves are llc's, not the generator's model of the convention: each entry loads every argument word into the register llc's probe reads it from, and llc's code may read no other register at the entry before writing it but the ones its dropped prologue pushes. Each call stores every argument word from the register llc passes it in (for a call of a unified definition, the one the definition reads it from) and loads each result word into the register llc takes it from, and llc's code may read no register after the call before writing it but those results and the saved registers. A call whose record says it takes the result from registers (RegResult) has no result loads: its target must be a lowered unified function, not a stub, a leaf or Go code, and the registers llc takes the result from must be the ones the callee's probe returns it in. Each return stores the registers llc returns and jumps to Go's epilogue, which touches only BP and SP, so those registers still hold the result after the RET. A leaf keeps C's registers, so of its facts only its calls' targets and the number of its calls and returns are checked. llc's own code must keep to what the build rules out: a call through memory fails, and so does a frame access wider than 8 bytes that is not an unaligned move. Each function's record must agree with the pass's signature for it, its list of leaves and its calling convention. Each indirect call must carry the symbol LLVM put at it in llc's object, of that function, and the signature it names must be the one the lowering marshalled; a symbol anywhere else fails. It compares engine_rodata and the data each function carries byte by byte, pointers by what they point to. It also holds each pool and table to its alignment at the address where it sits in the binary. -verify-all also fails when a function was not in the binary. GEN_EDIT_LOWERED edits the lowered text before it is assembled, for the check's own tests (TestPostLinkCheck, from C through the driver, and TestCheckCases, from hand-written llc code).

What the check takes on trust, and where it stops:

  • The pass's signatures, for the number of argument and result words only: the one it reports for each function, and the type id it gives each indirect call from the call's own IR type. That LLVM keeps the id on the right call is LLVM's (instruction selection copies it to the machine call and tail merging keeps calls of different ids apart), and the check reads it from llc's object, not from the text the lowering read.
  • llc's facts: that a probe gets the registers of the function it stands for (llc assigns them from the type, convention and attributes, which the probe copies), and that liveness as LLVM's LivePhysRegs computes it after the last machine pass is what the code reads. The facts of a call are matched to the calls of llc's object in order, and each names its target or type id, which the check compares.
  • The width of a narrow integer: the registers do not say which bits a callee relies on. The pass's widening makes every one a whole i32, and TestNarrowArgs runs it with garbage above each narrow value.
  • A Go frame of 4096 bytes or more has a stack check the check does not know, so such a function fails it.
  • Only a non-PIE linux binary: linux/amd64, or linux/arm64 for the arm64 engine. darwin/arm64 links the same code, so the linux/arm64 check covers it.

On arm64 the check (CheckEngineARM64, arm64_verify.go) is narrower. It holds each generated function in the linked binary to the object llvm-mc made of the lowered text: Go's prologue (a stack check exactly when the function is off the NOSPLIT list, whose load of the guard and branch to morestack it reads but not the compare between them, and the frame the TEXT asks for, below 4096 bytes or a multiple of 4096), the loads of the arguments, the no-ops before the body, the body word for word with each BL and ADRP/ADD at the target the object names, the stores of the result, Go's epilogue, one RET and the call of morestack. Each call in the body is the next one of the record, and it loads its result back from the Go stack unless the record says it takes it from x0, x1 or d0, which only a call to a lowered unified function may: the check holds each of those to storing its result from them and then running only Go's epilogue (x29, x30 and SP). A direct call must agree with the pass on whether its target is a leaf and, when it is not, on its signature, so no record can skip a load by calling a unified function a leaf or one that returns nothing. It compares engine_rodata byte for byte, pointers by what they point to. It does not read llc's own object: that the lowering kept the meaning of llc's code rests on the lowering's rules and llc's register facts, which the lowering checks as it goes. TestCheckARM64Function changes each part of a function and requires the check to refuse it.

Documentation

Overview

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Package gen is the phase-0 prototype of the new C-source engine generator. It is architecture independent in structure: an Arch describes the Go frame and the native calling convention, and the lowering and checks are written against that interface so arm64 can be added without touching the pipeline.

This file holds the shared vocabulary: the two function entry kinds, the value types the engine passes between functions, and the amd64 Arch.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

arm64 backend. Lowering edits llc's aarch64 assembly as text, llvm-mc re-assembles it, and emit writes Go assembly: the body as WORDs, every call a symbolic BL, the arguments loaded from FP after Go's prologue, one Go RET and the function table.

Frame: 0(RSP) is Go's saved LR, the outgoing ABI0 area starts at 8(RSP), the LLVM frame of F bytes sits at the next 16-byte boundary L, call-site save slots follow it, and the TEXT declares $(L+F+saves-8). Every SP move of LLVM's prologue and epilogues is dropped and each SP-relative access is rebased on the Go frame, tracking how much of the frame is allocated at that point, so any prologue shape and frames over 4096 bytes lower the same way.

A C call to a unified entry or a Go callback stores the argument registers into the outgoing area, calls the Go symbol and loads the result back, all inline at the call site. Go code and morestack clobber R19-R25 and F8-F15, which AAPCS code expects preserved. ARM64Mode picks how that is handled.

The arm64 lowering of the whole engine. It takes llc's aarch64 text of one function (preserve_none on every unified entry, the -mode pn of cmd/engine) and rewrites it as text for llvm-mc, as LowerARM64 does for the units of the prototype, plus what the engine has and the prototype does not: data whose address the code takes, jump tables and the computed goto of the interpreter, the label llc writes at each indirect call, and calls of the register-entry leaves (memcpy, memmove, memset).

The Go frame of a function with a frame: 0(RSP) is the LR Go saves, the outgoing ABI0 area starts at 8(RSP), and the LLVM frame of F bytes sits at the next 16-byte boundary L. The TEXT declares $(L+F-8), so that with the LR it is L+F, a multiple of 16, and the Go assembler adds 16 bytes on top for the frame pointer its callees save there. Every SP move of LLVM's prologue and epilogues is dropped, and every access [sp, #k] is rebased on the Go frame.

Data. llc writes the engine in the tiny code model, where every reference to data is one instruction that names the whole address: an adr of data or of a function, or a load of a label. Go assembly cannot attach a relocation to the bytes of an instruction, so the lowering writes each one on its own as an instruction that reaches the same address: an adr of other data or of a function becomes MOVD $sym(SB) in Go (an adrp and an add of the whole address), an adr of the function's own data (constant pools, jump tables, the table of the computed goto) stays an adr of it in the body, and a load of a label loads the data the plan put in the body, the function's own or a copy. No rewrite depends on any other instruction, so there is nothing to follow through the blocks. The page forms of the small code model (adrp and :lo12:), where the second instruction is right only while the register still holds the page of the first, are refused.

The post-link check of the arm64 engine. It reads a linked linux/arm64 binary of the generated package and holds every generated function to the object llvm-mc made of the lowered text, which the emit wrote as WORDs:

  • Go's prologue: a stack check exactly when the function is not on the NOSPLIT list, the frame the TEXT asks for (with the saved LR and frame pointer as cmd/internal/obj/arm64 rounds it, ARM64Stack.Local), and nothing else,
  • the loads of the arguments from the Go argument area into the registers of preserve_none, then no-ops up to where the body starts (16 bytes, or the line of 64 of -align64),
  • the body word for word: each BL the emit wrote goes to the function the object's call names, each ADRP and ADD to engine_rodata plus the object's offset or to the function it names, an indirect call is the same BLR, every other word and the function's own data are as in the object,
  • the stores of the result, Go's epilogue and one RET, and after it only the call of morestack of the stack check,

and engine_rodata byte for byte with each pointer at the function or data it names. It does not take llc's object: that the lowering kept the meaning of llc's code is what the lowering's own rules and llc's register facts hold (LowerARM64EngineFunc).

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Static checks. Each rule the plan lists is enforced mechanically and has a small example it must reject (checks_test.go). The lowering enforces the instruction-level rules (forbidden instructions, SP writes, escaping stack addresses, frame spills, indirect jumps); these functions enforce the rules that read the LLVM IR and the linked/assembled object.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Emit stage: read the re-assembled object with debug/elf, disassemble each function with x/arch/x86/x86asm, and write it as a Go assembly TEXT. Every call becomes a Go-visible CALL (direct ones symbolic through a relocation, indirect ones as CALL <reg>), so Go knows the frame, traceback and stack bookkeeping; all other instructions pass through as machine-code bytes. This is the only place the generated code enters the module, and verify.go checks the linker left those bytes unchanged.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Emit for the whole engine: the functions go into several .s files (one or more per C source file, each under a size bound), engine_rodata into its own file as Go DATA, and every function the generator could not lower becomes a stub that reports its name to Go.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

What llc itself decided about registers, as the plugin in pass/facts.cpp writes it (-goasm-facts). The post-link check holds the lowering to it instead of to the generator's own model of the calling convention, so a mistake in the model (a type it moves in the wrong register) shows as a difference instead of being checked against itself.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

The instructions engine code may not hold: the ones that call the system or read what the machine says (the time stamp counter, the hardware random source, the processor's identity), and the traps. The plan's list is syscall, sysenter, int 0x80, svc, rdtsc, rdtscp, rdrand, rdseed and cpuid. A trap (ud2, int3, hlt, brk, udf) is SIGILL, SIGTRAP or a general protection fault, and in Go SIGILL and SIGTRAP end the process whatever the program does, so engine code reaches host_trap instead (see pass/shadow.cpp). The lowering reads the assembly text for them and the post-link check the decoded machine code (verify_insn.go), so a mistake in one of the two readings does not let one through. arm64 has no post-link check yet: EmitARM64 reads the words of the assembled object.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

The signature LLVM binds to each indirect call. The pass gives the call a "kcfi" operand bundle whose type id encodes its signature code (-shadow-icall-sigs), llc keeps the id on the machine call, and the plugin (pass/icall.cpp) names a symbol at the call: goasm.icall.<function>.<n>.<id>. It shows as a label right before the call in llc's text and as a local symbol at the call's address in llc's object.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Lowering stage: turn the canonical LLVM GNU assembly of each engine function into a form Go's stack rules accept, as text, so re-assembly fixes every branch offset. Only external calls stay symbolic for elf2go.

The transform per unified function:

  • strip the canonical prologue (push BP; mov SP,BP; push <callee-saved>; reserve locals) and the single epilogue, recording the frame and the callee-saved set the body uses;
  • prepend an entry shim that loads each incoming argument from the Go argument area into its native argument register, after Go's prologue;
  • at each internal call, save the live callee-saved registers to the frame (the split callee's morestack does not preserve them), move the native argument registers into the outgoing argument area, call, move the result back, and restore;
  • replace every return with a jump to a common epilogue that stores the result into the Go result area and falls into Go's RET.

It rejects any shape it was not built to prove: a non-canonical prologue, a body that still touches BP (an unhandled frame spill), an SP write, an escaping stack address, a forbidden instruction or an unreviewed indirect jump. Failing loudly is the plan's rule: the generator never emits code it cannot account for.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Register-entry leaf lowering. A leaf neither grows the stack nor calls a split-checked function, so it is emitted NOSPLIT with its arguments left in the native argument registers and its result in the native result register. Only a direct call from a known caller may target it; the function table and indirect calls use the leaf's unified entry instead.

Functions without Go's stack check. Every unified entry starts with the stack check of the Go assembler, three instructions that compare the stack pointer with the guard in g and call morestack below it. A function whose TEXT says NOSPLIT has none: the Go documentation of the assembler gives it the room left above the guard, and the linker adds up every chain of NOSPLIT calls and fails the build when one needs more than that room (cmd/link/internal/ld/stackcheck.go: 800 bytes less one CALL on amd64, "nosplit stack over 792 byte limit"). The Go assembler marks a function NOSPLIT on its own when it calls nothing and its frame with the saved base pointer is under 128 bytes.

PickNoSplit chooses the functions whose TEXT says NOSPLIT. The list holds the leaves the assembler would mark anyway, named so that the post-link check knows every function without a stack check, the register-entry leaves, which are NOSPLIT by their form, and the functions it chooses on top of those. A chosen function

  • has no address taken, so that no call through a pointer reaches it: the linker takes the target of an indirect call for a function with a stack check and counts only the CALL,
  • makes no call through a pointer, so that every function a chain of NOSPLIT calls reaches is one the linker counts (an automatic NOSPLIT leaf whose address is taken is then reached only from functions with a stack check),
  • is on no cycle of direct calls, so that a recursion always goes through a stack check, which grows the stack and is where the Go runtime can stop the goroutine,
  • and fits the room, with what a fault and a Go caller add to it (below).

A chain of NOSPLIT calls starts below a function with a stack check, which leaves at least 800 bytes above the bottom of the goroutine's stack once its frame is made (internal/abi.StackNosplitBase, which the runtime's guard adds to the 128 bytes of abi.StackSmall). The room a chain may take is counted as the linker counts it: each frame with its saved base pointer, 8 bytes for each CALL, 8 for the CALL of morestack of a function with a stack check that a chain calls, and the height of each NOSPLIT callee. Two things the linker does not count are added:

  • A fault in a NOSPLIT function (a wild pointer of the C) becomes a call of runtime.sigpanic that the signal handler pushes onto the goroutine's stack where the fault is (runtime/signal_amd64.go, pushCall), and sigpanic has a stack check that may call morestack (runtime/signal_unix.go: it must not be nosplit "because the linker doesn't know where sigpanic calls can be injected"). morestack needs those two return addresses, 16 bytes, to fit above the bottom of the stack: newstack throws "split stack overflow", which ends the process, when they do not. FaultRoom is kept below the deepest frame of every chain for this.
  • A Go func value of an assembly function calls it through an ABI wrapper that the compiler marks NOSPLIT (cmd/compile/internal/ssagen/ abi.go), with a frame for the function's arguments: the linker counts the arguments, the saved base pointer and the CALL above the function, ArgBytes + 16 (a goasmkeep build takes the value of every function). Each chosen function leaves that room above it.

Go host callbacks are reached through such a wrapper too, the other way round, and are counted with CallbackHeight (the linker counted at most 72 bytes for those of internal/cengine: the wrapper of host_call).

The choice goes from the callees up: a function is taken when its chain fits with the callees taken so far, and taking it changes only the functions that call it, which come later. Nothing depends on how often a function is called, so the same C gives the same list.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Whole-program input: the engine's C compiled as one module. This file reads what the shadow-stack pass reports (the signature of every function, the address-taken ones, the call graph) and splits llc's assembly into functions and data objects, so the lowering can look a function or a data label up instead of scanning the whole text each time.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

The rewrites the lowering makes, written down per function as data. The post-link check (verify_insn.go) accepts a difference between llc's object and the linked code only where one of these declares it, and checks the declared rewrite against llc's object itself.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Read-only data of the whole engine. Go assembly cannot attach a relocation to the raw bytes of an instruction, so each data object of llc's text goes one of three ways:

  • into the body of the function that uses it, after its code, where the re-assembly resolves every PC-relative reference: constant pools, jump tables and the computed goto table (which hold offsets of the function's own labels), and a copy of each table a function reads with an instruction other than LEA (such as musl's exp data);
  • into one Go data symbol, engine_rodata, for everything whose address a function takes with LEA or another object holds: strings, function tables, Unicode tables. The lowering turns each LEA of it into a LEA of __goasm_rodata+offset, which emit writes as a Go LEAQ of ·engine_rodata. Pointers inside it become Go DATA relocations, and offsets between two of its objects are resolved here;
  • nowhere, if nothing uses it.

Writable globals go into engine_rodata as well, reached only through the GOT (see writable.go), and the symbol is then writable.

Anything else is an error: a direct reference to writable data, an absolute 32-bit address, a pointer in data that must sit inside a function, a non-LEA reference to data that holds pointers.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Post-link verification. After the Go linker has placed the generated code in a program, confirm it is still the code the generator produced: every body byte survived except the four displacement bytes of each symbolic CALL, each split-checked unified entry kept its stack-check prologue, the function table points only at unified entries, and the body holds no hidden call or cross-function tail jump.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT

Instruction-level post-link check of the whole engine. The lowering reads llc's text with a hand-written parser and rewrites it; a parser mistake that still assembles (a misread operand, a dropped line, a wrong frame offset or argument move) would reach the binary without any byte check noticing, since the generator's own re-assembly is then the reference. This check takes llc's object file instead, which llc writes from its own instructions without any text, disassembles it and the generated functions in the linked Go binary with x/arch, and matches them instruction by instruction. A difference is accepted only where the lowering declared a rewrite (FuncRecord) and the check confirms the rewrite's form itself: the prologue and epilogue it replaced, the frame mapping applied to each displacement, each call's saves, argument moves, call target and result, and each data reference resolving to the same llc label plus offset. Every other llc instruction must appear in order with the same meaning. The read- only data is checked the same way: every byte and every relocated pointer of engine_rodata in the binary, and every constant pool and jump table a function carries, against llc's object. The signatures are the pass's: EngineCheck.Sigs for each function and direct call. For an indirect call LLVM itself binds the signature to the call (a kcfi type id the pass sets, which llc keeps on the machine call and the plugin in pass/icall.cpp names in a symbol at it), and the check reads it from that symbol in llc's object. A signature gives only the number of argument and result words: which register holds each word is what llc decided (EngineCheck.Probes and Facts), and the check holds every entry load, argument move, result move and the registers llc's code reads at an entry and after a call to it.

Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT Use of this source code is governed by the MIT license in LICENSE at the root of this repository.

Writable data. SQLite keeps state in writable globals (its configuration, the state of its allocator and page cache, the tables of built-in functions it links together at start), shared by the whole process as a Go global is. They go into the one Go data symbol with the read-only data (DataPlan), which is then written without RODATA.

Go assembly cannot attach a relocation to the bytes of an instruction, so the lowering must rewrite every reference to data, and it can only rewrite one that takes an address. A direct load of a global (an arm64 ldr of a label, an amd64 movq sym(%rip)) would have to become a copy in the body, which is wrong for data that changes. So before llc runs, ExposeWritableData makes every writable global external and preemptible: position-independent code then reaches it only through the GOT, with one load of its address into a register (arm64 ldr x, :got:sym, amd64 movq sym@GOTPCREL(%rip), %reg), which the lowering turns into the address of the global in the data symbol.

Index

Constants

View Source
const (
	RodataSym = "__goasm_rodata"
	RodataGo  = "engine_rodata"
)

RodataSym is the symbol the lowering gives engine_rodata in its text, and RodataGo the Go symbol emit writes for it.

View Source
const (
	// NoSplitLimit is the room the linker gives a chain of NOSPLIT calls on
	// amd64: StackNosplit (800) less the CALL into the chain.
	NoSplitLimit = 792
	// FaultRoom is kept below the deepest frame of a chosen chain for the
	// call of sigpanic that a fault adds, and its call of morestack: 16
	// bytes, and as much again four times over in case a runtime needs more.
	FaultRoom = 64
	// CallbackHeight is what a call of a Go host callback adds below the
	// CALL: its ABI wrapper and the CALL of morestack of the Go function.
	CallbackHeight = 128
)
View Source
const (
	MinShadowSizeLog2 = 16
	MaxShadowSizeLog2 = 40
)

The shadow-stack region the engine is generated for is 2^n bytes, n from MinShadowSizeLog2 (64 KiB: the margin the C keeps free, and the smallest stack the host maps) to MaxShadowSizeLog2 (1 TiB, far inside the address space). pass/shadow.cpp holds the same two numbers.

View Source
const HostLayoutSymbol = "js_host_layout"

HostLayoutSymbol is the table of int64 that engine/quickjs.c defines with the layout of values a host reads and writes in place (JS_HOST_LAYOUT_* in engine/quickjs.h).

View Source
const HostPrefix = "host_"

HostPrefix starts the name of every Go callback the engine's C may call. They are declared in engine/host.h and written in Go in the package of the generated code, which the linker resolves them to.

View Source
const ICallPrefix = "goasm.icall."

ICallPrefix starts the name of every indirect-call symbol.

View Source
const ICallUnencodable = 0xffffffff

ICallUnencodable is the id the pass gives a signature code the id cannot hold.

View Source
const PackedEntry = "_block"

PackedEntry is the suffix of the Go name of a packed function's generated entry, whose plain name goes to the Go function PackedGo writes.

View Source
const PackedSlot = `` /* 164-byte string literal not displayed */

PackedSlot is the helper PackedGo's functions store through: a uintptr that points outside the Go heap made a pointer without a conversion vet would question.

View Source
const PollCounterOffset = 384

PollCounterOffset is where the poll counter is by default, from the base of the shadow-stack region (pass/poll.cpp, -poll-offset of cmd/engine). The host keeps the start of the region for what it passes the engine by address, and this word among it.

View Source
const ProbePrefix = "goasm.probe."

ProbePrefix starts the name of the probe of a unified definition.

Variables

View Source
var (
	EngineCFlags []string
	ARM64CFlags  []string
	FlagsErr     error
)

EngineCFlags and ARM64CFlags are the clang flags of engine code for the two targets: the generator's own (engineOnly, arm64Only) and the portable ones of engine/test/flags.sh. If sh could not read that file they hold the generator's part only and FlagsErr says why: CompileUnit and the drivers stop on it.

View Source
var AMD64Stack = StackRules{Arch: "amd64", Limit: 792, Call: 8,
	Wrapper: func(a int) int { return a + 16 },
	Local: func(frame int, calls bool) int {
		if frame == 0 && !calls {
			return 0
		}
		return frame + 8
	}}

AMD64Stack: the linker gives 800 bytes less one CALL (cmd/link/internal/ ld/stackcheck.go), a frame takes the saved base pointer on top, and the wrapper its arguments, the base pointer and its CALL.

View Source
var ARM64Stack = StackRules{Arch: "arm64", Limit: 792, Call: 0,
	Wrapper: func(a int) int { return a + 24 },
	Local: func(frame int, calls bool) int {
		if frame == 0 && !calls {
			return 0
		}
		s := frame + 8
		if s%16 == 8 {
			return s + 8
		}
		return s + 16
	}}

ARM64Stack: the linker gives 800 bytes less 8 for the frame pointer a callee saves below its SP, a call pushes nothing, and a frame is the TEXT size with the saved LR rounded up as cmd/internal/obj/arm64/obj7.go does: 8 more bytes when that is 8 past a multiple of 16, else 16, for the frame pointer. A leaf with no frame gets no frame at all (NOFRAME). The wrapper of a func value has its arguments, the LR and the frame pointer, rounded the same way: at most ArgBytes + 24.

View Source
var DispatchCopies = map[string]int{"JS_CallInternal": 175}

DispatchCopies is the least number of indirect jumps each function with a computed goto table must have in llc's output: one at the end of each handler, where llc copied the dispatch, without which the interpreter is much slower. A function with a table and no entry here fails, so that a new one gets a number of its own. llc makes the copies after register allocation, where it copies a computed goto of up to 10 instructions (before, the phis of the dispatch hold it back): the interpreter counts its bytecode in its handlers (pass/poll.cpp), so its dispatch has no tick and is 4 instructions there, with polls or without. JS_CallInternal has 204 on amd64 and 208 on arm64, and its least number is about a seventh below.

View Source
var EngineValueFlags = []string{"-fno-slp-vectorize", "-Xclang", "-target-feature", "-Xclang", "+slow-unaligned-mem-16"}

EngineValueFlags are the x86-64 flags of quickjs.c, the one engine file that holds JSValues, on top of EngineCFlags. A JSValue is 16 bytes. The interpreter writes one to the value stack as two 8-byte stores (the value is in a register and the tag is a constant), and LLVM copies one with a single 16-byte SSE move: the SLP vectorizer joins two 8-byte moves into it, and the expansion of the memcpy of a struct copy makes it. On Zen 3 a 16-byte load cannot be forwarded from two 8-byte stores (it waits for both to reach the cache), and 8-byte loads of a 16-byte store are slow to forward. So the file is built without the SLP vectorizer and with +slow-unaligned-mem-16, which makes LLVM expand a copy of an object aligned to 8 bytes into 8-byte moves. The feature needs the SLP vectorizer off: with it on, the feature made the regexp scripts about 3% slower. The small libc is left out, since its memcpy and memmove move 16 bytes at a time on purpose. arm64 is not given them.

View Source
var GeneratorCallbacks = []string{"host_poll", "host_trap"}

GeneratorCallbacks are the Go callbacks the shadow-stack pass itself puts into the engine: host_poll where the code polls, host_trap for a trap instruction (see pass/shadow.cpp).

View Source
var HostLayoutNames = []string{
	"layoutRefCountOffset",
	"layoutStringLenMask",
	"layoutStringWideBit",
	"layoutStringChars",
	"layoutStringLenMax",
	"atomEmptyString",
	"atomFileName",
	"atomLineNumber",
	"atomColumnNumber",
	"atomMessage",
	"atomStack",
	"atomName",
	"atomToString",
	"atomThen",
}

HostLayoutNames are the Go names of the entries of the table, in the order of JS_HOST_LAYOUT_* in engine/quickjs.h.

View Source
var KeepNarrowSigs bool

KeepNarrowSigs makes SigFromCode take a narrow integer for a word: the driver sets it for the negative build of GEN_KEEP_NARROW (see cmd/engine), and a test of that build sets it too. A normal run leaves it off.

View Source
var PostLinkOptArgs = []string{"-passes=default<O2>", "-inline-threshold=400", "-slp-threshold=1000000"}

PostLinkOptArgs are the options of the optimizer run over the linked module of the whole engine (the driver, cmd/engine), after the files are linked and before the shadow-stack pass. Each file was optimized alone by clang -O2, so this second -O2 sees the calls from one file to another (libc, tlsf and heap into the interpreter, the collector) and inlines the small functions it could not, which saves 1.5% to 1.7% of the instructions of a page. The inline threshold is the 400 of engine/test/flags.sh: the default 225 saves a little less, and 1000 makes the interpreter bigger and its frame 32 bytes taller.

The SLP vectorizer is off. -fno-slp-vectorize of EngineValueFlags is a clang option, not an attribute of the functions, so the optimizer cannot see that quickjs.c asked for it: with SLP on, a second -O2 joins the two 8-byte moves of a JSValue into one 16-byte move again (52 vector operations in JS_CallInternal), which EngineValueFlags exists to prevent. A threshold no tree can reach turns it off. +slow-unaligned-mem-16 and the other target options are attributes, so they stay with their file: quickjs.c and the small libc differ in them, and LLVM inlines a callee into a caller whose features include its own.

Functions

func AddCallee

func AddCallee(name string, sig Sig, kind Kind)

AddCallee registers one external callee (for example a host callback) that is referenced but not generated here.

func BodyDataAlign

func BodyDataAlign(align64 map[string]int, name string) int

BodyDataAlign is the alignment, as a power of 2, that FuncPlace keeps for the data in the body of function name: 16 bytes for a plain body, 64 for one put at its Align64 place.

func BuildExpr

func BuildExpr(tag, arch string) string

BuildExpr is the constraint of a generated file for arch: the architecture, and with a tag set, the tag as well.

func CheckBodyDataAlign

func CheckBodyDataAlign(plan *DataPlan, metas []Meta, align64 map[string]int) error

CheckBodyDataAlign fails when an object in the body of a lowered function asks for more alignment than BodyDataAlign keeps. Such an object would sit on its alignment in one link and off it in another, and only the post-link check of a build would see it, so the generator stops here.

func CheckDispatchCopies

func CheckDispatchCopies(name string, lines []string) (int, error)

CheckDispatchCopies counts the indirect jumps in the llc text of a function that has a computed goto table and fails when there are fewer than DispatchCopies asks.

func CheckIR

func CheckIR(ll string) error

CheckIR rejects IR shapes the ABI cannot carry across a movable Go stack: variadic functions, byval or sret aggregates, and long double.

func CheckObject

func CheckObject(objPath string, allow map[string]bool) error

CheckObject rejects a writable data section with content and any external relocation to a symbol outside the closed allow list (the engine's own functions plus the named host callbacks).

func CheckObjectData

func CheckObjectData(objPath string, allow map[string]bool, writable bool) error

CheckObjectData is CheckObject, which with writable takes writable data sections, and references to the globals in them, as the whole-engine driver does once ExposeWritableData has run (see writable.go). Thread-local data stays an error.

func CheckPollOffset

func CheckPollOffset(offset, sizeLog2 int) error

CheckPollOffset checks -poll-offset: a word inside the region.

func CheckShadowSizeLog2

func CheckShadowSizeLog2(n int) error

CheckShadowSizeLog2 refuses a -shadow-size-log2 outside that range, before any tool is called with it.

func ChooseNoSplit

func ChooseNoSplit(metas, stubs []Meta, addrTaken map[string]bool, plan *DataPlan, unit *AsmUnit, callbacks map[string]bool, force []string) (*NoSplitPick, []NoSplitFunc, error)

ChooseNoSplit runs PickNoSplit on the lowered functions and the stubs and marks the chosen ones in their Meta and its record, and returns what it read of them. A function's address is taken when addrTaken (the pass's report) says so, when code loads it and when engine_rodata holds it. The calls of a function name their targets as llc's text does, an alias (unit.Sets) going to the function it names.

func ChooseNoSplitFor

func ChooseNoSplitFor(r StackRules, metas, stubs []Meta, addrTaken map[string]bool, plan *DataPlan, unit *AsmUnit, callbacks map[string]bool, force []string) (*NoSplitPick, []NoSplitFunc, error)

ChooseNoSplitFor is ChooseNoSplit with the numbers of an architecture.

func Emit

func Emit(objPath string, metas []Meta, names NameMap, buildTag string) (string, error)

Emit writes the Go assembly for the functions described by metas.

func EmitARM64

func EmitARM64(objPath string, u *ARM64Unit, buildTag string) (string, error)

EmitARM64 writes the Go assembly of the lowered unit from its object.

func EmitARM64Engine

func EmitARM64Engine(objPath string, e EngineEmit) (map[string]string, error)

EmitARM64Engine writes the Go assembly of the lowered engine from the object llvm-mc made of the lowered text: one TEXT per function, the body as WORDs, every call a BL or CALL (R9) Go sees, each address of data or of a function a MOVD $sym(SB) (which the Go assembler writes as the adrp and add it stands for), the arguments loaded from the Go argument area after Go's prologue and the result stored before its RET.

func EmitEngine

func EmitEngine(objPath string, e EngineEmit) (map[string]string, error)

EmitEngine returns the generated .s files by name.

func EmitRodata

func EmitRodata(p *DataPlan, names NameMap, header string) string

EmitRodata writes engine_rodata as Go DATA: the bytes, the pointers to functions and to itself as relocations. Zero words are left to GLOBL.

func ExposeCopiedPointerTables

func ExposeCopiedPointerTables(ll string) (string, []string, error)

ExposeCopiedPointerTables does for constants what ExposeWritableData does for writable globals, for the constants that hold pointers and that a memcpy or memmove copies. Such a copy is how C initializes a local array or struct (FTS5's table of built-in tokenizers), and llc expands it into direct loads of the constant (an arm64 ldr q of a label, an amd64 movups sym(%rip)). The lowering cannot copy the constant into the body, as its pointers need relocations. Reached through the GOT, it is read through a register that holds its address in engine_rodata. Every other constant is left alone. It returns the new module and the names it changed.

func ExposeWritableData

func ExposeWritableData(ll string) (string, []string, error)

ExposeWritableData makes every writable global defined in the module ll external and not dso_local, so that llc reaches it through the GOT. It returns the new module and the names of the globals it changed. A constant, a declaration (external) and LLVM's own globals (llvm.*) are left alone. Any other linkage or mark (common, weak, hidden, thread_local, ...) is an error: it has not been reviewed.

func FuncPlace

func FuncPlace(align64 map[string]int, name string) string

FuncPlace is the assembly that puts function name in the lowered object where its body will sit in the binary, and ends with the line before its label. A body goes on a multiple of 16 bytes, or at its Align64 offset in a line of 64. The lowered text aligns each constant pool and jump table of a function with a .p2align, counted from the start of the function in the object, and the bytes are copied as they are. So the function has to start in the object at the place its body takes in the binary: then the alignment of an object holds where the code runs, and not only from the start of the body, up to 16 bytes for a plain body and up to 64 for an Align64 one. An object that asks for more is refused by CheckBodyDataAlign when the engine is generated. An SSE instruction with a pool for its memory operand faults when the pool is not on 16 bytes.

func FuncSizes

func FuncSizes(objPath string) (map[string]int, error)

FuncSizes returns the byte size of every function symbol of an ELF object.

func Functions

func Functions(asm string) []string

Functions lists the global functions an assembly file defines.

func HostLayout

func HostLayout(u *AsmUnit) ([]int64, error)

HostLayout reads the table HostLayoutSymbol from llc's text: one value per name of HostLayoutNames, from .quad lines and .zero runs of whole quads.

func HostLayoutDecls

func HostLayoutDecls(vs []int64) string

HostLayoutDecls writes the values of HostLayout as Go constants, nothing for none.

func ICallCode

func ICallCode(id uint32) (string, error)

ICallCode decodes a type id into its signature code.

func ICallID

func ICallID(code string) (id uint32, ok bool)

ICallID encodes a signature code the way the pass does (icallID in pass/shadow.cpp): bit 31 set, the parameter count in bits 27 to 30, the result in bits 24 to 26 and two bits per parameter from bit 0. ok is false for a code it cannot hold.

func IsHostCallback

func IsHostCallback(name string) bool

IsHostCallback says whether name is a Go callback of the host.

func Layout

func Layout(sig Sig) ([]int, int, int)

Layout gives the Go argument area of a signature: the offset of each parameter, of the result, and the total size.

func NoSplitReport

func NoSplitReport(p *NoSplitPick, funcs []NoSplitFunc) string

NoSplitReport is the list as the generator writes it down: one line per function, its kind (auto, register or chosen), frame and height.

func PackedGo

func PackedGo(name, entry, was string) (src string, ok bool, err error)

PackedGo writes the Go function a host calls a packed function by. The pass gives a function whose arguments need more integer registers than a unified entry has (six on arm64, ten on amd64) a block of them instead: it then takes the shadow pointer and the block's address. The Go host calls the engine's API by the C signature on every target, so the function takes the arguments of was (the signature code the function had, the shadow pointer first), stores them in a block at the shadow pointer, each in an 8-byte slot in order (16 bytes for a pair) as the pass lays the block out, and calls entry with the shadow pointer above the block. The block fits: the engine keeps 64 KiB of the region free past any shadow pointer it gives out, and a block is a few words. The function panics where the region would end inside the block all the same, which no host does, since the engine checks the room its own frames take but not this.

ok is false when was has an argument the host cannot pass by value (an aggregate passed by copy, a type the generator does not know): such a function gets no Go function, and a host call of it does not compile.

func ParseICallSymbol

func ParseICallSymbol(name string) (fn string, n int, id uint32, ok bool)

ParseICallSymbol splits an indirect-call symbol name into the function, the call's index in it and the type id.

func PostLinkOpt

func PostLinkOpt(llvmBin string, args []string, in, out string) error

PostLinkOpt runs the optimizer with args over the IR file in and writes the IR file out.

func PostLinkOptFor

func PostLinkOptFor(arch string) bool

PostLinkOptFor says whether the driver runs PostLinkOptArgs over the linked module of arch by default. It is on for both architectures.

func SetCallees

func SetCallees(specs []FuncSpec)

SetCallees records the signature and kind of every function the unit may call, for the lowering of internal calls.

func SharedCFlags

func SharedCFlags(flagsFile, triple string) ([]string, error)

SharedCFlags runs a flags.sh in sh, as the scripts of the C tree do, and returns its engine_flags followed by engine_target_flags for a clang target triple.

func SortedKeys

func SortedKeys[T any](m map[string]T) []string

SortedKeys returns the keys of a set in order.

func ValueFlagsFor

func ValueFlagsFor(arch, group string) []string

ValueFlagsFor gives the extra clang flags of one engine file: EngineValueFlags for quickjs.c on amd64, none for anything else (group is the name of the file without .c, or "libc").

Types

type AMD64Arch

type AMD64Arch struct{ PreserveNone, SaveAll, Poison bool }

amd64 implements Arch for linux/darwin/windows amd64 (GOAMD64=v1).

PreserveNone lowers code compiled with preserve_none on every unified entry: arguments arrive in R12, R13, R14, R15, RDI, RSI, RDX, RCX, R8, R9, R11, RAX and nothing survives a call but RSP and RBP, so call sites save nothing. The entry still loads the arguments only after Go's prologue, which uses R12 and R14. SaveAll saves all five SysV callee-saved registers around every unified call instead of the ones the body uses (for the cost comparison only). Poison is a negative build: it writes garbage into the registers at each entry, after each unified call and at each return (see poison in lower.go).

func AMD64

func AMD64() AMD64Arch

func (AMD64Arch) CalleeSaved

func (AMD64Arch) CalleeSaved() []string

func (AMD64Arch) GoFramePrologueBytes

func (AMD64Arch) GoFramePrologueBytes() int

Return address (8) plus the base pointer Go pushes (8): the incoming argument area sits that far above SP once the prologue has run.

func (AMD64Arch) IntArgRegs

func (AMD64Arch) IntArgRegs() []string

func (AMD64Arch) Name

func (AMD64Arch) Name() string

func (AMD64Arch) RetIntRegs

func (AMD64Arch) RetIntRegs() []string

func (AMD64Arch) RetSSEReg

func (AMD64Arch) RetSSEReg() string

func (AMD64Arch) SSEArgRegs

func (AMD64Arch) SSEArgRegs() []string

type ARM64CallStub

type ARM64CallStub struct {
	Name, Target string
	Sig          Sig
}

ARM64Stub is a call stub of the stub mode: a direct one to a Go symbol, or (Target "") an indirect one through R9.

type ARM64Env

type ARM64Env struct {
	Plan  *DataPlan
	Funcs map[string]bool
	// Facts are llc's registers of every function and call (facts.cpp),
	// Probes those of every unified definition's arguments. When set, each
	// entry must load its arguments into the registers llc's probe reads
	// them from, and each call must store its arguments from the registers
	// llc passes them in and load its result into the ones llc reads.
	Facts, Probes RegFacts
	// Variadic are the functions the pass left variadic: a call to one is
	// refused, since its arguments past the registers go on the stack.
	Variadic map[string]bool
	// Align64 is EngineEmit.Align64: where the body of each of these functions
	// starts in a line of 64 bytes, which FuncPlace puts the function at in
	// the lowered object.
	Align64 map[string]int
	// RegResult is LowerEnv.RegResult: the functions whose result a direct
	// call takes from x0, x1 or d0, where they stored it from.
	RegResult map[string]bool
}

ARM64Env is what lowering one function of the whole engine needs beyond its own text.

type ARM64Func

type ARM64Func struct {
	Meta
	Sig       Sig
	Leaf      bool // no stack use and no call
	LLVMFrame int  // F
	SaveSlots int  // call-site save slots after the LLVM frame
}

ARM64Func is one lowered function.

type ARM64Mode

type ARM64Mode int

ARM64Mode picks how a unified call keeps the AAPCS callee-saved registers.

const (
	// ARM64Stub calls a NOSPLIT stub per callee that saves and restores
	// all of R19-R25 and F8-F15 and moves the arguments (the first backend).
	ARM64Stub ARM64Mode = iota
	// ARM64SaveAll moves the arguments at the call site and saves all 15.
	ARM64SaveAll
	// ARM64SaveBody saves at each call site the callee-saved registers the
	// calling function's body uses: the ones LLVM's prologue saved.
	ARM64SaveBody
	// ARM64PreserveNone lowers code compiled with preserve_none on every
	// unified entry and host callback: LLVM keeps across a call only what is
	// live, in its frame, and no call site saves a register.
	ARM64PreserveNone
)

func ParseARM64Mode

func ParseARM64Mode(s string) (ARM64Mode, error)

type ARM64Unit

type ARM64Unit struct {
	Mode    ARM64Mode
	NoSaves bool // negative build: no call keeps a register
	Funcs   []ARM64Func
	Stubs   []ARM64CallStub
	Lowered string
	// Unified call sites and registers saved around them, over the unit.
	CallSites, SavedRegs int
}

ARM64Unit is the lowered unit.

func LowerARM64

func LowerARM64(engineAsm, leafAsm string, specs []FuncSpec, mode ARM64Mode, noSaves bool) (*ARM64Unit, error)

LowerARM64 lowers every spec'd function: unified ones from the engine unit's llc output, register-entry leaves from the leaf unit's.

type Arch

type Arch interface {
	Name() string
	// IntArgRegs / SSEArgRegs give the native argument registers in order.
	IntArgRegs() []string
	SSEArgRegs() []string
	// CalleeSaved are the native callee-saved GP registers a body may hold a
	// value in. A unified call clobbers them, so the caller saves the live
	// ones to its frame.
	CalleeSaved() []string
	// RetIntRegs / RetSSEReg give where a native body leaves its result.
	RetIntRegs() []string
	RetSSEReg() string
	// GoFramePrologueBytes is the distance Go's prologue adds between the
	// frame base and the incoming argument area (return address + saved BP).
	GoFramePrologueBytes() int
}

Arch abstracts the target so the lowering and checks are architecture independent. amd64 is implemented here; arm64 is a separate Arch.

type AsmFunc

type AsmFunc struct {
	Name  string
	Lines []string // raw lines after the label
}

AsmFunc is one function of llc's text: its body lines up to .Lfunc_end.

type AsmUnit

type AsmUnit struct {
	Funcs     map[string]*AsmFunc
	FuncOrder []string
	Data      map[string]*DataObj // by label and by every alias
	DataOrder []*DataObj
	// LabelFunc maps each local text label (.LBB, .Ltmp) to its function.
	LabelFunc map[string]string
	// Sets are the symbol assignments (name = target), such as musl's weak
	// aliases.
	Sets map[string]string
}

AsmUnit is llc's text split up.

func SplitAsm

func SplitAsm(asm, comment string) (*AsmUnit, error)

SplitAsm splits llc's x86-64 or aarch64 GNU text into functions and data objects. comment is the comment marker ("#" on x86-64, "//" on aarch64).

type BlobDiff

type BlobDiff struct {
	Off, Size int
	Pos, Neg  string
	K         int64
}

BlobDiff is a value inside engine_rodata that PlanData resolved from a difference of two labels (Pos - Neg + K, the relative tables LLVM builds), declared so the post-link check can recompute it in llc's object and in the linked blob.

type BlobReloc

type BlobReloc struct {
	Off  int
	Sym  string
	Add  int64
	Func bool
	// Label and LabelAdd are the pointer as llc's text wrote it (label plus
	// addend), for the post-link check to resolve in llc's object.
	Label    string
	LabelAdd int64
	// GOT marks the slot of a writable global (DataPlan.GOTSlot): a pointer
	// to Label in engine_rodata that no object of llc's holds.
	GOT bool
}

BlobReloc is an 8-byte pointer inside engine_rodata: to Sym+Add, where Sym is a function (Func) or engine_rodata itself.

type CallFacts

type CallFacts struct {
	Target            string
	Args, Defs, After []string
}

CallFacts is one call: its target (a symbol, or "*" and the kcfi type id of an indirect call), its argument registers in order, the registers it defines (its results) and the ones live right after it.

type CallKind

type CallKind int

CallKind says how the lowering rewrote one call.

const (
	CallUnified CallKind = iota // a direct call to a unified entry, marshalled
	CallLeaf                    // a direct call to a register-entry leaf, kept
	CallReg                     // a call through a register, marshalled
)

type CallRecord

type CallRecord struct {
	Kind      CallKind
	Target    string
	Sig       Sig
	RegResult bool
}

CallRecord is one call: its target (a C symbol, or the register of CallReg) and the signature it was marshalled with. RegResult says the call of CallUnified loads no result back from the Go stack: the callee is a lowered unified function, which the post-link check holds to storing its result from the registers the call reads it from and then running only Go's epilogue, so they still hold the result after its RET.

type CheckReport

type CheckReport struct {
	Checked map[string]int
	Skipped map[string][]string
}

CheckReport counts what was checked and lists what was not, by reason.

func CheckEngine

func CheckEngine(c EngineCheck) (*CheckReport, error)

CheckEngine runs the instruction-level and data checks.

func CheckEngineARM64

func CheckEngineARM64(c EngineCheck) (*CheckReport, error)

CheckEngineARM64 runs the check on c.Bin, a linked linux/arm64 binary of the package c.Pkg, against c.Lowered.

type DataObj

type DataObj struct {
	Label   string
	Aliases []string
	Section string
	Align   int // log2 of the alignment
	// AlignCond: the start alignment had a max-skip below its step, so llc
	// may have placed the object without it.
	AlignCond bool
	Lines     []string // data directives, cleaned
	Global    bool
	Order     int
}

DataObj is one labelled object of a data section in llc's text.

type DataPlan

type DataPlan struct {
	Blob    []byte
	BlobOff map[string]int // label or alias -> offset in engine_rodata
	Relocs  []BlobReloc
	Diffs   []BlobDiff
	// InBody lists, per function, the objects placed after its code: its
	// own (constant pools, jump tables, tables of its labels) and the
	// copies of objects it reads without LEA.
	InBody map[string][]*DataObj
	// Rename maps, per function, a data label to the label its body uses.
	Rename map[string]map[string]string
	// Errors are per function (key "" for data that belongs to none).
	Errors map[string][]error
	// Stats for the report.
	Objects, BlobObjects, InBodyObjects, DupObjects, Dropped int
	// Writable says engine_rodata holds writable globals: it is written
	// without RODATA.
	Writable bool
	// GOTSlot is, for each writable global whose GOT load x86 folded into a
	// compare or an add, the offset of an 8-byte slot at the end of
	// engine_rodata that holds its address: the lowering reads the slot
	// where llc read the GOT entry (writable.go).
	GOTSlot map[string]int
	// Pad is, per function and per copy label, the zero bytes written after
	// a copy that llc reads past its end. Such a read comes from code that
	// copies a variable length out of a constant (an inlined memcpy whose
	// source llc knows) and runs only when the length is longer than the
	// constant, which the C never does. llc's bytes past the object are
	// whatever follows it, so the copy reads zeros instead and the read stays
	// in the function's data.
	Pad map[string]map[string]int
	// contains filtered or unexported fields
}

DataPlan says where every data object goes.

func PlanData

func PlanData(u *AsmUnit, funcs map[string]bool) *DataPlan

PlanData classifies the data objects of an amd64 unit. funcs are the functions of the module (defined or external).

func PlanDataARM64

func PlanDataARM64(u *AsmUnit, funcs map[string]bool) *DataPlan

PlanDataARM64 does the same for an aarch64 unit of the tiny code model, whose code takes the address of data with an adr and reads it with a load of a label, and whose .word is 4 bytes.

func (*DataPlan) BodyData

func (p *DataPlan) BodyData(fn string) string

BodyData returns the assembly of the data objects placed in fn's body.

func (*DataPlan) RenameLine

func (p *DataPlan) RenameLine(fn, l string) string

RenameData rewrites the data labels a body line names, for the body of fn.

type EngineCheck

type EngineCheck struct {
	LLC     string // llc's object of the engine, with temporary labels kept
	Lowered string // the re-assembled lowered text, with temporary labels kept
	Bin     string // a linked Go binary of the generated package
	Pkg     string // the package's link symbol prefix
	Names   NameMap
	Unit    *AsmUnit
	Plan    *DataPlan
	Metas   []Meta // lowered functions (unified and register-entry)
	Stubs   []Meta // in missingNames order
	// Sigs and Leaves are the pass's signature of every function and the
	// register-entry leaves it was given, and PreserveNone the convention it
	// gave every unified entry: the check holds each function's record to
	// them.
	Sigs         map[string]Sig
	Leaves       map[string]bool
	PreserveNone bool
	// Probes and Facts are what llc itself decided about registers
	// (facts.go): Probes from llc's run over the probe module of
	// -shadow-probes, which gives the argument and result registers of each
	// unified definition, Facts from the run that wrote LLC, which gives
	// each function's calls, returns and live registers.
	Probes, Facts RegFacts
	// All requires every function to be in the binary and checked.
	All bool
	// Copied are the constants ExposeCopiedPointerTables sent through the
	// GOT: read-only, but reached as writable globals are.
	Copied map[string]bool
	// Align64 is EngineEmit.Align64: the body of each of these functions
	// starts that many bytes past a multiple of 64, after the PCALIGN $64
	// that replaces the usual PCALIGN $16 and the no-ops that follow it.
	Align64 map[string]int
	// HostPoll says the engine calls host_poll even though llc's object does
	// not name it (it is always required when the object does). The poll is
	// the only place the Go runtime can stop C code to collect garbage, so the
	// Go function must begin with a stack check: a function without one would
	// let a loop of C run on without a safe point.
	HostPoll bool
}

EngineCheck is what the post-link check reads.

type EngineEmit

type EngineEmit struct {
	Metas []Meta  // lowered functions, in output order
	Names NameMap // C symbol -> Go identifier
	// Group gives the file group of each function (its C source file);
	// MaxBytes bounds the size of one .s file.
	Group    map[string]string
	MaxBytes int
	// Stubs are functions emitted as a call to ·goasmMissing(index). With
	// NoStubs (a release build) any stub is an error.
	Stubs   []Meta
	NoStubs bool
	Plan    *DataPlan
	Arch    string // file name suffix and build tag
	// Tag, when set, is a build tag that every file requires besides the
	// architecture (see BuildExpr).
	Tag string
	// Align64 maps a function to where its body starts in a line of 64
	// bytes, a multiple of 8 below 64: a PCALIGN $64 takes the place of the
	// PCALIGN $16 after Go's prologue (which also puts the function itself
	// on 64 bytes), followed by that many bytes of 8-byte no-ops. The no-ops
	// run once per call. A change elsewhere in the program then moves the
	// body by whole lines of 64 bytes only.
	Align64 map[string]int
}

EngineEmit says what to write.

type FuncFacts

type FuncFacts struct {
	Args, Entry []string
	Calls       []CallFacts
	Rets        [][]string
}

FuncFacts is what llc decided for one function: the live-in registers of its arguments in order (only the ones its code reads, which for a probe of -shadow-probes is all of them), the registers its code reads before writing them, each call and each return in code order. Registers are llc's names, every name of a register that is live (eax and al beside rax).

type FuncRecord

type FuncRecord struct {
	Name                  string
	Kind                  Kind
	Sig                   Sig
	PreserveNone, SaveAll bool
	// NoSplit: the function is on the generator's NOSPLIT list (Meta.NoSplit),
	// so the binary has no stack check at its entry, and every unified
	// function that is not on it has one.
	NoSplit bool
	// The canonical prologue the lowering removed: the callee-saved
	// registers it pushed, in order, and the bytes it reserved below them.
	Pushed []string
	Local  int
	// The Go frame: locals at LocalBase, saved registers at SaveBase, the
	// outgoing argument area at 0, Frame bytes in all.
	LocalBase, SaveBase, Frame int
	// Saved are the registers saved around every unified call.
	Saved []string
	// Calls and Refs are in llc's order: every call, and every RIP-relative
	// reference to data or to a function.
	Calls []CallRecord
	Refs  []RefRecord
}

FuncRecord is every rewrite the lowering made in one function.

type FuncSpec

type FuncSpec struct {
	Name string
	Sig  Sig
	Kind Kind
	// Indirect lists, in source order, the signature of each call this
	// function makes through a function pointer. The function table and every
	// indirect call target a unified entry, so these marshal like a direct
	// unified call.
	Indirect []Sig
}

FuncSpec names a function in the compiled unit and how to generate it.

type IRViolation

type IRViolation struct {
	Func, Rule, Line string
}

IRViolation is one rule a function of the C breaks, as the IR shows it.

func CheckIRFuncs

func CheckIRFuncs(ll string) []IRViolation

CheckIRFuncs is CheckIR over a whole module that names the function of each violation instead of stopping at the first. A call of a variadic function counts as a violation of the caller too ("variadic call").

type Kind

type Kind int

Kind is one of the two entry conventions of the plan.

const (
	// Unified entry: arguments arrive on the Go stack (ABI0), the body reads
	// them after Go's split-check prologue, and the caller relies on no
	// callee-saved register surviving the call. The shadow-stack pointer is
	// the first stack argument. Function tables, address comparisons and
	// indirect calls all use this entry.
	Unified Kind = iota
	// Register entry: a NOSPLIT leaf that neither grows the stack nor calls a
	// split-checked function. Arguments stay in the native argument registers.
	// Only a direct call to a known leaf may target it.
	Register
)

type LowerEnv

type LowerEnv struct {
	Plan  *DataPlan
	Funcs map[string]bool
	// Variadic names the variadic functions. A call to one is rejected
	// unless LooseVarargs, which lowers it with its fixed arguments only
	// (wrong code, for measuring the output while the C still has them).
	Variadic     map[string]bool
	LooseVarargs bool
	// RegResult names the functions whose result a direct call takes from
	// the registers they return it in instead of loading it from the Go
	// stack (CallRecord.RegResult): the unified functions the driver lowers,
	// never a Go callback, a stub or a register-entry leaf. A poison build
	// takes none, since it writes over every register at a return.
	RegResult map[string]bool
}

LowerEnv is what lowering a function of the whole engine needs beyond its own text: where its data goes and which symbols are functions. The signature of each indirect call is the one LLVM bound to it, which llc writes as a goasm.icall label right before the call (icall.go).

type Meta

type Meta struct {
	Name     string
	Kind     Kind
	GoName   string // exported-to-asm Go identifier, e.g. recAsm
	Frame    int    // Go TEXT frame size
	ArgBytes int    // Go TEXT argument+result area size
	RetType  Type
	// NoSplit says the TEXT has NOSPLIT, so Go puts no stack check at the
	// entry (ChooseNoSplit). A register-entry leaf always has it.
	NoSplit bool
	// Rec is every rewrite the lowering made, for the instruction-level
	// post-link check (nil for the prototype's units).
	Rec *FuncRecord
}

Meta is what the emit stage needs after re-assembly.

func LowerARM64EngineFunc

func LowerARM64EngineFunc(lines []string, fn FuncSpec, env *ARM64Env) (string, Meta, error)

LowerARM64EngineFunc lowers one function of the whole engine from the body lines of llc's text (after its label, up to .Lfunc_end). The result is the text for llvm-mc, from the .globl of the function to its .size.

func LowerRegister

func LowerRegister(asm string, fn FuncSpec) (string, Meta, error)

LowerRegister transforms a frameless leaf (compiled with -fomit-frame-pointer) into a NOSPLIT body. It rejects anything with a frame, a call or a forbidden instruction.

func LowerUnified

func LowerUnified(a AMD64Arch, asm string, fn FuncSpec) (string, Meta, error)

LowerUnified transforms one function's GNU assembly. It returns the rewritten GNU text (one label, a trailing return label, calls left symbolic) and the Meta for emit.

func LowerUnifiedFunc

func LowerUnifiedFunc(a AMD64Arch, lines []string, fn FuncSpec, env *LowerEnv, asm string) (string, Meta, error)

LowerUnifiedFunc lowers the body lines of one function (after its label, up to .Lfunc_end). With env nil it is the prototype's unit: constant pools are found in asm and indirect calls take fn.Indirect in order.

type NameMap

type NameMap map[string]string

NameMap maps a C symbol to the Go identifier the assembly defines and calls.

func GoNames

func GoNames(syms []string) (NameMap, error)

GoNames maps C symbols to Go identifiers: dots (from LLVM's renaming of static functions with the same name) become underscores, and a name Go reserves gets a trailing underscore. Two C names mapping to one Go name is an error.

type NoSplitFunc

type NoSplitFunc struct {
	Name     string
	Kind     Kind
	Frame    int      // the Go TEXT frame, without the saved base pointer
	ArgBytes int      // the Go TEXT argument and result area
	Calls    []string // direct call targets, in order, repeats allowed
	Indirect bool     // calls through a register
	Stub     bool     // a stub of a function the generator could not take
}

NoSplitFunc is what the choice reads of one generated function.

type NoSplitPick

type NoSplitPick struct {
	// Set holds every function whose TEXT says NOSPLIT.
	Set map[string]bool
	// Auto are the leaves the assembler marks NOSPLIT by itself, Register
	// the register-entry leaves, Chosen the others in Set.
	Auto, Register, Chosen []string
	// Height is the room each function of Set takes below its entry,
	// FaultRoom included, as the choice counts it.
	Height map[string]int
	// Left are the functions that could not be chosen, by reason.
	Left map[string][]string
}

NoSplitPick is the choice: the functions without a stack check, and why.

func PickNoSplit

func PickNoSplit(funcs []NoSplitFunc, taken, callbacks map[string]bool, force []string) (*NoSplitPick, error)

PickNoSplit chooses the NOSPLIT functions of funcs. taken lists the functions whose address is used other than by a direct call, callbacks the Go functions the engine calls. force adds functions whatever the rules say, for the negative build that shows the linker refuses a recursion. It fails when a call goes to a function it does not know, or when the choice breaks its own rules.

func PickNoSplitFor

func PickNoSplitFor(r StackRules, funcs []NoSplitFunc, taken, callbacks map[string]bool, force []string) (*NoSplitPick, error)

PickNoSplitFor is PickNoSplit with the numbers of an architecture.

type RefRecord

type RefRecord struct {
	Label      string
	Add        int64
	Func, Blob bool
	// GOT is a load of the address of a writable global from the GOT that
	// the lowering made a LEA of it in engine_rodata (writable.go). With
	// Slot, x86 had folded the load into a compare or an add, and the
	// lowering reads the global's slot in engine_rodata instead
	// (DataPlan.GOTSlot).
	GOT, Slot bool
}

RefRecord is one RIP-relative reference as llc's text wrote it, label plus addend, and where the lowering pointed it: a function, engine_rodata, or the function's own copy of the data.

type RegFacts

type RegFacts map[string]*FuncFacts

RegFacts are the facts of one llc run, by function.

func ParseRegFacts

func ParseRegFacts(text string) (RegFacts, error)

ParseRegFacts reads the plugin's output.

type Report

type Report struct {
	// Sigs gives the code of every function the module defines or calls
	// (see SigFromCode), Defined whether the module defines it, Leaf
	// whether it is a register-entry leaf.
	Sigs    map[string]string
	Defined map[string]bool
	Leaf    map[string]bool
	// AddrTaken lists the functions whose address is used other than by a
	// direct call.
	AddrTaken map[string]bool
	// Calls is the direct call graph, ICalls the signature codes each
	// function calls through a pointer.
	Calls  map[string][]string
	ICalls map[string][]string
	// FrameAddr lists the functions that read the machine frame address
	// (QuickJS's stack overflow checks), Packed the ones the pass gave an
	// argument block, Rejected the ones the pass emptied, with the reason.
	FrameAddr map[string]bool
	Packed    map[string]bool
	// PackedWas is the signature code a packed function had before the
	// pass gave it the block.
	PackedWas    map[string]string
	PackedICalls map[string]int
	Rejected     map[string]string
	GotoTable    []string
	Mismatch     [][2]string
	// Varargs lists the variadic definitions -shadow-varargs rewrote.
	Varargs []string
}

Report is what the shadow-stack pass printed with -shadow-report.

func ParseReport

func ParseReport(text string) *Report

ParseReport reads the pass's report lines; other lines are ignored.

type Sig

type Sig struct {
	Params []Type
	Ret    Type
}

Sig is the signature of a generated function in logical order. By convention Params[0] of a Unified function is the shadow-stack pointer (Ptr).

func SigFromCode

func SigFromCode(code string) (Sig, error)

SigFromCode turns a pass signature code into a Sig. Every integer or pointer argument is one word, moved as the whole register, so whatever extension the C convention relies on survives the Go argument area. 'j' is a pair of 64-bit integers (a JSValue). A narrow integer ('n') is an error: the pass passes every one at the unified boundary as an i32, since preserve_none would put it in another register than the one the generator moves it in, and a C callee may rely on an extension Go does not make. A float ('f') is an error too: Go writes and reads each word as a whole float64, and no unified function of the engine takes or returns one. Variadic, byval ('B') and unknown ('?') types are errors: the IR check names the C that has them.

With KeepNarrowSigs, a negative build (the pass leaves narrow integers with -shadow-keep-narrow), 'n' is a word like 'i', as the generator once took it, and the post-link check must reject what comes of it.

type StackRules

type StackRules struct {
	Arch string
	// Limit is the room the linker gives a chain of NOSPLIT calls.
	Limit int
	// Call is what a call adds to the stack: the return address amd64
	// pushes, nothing on arm64, which keeps it in LR.
	Call int
	// Wrapper is what the ABI wrapper of a Go func value adds above a
	// function with argBytes of arguments and results.
	Wrapper func(argBytes int) int
	// Local is the bytes a frame of the TEXT size frame takes below the
	// entry, for a function that calls (or calls nothing).
	Local func(frame int, calls bool) int
}

StackRules are the numbers of the Go toolchain the choice counts with, which differ by architecture.

type Type

type Type int

Type is a value a generated function passes or returns. Every type maps to non-Go memory, so the paired Go declaration uses uintptr/int64/float64 and carries no Go pointer: the argument area needs no pointer map beyond the empty one.

const (
	Void Type = iota
	I64       // 64-bit integer, one GP register / one stack word
	Ptr       // engine pointer, treated as uintptr, one GP register / word
	F64       // double, one SSE register / one stack word
	JSV       // JSValue, 16 bytes, two GP registers, two stack words
)

type Unit

type Unit struct {
	IR, ShadowIR, Asm, Obj string // paths in the work dir
	Shadow                 string // the pass's report: one line per function
}

Unit is one C file compiled through the shadow-stack pass.

func CompileUnit

func CompileUnit(cfg UnitConfig, cfile string) (*Unit, error)

CompileUnit compiles cfile with clang, runs the IR checks and the shadow-stack pass, and writes assembly and an object with llc.

type UnitConfig

type UnitConfig struct {
	LLVM, Pass, Work string   // LLVM bin dir, built pass, work dir
	CFlags           []string // clang flags (EngineCFlags or ARM64CFlags)
	PassArgs         []string // extra shadow-stack pass options
	Leaves           []string // register-entry leaves: no shadow pointer
}

UnitConfig says how to compile a unit.

type VerifyReport

type VerifyReport struct {
	Func         string
	BodyBytes    int
	Calls        int
	HasMorestack bool
}

func VerifyLinked

func VerifyLinked(objPath, binPath, pkg string, metas []Meta) ([]VerifyReport, error)

VerifyLinked checks the functions of metas inside binPath against objPath.

Directories

Path Synopsis
cmd
engine command
gen command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL