Documentation
¶
Overview ¶
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Package gen is the phase-0 prototype of the new C-source engine generator. It is architecture independent in structure: an Arch describes the Go frame and the native calling convention, and the lowering and checks are written against that interface so arm64 can be added without touching the pipeline.
This file holds the shared vocabulary: the two function entry kinds, the value types the engine passes between functions, and the amd64 Arch.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
arm64 backend. Lowering edits llc's aarch64 assembly as text, llvm-mc re-assembles it, and emit writes Go assembly: the body as WORDs, every call a symbolic BL, the arguments loaded from FP after Go's prologue, one Go RET and the function table.
Frame: 0(RSP) is Go's saved LR, the outgoing ABI0 area starts at 8(RSP), the LLVM frame of F bytes sits at the next 16-byte boundary L, call-site save slots follow it, and the TEXT declares $(L+F+saves-8). Every SP move of LLVM's prologue and epilogues is dropped and each SP-relative access is rebased on the Go frame, tracking how much of the frame is allocated at that point, so any prologue shape and frames over 4096 bytes lower the same way.
A C call to a unified entry or a Go callback stores the argument registers into the outgoing area, calls the Go symbol and loads the result back, all inline at the call site. Go code and morestack clobber R19-R25 and F8-F15, which AAPCS code expects preserved. ARM64Mode picks how that is handled.
The arm64 lowering of the whole engine. It takes llc's aarch64 text of one function (preserve_none on every unified entry, the -mode pn of cmd/engine) and rewrites it as text for llvm-mc, as LowerARM64 does for the units of the prototype, plus what the engine has and the prototype does not: data whose address the code takes, jump tables and the computed goto of the interpreter, the label llc writes at each indirect call, and calls of the register-entry leaves (memcpy, memmove, memset).
The Go frame of a function with a frame: 0(RSP) is the LR Go saves, the outgoing ABI0 area starts at 8(RSP), and the LLVM frame of F bytes sits at the next 16-byte boundary L. The TEXT declares $(L+F-8), so that with the LR it is L+F, a multiple of 16, and the Go assembler adds 16 bytes on top for the frame pointer its callees save there. Every SP move of LLVM's prologue and epilogues is dropped, and every access [sp, #k] is rebased on the Go frame.
Data. llc writes the engine in the tiny code model, where every reference to data is one instruction that names the whole address: an adr of data or of a function, or a load of a label. Go assembly cannot attach a relocation to the bytes of an instruction, so the lowering writes each one on its own as an instruction that reaches the same address: an adr of other data or of a function becomes MOVD $sym(SB) in Go (an adrp and an add of the whole address), an adr of the function's own data (constant pools, jump tables, the table of the computed goto) stays an adr of it in the body, and a load of a label loads the data the plan put in the body, the function's own or a copy. No rewrite depends on any other instruction, so there is nothing to follow through the blocks. The page forms of the small code model (adrp and :lo12:), where the second instruction is right only while the register still holds the page of the first, are refused.
The post-link check of the arm64 engine. It reads a linked linux/arm64 binary of the generated package and holds every generated function to the object llvm-mc made of the lowered text, which the emit wrote as WORDs:
- Go's prologue: a stack check exactly when the function is not on the NOSPLIT list, the frame the TEXT asks for (with the saved LR and frame pointer as cmd/internal/obj/arm64 rounds it, ARM64Stack.Local), and nothing else,
- the loads of the arguments from the Go argument area into the registers of preserve_none, then no-ops up to where the body starts (16 bytes, or the line of 64 of -align64),
- the body word for word: each BL the emit wrote goes to the function the object's call names, each ADRP and ADD to engine_rodata plus the object's offset or to the function it names, an indirect call is the same BLR, every other word and the function's own data are as in the object,
- the stores of the result, Go's epilogue and one RET, and after it only the call of morestack of the stack check,
and engine_rodata byte for byte with each pointer at the function or data it names. It does not take llc's object: that the lowering kept the meaning of llc's code is what the lowering's own rules and llc's register facts hold (LowerARM64EngineFunc).
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Static checks. Each rule the plan lists is enforced mechanically and has a small example it must reject (checks_test.go). The lowering enforces the instruction-level rules (forbidden instructions, SP writes, escaping stack addresses, frame spills, indirect jumps); these functions enforce the rules that read the LLVM IR and the linked/assembled object.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Emit stage: read the re-assembled object with debug/elf, disassemble each function with x/arch/x86/x86asm, and write it as a Go assembly TEXT. Every call becomes a Go-visible CALL (direct ones symbolic through a relocation, indirect ones as CALL <reg>), so Go knows the frame, traceback and stack bookkeeping; all other instructions pass through as machine-code bytes. This is the only place the generated code enters the module, and verify.go checks the linker left those bytes unchanged.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Emit for the whole engine: the functions go into several .s files (one or more per C source file, each under a size bound), engine_rodata into its own file as Go DATA, and every function the generator could not lower becomes a stub that reports its name to Go.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
What llc itself decided about registers, as the plugin in pass/facts.cpp writes it (-goasm-facts). The post-link check holds the lowering to it instead of to the generator's own model of the calling convention, so a mistake in the model (a type it moves in the wrong register) shows as a difference instead of being checked against itself.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
The instructions engine code may not hold: the ones that call the system or read what the machine says (the time stamp counter, the hardware random source, the processor's identity), and the traps. The plan's list is syscall, sysenter, int 0x80, svc, rdtsc, rdtscp, rdrand, rdseed and cpuid. A trap (ud2, int3, hlt, brk, udf) is SIGILL, SIGTRAP or a general protection fault, and in Go SIGILL and SIGTRAP end the process whatever the program does, so engine code reaches host_trap instead (see pass/shadow.cpp). The lowering reads the assembly text for them and the post-link check the decoded machine code (verify_insn.go), so a mistake in one of the two readings does not let one through. arm64 has no post-link check yet: EmitARM64 reads the words of the assembled object.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
The signature LLVM binds to each indirect call. The pass gives the call a "kcfi" operand bundle whose type id encodes its signature code (-shadow-icall-sigs), llc keeps the id on the machine call, and the plugin (pass/icall.cpp) names a symbol at the call: goasm.icall.<function>.<n>.<id>. It shows as a label right before the call in llc's text and as a local symbol at the call's address in llc's object.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Lowering stage: turn the canonical LLVM GNU assembly of each engine function into a form Go's stack rules accept, as text, so re-assembly fixes every branch offset. Only external calls stay symbolic for elf2go.
The transform per unified function:
- strip the canonical prologue (push BP; mov SP,BP; push <callee-saved>; reserve locals) and the single epilogue, recording the frame and the callee-saved set the body uses;
- prepend an entry shim that loads each incoming argument from the Go argument area into its native argument register, after Go's prologue;
- at each internal call, save the live callee-saved registers to the frame (the split callee's morestack does not preserve them), move the native argument registers into the outgoing argument area, call, move the result back, and restore;
- replace every return with a jump to a common epilogue that stores the result into the Go result area and falls into Go's RET.
It rejects any shape it was not built to prove: a non-canonical prologue, a body that still touches BP (an unhandled frame spill), an SP write, an escaping stack address, a forbidden instruction or an unreviewed indirect jump. Failing loudly is the plan's rule: the generator never emits code it cannot account for.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Register-entry leaf lowering. A leaf neither grows the stack nor calls a split-checked function, so it is emitted NOSPLIT with its arguments left in the native argument registers and its result in the native result register. Only a direct call from a known caller may target it; the function table and indirect calls use the leaf's unified entry instead.
Functions without Go's stack check. Every unified entry starts with the stack check of the Go assembler, three instructions that compare the stack pointer with the guard in g and call morestack below it, unless its TEXT says NOSPLIT. The generator chooses no function to go without it. The functions without one are those that cannot have one:
- the register-entry leaves (memcpy, memmove, memset), which are NOSPLIT by their form: they take their arguments in registers that morestack would not keep,
- the leaves the Go assembler marks NOSPLIT by itself: a function that calls nothing and whose frame with the saved base pointer is under 128 bytes (abi.StackSmall, cmd/internal/obj/x86/obj6.go and cmd/internal/obj/arm64/obj7.go).
The generator names both in their TEXT, so that the list is written down and the post-link check holds every entry to it: a function on it must have no stack check, every other one must have one. A Go assembler that marks other leaves by itself, or no longer marks these, fails that check.
An automatic leaf calls nothing. A register-entry leaf calls nothing but other register-entry leaves, whose arguments stay in registers (the engine's three call nothing), and MarkNoSplit fails when a function on the list makes another call, when register-entry leaves call each other in a cycle, or when one takes 128 bytes or more below its entry with the leaves it calls. A leaf is entered from a function that passed its stack check, which leaves at least 800 bytes above the bottom of the goroutine's stack once its frame is made (internal/abi.StackNosplitBase with the 128 bytes of abi.StackSmall in the guard). A fault in the leaf (a wild pointer of the C) becomes a call of runtime.sigpanic that the signal handler pushes where the fault is, and sigpanic's stack check may call morestack: that takes 16 bytes on amd64 and 24 on arm64, far below what the leaf leaves of the 800. The linker still adds up every chain of NOSPLIT calls on each build (cmd/link/internal/ld/stackcheck.go), the ABI wrapper of a Go func value that calls a leaf included.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Whole-program input: the engine's C compiled as one module. This file reads what the shadow-stack pass reports (the signature of every function, the address-taken ones, the call graph) and splits llc's assembly into functions and data objects, so the lowering can look a function or a data label up instead of scanning the whole text each time.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
The rewrites the lowering makes, written down per function as data. The post-link check (verify_insn.go) accepts a difference between llc's object and the linked code only where one of these declares it, and checks the declared rewrite against llc's object itself.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Read-only data of the whole engine. Go assembly cannot attach a relocation to the raw bytes of an instruction, so each data object of llc's text goes one of three ways:
- into the body of the function that uses it, after its code, where the re-assembly resolves every PC-relative reference: constant pools, jump tables and the computed goto table (which hold offsets of the function's own labels), and a copy of each table a function reads with an instruction other than LEA (such as musl's exp data);
- into one Go data symbol, engine_rodata, for everything whose address a function takes with LEA or another object holds: strings, function tables, Unicode tables. The lowering turns each LEA of it into a LEA of __goasm_rodata+offset, which emit writes as a Go LEAQ of ·engine_rodata. Pointers inside it become Go DATA relocations, and offsets between two of its objects are resolved here;
- nowhere, if nothing uses it.
Writable globals go into engine_rodata as well, reached only through the GOT (see writable.go), and the symbol is then writable.
Anything else is an error: a direct reference to writable data, an absolute 32-bit address, a pointer in data that must sit inside a function, a non-LEA reference to data that holds pointers.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Post-link verification. After the Go linker has placed the generated code in a program, confirm it is still the code the generator produced: every body byte survived except the four displacement bytes of each symbolic CALL, each split-checked unified entry kept its stack-check prologue, the function table points only at unified entries, and the body holds no hidden call or cross-function tail jump.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT
Instruction-level post-link check of the whole engine. The lowering reads llc's text with a hand-written parser and rewrites it; a parser mistake that still assembles (a misread operand, a dropped line, a wrong frame offset or argument move) would reach the binary without any byte check noticing, since the generator's own re-assembly is then the reference. This check takes llc's object file instead, which llc writes from its own instructions without any text, disassembles it and the generated functions in the linked Go binary with x/arch, and matches them instruction by instruction. A difference is accepted only where the lowering declared a rewrite (FuncRecord) and the check confirms the rewrite's form itself: the prologue and epilogue it replaced, the frame mapping applied to each displacement, each call's saves, argument moves, call target and result, and each data reference resolving to the same llc label plus offset. Every other llc instruction must appear in order with the same meaning. The read- only data is checked the same way: every byte and every relocated pointer of engine_rodata in the binary, and every constant pool and jump table a function carries, against llc's object. The signatures are the pass's: EngineCheck.Sigs for each function and direct call. For an indirect call LLVM itself binds the signature to the call (a kcfi type id the pass sets, which llc keeps on the machine call and the plugin in pass/icall.cpp names in a symbol at it), and the check reads it from that symbol in llc's object. A signature gives only the number of argument and result words: which register holds each word is what llc decided (EngineCheck.Probes and Facts), and the check holds every entry load, argument move, result move and the registers llc's code reads at an entry and after a call to it.
Copyright (c) 2026 The liustack/sqlite Authors. SPDX-License-Identifier: MIT Use of this source code is governed by the MIT license in LICENSE at the root of this repository.
Writable data. SQLite keeps state in writable globals (its configuration, the state of its allocator and page cache, the tables of built-in functions it links together at start), shared by the whole process as a Go global is. They go into the one Go data symbol with the read-only data (DataPlan), which is then written without RODATA.
Go assembly cannot attach a relocation to the bytes of an instruction, so the lowering must rewrite every reference to data, and it can only rewrite one that takes an address. A direct load of a global (an arm64 ldr of a label, an amd64 movq sym(%rip)) would have to become a copy in the body, which is wrong for data that changes. So before llc runs, ExposeWritableData makes every writable global external and preemptible: position-independent code then reaches it only through the GOT, with one load of its address into a register (arm64 ldr x, :got:sym, amd64 movq sym@GOTPCREL(%rip), %reg), which the lowering turns into the address of the global in the data symbol.
Index ¶
- Constants
- Variables
- func AddCallee(name string, sig Sig, kind Kind)
- func BareName(name string) string
- func BodyDataAlign(align64 map[string]int, name string) int
- func BuildExpr(tag, arch string) string
- func CheckBodyDataAlign(plan *DataPlan, metas []Meta, align64 map[string]int) error
- func CheckDispatchCopies(name string, lines []string) (int, error)
- func CheckIR(ll string) error
- func CheckObject(objPath string, allow map[string]bool) error
- func CheckObjectData(objPath string, allow map[string]bool, writable bool) error
- func CheckPollOffset(offset, sizeLog2 int) error
- func CheckShadowSizeLog2(n int) error
- func Emit(objPath string, metas []Meta, names NameMap, buildTag string) (string, error)
- func EmitARM64(objPath string, u *ARM64Unit, buildTag string) (string, error)
- func EmitARM64Engine(objPath string, e EngineEmit) (map[string]string, error)
- func EmitEngine(objPath string, e EngineEmit) (map[string]string, error)
- func EmitRodata(p *DataPlan, names NameMap, header string) string
- func ExposeCopiedPointerTables(ll string) (string, []string, error)
- func ExposeWritableData(ll string) (string, []string, error)
- func FuncPlace(align64 map[string]int, name string) string
- func FuncSizes(objPath string) (map[string]int, error)
- func Functions(asm string) []string
- func HostLayout(u *AsmUnit) ([]int64, error)
- func HostLayoutDecls(vs []int64) string
- func ICallCode(id uint32) (string, error)
- func ICallID(code string) (id uint32, ok bool)
- func InlineRemarks(stderr string) string
- func IsHostCallback(name string) bool
- func Layout(sig Sig) ([]int, int, int)
- func MisexpectLines(stderr string) []string
- func NoSplitReport(l *NoSplitList, metas []Meta) string
- func PackedGo(name, entry, was string) (src string, ok bool, err error)
- func ParseICallSymbol(name string) (fn string, n int, id uint32, ok bool)
- func PostLinkOpt(llvmBin string, args []string, in, out string) error
- func PostLinkOptFor(arch string) bool
- func ProfileFlags(indexed string) []string
- func ProfileNames(text string) map[string]bool
- func ReplayClangArgs(remarks string) []string
- func ReplayOptArgs(remarks string) []string
- func SetCallees(specs []FuncSpec)
- func SharedCFlags(flagsFile, triple string) ([]string, error)
- func SortedKeys[T any](m map[string]T) []string
- func StaleWarnings(stale, fresh, required []string, max int) []string
- func StripCGProfile(ll string) string
- func UnprofiledFunctions(feIR string, profile map[string]bool) (stale, fresh []string)
- func ValueFlagsFor(arch, group string) []string
- type AMD64Arch
- type ARM64CallStub
- type ARM64Env
- type ARM64Func
- type ARM64Mode
- type ARM64Unit
- type Arch
- type AsmFunc
- type AsmUnit
- type BlobDiff
- type BlobReloc
- type CallFacts
- type CallKind
- type CallRecord
- type CheckReport
- type DataObj
- type DataPlan
- type EngineCheck
- type EngineEmit
- type FuncFacts
- type FuncRecord
- type FuncSpec
- type IRViolation
- type Kind
- type LowerEnv
- type Meta
- func LowerARM64EngineFunc(lines []string, fn FuncSpec, env *ARM64Env) (string, Meta, error)
- func LowerRegister(asm string, fn FuncSpec) (string, Meta, error)
- func LowerUnified(a AMD64Arch, asm string, fn FuncSpec) (string, Meta, error)
- func LowerUnifiedFunc(a AMD64Arch, lines []string, fn FuncSpec, env *LowerEnv, asm string) (string, Meta, error)
- type NameMap
- type NoSplitList
- type RefRecord
- type RegFacts
- type Report
- type Sig
- type StackRules
- type Type
- type Unit
- type UnitConfig
- type VerifyReport
Constants ¶
const ( RodataSym = "__goasm_rodata" RodataGo = "engine_rodata" )
RodataSym is the symbol the lowering gives engine_rodata in its text, and RodataGo the Go symbol emit writes for it.
const ( MinShadowSizeLog2 = 16 MaxShadowSizeLog2 = 40 )
The shadow-stack region the engine is generated for is 2^n bytes, n from MinShadowSizeLog2 (64 KiB: the margin the C keeps free, and the smallest stack the host maps) to MaxShadowSizeLog2 (1 TiB, far inside the address space). pass/shadow.cpp holds the same two numbers.
const HostLayoutSymbol = "js_host_layout"
HostLayoutSymbol is the table of int64 that engine/quickjs.c defines with the layout of values a host reads and writes in place (JS_HOST_LAYOUT_* in engine/quickjs.h).
const HostPrefix = "host_"
HostPrefix starts the name of every Go callback the engine's C may call. They are declared in engine/host.h and written in Go in the package of the generated code, which the linker resolves them to.
const ICallPrefix = "goasm.icall."
ICallPrefix starts the name of every indirect-call symbol.
const ICallUnencodable = 0xffffffff
ICallUnencodable is the id the pass gives a signature code the id cannot hold.
const PGOStaleMax = 30
PGOStaleMax is how many functions of the engine may have a profile that no longer fits their C before make gen warns that the profile is old: a few are the normal state between two trainings.
const PackedEntry = "_block"
PackedEntry is the suffix of the Go name of a packed function's generated entry, whose plain name goes to the Go function PackedGo writes.
const PackedSlot = `` /* 164-byte string literal not displayed */
PackedSlot is the helper PackedGo's functions store through: a uintptr that points outside the Go heap made a pointer without a conversion vet would question.
const PollCounterOffset = 384
PollCounterOffset is where the poll counter is by default, from the base of the shadow-stack region (pass/poll.cpp, -poll-offset of cmd/engine). The host keeps the start of the region for what it passes the engine by address, and this word among it.
const ProbePrefix = "goasm.probe."
ProbePrefix starts the name of the probe of a unified definition.
Variables ¶
var ( PGOOptFlags = PGOPinned PGOLLCFlags = []string{"-pgso=false"} )
PGOOptFlags pin the post-link optimizer as ProfileFlags pin clang, and PGOLLCFlags turn size optimization of cold code off in llc.
var ( EngineCFlags []string ARM64CFlags []string FlagsErr error )
EngineCFlags and ARM64CFlags are the clang flags of engine code for the two targets: the generator's own (engineOnly, arm64Only) and the portable ones of engine/test/flags.sh. If sh could not read that file they hold the generator's part only and FlagsErr says why: CompileUnit and the drivers stop on it.
var AMD64Stack = StackRules{Arch: "amd64", Local: func(frame int, calls bool) int { if frame == 0 && !calls { return 0 } return frame + 8 }}
AMD64Stack: a frame takes the saved base pointer on top, and a leaf with no frame takes nothing.
var ARM64Stack = StackRules{Arch: "arm64", Local: func(frame int, calls bool) int { if frame == 0 && !calls { return 0 } s := frame + 8 if s%16 == 8 { return s + 8 } return s + 16 }}
ARM64Stack: a frame is the TEXT size with the saved LR rounded up as cmd/internal/obj/arm64/obj7.go does: 8 more bytes when that is 8 past a multiple of 16, else 16, for the frame pointer. A leaf with no frame gets no frame at all (NOFRAME).
var DispatchCopies = map[string]int{"JS_CallInternal": 175}
DispatchCopies is the least number of indirect jumps each function with a computed goto table must have in llc's output: one at the end of each handler, where llc copied the dispatch, without which the interpreter is much slower. A function with a table and no entry here fails, so that a new one gets a number of its own. llc makes the copies after register allocation, where it copies a computed goto of up to 10 instructions (before, the phis of the dispatch hold it back): the interpreter counts its bytecode in its handlers (pass/poll.cpp), so its dispatch has no tick and is 4 instructions there, with polls or without. JS_CallInternal has 204 on amd64 and 208 on arm64, and its least number is about a seventh below.
var EngineValueFlags = []string{"-fno-slp-vectorize", "-Xclang", "-target-feature", "-Xclang", "+slow-unaligned-mem-16"}
EngineValueFlags are the x86-64 flags of quickjs.c, the one engine file that holds JSValues, on top of EngineCFlags. A JSValue is 16 bytes. The interpreter writes one to the value stack as two 8-byte stores (the value is in a register and the tag is a constant), and LLVM copies one with a single 16-byte SSE move: the SLP vectorizer joins two 8-byte moves into it, and the expansion of the memcpy of a struct copy makes it. On Zen 3 a 16-byte load cannot be forwarded from two 8-byte stores (it waits for both to reach the cache), and 8-byte loads of a 16-byte store are slow to forward. So the file is built without the SLP vectorizer and with +slow-unaligned-mem-16, which makes LLVM expand a copy of an object aligned to 8 bytes into 8-byte moves. The feature needs the SLP vectorizer off: with it on, the feature made the regexp scripts about 3% slower. The small libc is left out, since its memcpy and memmove move 16 bytes at a time on purpose. arm64 is not given them.
var GeneratorCallbacks = []string{"host_poll", "host_trap"}
GeneratorCallbacks are the Go callbacks the shadow-stack pass itself puts into the engine: host_poll where the code polls, host_trap for a trap instruction (see pass/shadow.cpp).
var HostLayoutNames = []string{
"layoutRefCountOffset",
"layoutStringLenMask",
"layoutStringWideBit",
"layoutStringChars",
"layoutStringLenMax",
"atomEmptyString",
"atomFileName",
"atomLineNumber",
"atomColumnNumber",
"atomMessage",
"atomStack",
"atomName",
"atomToString",
"atomThen",
}
HostLayoutNames are the Go names of the entries of the table, in the order of JS_HOST_LAYOUT_* in engine/quickjs.h.
var KeepNarrowSigs bool
KeepNarrowSigs makes SigFromCode take a narrow integer for a word: the driver sets it for the negative build of GEN_KEEP_NARROW (see cmd/engine), and a test of that build sets it too. A normal run leaves it off.
var PGOPinned = []string{
"-pgso=false",
"-loop-vectorize-with-block-frequency=false",
}
PGOPinned are the options of LLVM that keep a decision from the profile whose cost depends on the data, which the training cannot know.
var PGORequired = []string{"sqlite3VdbeExec"}
PGORequired are the functions make gen names when their profile no longer fits their C: the bytecode loop of SQLite, and a change of its C drops its whole profile.
var PostLinkOptArgs = []string{"-passes=default<O2>", "-inline-threshold=400", "-slp-threshold=1000000"}
PostLinkOptArgs are the options of the optimizer run over the linked module of the whole engine (the driver, cmd/engine), after the files are linked and before the shadow-stack pass. Each file was optimized alone by clang -O2, so this second -O2 sees the calls from one file to another (libc, tlsf and heap into the interpreter, the collector) and inlines the small functions it could not, which saves 1.5% to 1.7% of the instructions of a page. The inline threshold is the 400 of engine/test/flags.sh: the default 225 saves a little less, and 1000 makes the interpreter bigger and its frame 32 bytes taller.
The SLP vectorizer is off. -fno-slp-vectorize of EngineValueFlags is a clang option, not an attribute of the functions, so the optimizer cannot see that quickjs.c asked for it: with SLP on, a second -O2 joins the two 8-byte moves of a JSValue into one 16-byte move again (52 vector operations in JS_CallInternal), which EngineValueFlags exists to prevent. A threshold no tree can reach turns it off. +slow-unaligned-mem-16 and the other target options are attributes, so they stay with their file: quickjs.c and the small libc differ in them, and LLVM inlines a callee into a caller whose features include its own.
var RemarksFlags = []string{"-gline-tables-only", "-Rpass=inline"}
RemarksFlags are the flags of the first compile: the inlining decisions as remarks, with the line and column of each call site, which the replay names.
Functions ¶
func AddCallee ¶
AddCallee registers one external callee (for example a host callback) that is referenced but not generated here.
func BodyDataAlign ¶
BodyDataAlign is the alignment, as a power of 2, that FuncPlace keeps for the data in the body of function name: 16 bytes for a plain body, 64 for one put at its Align64 place.
func BuildExpr ¶
BuildExpr is the constraint of a generated file for arch: the architecture, and with a tag set, the tag as well.
func CheckBodyDataAlign ¶
CheckBodyDataAlign fails when an object in the body of a lowered function asks for more alignment than BodyDataAlign keeps. Such an object would sit on its alignment in one link and off it in another, and only the post-link check of a build would see it, so the generator stops here.
func CheckDispatchCopies ¶
CheckDispatchCopies counts the indirect jumps in the llc text of a function that has a computed goto table and fails when there are fewer than DispatchCopies asks.
func CheckIR ¶
CheckIR rejects IR shapes the ABI cannot carry across a movable Go stack: variadic functions, byval or sret aggregates, and long double.
func CheckObject ¶
CheckObject rejects a writable data section with content and any external relocation to a symbol outside the closed allow list (the engine's own functions plus the named host callbacks).
func CheckObjectData ¶
CheckObjectData is CheckObject, which with writable takes writable data sections, and references to the globals in them, as the whole-engine driver does once ExposeWritableData has run (see writable.go). Thread-local data stays an error.
func CheckPollOffset ¶
CheckPollOffset checks -poll-offset: a word inside the region.
func CheckShadowSizeLog2 ¶
CheckShadowSizeLog2 refuses a -shadow-size-log2 outside that range, before any tool is called with it.
func EmitARM64Engine ¶
func EmitARM64Engine(objPath string, e EngineEmit) (map[string]string, error)
EmitARM64Engine writes the Go assembly of the lowered engine from the object llvm-mc made of the lowered text: one TEXT per function, the body as WORDs, every call a BL or CALL (R9) Go sees, each address of data or of a function a MOVD $sym(SB) (which the Go assembler writes as the adrp and add it stands for), the arguments loaded from the Go argument area after Go's prologue and the result stored before its RET.
func EmitEngine ¶
func EmitEngine(objPath string, e EngineEmit) (map[string]string, error)
EmitEngine returns the generated .s files by name.
func EmitRodata ¶
EmitRodata writes engine_rodata as Go DATA: the bytes, the pointers to functions and to itself as relocations. Zero words are left to GLOBL.
func ExposeCopiedPointerTables ¶
ExposeCopiedPointerTables does for constants what ExposeWritableData does for writable globals, for the constants that hold pointers and that a memcpy or memmove copies. Such a copy is how C initializes a local array or struct (FTS5's table of built-in tokenizers), and llc expands it into direct loads of the constant (an arm64 ldr q of a label, an amd64 movups sym(%rip)). The lowering cannot copy the constant into the body, as its pointers need relocations. Reached through the GOT, it is read through a register that holds its address in engine_rodata. Every other constant is left alone. It returns the new module and the names it changed.
func ExposeWritableData ¶
ExposeWritableData makes every writable global defined in the module ll external and not dso_local, so that llc reaches it through the GOT. It returns the new module and the names of the globals it changed. A constant, a declaration (external) and LLVM's own globals (llvm.*) are left alone. Any other linkage or mark (common, weak, hidden, thread_local, ...) is an error: it has not been reviewed.
func FuncPlace ¶
FuncPlace is the assembly that puts function name in the lowered object where its body will sit in the binary, and ends with the line before its label. A body goes on a multiple of 16 bytes, or at its Align64 offset in a line of 64. The lowered text aligns each constant pool and jump table of a function with a .p2align, counted from the start of the function in the object, and the bytes are copied as they are. So the function has to start in the object at the place its body takes in the binary: then the alignment of an object holds where the code runs, and not only from the start of the body, up to 16 bytes for a plain body and up to 64 for an Align64 one. An object that asks for more is refused by CheckBodyDataAlign when the engine is generated. An SSE instruction with a pool for its memory operand faults when the pool is not on 16 bytes.
func HostLayout ¶
HostLayout reads the table HostLayoutSymbol from llc's text: one value per name of HostLayoutNames, from .quad lines and .zero runs of whole quads.
func HostLayoutDecls ¶
HostLayoutDecls writes the values of HostLayout as Go constants, nothing for none.
func ICallID ¶
ICallID encodes a signature code the way the pass does (icallID in pass/shadow.cpp): bit 31 set, the parameter count in bits 27 to 30, the result in bits 24 to 26 and two bits per parameter from bit 0. ok is false for a code it cannot hold.
func InlineRemarks ¶ added in v0.1.5
InlineRemarks keeps the lines of remarks that say a call was inlined: the input of -cgscc-inline-replay. The rest (the remarks of calls not inlined, the source lines clang prints under each remark, warnings) goes: a line of the C that holds the words of a remark is not one.
func IsHostCallback ¶
IsHostCallback says whether name is a Go callback of the host.
func Layout ¶
Layout gives the Go argument area of a signature: the offset of each parameter, of the result, and the total size.
func MisexpectLines ¶ added in v0.1.5
MisexpectLines keeps the warnings of -Wmisexpect from the output of clang: a likely() or unlikely() of the C that the training ran the other way.
func NoSplitReport ¶
func NoSplitReport(l *NoSplitList, metas []Meta) string
NoSplitReport is the list as the generator writes it down: one line per function, its kind (auto or register), frame and arguments.
func PackedGo ¶
PackedGo writes the Go function a host calls a packed function by. The pass gives a function whose arguments need more integer registers than a unified entry has (six on arm64, ten on amd64) a block of them instead: it then takes the shadow pointer and the block's address. The Go host calls the engine's API by the C signature on every target, so the function takes the arguments of was (the signature code the function had, the shadow pointer first), stores them in a block at the shadow pointer, each in an 8-byte slot in order (16 bytes for a pair) as the pass lays the block out, and calls entry with the shadow pointer above the block. The block fits: the engine keeps 64 KiB of the region free past any shadow pointer it gives out, and a block is a few words. The function panics where the region would end inside the block all the same, which no host does, since the engine checks the room its own frames take but not this.
ok is false when was has an argument the host cannot pass by value (an aggregate passed by copy, a type the generator does not know): such a function gets no Go function, and a host call of it does not compile.
func ParseICallSymbol ¶
ParseICallSymbol splits an indirect-call symbol name into the function, the call's index in it and the type id.
func PostLinkOpt ¶
PostLinkOpt runs the optimizer with args over the IR file in and writes the IR file out.
func PostLinkOptFor ¶
PostLinkOptFor says whether the driver runs PostLinkOptArgs over the linked module of arch by default. It is on for both architectures.
func ProfileFlags ¶ added in v0.1.5
ProfileFlags are the flags of the second compile besides the replay. The line tables keep the call sites of the replay matching those of the remarks and are stripped after the post-link optimizer. -Wmisexpect names each likely() or unlikely() the training ran the other way more often than not (the tolerance of 50 percent), see MisexpectLines.
func ProfileNames ¶ added in v0.1.5
ProfileNames lists the functions of a text profile (llvm-profdata merge --text): the line before each "# Func Hash:". A static function is named file:name, as !PGOFuncName names it.
func ReplayClangArgs ¶ added in v0.1.5
ReplayClangArgs makes clang inline exactly the calls of remarks and no other.
func ReplayOptArgs ¶ added in v0.1.5
ReplayOptArgs is ReplayClangArgs for opt.
func SetCallees ¶
func SetCallees(specs []FuncSpec)
SetCallees records the signature and kind of every function the unit may call, for the lowering of internal calls.
func SharedCFlags ¶
SharedCFlags runs a flags.sh in sh, as the scripts of the C tree do, and returns its engine_flags followed by engine_target_flags for a clang target triple.
func SortedKeys ¶
SortedKeys returns the keys of a set in order.
func StaleWarnings ¶ added in v0.1.5
StaleWarnings are the lines make gen prints when the profile is behind the C: a function of required that has no profile (the branches and loops of its C changed since the training, or it is new), and more than max functions whose profile no longer fits. make gen goes on: a function without a profile compiles as the build without one does and only loses the gain. When to train again is in docs/upstream.md.
func StripCGProfile ¶ added in v0.1.5
StripCGProfile takes the "CG Profile" module flag out of a module. clang writes one into each file it compiles with a profile and llvm-link merges them, then the post-link optimizer adds a second, which the verifier refuses (two flags of one name). Left in after the optimizer, it makes llc write .cg_profile directives, which the lowering does not take. LLVM 22 has no switch for it. The flag only orders functions for a linker, which the Go linker is not.
func UnprofiledFunctions ¶ added in v0.1.5
UnprofiledFunctions reads the IR clang writes with a profile before any optimization (-Xclang -disable-llvm-passes) and gives the functions that have no profile: those the profile names have one that no longer fits their C (stale), the others are new since the training. Names are the profile's: file:name for a static function.
func ValueFlagsFor ¶
ValueFlagsFor gives the extra clang flags of one engine file: EngineValueFlags for quickjs.c on amd64, none for anything else (group is the name of the file without .c, or "libc").
Types ¶
type AMD64Arch ¶
type AMD64Arch struct{ PreserveNone, SaveAll, Poison bool }
amd64 implements Arch for linux/darwin/windows amd64 (GOAMD64=v1).
PreserveNone lowers code compiled with preserve_none on every unified entry: arguments arrive in R12, R13, R14, R15, RDI, RSI, RDX, RCX, R8, R9, R11, RAX and nothing survives a call but RSP and RBP, so call sites save nothing. The entry still loads the arguments only after Go's prologue, which uses R12 and R14. SaveAll saves all five SysV callee-saved registers around every unified call instead of the ones the body uses (for the cost comparison only). Poison is a negative build: it writes garbage into the registers at each entry, after each unified call and at each return (see poison in lower.go).
func (AMD64Arch) CalleeSaved ¶
func (AMD64Arch) GoFramePrologueBytes ¶
Return address (8) plus the base pointer Go pushes (8): the incoming argument area sits that far above SP once the prologue has run.
func (AMD64Arch) IntArgRegs ¶
func (AMD64Arch) RetIntRegs ¶
func (AMD64Arch) SSEArgRegs ¶
type ARM64CallStub ¶
ARM64Stub is a call stub of the stub mode: a direct one to a Go symbol, or (Target "") an indirect one through R9.
type ARM64Env ¶
type ARM64Env struct {
Plan *DataPlan
Funcs map[string]bool
// Facts are llc's registers of every function and call (facts.cpp),
// Probes those of every unified definition's arguments. When set, each
// entry must load its arguments into the registers llc's probe reads
// them from, and each call must store its arguments from the registers
// llc passes them in and load its result into the ones llc reads.
Facts, Probes RegFacts
// Variadic are the functions the pass left variadic: a call to one is
// refused, since its arguments past the registers go on the stack.
Variadic map[string]bool
// Align64 is EngineEmit.Align64: where the body of each of these functions
// starts in a line of 64 bytes, which FuncPlace puts the function at in
// the lowered object.
Align64 map[string]int
// RegResult is LowerEnv.RegResult: the functions whose result a direct
// call takes from x0, x1 or d0, where they stored it from.
RegResult map[string]bool
}
ARM64Env is what lowering one function of the whole engine needs beyond its own text.
type ARM64Func ¶
type ARM64Func struct {
Meta
Sig Sig
Leaf bool // no stack use and no call
LLVMFrame int // F
SaveSlots int // call-site save slots after the LLVM frame
}
ARM64Func is one lowered function.
type ARM64Mode ¶
type ARM64Mode int
ARM64Mode picks how a unified call keeps the AAPCS callee-saved registers.
const ( // ARM64Stub calls a NOSPLIT stub per callee that saves and restores // all of R19-R25 and F8-F15 and moves the arguments (the first backend). ARM64Stub ARM64Mode = iota // ARM64SaveAll moves the arguments at the call site and saves all 15. ARM64SaveAll // ARM64SaveBody saves at each call site the callee-saved registers the // calling function's body uses: the ones LLVM's prologue saved. ARM64SaveBody // ARM64PreserveNone lowers code compiled with preserve_none on every // unified entry and host callback: LLVM keeps across a call only what is // live, in its frame, and no call site saves a register. ARM64PreserveNone )
func ParseARM64Mode ¶
type ARM64Unit ¶
type ARM64Unit struct {
Mode ARM64Mode
NoSaves bool // negative build: no call keeps a register
Funcs []ARM64Func
Stubs []ARM64CallStub
Lowered string
// Unified call sites and registers saved around them, over the unit.
CallSites, SavedRegs int
}
ARM64Unit is the lowered unit.
type Arch ¶
type Arch interface {
Name() string
// IntArgRegs / SSEArgRegs give the native argument registers in order.
IntArgRegs() []string
SSEArgRegs() []string
// CalleeSaved are the native callee-saved GP registers a body may hold a
// value in. A unified call clobbers them, so the caller saves the live
// ones to its frame.
CalleeSaved() []string
// RetIntRegs / RetSSEReg give where a native body leaves its result.
RetIntRegs() []string
RetSSEReg() string
// GoFramePrologueBytes is the distance Go's prologue adds between the
// frame base and the incoming argument area (return address + saved BP).
GoFramePrologueBytes() int
}
Arch abstracts the target so the lowering and checks are architecture independent. amd64 is implemented here; arm64 is a separate Arch.
type AsmUnit ¶
type AsmUnit struct {
Funcs map[string]*AsmFunc
FuncOrder []string
Data map[string]*DataObj // by label and by every alias
DataOrder []*DataObj
// LabelFunc maps each local text label (.LBB, .Ltmp) to its function.
LabelFunc map[string]string
// Sets are the symbol assignments (name = target), such as musl's weak
// aliases.
Sets map[string]string
}
AsmUnit is llc's text split up.
type BlobDiff ¶
BlobDiff is a value inside engine_rodata that PlanData resolved from a difference of two labels (Pos - Neg + K, the relative tables LLVM builds), declared so the post-link check can recompute it in llc's object and in the linked blob.
type BlobReloc ¶
type BlobReloc struct {
Off int
Sym string
Add int64
Func bool
// Label and LabelAdd are the pointer as llc's text wrote it (label plus
// addend), for the post-link check to resolve in llc's object.
Label string
LabelAdd int64
// GOT marks the slot of a writable global (DataPlan.GOTSlot): a pointer
// to Label in engine_rodata that no object of llc's holds.
GOT bool
}
BlobReloc is an 8-byte pointer inside engine_rodata: to Sym+Add, where Sym is a function (Func) or engine_rodata itself.
type CallFacts ¶
CallFacts is one call: its target (a symbol, or "*" and the kcfi type id of an indirect call), its argument registers in order, the registers it defines (its results) and the ones live right after it.
type CallRecord ¶
CallRecord is one call: its target (a C symbol, or the register of CallReg) and the signature it was marshalled with. RegResult says the call of CallUnified loads no result back from the Go stack: the callee is a lowered unified function, which the post-link check holds to storing its result from the registers the call reads it from and then running only Go's epilogue, so they still hold the result after its RET.
type CheckReport ¶
CheckReport counts what was checked and lists what was not, by reason.
func CheckEngine ¶
func CheckEngine(c EngineCheck) (*CheckReport, error)
CheckEngine runs the instruction-level and data checks.
func CheckEngineARM64 ¶
func CheckEngineARM64(c EngineCheck) (*CheckReport, error)
CheckEngineARM64 runs the check on c.Bin, a linked linux/arm64 binary of the package c.Pkg, against c.Lowered.
type DataObj ¶
type DataObj struct {
Label string
Aliases []string
Section string
Align int // log2 of the alignment
// AlignCond: the start alignment had a max-skip below its step, so llc
// may have placed the object without it.
AlignCond bool
Lines []string // data directives, cleaned
Global bool
Order int
}
DataObj is one labelled object of a data section in llc's text.
type DataPlan ¶
type DataPlan struct {
Blob []byte
BlobOff map[string]int // label or alias -> offset in engine_rodata
Relocs []BlobReloc
Diffs []BlobDiff
// InBody lists, per function, the objects placed after its code: its
// own (constant pools, jump tables, tables of its labels) and the
// copies of objects it reads without LEA.
InBody map[string][]*DataObj
// Rename maps, per function, a data label to the label its body uses.
Rename map[string]map[string]string
// Errors are per function (key "" for data that belongs to none).
Errors map[string][]error
// Stats for the report.
Objects, BlobObjects, InBodyObjects, DupObjects, Dropped int
// Writable says engine_rodata holds writable globals: it is written
// without RODATA.
Writable bool
// GOTSlot is, for each writable global whose GOT load x86 folded into a
// compare or an add, the offset of an 8-byte slot at the end of
// engine_rodata that holds its address: the lowering reads the slot
// where llc read the GOT entry (writable.go).
GOTSlot map[string]int
// Pad is, per function and per copy label, the zero bytes written after
// a copy that llc reads past its end. Such a read comes from code that
// copies a variable length out of a constant (an inlined memcpy whose
// source llc knows) and runs only when the length is longer than the
// constant, which the C never does. llc's bytes past the object are
// whatever follows it, so the copy reads zeros instead and the read stays
// in the function's data.
Pad map[string]map[string]int
// contains filtered or unexported fields
}
DataPlan says where every data object goes.
func PlanData ¶
PlanData classifies the data objects of an amd64 unit. funcs are the functions of the module (defined or external).
func PlanDataARM64 ¶
PlanDataARM64 does the same for an aarch64 unit of the tiny code model, whose code takes the address of data with an adr and reads it with a load of a label, and whose .word is 4 bytes.
func (*DataPlan) RenameLine ¶
RenameData rewrites the data labels a body line names, for the body of fn.
type EngineCheck ¶
type EngineCheck struct {
LLC string // llc's object of the engine, with temporary labels kept
Lowered string // the re-assembled lowered text, with temporary labels kept
Bin string // a linked Go binary of the generated package
Pkg string // the package's link symbol prefix
Names NameMap
Unit *AsmUnit
Plan *DataPlan
Metas []Meta // lowered functions (unified and register-entry)
Stubs []Meta // in missingNames order
// Sigs and Leaves are the pass's signature of every function and the
// register-entry leaves it was given, and PreserveNone the convention it
// gave every unified entry: the check holds each function's record to
// them.
Sigs map[string]Sig
Leaves map[string]bool
PreserveNone bool
// Probes and Facts are what llc itself decided about registers
// (facts.go): Probes from llc's run over the probe module of
// -shadow-probes, which gives the argument and result registers of each
// unified definition, Facts from the run that wrote LLC, which gives
// each function's calls, returns and live registers.
Probes, Facts RegFacts
// All requires every function to be in the binary and checked.
All bool
// Copied are the constants ExposeCopiedPointerTables sent through the
// GOT: read-only, but reached as writable globals are.
Copied map[string]bool
// Align64 is EngineEmit.Align64: the body of each of these functions
// starts that many bytes past a multiple of 64, after the PCALIGN $64
// that replaces the usual PCALIGN $16 and the no-ops that follow it.
Align64 map[string]int
// HostPoll says the engine calls host_poll even though llc's object does
// not name it (it is always required when the object does). The poll is
// the only place the Go runtime can stop C code to collect garbage, so the
// Go function must begin with a stack check: a function without one would
// let a loop of C run on without a safe point.
HostPoll bool
}
EngineCheck is what the post-link check reads.
type EngineEmit ¶
type EngineEmit struct {
Metas []Meta // lowered functions, in output order
Names NameMap // C symbol -> Go identifier
// Group gives the file group of each function (its C source file);
// MaxBytes bounds the size of one .s file.
Group map[string]string
MaxBytes int
// Stubs are functions emitted as a call to ·goasmMissing(index). With
// NoStubs (a release build) any stub is an error.
Stubs []Meta
NoStubs bool
Plan *DataPlan
Arch string // file name suffix and build tag
// Tag, when set, is a build tag that every file requires besides the
// architecture (see BuildExpr).
Tag string
// Align64 maps a function to where its body starts in a line of 64
// bytes, a multiple of 8 below 64: a PCALIGN $64 takes the place of the
// PCALIGN $16 after Go's prologue (which also puts the function itself
// on 64 bytes), followed by that many bytes of 8-byte no-ops. The no-ops
// run once per call. A change elsewhere in the program then moves the
// body by whole lines of 64 bytes only.
Align64 map[string]int
}
EngineEmit says what to write.
type FuncFacts ¶
FuncFacts is what llc decided for one function: the live-in registers of its arguments in order (only the ones its code reads, which for a probe of -shadow-probes is all of them), the registers its code reads before writing them, each call and each return in code order. Registers are llc's names, every name of a register that is live (eax and al beside rax).
type FuncRecord ¶
type FuncRecord struct {
Name string
Kind Kind
Sig Sig
PreserveNone, SaveAll bool
// NoSplit: the function is on the generator's NOSPLIT list (Meta.NoSplit),
// so the binary has no stack check at its entry, and every unified
// function that is not on it has one.
NoSplit bool
// The canonical prologue the lowering removed: the callee-saved
// registers it pushed, in order, and the bytes it reserved below them.
Pushed []string
Local int
// The Go frame: locals at LocalBase, saved registers at SaveBase, the
// outgoing argument area at 0, Frame bytes in all.
LocalBase, SaveBase, Frame int
// Saved are the registers saved around every unified call.
Saved []string
// Calls and Refs are in llc's order: every call, and every RIP-relative
// reference to data or to a function.
Calls []CallRecord
Refs []RefRecord
}
FuncRecord is every rewrite the lowering made in one function.
type FuncSpec ¶
type FuncSpec struct {
Name string
Sig Sig
Kind Kind
// Indirect lists, in source order, the signature of each call this
// function makes through a function pointer. The function table and every
// indirect call target a unified entry, so these marshal like a direct
// unified call.
Indirect []Sig
}
FuncSpec names a function in the compiled unit and how to generate it.
type IRViolation ¶
type IRViolation struct {
Func, Rule, Line string
}
IRViolation is one rule a function of the C breaks, as the IR shows it.
func CheckIRFuncs ¶
func CheckIRFuncs(ll string) []IRViolation
CheckIRFuncs is CheckIR over a whole module that names the function of each violation instead of stopping at the first. A call of a variadic function counts as a violation of the caller too ("variadic call").
type Kind ¶
type Kind int
Kind is one of the two entry conventions of the plan.
const ( // Unified entry: arguments arrive on the Go stack (ABI0), the body reads // them after Go's split-check prologue, and the caller relies on no // callee-saved register surviving the call. The shadow-stack pointer is // the first stack argument. Function tables, address comparisons and // indirect calls all use this entry. Unified Kind = iota // Register entry: a NOSPLIT leaf that neither grows the stack nor calls a // split-checked function. Arguments stay in the native argument registers. // Only a direct call to a known leaf may target it. Register )
type LowerEnv ¶
type LowerEnv struct {
Plan *DataPlan
Funcs map[string]bool
// Variadic names the variadic functions. A call to one is rejected
// unless LooseVarargs, which lowers it with its fixed arguments only
// (wrong code, for measuring the output while the C still has them).
Variadic map[string]bool
LooseVarargs bool
// RegResult names the functions whose result a direct call takes from
// the registers they return it in instead of loading it from the Go
// stack (CallRecord.RegResult): the unified functions the driver lowers,
// never a Go callback, a stub or a register-entry leaf. A poison build
// takes none, since it writes over every register at a return.
RegResult map[string]bool
}
LowerEnv is what lowering a function of the whole engine needs beyond its own text: where its data goes and which symbols are functions. The signature of each indirect call is the one LLVM bound to it, which llc writes as a goasm.icall label right before the call (icall.go).
type Meta ¶
type Meta struct {
Name string
Kind Kind
GoName string // exported-to-asm Go identifier, e.g. recAsm
Frame int // Go TEXT frame size
ArgBytes int // Go TEXT argument+result area size
RetType Type
// NoSplit says the TEXT has NOSPLIT, so Go puts no stack check at the
// entry (MarkNoSplit): a register-entry leaf, or a leaf the Go
// assembler marks NOSPLIT by itself.
NoSplit bool
// Rec is every rewrite the lowering made, for the instruction-level
// post-link check (nil for the prototype's units).
Rec *FuncRecord
}
Meta is what the emit stage needs after re-assembly.
func LowerARM64EngineFunc ¶
LowerARM64EngineFunc lowers one function of the whole engine from the body lines of llc's text (after its label, up to .Lfunc_end). The result is the text for llvm-mc, from the .globl of the function to its .size.
func LowerRegister ¶
LowerRegister transforms a frameless leaf (compiled with -fomit-frame-pointer) into a NOSPLIT body. It rejects anything with a frame, a call or a forbidden instruction.
func LowerUnified ¶
LowerUnified transforms one function's GNU assembly. It returns the rewritten GNU text (one label, a trailing return label, calls left symbolic) and the Meta for emit.
func LowerUnifiedFunc ¶
func LowerUnifiedFunc(a AMD64Arch, lines []string, fn FuncSpec, env *LowerEnv, asm string) (string, Meta, error)
LowerUnifiedFunc lowers the body lines of one function (after its label, up to .Lfunc_end). With env nil it is the prototype's unit: constant pools are found in asm and indirect calls take fn.Indirect in order.
type NoSplitList ¶ added in v0.1.5
type NoSplitList struct {
// Set holds every one of them, Auto the leaves the assembler marks
// NOSPLIT by itself and Register the register-entry leaves.
Set map[string]bool
Auto, Register []string
}
NoSplitList is the functions whose TEXT says NOSPLIT.
func MarkNoSplit ¶ added in v0.1.5
func MarkNoSplit(r StackRules, metas []Meta) (*NoSplitList, error)
MarkNoSplit sets Meta.NoSplit, and NoSplit of its record, on the functions without a stack check: the register-entry leaves and the unified leaves the Go assembler marks NOSPLIT by itself. It fails when a register-entry leaf calls anything but another one, when they call each other in a cycle, or when one takes autoNoSplitFrame bytes or more below its entry with the leaves it calls (8 bytes for each call, on arm64 too).
type RefRecord ¶
type RefRecord struct {
Label string
Add int64
Func, Blob bool
// GOT is a load of the address of a writable global from the GOT that
// the lowering made a LEA of it in engine_rodata (writable.go). With
// Slot, x86 had folded the load into a compare or an add, and the
// lowering reads the global's slot in engine_rodata instead
// (DataPlan.GOTSlot).
GOT, Slot bool
}
RefRecord is one RIP-relative reference as llc's text wrote it, label plus addend, and where the lowering pointed it: a function, engine_rodata, or the function's own copy of the data.
type RegFacts ¶
RegFacts are the facts of one llc run, by function.
func ParseRegFacts ¶
ParseRegFacts reads the plugin's output.
type Report ¶
type Report struct {
// Sigs gives the code of every function the module defines or calls
// (see SigFromCode), Defined whether the module defines it, Leaf
// whether it is a register-entry leaf.
Sigs map[string]string
Defined map[string]bool
Leaf map[string]bool
// AddrTaken lists the functions whose address is used other than by a
// direct call.
AddrTaken map[string]bool
// Calls is the direct call graph, ICalls the signature codes each
// function calls through a pointer.
Calls map[string][]string
ICalls map[string][]string
// FrameAddr lists the functions that read the machine frame address
// (QuickJS's stack overflow checks), Packed the ones the pass gave an
// argument block, Rejected the ones the pass emptied, with the reason.
FrameAddr map[string]bool
Packed map[string]bool
// PackedWas is the signature code a packed function had before the
// pass gave it the block.
PackedWas map[string]string
PackedICalls map[string]int
Rejected map[string]string
GotoTable []string
Mismatch [][2]string
// Varargs lists the variadic definitions -shadow-varargs rewrote.
Varargs []string
}
Report is what the shadow-stack pass printed with -shadow-report.
func ParseReport ¶
ParseReport reads the pass's report lines; other lines are ignored.
type Sig ¶
Sig is the signature of a generated function in logical order. By convention Params[0] of a Unified function is the shadow-stack pointer (Ptr).
func SigFromCode ¶
SigFromCode turns a pass signature code into a Sig. Every integer or pointer argument is one word, moved as the whole register, so whatever extension the C convention relies on survives the Go argument area. 'j' is a pair of 64-bit integers (a JSValue). A narrow integer ('n') is an error: the pass passes every one at the unified boundary as an i32, since preserve_none would put it in another register than the one the generator moves it in, and a C callee may rely on an extension Go does not make. A float ('f') is an error too: Go writes and reads each word as a whole float64, and no unified function of the engine takes or returns one. Variadic, byval ('B') and unknown ('?') types are errors: the IR check names the C that has them.
With KeepNarrowSigs, a negative build (the pass leaves narrow integers with -shadow-keep-narrow), 'n' is a word like 'i', as the generator once took it, and the post-link check must reject what comes of it.
type StackRules ¶
type StackRules struct {
Arch string
// Local is the bytes a frame of the TEXT size frame takes below the
// entry, for a function that calls (or calls nothing).
Local func(frame int, calls bool) int
}
StackRules is how the Go assembler of an architecture lays out a frame.
type Type ¶
type Type int
Type is a value a generated function passes or returns. Every type maps to non-Go memory, so the paired Go declaration uses uintptr/int64/float64 and carries no Go pointer: the argument area needs no pointer map beyond the empty one.
type Unit ¶
type Unit struct {
IR, ShadowIR, Asm, Obj string // paths in the work dir
Shadow string // the pass's report: one line per function
}
Unit is one C file compiled through the shadow-stack pass.
func CompileUnit ¶
func CompileUnit(cfg UnitConfig, cfile string) (*Unit, error)
CompileUnit compiles cfile with clang, runs the IR checks and the shadow-stack pass, and writes assembly and an object with llc.
type UnitConfig ¶
type UnitConfig struct {
LLVM, Pass, Work string // LLVM bin dir, built pass, work dir
CFlags []string // clang flags (EngineCFlags or ARM64CFlags)
PassArgs []string // extra shadow-stack pass options
Leaves []string // register-entry leaves: no shadow pointer
}
UnitConfig says how to compile a unit.
type VerifyReport ¶
func VerifyLinked ¶
func VerifyLinked(objPath, binPath, pkg string, metas []Meta) ([]VerifyReport, error)
VerifyLinked checks the functions of metas inside binPath against objPath.