hex

hex

A drop-in fast path for hexadecimal (base16) encoding and decoding, byte- and error-identical to the standard library's encoding/hex. Both directions run a SIMD kernel generated by go-asmgen; the short tail reuses an encoding/hex-equivalent scalar loop, so output and errors match exactly. Pure Go, CGO_ENABLED=0, stable Go, no GOEXPERIMENT.

s := hex.EncodeToString(data)         // same bytes as encoding/hex.EncodeToString
b, err := hex.DecodeString(s)         // same bytes + same errors as encoding/hex

API mirrors encoding/hex: Encode, EncodeToString, Decode, DecodeString, EncodedLen, DecodedLen.

Six architectures are wired into go-asmgen; three of them now have hand-tuned SIMD hex kernels (amd64, ppc64le VSX, s390x vector facility), the rest fall back to the encoding/hex scalar loop. The ppc64le and s390x kernels are QEMU-validated (full table + fuzz, byte- and error-identical to encoding/hex); s390x is additionally measured on real IBM z15 (VXE2), 2026-07-03, -count=6: encode ~18×, decode ~3.1× the encoding/hex scalar path. ppc64le native throughput is pending real POWER hardware.

How encode works

Per 16 input bytes (SSE; 32 for AVX2): split each byte into its high and low nibble (PSRLW $4 + PAND 0x0f for the high nibble, PAND 0x0f for the low), map each nibble 0..15 to its ASCII digit via a PSHUFB lookup of the table "0123456789abcdef", then interleave the two nibble streams with PUNPCKLBW/PUNPCKHBW so each input byte's (hi, lo) characters land adjacent, and store. The AVX2 path doubles the width and recombines the two 128-bit lanes with VPERM2I128. Constants are emitted via go-asmgen's emit.File.Data. Verified against encoding/hex (table + fuzz, native arm64 + amd64 under Rosetta).

How decode works

Per 32 input chars (SSE; 64 for AVX2) the decoder maps each ASCII hex char to its 0..15 nibble value and, in parallel, detects any invalid char:

• Validity — a char is valid iff it lies in ['0','9'] OR ['A','F'] OR ['a','f']. Each range is two PCMPGTB (signed; all hex chars are < 0x80 so the sign bit is irrelevant) AND'd together; the three range masks are OR'd into a per-char "valid" mask.
• Nibble — digit → c-'0', upper → c-('A'-10), lower → c-('a'-10); each subtraction is masked by its range mask and OR'd, so every valid char yields its 0..15 nibble.
• Fuse — PMADDUBSW with a {16,1,16,1,…} multiplier turns each adjacent (hi,lo) nibble pair into one byte hi*16+lo == (hi<<4)|lo (no overflow, each nibble < 16); PACKUSWB packs the two halves into the output bytes. The AVX2 path fixes the per-lane PACKUSWB/PMADDUBSW interleave with a single VPERMQ $0xD8.

The kernel never tries to pin-point an invalid byte in SIMD. The per-char "valid" masks are inverted and accumulated; if a 32-char (or 64-char) block contains any invalid char the kernel stops and returns the count of fully decoded blocks, and a scalar encoding/hex-equivalent loop resumes from there. That loop produces the exact InvalidByteError offset and ErrLength (odd input decodes floor(n/2) bytes before erroring) — so Decode / DecodeString are byte- and error-identical to the standard library. Verified against encoding/hex (table + fuzz: invalid bytes at every offset in both nibble positions, across block boundaries; native arm64 scalar + amd64 SSE under Rosetta; AVX2 force-tested on native CI hardware).

ppc64le (VSX) and s390x (vector facility)

Both ports map the PSHUFB/PMADDUBSW amd64 logic onto vector permute:

• Encode — load 16 source bytes, split into nibbles (VAND 0x0f for the low nibble; VSRB/VESRLB $4 for the high), map each nibble to its ASCII digit with a single VPERM of the 16-entry table "0123456789abcdef", then interleave the hi/lo ASCII streams with two VPERM passes driven by control vectors so each input byte's (hi, lo) characters land adjacent, and store.
• Decode — load two 16-char halves, range-check each char with unsigned compares (VCMPGTUB on ppc64le, VCHLB on s390x) for ['0','9']/['A','F']/ ['a','f'], derive the nibble by masked subtraction, accumulate a "bad" mask and bail to the scalar tail on any invalid char (so the InvalidByteError offset stays bit-exact), then VPERM-gather the even/odd nibbles, hi<<4 | lo (VSLB/VESLB $4 + VOR), and store.

ppc64le uses LXVB16X/STXVB16X (natural memory byte order on little-endian, sidestepping the LXVD2X doubleword swap), minding the VSX↔VMX Vn == VS(32+n) aliasing. s390x is big-endian: VL puts the first memory byte into lane 0 (the leftmost/high-order lane), so the per-byte high-nibble- first ordering is fixed by the VPERM control vectors, not by endianness — a position-dependent test (0x12 0x34 → "1234") guards it. Both ports are validated under QEMU with the full differential table test and coverage-guided FuzzEncode/FuzzDecode (byte- and error-identical to encoding/hex); see the qemu CI job.

Performance

Encode/decode throughput on a 1 MiB random buffer, native amd64 (GitHub Actions ubuntu-latest), -count=6, median MB/s — see .github/workflows/bench.yml. The dev box is arm64, where this package's amd64 kernel only runs under Rosetta (unrepresentative), so these numbers come from CI on native hardware (GitHub Actions ubuntu-latest, AMD EPYC, AVX2-capable, GOAMD64=v1), median of 6.

On decode this package is 1.37× faster than tmthrgd/go-hex and 6.24× over stdlib. The forced SSE-only paths are ~16800 MB/s encode and ~6329 MB/s decode; the AVX2 dispatch above is the default on AVX2 hardware (~13063 MB/s decode).

Re-bench ratios (as of 2026-06-14, -count=6 medians, kernels unchanged). amd64 measured on the local x86_64 QEMU VM (absolute MB/s run low; ratios are the signal); arm64 native on Apple Silicon:

Verdict holds: on amd64 this package still edges tmthrgd/go-hex on encode and clearly leads on decode. arm64 has no SIMD kernel yet (NEON planned), so encode/decode track stdlib; the ~9.7× over tmthrgd on arm64 decode is because tmthrgd has no arm64 SIMD path either and its scalar decode is much slower than encoding/hex. Native-CI absolute numbers above are unchanged (gh logs unavailable in this env).

s390x — measured on real IBM z15; ppc64le — llvm-mca cycle-model estimate

s390x — measured on real IBM z15 (VXE2), native execution, 2026-07-03, -count=6: the vector-facility hex kernel runs encode ~18× and decode ~3.1× the encoding/hex scalar path — hex encode is the SIMD sweet spot (a branchless VPERM table lookup). This supersedes the s390x cycle-model estimate below (which projected ~16×).

For ppc64le the row below is static analysis, NOT a hardware measurement; native POWER perf pending real silicon. qemu's TCG is not cycle-accurate, so the cycle model is the only defensible signal. Numbers from llvm-mca (LLVM 22) fed the straight-line encode inner loop (encodeBlocksVSX / encodeBlocksVX, 16 input bytes → 32 hex bytes/iter) translated to LLVM asm; no data-dependent branch, so the model is the whole story. Scalar baseline: the per-input-byte two-nibble-LUT loop modeled the same.

Hex encode is the SIMD sweet spot: a pure VPERM table-lookup with no branches, so the VSX kernel is estimated ~6× (ppc64le) on the cycle model, and the s390x kernel — modeled at ~16× — now measures ~18× encode on real z15 (its z14 model retires the four VPERMs + load + two stores in just 3 cycles). Caveats: llvm-mca idealizes the frontend (perfect dispatch, no branch misprediction, no cache/store-buffer stalls), so these are compute upper bounds; the scalar baseline's nibble-LUT loads are assumed L1-resident. The decode kernels (with their VCHLB range validation) would model lower but were not isolated here — encode is the cleaner straight-line loop. All instructions were accepted and modeled by llvm-mca (no fallbacks).

Notes:

• tmthrgd/go-hex is the prior pure-Go SIMD hex codec (hand-written Plan9 asm, SSE/AVX). It was archived upstream in September 2025. It also ships a SIMD decoder; this package now has its own SIMD decoder (SSE2/SSSE3 + AVX2) too, so encode and decode are both vectorised and competitive.
• Go's standard encoding/hex is scalar (byte-by-byte); see golang/go#68188, the open proposal to add SIMD to the standard library.

Regenerating the assembly

The per-arch .s kernels are committed. To regenerate (go-asmgen is a build-time tool, not a runtime dependency):

go get github.com/go-asmgen/asmgen@v0.5.0
go run encode_gen.go            # amd64
go run decode_gen.go            # amd64
go run encode_ppc64le_gen.go    # ppc64le VSX
go run decode_ppc64le_gen.go    # ppc64le VSX
go run encode_s390x_gen.go      # s390x vector facility
go run decode_s390x_gen.go      # s390x vector facility
go mod edit -droprequire github.com/go-asmgen/asmgen
go mod edit -go=1.20
go mod tidy

Coverage

The CI gate enforces 100% coverage of the Go code on every arch job: native amd64 + native arm64, plus the QEMU-emulated ppc64le and s390x jobs (the generic fallback compiles and is measured on arm64; the VSX and vector-facility dispatch + kernels are measured under emulation). Coverage is of the Go statements only: the generated .s SIMD kernels are not measured by go test -cover — they are validated by differential tests against the scalar encoding/hex reference (including invalid bytes at every offset) plus fuzzing.

License

BSD-3-Clause.