hex

package
v1.3.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2026 License: Apache-2.0 Imports: 4 Imported by: 0

README

encoding/hex

Standard-library-compatible hexadecimal encoding and decoding with separate SIMD backends for Go and TinyGo. Change the import:

import "github.com/shibukawa/tinygodriver/encoding/hex"

encoded := hex.EncodeToString(payload)
decoded, err := hex.DecodeString(encoded)

Encode, Decode, AppendEncode, AppendDecode, EncodeToString and DecodeString use the fast path. Encoding emits lowercase; decoding accepts both letter cases. InvalidByteError and ErrLength retain standard-library identity, including partial output and the invalid-character precedence for odd-length input. In-place decoding is supported.

NewEncoder, NewDecoder, Dump and Dumper delegate to the standard library. Their internal streaming/formatting loops are not SIMD accelerated. DecodeString borrows its immutable input without a byte-slice copy; all decode kernels read their source without modifying it.

Backend selection

Build Backend
Go, GOEXPERIMENT=simd, arm64 simd/archsimd, NEON
Go, GOEXPERIMENT=simd, amd64 with AVX2 simd/archsimd, runtime CPU check
TinyGo with cgo, darwin/linux/windows arm64 Clang vectors, NEON
TinyGo with cgo, darwin/linux/windows amd64 Clang vectors, baseline SSE2
Other builds Standard library
GOEXPERIMENT=simd go build ./your/app
tinygo build -opt=2 -o app ./your/app

SIMD starts at 32 raw bytes for encoding or 64 hex characters for decoding. The encoder reads 16 bytes and writes exactly 32 characters per step. The decoder validates all 32 input characters before writing their 16 decoded bytes; rejected batches and fringes return to the standard decoder. Short destinations also go through the standard implementation. No vector read or write crosses the slice's length, including unaligned buffers.

Go's NEON encoder uses alphabet lookups and interleaved stores; its decoder packs even/odd nibbles. Go's amd64 encoder widens bytes and maps nibbles arithmetically. The C kernel uses Clang vectors without intrinsic headers, assembly libraries or instruction sets beyond SSE2 on amd64. The optional hex_c tag selects this C kernel under host Go for additional validation.

The repository's AWS signing, S3 hashing, zstd ETag and vendored PostgreSQL helpers use this package. PostgreSQL bytea text encoding appends directly into the caller's buffer with AppendEncode, avoiding an intermediate hex string.

Verification

go test ./encoding/hex
GOEXPERIMENT=simd go test ./encoding/hex
CC=clang go test -tags hex_c ./encoding/hex
tinygo test ./encoding/hex
tinygo test -opt=2 ./encoding/hex
clang -O2 -g -fsanitize=address,undefined \
  encoding/hex/testdata/vector_bounds.c -o /tmp/hex-bounds
/tmp/hex-bounds
GOEXPERIMENT=simd go test ./encoding/hex -run '^$' \
  -fuzz '^FuzzCompatibility$' -fuzztime=15s -parallel=2

Tests compare every byte value in all 128 positions of a later vector batch, partial output, error identity, odd-length input, exact-length buffers, canaries, in-place decoding, append APIs, and streaming adapters. The C harness uses exact allocations under AddressSanitizer and UndefinedBehaviorSanitizer. The PostgreSQL bytea regression test checks prefixes, nil versus empty values, the valuer interface and valuer error propagation.

Native arm64 tests passed under Go, Go SIMD, Clang, and TinyGo at both optimization settings. amd64 Go/C test binaries also ran under Rosetta; this does not establish native amd64 performance. Go and TinyGo WASI fallback builds passed. Go SIMD and C fuzz runs completed 871,356 and 1,474,394 executions.

AWS/S3/zstd caller tests passed under Go, Go SIMD and TinyGo. PostgreSQL package tests and the vendored bytea test passed under Go; the bytea test also passed under TinyGo. PostgreSQL server integration requires a separately configured server and was not part of this run.

Measurements

Measured on Apple M3, Go 1.27.0 and TinyGo 0.42.0 / LLVM 22.1.4 on 2026-10-20. Medians of five 300 ms samples, one benchmark at a time with GOMAXPROCS=1. Each cell is standard → selected decimal GB/s for 64 KiB:

Operation Go SIMD TinyGo -opt=z TinyGo -opt=2
Encode 1.73 → 17.64 (10.18×) 1.85 → 24.35 (13.20×) 1.86 → 24.31 (13.06×)
Decode 1.54 → 7.74 (5.04×) 1.50 → 9.55 (6.39×) 1.24 → 9.56 (7.74×)

At 32 raw bytes, encoding improves 4.40× under Go SIMD, 2.77× under default TinyGo and 5.39× with -opt=2. Sixteen-byte inputs retain scalar handling and can pay up to about 1.7 ns extra under TinyGo's default size optimization. Every benchmark sample reports 0 B/op and 0 allocs/op after setup. These are in-memory conversion measurements, not complete application throughput.

See the combined measurements and the raw samples in testdata/bench-20261020.txt. The benchmark reuses destination buffers and compares each compiler's standard library with this package in the same binary. Throughput counts raw binary bytes for both encoding and decoding.

GOMAXPROCS=1 GOEXPERIMENT=simd go test ./encoding/hex -run '^$' \
  -bench '^BenchmarkHex$' -benchmem -benchtime=300ms -count=5
GOMAXPROCS=1 tinygo test -run '^$' -bench '^BenchmarkHex$' \
  -benchmem -benchtime=300ms -count=5 ./encoding/hex

Documentation

Overview

Package hex implements hexadecimal encoding and decoding with the standard encoding/hex API and optional bounded SIMD kernels for Go and TinyGo.

Index

Constants

This section is empty.

Variables

View Source
var ErrLength = std.ErrLength

Functions

func AppendDecode

func AppendDecode(dst, src []byte) ([]byte, error)

func AppendEncode

func AppendEncode(dst, src []byte) []byte

func Decode

func Decode(dst, src []byte) (int, error)

Decode accepts either hex letter case and returns the decoded prefix on error.

func DecodeString

func DecodeString(s string) ([]byte, error)

func DecodedLen

func DecodedLen(n int) int

func Dump

func Dump(data []byte) string

func Dumper

func Dumper(w io.Writer) io.WriteCloser

func Encode

func Encode(dst, src []byte) int

Encode writes lowercase hex and returns EncodedLen(len(src)).

func EncodeToString

func EncodeToString(src []byte) string

func EncodedLen

func EncodedLen(n int) int

func NewDecoder

func NewDecoder(r io.Reader) io.Reader

func NewEncoder

func NewEncoder(w io.Writer) io.Writer

Streaming and dump formatting retain the standard library's adapters.

Types

type InvalidByteError

type InvalidByteError = std.InvalidByteError

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL