normalize

unicode-normalize — go-ruby-unicode-normalize

A pure-Go (no cgo) reimplementation of Ruby's String#unicode_normalize and String#unicode_normalized? — the unicode_normalize standard library — matching MRI 4.0.5 (Unicode 17.0.0) byte-for-byte. It normalizes UTF-8 text to any of the four Unicode Standard Annex #15 forms (NFC, NFD, NFKC, NFKD) and reports whether a string is already normalized, without any Ruby runtime.

It is the Unicode-normalization backend for go-embedded-ruby, but is a standalone, reusable module — a sibling of go-ruby-regexp (the Onigmo engine), go-ruby-marshal and go-ruby-yaml.

API

import normalize "github.com/go-ruby-unicode-normalize/unicode-normalize"

// Normalize returns s in the given form (NFC is the default form for Ruby's
// unicode_normalize when called with no argument).
func Normalize(s string, form normalize.Form) string

// IsNormalized reports whether s is already in the given form
// (Ruby's unicode_normalized?).
func IsNormalized(s string, form normalize.Form) bool

normalize.Normalize("é", normalize.NFC)   // "é"  (composes e + acute)
normalize.Normalize("é", normalize.NFD)          // "é" (decomposes)
normalize.Normalize("fi", normalize.NFKC)         // "fi"  (ligature -> ascii)
normalize.Normalize("①", normalize.NFKC)         // "1"
normalize.Normalize("Ａ", normalize.NFKC)         // "A"   (full-width -> ascii)
normalize.IsNormalized("é", normalize.NFC)       // true  (composed)
normalize.IsNormalized("é", normalize.NFC) // false (decomposed)

The four forms are the normalize.Form values NFC, NFD, NFKC, NFKD, corresponding to Ruby's :nfc, :nfd, :nfkc, :nfkd. The zero value is NFC, mirroring Ruby's default. Form.String returns the Ruby symbol name (":nfc", …) for diagnostics and for the rbgo binding layer.

What the host binds. This library is the deterministic, interpreter-free core. The rbgo binding wires Normalize to String#unicode_normalize and IsNormalized to String#unicode_normalized?, defaults the form to :nfc, maps the :nfc/:nfd/:nfkc/:nfkd symbols to Form, raises ArgumentError on an unknown form symbol and on invalid UTF-8 byte sequences (matching MRI), and otherwise hands the bytes straight through.

How it works

The heavy lifting is delegated to golang.org/x/text/unicode/norm, a pure-Go (cgo-free) implementation of the same Unicode standard. Both MRI 4.0.5 and x/text target Unicode 17.0.0, so they agree across the official NormalizationTest corpus and the Ruby-specific edge cases (Hangul composition, full/half-width and ligature compatibility, combining-mark ordering).

x/text selects its Unicode tables by build tag: Unicode 17.0.0 under go1.27, Unicode 15.0.0 otherwise. This module requires Go 1.27.1, so it is always built against the Unicode 17.0.0 tables and needs no local patching; TestUnicodeVersion fails if that precondition ever changes. (Before the 1.27.1 floor, a small override layer patched the Unicode 16.0/17.0 additions into the 15.0.0 tables; it became a no-op under the 17.0.0 tables and was removed.)

Tests & coverage

Two complementary layers, both run in CI on every supported platform:

• Deterministic, ruby-free tests pin the MRI-observed results and alone keep statement coverage at 100%, so the Windows and cross-architecture lanes (where MRI is absent) still pass the gate.
• MRI oracle tests run differentially against the ruby binary: a curated edge-case corpus, a combining-sequence fuzz, and an exhaustive sweep of every assigned scalar value × all four forms. They skip themselves when ruby is absent and gate on RUBY_VERSION >= "4.0" (Unicode 17.0.0); the oracle script binmodes stdin/stdout so the UTF-8 bytes survive intact.

go test -race -coverpkg=./... -coverprofile=cover.out ./...   # 100.0%
go test -short ./...                                          # skip the exhaustive sweep

CI enforces the 100% coverage gate plus builds and tests on three operating systems (Linux, macOS, Windows) and all six supported 64-bit architectures (amd64, arm64, riscv64, loong64, ppc64le, s390x). The package is cgo-free (CGO_ENABLED=0).

License

BSD-3-Clause — see LICENSE. Copyright (c) 2026, the go-ruby-unicode-normalize/unicode-normalize authors.

WebAssembly

Being pure Go (CGO=0), this library also compiles to WebAssembly — both GOOS=js GOARCH=wasm (browser / Node.js) and GOOS=wasip1 GOARCH=wasm (WASI). CI builds both targets on every push, alongside the six 64-bit native/qemu arches.

GOOS=js     GOARCH=wasm go build ./...   # browser / Node
GOOS=wasip1 GOARCH=wasm go build ./...   # WASI (wasmtime, wasmer, wasmedge, …)