uucode

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 12, 2026 License: MIT Imports: 1 Imported by: 0

README

go-uucode

go-uucode is a small Go Unicode segmentation and width package inspired by Jacob Sandlund's excellent uucode. Jacob's Zig implementation does the hard architectural work here: generated Unicode tables, compact property rows, and a fast staged lookup strategy. This package ports that table-first approach to Go.

It provides:

  • extended grapheme cluster iteration over Go strings
  • grapheme-aware terminal cell width with StringWidth
  • narrow lookup APIs for generated Unicode category, break, binary, emoji, width, and case properties
  • no runtime UCD parser, cache, or fallback path

Usage

package main

import (
	"fmt"

	"github.com/rockorager/go-uucode"
)

func main() {
	s := "👩🏽‍🚀🇨🇭A\u0300"

	it := uucode.NewGraphemeIterator(s)
	for {
		g, ok := it.Next()
		if !ok {
			break
		}
		fmt.Printf("%q [%d:%d]\n", s[g.Start:g.End], g.Start, g.End)
	}

	fmt.Println(uucode.StringWidth("ò👨🏻‍❤️‍👨🏿_"))
	fmt.Println(uucode.IsLetter('界'), uucode.WordBreak('A'), uucode.LineBreak(' '))
}

Benchmarks

Benchmarks below were run on an Apple M4 Max with Go 1.26.1. Both libraries reported 0 B/op and 0 allocs/op.

Public API benchmark go-uucode ns/op uniseg ns/op Speedup
Grapheme ASCII 361.6 3326 9.20x
Grapheme Combining 254.6 1810 7.11x
Grapheme Emoji 184.7 1863 10.09x
Grapheme Mixed 255.5 2452 9.60x
Width ASCII 33.75 489.2 14.49x
Width Combining 286.6 331.2 1.16x
Width Emoji 217.3 444.1 2.04x
Width Mixed 250.9 500.1 1.99x

Predicate APIs are benchmarked against Go's unicode package on a rotating 32-rune corpus. These are speed comparisons against the public stdlib APIs; the local Go toolchain reports unicode.Version == "15.0.0" while go-uucode ships Unicode 17 data. The rows below show the mean of the six benchmark subcases:

Predicate benchmark go-uucode ns/op stdlib ns/op Speedup
IsUpper 1.85 5.48 2.97x
IsLower 1.80 5.87 3.27x
IsTitle 2.00 2.26 1.13x
IsLetter 1.69 6.24 3.70x
IsNumber 1.48 5.05 3.41x
IsDigit 1.76 4.71 2.68x
IsMark 1.96 6.25 3.19x
IsPunct 2.63 6.56 2.49x
IsSymbol 2.25 6.39 2.84x
IsGraphic 2.66 22.83 8.59x
IsPrint 2.55 21.70 8.50x
IsControl 0.38 0.68 1.77x
IsSpace 1.76 3.30 1.87x

Generated binary property APIs are benchmarked against unicode.Is with the matching stdlib range table on a property-focused 32-rune corpus:

Binary property benchmark go-uucode ns/op stdlib ns/op Speedup
IsASCIIHexDigit 2.44 2.58 1.06x
IsHexDigit 2.45 3.45 1.41x
IsDash 2.47 3.83 1.55x
IsDiacritic 2.45 5.25 2.14x
IsQuotationMark 2.44 3.85 1.57x
IsPatternSyntax 2.45 4.40 1.80x
IsPatternWhiteSpace 2.44 3.31 1.36x
IsVariationSelector 2.42 3.04 1.26x
IsNoncharacter 2.44 3.13 1.28x
IsUnifiedIdeograph 2.47 2.95 1.20x

Simple case mapping APIs are benchmarked against the matching unicode functions on a case-focused 32-rune corpus:

Case mapping benchmark go-uucode ns/op stdlib ns/op Speedup
ToUpper 2.01 6.86 3.42x
ToLower 1.92 6.83 3.56x
ToTitle 1.92 6.85 3.57x
SimpleFold 1.93 6.37 3.30x

String case folding is benchmarked against strings.EqualFold:

EqualFold benchmark go-uucode ns/op stdlib ns/op Speedup
ASCII equal 12.38 13.00 1.05x
ASCII miss 8.92 9.02 1.01x
Kelvin 9.80 9.29 0.95x
Greek sigma 18.72 28.79 1.54x
Mixed Unicode 31.94 41.27 1.29x
Length miss 2.68 2.63 0.98x

Run the package benchmarks:

go test -run '^$' -bench . -benchmem

The comparison against github.com/rivo/uniseg lives in a separate nested module so uniseg is not a dependency of this package:

cd bench/uniseg
go test -run '^$' -bench . -benchmem

Generated Tables

The package ships Unicode 17 source files and generates packed runtime tables. The hot path uses three stages:

  • runtimeStage1 indexes 256-code-point blocks by cp >> 8
  • runtimeStage2 indexes the low byte within deduplicated blocks
  • runtimeStage3 stores deduplicated packed property rows

Regenerate after changing UCD files or generator logic:

go generate ./...

The generated runtime rows store compact fields for grapheme segmentation, terminal-width calculation, general category predicates, word/sentence/line break properties, East Asian Width, PropList binary properties, simple case mapping, simple case folding, and emoji properties used by the public lookup functions.

Attribution

The design is based on the real jacobsandlund/uucode. If you are interested in the original implementation, Unicode table generation strategy, or a Zig library for this problem space, start there.

Documentation

Overview

Package uucode is a Go port of jacobsandlund/uucode's core Unicode APIs.

The package uses generated Unicode Character Database tables and exposes string grapheme cluster iteration and terminal cell width helpers.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ComputeGraphemeBreak 

func ComputeGraphemeBreak(gb1, gb2 GraphemeBreak, state *BreakState) bool

ComputeGraphemeBreak reports whether there is a grapheme cluster boundary between two grapheme break properties, updating state for rules that depend on previous properties.

func EastAsianWidth 

func EastAsianWidth(r rune) string

EastAsianWidth returns the Unicode East Asian Width abbreviation for r.

func EqualFold 

func EqualFold(s, t string) bool

EqualFold reports whether s and t are equal under Unicode simple case folding.

func GeneralCategory 

func GeneralCategory(r rune) string

GeneralCategory returns the Unicode general category abbreviation for r.

func IsASCIIHexDigit 

func IsASCIIHexDigit(r rune) bool

IsASCIIHexDigit reports whether r has the Unicode ASCII_Hex_Digit property.

func IsBreak 

func IsBreak(cp1, cp2 rune, state *BreakState) bool

IsBreak reports whether there is a grapheme cluster boundary between cp1 and cp2, updating state for rules that depend on previous code points.

func IsControl 

func IsControl(r rune) bool

IsControl reports whether r has general category Cc.

func IsDash 

func IsDash(r rune) bool

IsDash reports whether r has the Unicode Dash property.

func IsDiacritic 

func IsDiacritic(r rune) bool

IsDiacritic reports whether r has the Unicode Diacritic property.

func IsDigit 

func IsDigit(r rune) bool

IsDigit reports whether r has general category Nd.

func IsEmojiPresentation 

func IsEmojiPresentation(r rune) bool

IsEmojiPresentation reports whether r has emoji presentation by default.

func IsExtendedPictographic 

func IsExtendedPictographic(r rune) bool

IsExtendedPictographic reports whether r has the Extended_Pictographic property.

func IsGraphic 

func IsGraphic(r rune) bool

IsGraphic reports whether r is defined as a Graphic by Go's unicode package.

func IsHexDigit 

func IsHexDigit(r rune) bool

IsHexDigit reports whether r has the Unicode Hex_Digit property.

func IsLetter 

func IsLetter(r rune) bool

IsLetter reports whether r has a Unicode letter general category.

func IsLower 

func IsLower(r rune) bool

IsLower reports whether r has general category Ll.

func IsMark 

func IsMark(r rune) bool

IsMark reports whether r has a Unicode mark general category.

func IsNoncharacter 

func IsNoncharacter(r rune) bool

IsNoncharacter reports whether r has the Unicode Noncharacter_Code_Point property.

func IsNumber 

func IsNumber(r rune) bool

IsNumber reports whether r has a Unicode number general category.

func IsPatternSyntax 

func IsPatternSyntax(r rune) bool

IsPatternSyntax reports whether r has the Unicode Pattern_Syntax property.

func IsPatternWhiteSpace 

func IsPatternWhiteSpace(r rune) bool

IsPatternWhiteSpace reports whether r has the Unicode Pattern_White_Space property.

func IsPrint 

func IsPrint(r rune) bool

IsPrint reports whether r is defined as printable by Go's unicode package.

func IsPunct 

func IsPunct(r rune) bool

IsPunct reports whether r has a Unicode punctuation general category.

func IsQuotationMark 

func IsQuotationMark(r rune) bool

IsQuotationMark reports whether r has the Unicode Quotation_Mark property.

func IsSpace 

func IsSpace(r rune) bool

IsSpace reports whether r has the Unicode White_Space property.

func IsSymbol 

func IsSymbol(r rune) bool

IsSymbol reports whether r has a Unicode symbol general category.

func IsTitle 

func IsTitle(r rune) bool

IsTitle reports whether r has general category Lt.

func IsUnifiedIdeograph 

func IsUnifiedIdeograph(r rune) bool

IsUnifiedIdeograph reports whether r has the Unicode Unified_Ideograph property.

func IsUpper 

func IsUpper(r rune) bool

IsUpper reports whether r has general category Lu.

func IsVariationSelector 

func IsVariationSelector(r rune) bool

IsVariationSelector reports whether r has the Unicode Variation_Selector property.

func LineBreak 

func LineBreak(r rune) string

LineBreak returns the Unicode line break property abbreviation for r.

func RuneWidth 

func RuneWidth(r rune) int

RuneWidth returns the terminal cell width for r by itself.

func SentenceBreak 

func SentenceBreak(r rune) string

SentenceBreak returns the Unicode sentence break property name for r.

func SimpleFold 

func SimpleFold(r rune) rune

SimpleFold returns the next rune equivalent to r under simple case folding.

func StringWidth 

func StringWidth(s string) int

StringWidth returns the grapheme-aware terminal cell width of s.

The result follows the width rules generated into this package's runtime tables and treats extended grapheme clusters as display units.

func ToLower 

func ToLower(r rune) rune

ToLower maps r to its simple lowercase mapping.

func ToTitle 

func ToTitle(r rune) rune

ToTitle maps r to its simple titlecase mapping.

func ToUpper 

func ToUpper(r rune) rune

ToUpper maps r to its simple uppercase mapping.

func WordBreak 

func WordBreak(r rune) string

WordBreak returns the Unicode word break property name for r.

Types

type BreakState 

type BreakState string

BreakState carries state between adjacent grapheme break decisions.

Most callers should use GraphemeIterator instead of managing BreakState directly. 

const (
	BreakStateDefault                BreakState = "default"
	BreakStateRegionalIndicator      BreakState = "regional_indicator"
	BreakStateExtendedPictographic   BreakState = "extended_pictographic"
	BreakStateIndicConjunctConsonant BreakState = "indic_conjunct_break_consonant"
	BreakStateIndicConjunctLinker    BreakState = "indic_conjunct_break_linker"
)

Grapheme break state values.

type Grapheme 

type Grapheme struct {
	// Start is the byte offset of the first byte in the grapheme cluster.
	Start int
	// End is the byte offset just after the grapheme cluster.
	End int
}

Grapheme identifies a grapheme cluster by byte offsets into the original string.

type GraphemeBreak 

type GraphemeBreak string

GraphemeBreak is a Unicode grapheme break property value. 

const (
	GraphemeOther                  GraphemeBreak = "other"
	GraphemeControl                GraphemeBreak = "control"
	GraphemePrepend                GraphemeBreak = "prepend"
	GraphemeCR                     GraphemeBreak = "cr"
	GraphemeLF                     GraphemeBreak = "lf"
	GraphemeRegionalIndicator      GraphemeBreak = "regional_indicator"
	GraphemeSpacingMark            GraphemeBreak = "spacing_mark"
	GraphemeL                      GraphemeBreak = "l"
	GraphemeV                      GraphemeBreak = "v"
	GraphemeT                      GraphemeBreak = "t"
	GraphemeLV                     GraphemeBreak = "lv"
	GraphemeLVT                    GraphemeBreak = "lvt"
	GraphemeZWJ                    GraphemeBreak = "zwj"
	GraphemeZWNJ                   GraphemeBreak = "zwnj"
	GraphemeExtendedPictographic   GraphemeBreak = "extended_pictographic"
	GraphemeEmojiModifierBase      GraphemeBreak = "emoji_modifier_base"
	GraphemeEmojiModifier          GraphemeBreak = "emoji_modifier"
	GraphemeIndicConjunctExtend    GraphemeBreak = "indic_conjunct_break_extend"
	GraphemeIndicConjunctLinker    GraphemeBreak = "indic_conjunct_break_linker"
	GraphemeIndicConjunctConsonant GraphemeBreak = "indic_conjunct_break_consonant"
)

Grapheme break property values.

func GraphemeBreakProperty 

func GraphemeBreakProperty(r rune) GraphemeBreak

GraphemeBreakProperty returns the Unicode grapheme break property for r.

type GraphemeIterator 

type GraphemeIterator struct {
	// contains filtered or unexported fields
}

GraphemeIterator iterates over extended grapheme clusters in a string.

func NewGraphemeIterator 

func NewGraphemeIterator(s string) GraphemeIterator

NewGraphemeIterator returns a grapheme cluster iterator for s.

func (*GraphemeIterator) Next 

func (it *GraphemeIterator) Next() (Grapheme, bool)

Next returns the next grapheme cluster.

The returned Grapheme contains byte offsets into the original string. ok is false after the iterator is exhausted.

func (GraphemeIterator) Peek 

func (it GraphemeIterator) Peek() (Grapheme, bool)

Peek returns the next grapheme cluster without advancing the iterator.

Source Files

View all Source files

Directories

Path Synopsis
cmd
uucodegen command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.