uucode

package module
v1.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2026 License: MIT Imports: 1 Imported by: 0

README

go-uucode

go-uucode is a small Go Unicode segmentation and width package inspired by Jacob Sandlund's excellent uucode. Jacob's Zig implementation does the hard architectural work here: generated Unicode tables, compact property rows, and a fast staged lookup strategy. This package ports that table-first approach to Go.

It provides:

  • extended grapheme cluster iteration over Go strings
  • UAX #14 line break opportunity iteration over Go strings
  • grapheme-aware terminal cell width with StringWidth
  • typed lookup APIs for generated Unicode category, break, binary, emoji, width, and case properties
  • no runtime UCD parser, cache, or fallback path

Usage

package main

import (
	"fmt"

	"github.com/rockorager/go-uucode"
)

func main() {
	s := "👩🏽‍🚀🇨🇭A\u0300"

	it := uucode.NewGraphemeIterator(s)
	for {
		g, ok := it.Next()
		if !ok {
			break
		}
		fmt.Printf("%q [%d:%d]\n", s[g.Start:g.End], g.Start, g.End)
	}

	text := "hello, 世界\nnext"
	lines := uucode.NewLineIterator(text)
	for {
		seg, ok := lines.Next()
		if !ok {
			break
		}
		fmt.Printf("%q %s\n", text[seg.Start:seg.End], seg.Break)
	}

	fmt.Println(uucode.StringWidth("ò👨🏻‍❤️‍👨🏿_"))
	fmt.Println(uucode.IsLetter('界'), uucode.WordBreak('A'), uucode.LineBreak(' '))
}

Benchmarks

Benchmarks below were run on an Apple M4 Max with Go 1.26.1. Both libraries reported 0 B/op and 0 allocs/op.

Public API benchmark go-uucode ns/op uniseg ns/op Speedup
Grapheme ASCII 361.6 3326 9.20x
Grapheme Combining 254.6 1810 7.11x
Grapheme Emoji 184.7 1863 10.09x
Grapheme Mixed 255.5 2452 9.60x
Width ASCII 33.75 489.2 14.49x
Width Combining 286.6 331.2 1.16x
Width Emoji 217.3 444.1 2.04x
Width Mixed 250.9 500.1 1.99x

Predicate APIs are benchmarked against Go's unicode package on a rotating 32-rune corpus. These are speed comparisons against the public stdlib APIs; the local Go toolchain reports unicode.Version == "15.0.0" while go-uucode ships Unicode 17 data. The rows below show the mean of the six benchmark subcases:

Predicate benchmark go-uucode ns/op stdlib ns/op Speedup
IsUpper 1.67 5.73 3.42x
IsLower 1.68 5.77 3.44x
IsTitle 2.63 2.26 0.86x
IsLetter 1.75 6.50 3.72x
IsNumber 1.69 5.24 3.11x
IsDigit 1.68 4.68 2.78x
IsMark 1.75 6.52 3.72x
IsPunct 2.52 6.24 2.47x
IsSymbol 2.54 6.31 2.48x
IsGraphic 2.56 22.23 8.69x
IsPrint 2.62 22.26 8.50x
IsControl 0.37 0.68 1.83x
IsSpace 2.66 3.33 1.25x

Generated binary property APIs are benchmarked against unicode.Is with the matching stdlib range table on a property-focused 32-rune corpus:

Binary property benchmark go-uucode ns/op stdlib ns/op Speedup
IsASCIIHexDigit 2.44 2.58 1.06x
IsHexDigit 2.45 3.45 1.41x
IsDash 2.47 3.83 1.55x
IsDiacritic 2.45 5.25 2.14x
IsQuotationMark 2.44 3.85 1.57x
IsPatternSyntax 2.45 4.40 1.80x
IsPatternWhiteSpace 2.44 3.31 1.36x
IsVariationSelector 2.42 3.04 1.26x
IsNoncharacter 2.44 3.13 1.28x
IsUnifiedIdeograph 2.47 2.95 1.20x

Simple case mapping APIs are benchmarked against the matching unicode functions on a case-focused 32-rune corpus:

Case mapping benchmark go-uucode ns/op stdlib ns/op Speedup
ToUpper 2.01 6.86 3.42x
ToLower 1.92 6.83 3.56x
ToTitle 1.92 6.85 3.57x
SimpleFold 1.93 6.37 3.30x

String case folding is benchmarked against strings.EqualFold:

EqualFold benchmark go-uucode ns/op stdlib ns/op Speedup
ASCII equal 12.38 13.00 1.05x
ASCII miss 8.92 9.02 1.01x
Kelvin 9.80 9.29 0.95x
Greek sigma 18.72 28.79 1.54x
Mixed Unicode 31.94 41.27 1.29x
Length miss 2.68 2.63 0.98x

Run the package benchmarks:

go test -run '^$' -bench . -benchmem

The comparison against github.com/rivo/uniseg lives in a separate nested module so uniseg is not a dependency of this package:

cd bench/uniseg
go test -run '^$' -bench . -benchmem

Generated Tables

The package ships Unicode 17 source files and generates packed runtime tables. The hot path uses three stages:

  • runtimeStage1 indexes 256-code-point blocks by cp >> 8
  • runtimeStage2 indexes the low byte within deduplicated blocks
  • runtimeStage3 stores deduplicated packed property rows

Regenerate after changing UCD files or generator logic:

go generate ./...

The generated runtime rows store compact fields for grapheme segmentation, terminal-width calculation, general category predicates, word/sentence/line break properties, East Asian Width, PropList binary properties, simple case mapping, simple case folding, and emoji properties used by the public lookup functions.

Attribution

The design is based on the real jacobsandlund/uucode. If you are interested in the original implementation, Unicode table generation strategy, or a Zig library for this problem space, start there.

Documentation

Overview

Package uucode is a Go port of jacobsandlund/uucode's core Unicode APIs.

The package uses generated Unicode Character Database tables and exposes string grapheme cluster iteration, line break opportunity iteration, and terminal cell width helpers.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ComputeGraphemeBreak 

func ComputeGraphemeBreak(gb1, gb2 GraphemeBreak, state *BreakState) bool

ComputeGraphemeBreak reports whether there is a grapheme cluster boundary between two grapheme break properties, updating state for rules that depend on previous properties.

func EqualFold 

func EqualFold(s, t string) bool

EqualFold reports whether s and t are equal under Unicode simple case folding.

func IsASCIIHexDigit 

func IsASCIIHexDigit(r rune) bool

IsASCIIHexDigit reports whether r has the Unicode ASCII_Hex_Digit property.

func IsBreak 

func IsBreak(cp1, cp2 rune, state *BreakState) bool

IsBreak reports whether there is a grapheme cluster boundary between cp1 and cp2, updating state for rules that depend on previous code points.

func IsControl 

func IsControl(r rune) bool

IsControl reports whether r has general category Cc.

func IsDash 

func IsDash(r rune) bool

IsDash reports whether r has the Unicode Dash property.

func IsDiacritic 

func IsDiacritic(r rune) bool

IsDiacritic reports whether r has the Unicode Diacritic property.

func IsDigit 

func IsDigit(r rune) bool

IsDigit reports whether r has general category Nd.

func IsEmojiPresentation 

func IsEmojiPresentation(r rune) bool

IsEmojiPresentation reports whether r has emoji presentation by default.

func IsEmojiVariationBase added in v1.1.0 

func IsEmojiVariationBase(r rune) bool

IsEmojiVariationBase reports whether r is a base for an emoji variation sequence.

func IsExtendedPictographic 

func IsExtendedPictographic(r rune) bool

IsExtendedPictographic reports whether r has the Extended_Pictographic property.

func IsGraphic 

func IsGraphic(r rune) bool

IsGraphic reports whether r is defined as a Graphic by Go's unicode package.

func IsHexDigit 

func IsHexDigit(r rune) bool

IsHexDigit reports whether r has the Unicode Hex_Digit property.

func IsLetter 

func IsLetter(r rune) bool

IsLetter reports whether r has a Unicode letter general category.

func IsLower 

func IsLower(r rune) bool

IsLower reports whether r has general category Ll.

func IsMark 

func IsMark(r rune) bool

IsMark reports whether r has a Unicode mark general category.

func IsNoncharacter 

func IsNoncharacter(r rune) bool

IsNoncharacter reports whether r has the Unicode Noncharacter_Code_Point property.

func IsNumber 

func IsNumber(r rune) bool

IsNumber reports whether r has a Unicode number general category.

func IsPatternSyntax 

func IsPatternSyntax(r rune) bool

IsPatternSyntax reports whether r has the Unicode Pattern_Syntax property.

func IsPatternWhiteSpace 

func IsPatternWhiteSpace(r rune) bool

IsPatternWhiteSpace reports whether r has the Unicode Pattern_White_Space property.

func IsPrint 

func IsPrint(r rune) bool

IsPrint reports whether r is defined as printable by Go's unicode package.

func IsPunct 

func IsPunct(r rune) bool

IsPunct reports whether r has a Unicode punctuation general category.

func IsQuotationMark 

func IsQuotationMark(r rune) bool

IsQuotationMark reports whether r has the Unicode Quotation_Mark property.

func IsSpace 

func IsSpace(r rune) bool

IsSpace reports whether r has the Unicode White_Space property.

func IsSymbol 

func IsSymbol(r rune) bool

IsSymbol reports whether r has a Unicode symbol general category.

func IsTitle 

func IsTitle(r rune) bool

IsTitle reports whether r has general category Lt.

func IsUnifiedIdeograph 

func IsUnifiedIdeograph(r rune) bool

IsUnifiedIdeograph reports whether r has the Unicode Unified_Ideograph property.

func IsUpper 

func IsUpper(r rune) bool

IsUpper reports whether r has general category Lu.

func IsVariationSelector 

func IsVariationSelector(r rune) bool

IsVariationSelector reports whether r has the Unicode Variation_Selector property.

func RuneWidth 

func RuneWidth(r rune) int

RuneWidth returns the terminal cell width for r by itself.

func SimpleFold 

func SimpleFold(r rune) rune

SimpleFold returns the next rune equivalent to r under simple case folding.

func StringWidth 

func StringWidth(s string) int

StringWidth returns the grapheme-aware terminal cell width of s.

The result follows the width rules generated into this package's runtime tables and treats extended grapheme clusters as display units.

func ToLower 

func ToLower(r rune) rune

ToLower maps r to its simple lowercase mapping.

func ToTitle 

func ToTitle(r rune) rune

ToTitle maps r to its simple titlecase mapping.

func ToUpper 

func ToUpper(r rune) rune

ToUpper maps r to its simple uppercase mapping.

Types

type BreakState 

type BreakState uint8

BreakState carries state between adjacent grapheme break decisions.

Most callers should use GraphemeIterator instead of managing BreakState directly. 

const (
	BreakStateDefault BreakState = iota
	BreakStateRegionalIndicator
	BreakStateExtendedPictographic
	BreakStateIndicConjunctConsonant
	BreakStateIndicConjunctLinker
)

Grapheme break state values.

func (BreakState) String added in v1.0.2 

func (state BreakState) String() string

String returns the state name for state.

type EastAsianWidthClass added in v1.0.2 

type EastAsianWidthClass uint8

EastAsianWidthClass is a Unicode East_Asian_Width property value. 

const (
	EastAsianWidthN EastAsianWidthClass = iota
	EastAsianWidthNa
	EastAsianWidthA
	EastAsianWidthW
	EastAsianWidthH
	EastAsianWidthF
)

Unicode East_Asian_Width property values.

func EastAsianWidth 

func EastAsianWidth(r rune) EastAsianWidthClass

EastAsianWidth returns the Unicode East Asian Width property for r.

func (EastAsianWidthClass) String added in v1.0.2 

func (eaw EastAsianWidthClass) String() string

String returns the Unicode East_Asian_Width abbreviation for eaw.

type GeneralCategoryClass added in v1.0.2 

type GeneralCategoryClass uint8

GeneralCategoryClass is a Unicode General_Category property value. 

const (
	GeneralCategoryCn GeneralCategoryClass = iota
	GeneralCategoryCc
	GeneralCategoryZs
	GeneralCategoryPo
	GeneralCategorySc
	GeneralCategoryPs
	GeneralCategoryPe
	GeneralCategorySm
	GeneralCategoryPd
	GeneralCategoryNd
	GeneralCategoryLu
	GeneralCategorySk
	GeneralCategoryPc
	GeneralCategoryLl
	GeneralCategorySo
	GeneralCategoryLo
	GeneralCategoryPi
	GeneralCategoryCf
	GeneralCategoryNo
	GeneralCategoryPf
	GeneralCategoryLt
	GeneralCategoryLm
	GeneralCategoryMn
	GeneralCategoryMe
	GeneralCategoryMc
	GeneralCategoryNl
	GeneralCategoryZl
	GeneralCategoryZp
	GeneralCategoryCs
	GeneralCategoryCo
)

Unicode General_Category property values.

func GeneralCategory 

func GeneralCategory(r rune) GeneralCategoryClass

GeneralCategory returns the Unicode general category for r.

func (GeneralCategoryClass) String added in v1.0.2 

func (gc GeneralCategoryClass) String() string

String returns the Unicode General_Category abbreviation for gc.

type Grapheme 

type Grapheme struct {
	// Start is the byte offset of the first byte in the grapheme cluster.
	Start int
	// End is the byte offset just after the grapheme cluster.
	End int
}

Grapheme identifies a grapheme cluster by byte offsets into the original string.

type GraphemeBreak 

type GraphemeBreak uint8

GraphemeBreak is a Unicode grapheme break property value. 

const (
	GraphemeOther GraphemeBreak = iota
	GraphemeControl
	GraphemePrepend
	GraphemeCR
	GraphemeLF
	GraphemeRegionalIndicator
	GraphemeSpacingMark
	GraphemeL
	GraphemeV
	GraphemeT
	GraphemeLV
	GraphemeLVT
	GraphemeZWJ
	GraphemeZWNJ
	GraphemeExtendedPictographic
	GraphemeEmojiModifierBase
	GraphemeEmojiModifier
	GraphemeIndicConjunctExtend
	GraphemeIndicConjunctLinker
	GraphemeIndicConjunctConsonant
)

Grapheme break property values.

func GraphemeBreakProperty 

func GraphemeBreakProperty(r rune) GraphemeBreak

GraphemeBreakProperty returns the Unicode grapheme break property for r.

func (GraphemeBreak) String added in v1.0.2 

func (gb GraphemeBreak) String() string

String returns the Unicode grapheme break property name for gb.

type GraphemeIterator 

type GraphemeIterator struct {
	// contains filtered or unexported fields
}

GraphemeIterator iterates over extended grapheme clusters in a string.

func NewGraphemeIterator 

func NewGraphemeIterator(s string) GraphemeIterator

NewGraphemeIterator returns a grapheme cluster iterator for s.

func (*GraphemeIterator) Next 

func (it *GraphemeIterator) Next() (Grapheme, bool)

Next returns the next grapheme cluster.

The returned Grapheme contains byte offsets into the original string. ok is false after the iterator is exhausted.

func (GraphemeIterator) Peek 

func (it GraphemeIterator) Peek() (Grapheme, bool)

Peek returns the next grapheme cluster without advancing the iterator.

type GraphemeWidthIterator added in v1.2.0 

type GraphemeWidthIterator struct {
	// contains filtered or unexported fields
}

GraphemeWidthIterator iterates over extended grapheme clusters in a string and computes each cluster's terminal cell width in the same pass.

func NewGraphemeWidthIterator added in v1.2.0 

func NewGraphemeWidthIterator(s string) GraphemeWidthIterator

NewGraphemeWidthIterator returns a grapheme cluster iterator for s that also computes display width for each cluster.

func (*GraphemeWidthIterator) Next added in v1.2.0 

func (it *GraphemeWidthIterator) Next() (GraphemeWithWidth, bool)

Next returns the next grapheme cluster with its terminal cell width.

The returned GraphemeWithWidth contains byte offsets into the original string. ok is false after the iterator is exhausted.

func (GraphemeWidthIterator) Peek added in v1.2.0 

func (it GraphemeWidthIterator) Peek() (GraphemeWithWidth, bool)

Peek returns the next grapheme cluster with width without advancing the iterator.

type GraphemeWithWidth added in v1.2.0 

type GraphemeWithWidth struct {
	// Start is the byte offset of the first byte in the grapheme cluster.
	Start int
	// End is the byte offset just after the grapheme cluster.
	End int
	// Width is the terminal cell width of the grapheme cluster.
	Width int
}

GraphemeWithWidth identifies a grapheme cluster by byte offsets and includes its terminal cell width.

type LineBreakClass added in v1.0.2 

type LineBreakClass uint8

LineBreakClass is a Unicode Line_Break property value. 

const (
	LineBreakXX LineBreakClass = iota
	LineBreakCM
	LineBreakBA
	LineBreakLF
	LineBreakBK
	LineBreakCR
	LineBreakSP
	LineBreakEX
	LineBreakQU
	LineBreakAL
	LineBreakPR
	LineBreakPO
	LineBreakOP
	LineBreakCP
	LineBreakIS
	LineBreakHY
	LineBreakSY
	LineBreakNU
	LineBreakCL
	LineBreakNL
	LineBreakGL
	LineBreakAI
	LineBreakBB
	LineBreakHH
	LineBreakHL
	LineBreakSA
	LineBreakJL
	LineBreakJV
	LineBreakJT
	LineBreakNS
	LineBreakAK
	LineBreakVI
	LineBreakAS
	LineBreakID
	LineBreakVF
	LineBreakZW
	LineBreakZWJ
	LineBreakB2
	LineBreakIN
	LineBreakWJ
	LineBreakEB
	LineBreakCJ
	LineBreakH2
	LineBreakH3
	LineBreakSG
	LineBreakCB
	LineBreakAP
	LineBreakRI
	LineBreakEM
)

Unicode Line_Break property values.

func LineBreak 

func LineBreak(r rune) LineBreakClass

LineBreak returns the Unicode line break property for r.

func (LineBreakClass) String added in v1.0.2 

func (lb LineBreakClass) String() string

String returns the Unicode Line_Break abbreviation for lb.

type LineBreakKind added in v1.0.2 

type LineBreakKind uint8

LineBreakKind describes the line break opportunity after a line segment. 

const (
	// LineDontBreak means a line must not be broken at this boundary.
	LineDontBreak LineBreakKind = iota
	// LineCanBreak means a line may be broken at this boundary.
	LineCanBreak
	// LineMustBreak means a line must be broken at this boundary.
	LineMustBreak
)

Line break opportunity values.

func (LineBreakKind) String added in v1.0.2 

func (kind LineBreakKind) String() string

String returns the line break opportunity name for kind.

type LineIterator added in v1.0.2 

type LineIterator struct {
	// contains filtered or unexported fields
}

LineIterator iterates over non-breaking line segments in a string according to the Unicode Line Breaking Algorithm (UAX #14).

func NewLineIterator added in v1.0.2 

func NewLineIterator(s string) LineIterator

NewLineIterator returns a line break iterator for s.

func (*LineIterator) Next added in v1.0.2 

func (it *LineIterator) Next() (LineSegment, bool)

Next returns the next non-breaking line segment.

The returned LineSegment contains byte offsets into the original string. ok is false after the iterator is exhausted. In accordance with UAX #14 LB3, the final segment is returned with Break set to LineMustBreak.

func (LineIterator) Peek added in v1.0.2 

func (it LineIterator) Peek() (LineSegment, bool)

Peek returns the next line segment without advancing the iterator.

type LineSegment added in v1.0.2 

type LineSegment struct {
	// Start is the byte offset of the first byte in the segment.
	Start int
	// End is the byte offset just after the segment.
	End int
	// Break is the line break opportunity after End.
	Break LineBreakKind
}

LineSegment identifies a non-breaking line segment by byte offsets into the original string. Break describes the line break opportunity after End.

type SentenceBreakClass added in v1.0.2 

type SentenceBreakClass uint8

SentenceBreakClass is a Unicode Sentence_Break property value. 

const (
	SentenceBreakOther SentenceBreakClass = iota
	SentenceBreakSp
	SentenceBreakLF
	SentenceBreakCR
	SentenceBreakSTerm
	SentenceBreakClose
	SentenceBreakSContinue
	SentenceBreakATerm
	SentenceBreakNumeric
	SentenceBreakUpper
	SentenceBreakLower
	SentenceBreakSep
	SentenceBreakFormat
	SentenceBreakOLetter
	SentenceBreakExtend
)

Unicode Sentence_Break property values.

func SentenceBreak 

func SentenceBreak(r rune) SentenceBreakClass

SentenceBreak returns the Unicode sentence break property for r.

func (SentenceBreakClass) String added in v1.0.2 

func (sb SentenceBreakClass) String() string

String returns the Unicode Sentence_Break property name for sb.

type WordBreakClass added in v1.0.2 

type WordBreakClass uint8

WordBreakClass is a Unicode Word_Break property value. 

const (
	WordBreakOther WordBreakClass = iota
	WordBreakLF
	WordBreakNewline
	WordBreakCR
	WordBreakWSegSpace
	WordBreakDoubleQuote
	WordBreakSingleQuote
	WordBreakMidNum
	WordBreakMidNumLet
	WordBreakNumeric
	WordBreakMidLetter
	WordBreakALetter
	WordBreakExtendNumLet
	WordBreakFormat
	WordBreakExtend
	WordBreakHebrewLetter
	WordBreakZWJ
	WordBreakKatakana
	WordBreakRegionalIndicator
)

Unicode Word_Break property values.

func WordBreak 

func WordBreak(r rune) WordBreakClass

WordBreak returns the Unicode word break property for r.

func (WordBreakClass) String added in v1.0.2 

func (wb WordBreakClass) String() string

String returns the Unicode Word_Break property name for wb.

Source Files

View all Source files

Directories

Path Synopsis
cmd
uucodegen command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.