typewriter

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 12, 2026 License: MIT Imports: 3 Imported by: 0

README

typewriter

Go Reference Build Status

Converts typographic ("smart") Unicode characters back to their plain ASCII equivalents, and normalises Unicode style variants (bold, italic, monospace, superscript, subscript) to plain text — optionally wrapping runs with configurable markup.

  • Requires Go 1.22+
  • Uses only stdlib, no dependencies
  • Safe for concurrent use
  • For a goldmark extension see goldmark-typewriter.
  • For emoticon/emoji/shortcode conversion see demoji.

Installation

go get github.com/client9/typewriter

Quick start

Package-level functions use all [Default] categories and require no configuration:

import "github.com/client9/typewriter"

clean := typewriter.Replace(s)       // string → string
cleanB := typewriter.ReplaceBytes(b) // []byte → []byte

For custom behaviour, create a [Replacer] with [New]. It is safe for concurrent use and should be built once and reused:

r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default,
    Runs: []typewriter.RunStyle{
        {Style: typewriter.Bold, Prefix: "**", Suffix: "**"},
    },
})
clean := r.Replace(s)

What it converts

Character substitutions

All categories are active by default.

Category Examples Result
Quotes " " ' ' « » „ " ' << >>
Dashes em dash —, en dash –, minus − --- -- -
Ellipsis … ...
Fractions ½ ¼ ¾ ⅓ ⅛ 1/2 1/4 3/4 1/3 1/8
Symbols © ® ™ § (c) (r) (tm) S. P.
Math × ÷ ≠ ≤ ≥ → x / != <= >= ->
Ligatures fi fl ff ffi fi fl ff ffi
Bullets • † → *   ‡ → **   · → . * ** .
Spaces NBSP, thin, en, em, figure, hair, U+2028, U+2029 plain space
Unicode style variants (run-based)

Runs of styled characters are detected and converted together, so the whole run can be wrapped with a prefix and suffix.

Style Example Default (strip) Markdown HTML
Bold 𝗛𝗲𝗹𝗹𝗼 Hello **Hello** <b>Hello</b>
Italic 𝘸𝘰𝘳𝘭𝘥 world _world_ <i>world</i>
BoldItalic 𝙃𝙚𝙡𝙡𝙤 Hello ***Hello***
Monospace 𝙷𝚎𝚕𝚕𝚘 Hello `Hello`
Superscript E=mc² E=mc2 E=mc^2
Subscript H₂O H2O

Style variants are not active by default — configure with Config.Runs.

Configuration

Config has three fields: Categories selects which built-in conversion groups are active; Overrides adds, changes, or excludes individual character mappings; Runs configures Unicode style run detection. RunStyle.Prefix and RunStyle.Suffix wrap each detected run — leave them empty to strip styled characters to plain ASCII.

Enable only specific categories

Categories is a bitfield — set it to exactly the categories you want:

// Only convert dashes and ellipses.
r := typewriter.New(typewriter.Config{
    Categories: typewriter.Dashes | typewriter.Ellipsis,
})
Disable specific categories

Use bit-clear to remove from the default set:

r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default &^ typewriter.Math,
})
Override or exclude individual characters
r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default,
    Overrides: map[string]string{
        "—":   "--",   // prefer -- over --- for em dash
        "×":   "",     // leave × unchanged (empty = pass through)
        "°":   "deg",  // add a mapping not in builtins
    },
})
Convert Unicode bold/italic to markdown
r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default,
    Runs: []typewriter.RunStyle{
        {Style: typewriter.Bold,   Prefix: "**", Suffix: "**"},
        {Style: typewriter.Italic, Prefix: "_",  Suffix: "_"},
    },
})
r.Replace("𝗛𝗲𝗹𝗹𝗼 𝘸𝘰𝘳𝘭𝘥")  // → "**Hello** _world_"
Convert Unicode bold/italic to HTML
r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default,
    Runs: []typewriter.RunStyle{
        {Style: typewriter.Bold,   Prefix: "<b>",  Suffix: "</b>"},
        {Style: typewriter.Italic, Prefix: "<i>",  Suffix: "</i>"},
    },
})
Superscripts and subscripts
r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default,
    Runs: []typewriter.RunStyle{
        {Style: typewriter.Superscript, Prefix: "^"},   // E=mc² → E=mc^2
        {Style: typewriter.Subscript},                  // H₂O  → H2O
    },
})

Normalising before Goldmark's Typographer

Goldmark's Typographer extension converts ASCII punctuation to smart Unicode characters (-- → –, "..." → "…", etc.). Markdown from mixed sources (hand-authored, Word, AI-generated) arrives with inconsistent typography, so a Typographer pass produces inconsistent output: content already containing "Hello" passes through unchanged while "Hello" gets converted.

The fix is to strip everything to a clean ASCII baseline with typewriter first:

import (
    "log"

    "github.com/client9/typewriter"
    "github.com/yuin/goldmark"
    "github.com/yuin/goldmark/extension"
)

clean := typewriter.ReplaceBytes(src)

md := goldmark.New(goldmark.WithExtensions(extension.Typographer))
if err := md.Convert(clean, &buf); err != nil {
    log.Fatal(err)
}

For direct Goldmark integration see goldmark-typewriter.

License

MIT

Documentation

Overview

Package typewriter converts typographic ("smart") Unicode characters back to their plain ASCII typewriter equivalents, and normalises Unicode style variants (bold, italic, monospace, superscript, subscript) back to plain letters.

It is designed for cleaning up text that has passed through a word processor, rich-text editor, or AI-generated content pipeline: curly quotes, em dashes, ligatures, non-breaking spaces, styled Unicode alphabets, and similar characters that look fine on screen but cause problems in plain-text contexts such as source code, configuration files, command-line arguments, and Markdown documents.

Quick start

The package-level functions use all Default conversions and require no configuration: 

clean := typewriter.Replace(s)
clean := typewriter.ReplaceBytes(b)

Creating a Replacer

For custom behaviour, create a Replacer with New: 

r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default,
})
clean := r.Replace(s)

A Replacer is safe for concurrent use and should be created once and reused.

Categories

Built-in conversions are grouped into Category bitfields. Default (equivalently CategoryAll) enables all of them. Use bitwise operations to select a subset: 

// Only ellipsis and dashes:
typewriter.Config{Categories: typewriter.Ellipsis | typewriter.Dashes}

// Everything except math symbols:
typewriter.Config{Categories: typewriter.Default &^ typewriter.Math}

The defined categories are Quotes, Dashes, Ellipsis, Fractions, Symbols, Math, Ligatures, Bullets, and Spaces.

Overrides

Config.Overrides customises or extends the built-in table on a character-by-character basis. Overrides are applied before built-ins. An empty target string excludes the source from conversion: 

typewriter.Config{
    Categories: typewriter.Default,
    Overrides: map[string]string{
        "—": "--",  // prefer -- over the default ---
        "×": "",    // leave × unchanged
        "°": "deg", // add a mapping not in any built-in category
    },
}

Unicode style runs

Social-media and AI-generated text frequently contains styled Unicode variants: 𝗯𝗼𝗹𝗱, 𝘪𝘵𝘢𝘭𝘪𝘤, 𝚖𝚘𝚗𝚘𝚜𝚙𝚊𝚌𝚎, ˢᵘᵖᵉʳˢᶜʳⁱᵖᵗ, ₛᵤᵦₛ꜀ᵣᵢₚₜ. Config.Runs maps contiguous runs of styled characters to plain ASCII, optionally wrapping the run with a configurable prefix and suffix: 

r := typewriter.New(typewriter.Config{
    Categories: typewriter.Default,
    Runs: []typewriter.RunStyle{
        {Style: typewriter.Bold,   Prefix: "**", Suffix: "**"},
        {Style: typewriter.Italic, Prefix: "_",  Suffix: "_"},
    },
})

With empty RunStyle.Prefix and RunStyle.Suffix (the zero value), styled runs are stripped to plain ASCII with no added markup. Character substitutions (quotes, dashes, etc.) and run detection compose correctly in a single pass.

Example 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	fmt.Println(typewriter.Replace(`"Hello" — wait…`))
}

Output:
"Hello" --- wait...

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Replace 

func Replace(s string) string

Replace returns s with all Default conversions applied.

Example 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	fmt.Println(typewriter.Replace("© 2024 — all rights reserved™"))
}

Output:
(c) 2024 --- all rights reserved(tm)

func ReplaceBytes 

func ReplaceBytes(b []byte) []byte

ReplaceBytes returns a copy of b with all Default conversions applied.

Example 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	fmt.Println(string(typewriter.ReplaceBytes([]byte("file ½ done…"))))
}

Output:
file 1/2 done...

Types

type Category 

type Category uint

Category is a bitmask that selects groups of character substitutions. Combine groups with |; exclude a group from Default with &^. 

const (
	Quotes    Category = 1 << iota // curly/angle quotes → straight ASCII
	Dashes                         // em/en dashes → ---/--
	Ellipsis                       // … → ...
	Fractions                      // ½ ¼ ¾ → 1/2 1/4 3/4
	Symbols                        // © ® ™ § ¶ → (c) (r) (tm) S. P.
	Math                           // × ÷ ≠ ≤ ≥ → x / != <= >=
	Ligatures                      // fi fl ff → fi fl ff
	Bullets                        // • · † ‡ → * . * **
	Spaces                         // non-breaking and width-variant spaces → plain space

	// Default is all defined categories.
	Default = Quotes | Dashes | Ellipsis | Fractions | Symbols | Math | Ligatures | Bullets | Spaces

	// CategoryAll is an alias for Default, kept for forward compatibility.
	CategoryAll = Default
)

type Config 

type Config struct {
	// Categories selects which built-in conversion groups are active.
	// Use [Default] to enable all groups.
	Categories Category

	// Overrides adjusts individual mappings before the built-in table is
	// consulted. The key is the Unicode source string; the value is the ASCII
	// target. An empty value suppresses the built-in mapping for that key.
	Overrides map[string]string

	// Runs configures detection of contiguous Unicode-styled character runs
	// (bold, italic, monospace, etc.) and the Prefix/Suffix used to wrap
	// the recovered ASCII text. See [RunStyle].
	Runs []RunStyle
}

Config configures a Replacer. Pass as a value to New.

type Replacer 

type Replacer struct {
	// contains filtered or unexported fields
}

Replacer applies typographic-to-ASCII conversions configured by New. A Replacer is safe for concurrent use by multiple goroutines.

func New 

func New(cfg Config) *Replacer

New returns a Replacer configured by cfg.

Example 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
	})
	fmt.Println(r.Replace("½ price — today only…"))
}

Output:
1/2 price --- today only...
Example (CategoryBlacklist) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Convert everything except math symbols.
	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default &^ typewriter.Math,
	})
	fmt.Println(r.Replace("10× better…"))
}

Output:
10× better...
Example (CategoryWhitelist) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Convert only ellipsis; leave everything else untouched.
	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Ellipsis,
	})
	fmt.Println(r.Replace(`"wait…"`))
}

Output:
"wait..."
Example (OverrideAdd) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Add a mapping not covered by any built-in category.
	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Overrides:  map[string]string{"°": "deg"},
	})
	fmt.Println(r.Replace("90°"))
}

Output:
90deg
Example (OverrideExclude) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Leave the multiplication sign unchanged.
	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Overrides:  map[string]string{"×": ""},
	})
	fmt.Println(r.Replace("10×"))
}

Output:
10×
Example (OverrideValue) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Use double-hyphen for em dash instead of the default triple-hyphen.
	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Overrides:  map[string]string{"—": "--"},
	})
	fmt.Println(r.Replace("one—two"))
}

Output:
one--two
Example (RunsAndSubstitutions) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Run detection and character substitutions compose in a single pass.
	// 𝗛𝗲𝗹𝗹𝗼 = sans-serif bold "Hello" (U+1D5DB … U+1D5FC)
	// 𝘄𝗼𝗿𝗹𝗱 = sans-serif bold "world" (U+1D604 … U+1D5F1)
	boldHello := "\U0001d5db\U0001d5f2\U0001d5f9\U0001d5f9\U0001d5fc"
	boldWorld := "\U0001d604\U0001d5fc\U0001d5ff\U0001d5f9\U0001d5f1"

	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Runs:       []typewriter.RunStyle{{Style: typewriter.Bold, Prefix: "**", Suffix: "**"}},
	})
	fmt.Println(r.Replace(boldHello + " © " + boldWorld))
}

Output:
**Hello** (c) **world**
Example (RunsHTML) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Convert styled Unicode variants to HTML.
	// 𝗛𝗲𝗹𝗹𝗼 = sans-serif bold "Hello" (U+1D5DB … U+1D5FC)
	bold := "\U0001d5db\U0001d5f2\U0001d5f9\U0001d5f9\U0001d5fc"

	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Runs: []typewriter.RunStyle{
			{Style: typewriter.Bold, Prefix: "<b>", Suffix: "</b>"},
		},
	})
	fmt.Println(r.Replace(bold + " world"))
}

Output:
<b>Hello</b> world
Example (RunsMarkdown) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Convert styled Unicode variants to Markdown.
	// 𝗛𝗲𝗹𝗹𝗼 = sans-serif bold "Hello" (U+1D5DB … U+1D5FC)
	// 𝘸𝘰𝘳𝘭𝘥 = sans-serif italic "world" (U+1D638 … U+1D625)
	bold := "\U0001d5db\U0001d5f2\U0001d5f9\U0001d5f9\U0001d5fc"
	italic := "\U0001d638\U0001d630\U0001d633\U0001d62d\U0001d625"

	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Runs: []typewriter.RunStyle{
			{Style: typewriter.Bold, Prefix: "**", Suffix: "**"},
			{Style: typewriter.Italic, Prefix: "_", Suffix: "_"},
		},
	})
	fmt.Println(r.Replace(bold + " " + italic))
}

Output:
**Hello** _world_
Example (RunsStrip) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Strip styled Unicode variants to plain ASCII.
	// 𝗛𝗲𝗹𝗹𝗼 = sans-serif bold "Hello" (U+1D5DB … U+1D5FC)
	// 𝘸𝘰𝘳𝘭𝘥 = sans-serif italic "world" (U+1D638 … U+1D625)
	bold := "\U0001d5db\U0001d5f2\U0001d5f9\U0001d5f9\U0001d5fc"
	italic := "\U0001d638\U0001d630\U0001d633\U0001d62d\U0001d625"

	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Runs: []typewriter.RunStyle{
			{Style: typewriter.Bold},
			{Style: typewriter.Italic},
		},
	})
	fmt.Println(r.Replace(bold + " " + italic))
}

Output:
Hello world
Example (RunsSuperscript) 
package main

import (
	"fmt"

	"github.com/client9/typewriter"
)

func main() {
	// Render superscript digits with a caret (common in plain-text math).
	r := typewriter.New(typewriter.Config{
		Categories: typewriter.Default,
		Runs:       []typewriter.RunStyle{{Style: typewriter.Superscript, Prefix: "^"}},
	})
	fmt.Println(r.Replace("E=mc²"))
}

Output:
E=mc^2

func (*Replacer) Replace 

func (r *Replacer) Replace(s string) string

Replace returns s with all active conversions applied. If r is nil, the package-level Default replacer is used.

func (*Replacer) ReplaceBytes 

func (r *Replacer) ReplaceBytes(b []byte) []byte

ReplaceBytes returns a copy of b with all active conversions applied. If r is nil, the package-level Default replacer is used.

type RunStyle 

type RunStyle struct {
	Style  UnicodeStyle // style variant to detect; must not be StyleUnknown
	Prefix string       // prepended to the recovered ASCII text
	Suffix string       // appended to the recovered ASCII text
}

RunStyle configures how a contiguous run of styled Unicode characters is converted. The recovered ASCII text is wrapped with Prefix and Suffix. When both are empty the run is stripped to plain ASCII with no added markup.

Style must be set explicitly; StyleUnknown (zero value) matches nothing. If the same Style value appears more than once in Config.Runs, only the first occurrence is used; subsequent duplicates are silently ignored.

type UnicodeStyle 

type UnicodeStyle int

UnicodeStyle identifies a typographic Unicode style variant used in mathematical notation and social-media text.

The zero value is StyleUnknown. Callers must set Style explicitly when constructing a RunStyle; a zero-value RunStyle{} has no defined style. 

const (
	StyleUnknown UnicodeStyle = iota // zero value; not a valid style
	Bold                             // sans-serif bold: 𝗔𝗕𝗖 → ABC
	Italic                           // sans-serif italic: 𝘈𝘉𝘊 → ABC
	BoldItalic                       // sans-serif bold-italic: 𝘼𝘽𝘾 → ABC
	Monospace                        // monospace: 𝙰𝙱𝙲 → ABC
	Superscript                      // superscript digits/letters: ²⁴ → 24
	Subscript                        // subscript digits: ₂₄ → 24
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.