text

package
v2.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 27, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package text provides functionalities to manipulate texts.

Index

Constants

View Source
const EOF = byte(0xff)

EOF indicates the end of file.

Variables

This section is empty.

Functions

This section is empty.

Types

type BlockReader

type BlockReader interface {
	Reader
	// Reset resets current state and sets new segments to the reader.
	Reset(segs []Segment)
}

A BlockReader interface is a reader that is optimized for Blocks.

func NewBlockReader

func NewBlockReader(source []byte, segs []Segment, decoder Decoder) BlockReader

NewBlockReader returns a new BlockReader, decoding constructed Values using the given Decoder.

type Decoder

type Decoder interface {
	// Decode decodes the given byte slice and returns the decoded bytes.
	Decode(b []byte) []byte

	// DecodeTo decodes the given byte slice and writes the decoded bytes to w.
	DecodeTo(w io.Writer, b []byte) (int, error)
}

A Decoder decodes a byte slice, e.g. resolving backslash escapes and character references, and returns or writes out the decoded bytes.

Note that this interface is not intended to 'normalize' a byte slice, but to decode it. For example, CommonMark requires that code spans normalize leading spaces, trailing spaces and newlines, but this interface does not do that.

var IdentityDecoder Decoder = &identityDecoder{}

IdentityDecoder is a decoder that does not perform any decoding and returns the bytes as is.

type DecoderOption

type DecoderOption func(*decoderConfig)

DecoderOption is a function that configures a decoder.

func WithEscapedSpace

func WithEscapedSpace() DecoderOption

WithEscapedSpace configures the decoder to treat escaped spaces as an empty character.

type DefaultDecoder

type DefaultDecoder struct {
	// contains filtered or unexported fields
}

DefaultDecoder is the default implementation of the Decoder interface.

CommonMark defines texts except some inline elements (e.g., code span, auto link, etc) can contain character references and backslash escapes. This decoder decodes them.

- decodes entity references (e.g., `& -> &`) - decodes numeric character references (e.g., `& -> &`) - decodes `\` escaped punctuations (e.g., `\* -> *`)

  • If WithEscapedSpace is set, it will also decode `\ ` to an empty character.

func NewDecoder

func NewDecoder(opts ...DecoderOption) *DefaultDecoder

NewDecoder creates a new Decoder with the given options.

func (*DefaultDecoder) Decode

func (d *DefaultDecoder) Decode(b []byte) []byte

Decode implements the Decoder interface.

func (*DefaultDecoder) DecodeTo

func (d *DefaultDecoder) DecodeTo(w io.Writer, b []byte) (int, error)

DecodeTo implements the Decoder interface.

type Index

type Index struct {
	Start int
	Stop  int
}

An Index represents a position range in a source byte slice. Unlike Segment, Index does not carry parsing metadata such as Padding or ForceNewline.

func NewIndex

func NewIndex(start, stop int) Index

NewIndex returns a new Index.

func NewIndexFromSegment

func NewIndexFromSegment(seg Segment) Index

NewIndexFromSegment returns a new Index derived from the given Segment, dropping any padding and ignoring ForceNewline.

func (Index) IsEmpty

func (i Index) IsEmpty() bool

IsEmpty returns true if this index is empty, otherwise false.

type Lines

type Lines struct {
	// contains filtered or unexported fields
}

A Lines value holds the raw content of a block node. It is either an owned string or a sequence of Segment values copied from the parser's source-segment list.

Lines is used for block nodes whose rendered content is taken directly from the source (e.g. CodeBlock, FencedCodeBlock, HTMLBlock).

Lines is always 'raw'; it does not perform any decoding of the source content.

func NewLines

func NewLines[T LinesInput](v T) Lines

NewLines returns a Lines from the given input, which may be a string, byte slice, or slice of Segment.

func NewLinesFromSegments

func NewLinesFromSegments(segs []Segment) Lines

NewLinesFromSegments returns a Lines backed by source segments.

func NewLinesFromString

func NewLinesFromString(s string) Lines

NewLinesFromString returns a Lines backed by an owned string.

func (*Lines) AppendSegment

func (l *Lines) AppendSegment(seg Segment)

AppendSegment appends a Segment to the internal segment list.

func (Lines) Bytes

func (l Lines) Bytes(source []byte) []byte

Bytes returns the concatenated byte content of all segments, or the owned string. The returned byte slice is read-only and should not be modified.

func (Lines) IsOwned

func (l Lines) IsOwned() bool

IsOwned returns true if this value is an owned string not derived from the source byte slice.

func (Lines) Segments

func (l Lines) Segments() []Segment

Segments returns the internal segment slice, allowing callers to read and otherwise inspect the segment list.

func (Lines) Str

func (l Lines) Str(source []byte) string

Str returns the string representation of this value. The returned string is read-only and should not be modified. See Bytes() for details on how the string is constructed from source segments.

func (Lines) WriteTo

func (l Lines) WriteTo(w io.Writer, source []byte) (int, error)

WriteTo writes the value to the given buffer, using the Lines.Bytes.

type LinesInput

type LinesInput interface {
	string | []byte | []Segment | Lines
}

LinesInput is a type constraint for types that can be converted to a Lines.

type MultiLineValue

type MultiLineValue struct {
	// contains filtered or unexported fields
}

A MultiLineValue holds a potentially multiline inline value that is either an owned string or a set of source position ranges. Use MultiLineValue for data that may span multiple lines per the CommonMark spec (e.g. link labels).

When constructing the byte value from source positions, the ranges are simply concatenated verbatim; no newline folding or other transformation is applied.

func NewMultiLineValue

func NewMultiLineValue[T MultiLineValueInput](v T, decoder Decoder) MultiLineValue

NewMultiLineValue returns a MultiLineValue from the given input, which may be a string, byte slice, Index, or slice of Index, bound to the given Decoder.

func NewMultiLineValueFromIndex

func NewMultiLineValueFromIndex(i Index, decoder Decoder) MultiLineValue

NewMultiLineValueFromIndex returns a MultiLineValue backed by a single source position, bound to the given Decoder.

func NewMultiLineValueFromIndices

func NewMultiLineValueFromIndices(indices []Index, decoder Decoder) MultiLineValue

NewMultiLineValueFromIndices returns a MultiLineValue backed by source positions, bound to the given Decoder.

func NewMultiLineValueFromString

func NewMultiLineValueFromString(s string, decoder Decoder) MultiLineValue

NewMultiLineValueFromString returns a MultiLineValue backed by an owned string, bound to the given Decoder.

func (MultiLineValue) Bytes

func (v MultiLineValue) Bytes(source []byte) []byte

Bytes implements Value.Bytes .

func (MultiLineValue) Index

func (v MultiLineValue) Index() Index

Index implements Value.Index .

func (MultiLineValue) Indices

func (v MultiLineValue) Indices() []Index

Indices implements Value.Indices .

func (MultiLineValue) IsEmpty

func (v MultiLineValue) IsEmpty() bool

IsEmpty implements Value.IsEmpty .

func (MultiLineValue) IsOwned

func (v MultiLineValue) IsOwned() bool

IsOwned implements Value.IsOwned .

func (MultiLineValue) Str

func (v MultiLineValue) Str(source []byte) string

Str implements Value.Str .

func (MultiLineValue) Value

func (v MultiLineValue) Value(source []byte) string

Value implements Value.Value .

func (MultiLineValue) WriteTo

func (v MultiLineValue) WriteTo(w io.Writer, source []byte) (int, error)

WriteTo implements Value.WriteTo .

type MultiLineValueInput

type MultiLineValueInput interface {
	string | []byte | Index | []Index
}

MultiLineValueInput is a type constraint for types that can be converted to a MultiLineValue.

type Reader

type Reader interface {
	io.RuneReader

	// Source returns a source of the reader.
	Source() []byte

	// ResetPosition resets positions.
	ResetPosition()

	// Peek returns a byte at current position without advancing the internal pointer.
	Peek() byte

	// PeekLine returns the current line without advancing the internal pointer.
	PeekLine() ([]byte, Segment)

	// PrecedingCharacter returns a character just before current internal pointer.
	PrecedingCharacter() rune

	// ValueBetween returns a MultiLineValue covering the given [start, stop)
	// byte range within the source.
	//
	// Value will be decoded using the Decoder bound to this reader.
	ValueBetween(start, stop int) MultiLineValue

	// Decoder returns the Decoder bound to this reader. Values constructed
	// from positions within this reader's source should be bound to this
	// Decoder.
	Decoder() Decoder

	// LineOffset returns a distance from the line head to current position.
	LineOffset() int

	// Position returns current line number and position.
	Position() (int, Segment)

	// SetPosition sets current line number and position.
	SetPosition(int, Segment)

	// SetPadding sets padding to the reader.
	SetPadding(int)

	// Advance advances the internal pointer.
	Advance(int)

	// AdvanceAndSetPadding advances the internal pointer and add padding to the
	// reader.
	AdvanceAndSetPadding(int, int)

	// AdvanceToEOL advances the internal pointer to the end of line.
	// If the line ends with a newline, it will be included in the segment.
	// If the line ends with EOF, it will not be included in the segment.
	AdvanceToEOL()

	// AdvanceLine advances the internal pointer to the next line head.
	AdvanceLine()

	// SkipSpaces skips space characters and returns a non-blank line.
	// If it reaches EOF, returns false.
	SkipSpaces() (Segment, int, bool)

	// SkipBlankLines skips blank lines and returns a non-blank line.
	// If it reaches EOF, returns false.
	SkipBlankLines() (Segment, int, bool)

	// Match performs regular expression matching to current line.
	Match(reg *regexp.Regexp) bool

	// Match performs regular expression searching to current line.
	FindSubMatch(reg *regexp.Regexp) [][]byte
}

A Reader interface provides abstracted method for reading text.

func NewReader

func NewReader(b []byte, decoder Decoder) Reader

NewReader return a new Reader that can read UTF-8 bytes, decoding constructed Values using the given Decoder. b need not be a Markdown document's source; it may be any byte slice, e.g. a substring extracted from a Value.

type Segment

type Segment struct {
	// Start is a start position of the segment.
	Start int

	// Stop is a stop position of the segment.
	// This value should be excluded.
	Stop int

	// Padding is a padding length of the segment.
	Padding int

	// ForceNewline is true if the segment should be ended with a newline.
	// Some elements(i.e. CodeBlock, FencedCodeBlock) does not trim trailing
	// newlines. Spec defines that EOF is treated as a newline, so we need to
	// add a newline to the end of the segment if it is not empty.
	//
	// i.e.:
	//
	//     “`go
	//     const test = "test"
	//
	// This code does not close the code block and ends with EOF. In this case,
	// we need to add a newline to the end of the last line like `const test = "test"\n`.
	ForceNewline bool
}

A Segment struct holds information about source positions.

func NewSegment

func NewSegment(start, stop int) Segment

NewSegment return a new Segment.

func NewSegmentPadding

func NewSegmentPadding(start, stop, padding int) Segment

NewSegmentPadding returns a new Segment with the given padding.

func (Segment) Between

func (t Segment) Between(other Segment) Segment

Between returns a segment between this segment and the given segment.

func (Segment) Bytes

func (t Segment) Bytes(source []byte) []byte

Bytes returns bytes of the segment.

func (Segment) IsEmpty

func (t Segment) IsEmpty() bool

IsEmpty returns true if this segment is empty, otherwise false.

func (Segment) Len

func (t Segment) Len() int

Len returns a length of the segment.

func (Segment) Str

func (t Segment) Str(source []byte) string

Str returns a string of the segment.

func (Segment) TrimLeftSpace

func (t Segment) TrimLeftSpace(source []byte) Segment

TrimLeftSpace returns a new segment by slicing off all leading space characters including padding.

func (Segment) TrimLeftSpaceWidth

func (t Segment) TrimLeftSpaceWidth(width int, source []byte) Segment

TrimLeftSpaceWidth returns a new segment by slicing off leading space characters until the given width.

func (Segment) TrimRightSpace

func (t Segment) TrimRightSpace(source []byte) Segment

TrimRightSpace returns a new segment by slicing off all trailing space characters.

func (Segment) WithStart

func (t Segment) WithStart(v int) Segment

WithStart returns a new Segment with the given Start and the same Stop. Padding and ForceNewline are reset to their zero values, since Padding describes space that precedes the original Start position.

func (Segment) WithStop

func (t Segment) WithStop(v int) Segment

WithStop returns a new Segment with the given Stop and the same Start and Padding. ForceNewline is reset to false.

type SingleLineValue

type SingleLineValue struct {
	// contains filtered or unexported fields
}

A SingleLineValue holds a single-line inline value that is either an owned string or a position range within a source byte slice. Use SingleLineValue for data that must not contain newlines per the CommonMark spec (e.g. link destinations, fenced code block info strings).

func NewSingleLineValue

func NewSingleLineValue[T SingleLineValueInput](v T, decoder Decoder) SingleLineValue

NewSingleLineValue returns a Value from the given input, bound to the given Decoder.

func NewSingleLineValueFromIndex

func NewSingleLineValueFromIndex(i Index, decoder Decoder) SingleLineValue

NewSingleLineValueFromIndex returns a Value backed by a source position, bound to the given Decoder.

func NewSingleLineValueFromSegment

func NewSingleLineValueFromSegment(seg Segment, decoder Decoder) SingleLineValue

NewSingleLineValueFromSegment returns a Value backed by a source position derived from the given Segment, bound to the given Decoder.

func NewSingleLineValueFromString

func NewSingleLineValueFromString(s string, decoder Decoder) SingleLineValue

NewSingleLineValueFromString returns a Value backed by an owned string, bound to the given Decoder. This function does not check whether the string contains newlines; it is the caller's responsibility to ensure that the string is single-line.

ValueBuilder.Build will automatically choose between SingleLineValue and MultiLineValue based on the presence of newlines in the string.

func (SingleLineValue) Bytes

func (v SingleLineValue) Bytes(source []byte) []byte

Bytes implements Value.Bytes .

func (SingleLineValue) Index

func (v SingleLineValue) Index() Index

Index implements Value.Index .

func (SingleLineValue) Indices

func (v SingleLineValue) Indices() []Index

Indices implements Value.Indices .

func (SingleLineValue) IsEmpty

func (v SingleLineValue) IsEmpty() bool

IsEmpty implements Value.IsEmpty .

func (SingleLineValue) IsOwned

func (v SingleLineValue) IsOwned() bool

IsOwned implements Value.IsOwned .

func (SingleLineValue) Str

func (v SingleLineValue) Str(source []byte) string

Str implements Value.Str .

func (SingleLineValue) Value

func (v SingleLineValue) Value(source []byte) string

Value implements Value.Value .

func (SingleLineValue) WithStop

func (v SingleLineValue) WithStop(stop int) SingleLineValue

WithStop returns a new SingleLineValue with the same start index but a different stop index. This method panics if the value is owned.

func (SingleLineValue) WriteTo

func (v SingleLineValue) WriteTo(w io.Writer, source []byte) (int, error)

WriteTo implements Value.WriteTo .

type SingleLineValueInput

type SingleLineValueInput interface {
	string | []byte | Index
}

SingleLineValueInput is a type constraint for types that can be converted to a Value.

type Value

type Value interface {
	// Value returns the decoded string representation of this value.
	Value(source []byte) string

	// Bytes returns the source byte slice corresponding to this value.
	Bytes(source []byte) []byte

	// Str returns the string representation of this value.
	Str(source []byte) string

	// IsOwned returns true if this value is an owned string not derived from the source byte slice.
	IsOwned() bool

	// IsEmpty returns true if this value is empty, otherwise false.
	IsEmpty() bool

	// Index returns the source position of this value.
	//
	// The result is meaningful only when IsOwned() returns false.
	// If Value is backed by multiple source positions, Index returns the first position.
	Index() Index

	// Indices returns the slice of Index values in this value.
	//
	// The result is meaningful only when IsOwned() returns false.
	Indices() []Index

	// WriteTo writes the value to the given buffer, using the [Value].Value.
	WriteTo(w io.Writer, source []byte) (int, error)
}

A Value represents an inline value.

type ValueBuilder

type ValueBuilder struct {
	// contains filtered or unexported fields
}

ValueBuilder is a helper for building a Value.

func (*ValueBuilder) AddIndex

func (b *ValueBuilder) AddIndex(i Index) *ValueBuilder

AddIndex adds an Index to the builder.

func (*ValueBuilder) AddSegment

func (b *ValueBuilder) AddSegment(seg Segment) *ValueBuilder

AddSegment adds an Index derived from the given Segment to the builder.

func (*ValueBuilder) Build

func (b *ValueBuilder) Build() Value

Build returns a Value from the accumulated state. If OwnedString or OwnedBytes was called, the result is backed by that string. Otherwise, the result is backed by the accumulated indices.

func (*ValueBuilder) BuildMultiLine

func (b *ValueBuilder) BuildMultiLine() MultiLineValue

BuildMultiLine returns a MultiLineValue from the accumulated state. If OwnedString or OwnedBytes was called, the result is backed by that string. Otherwise, the result is backed by the accumulated indices.

func (*ValueBuilder) BuildSingleLine

func (b *ValueBuilder) BuildSingleLine() SingleLineValue

BuildSingleLine returns a SingleLineValue from the accumulated state. If OwnedString or OwnedBytes was called, the result is backed by that string. Otherwise, the result is backed by the first accumulated index.

func (*ValueBuilder) Decoder

func (b *ValueBuilder) Decoder(d Decoder) *ValueBuilder

Decoder sets the Decoder to be bound to the Value produced by Build, BuildSingleLine, or BuildMultiLine. If not set, the produced Value is bound to IdentityDecoder.

func (*ValueBuilder) OwnedBytes

func (b *ValueBuilder) OwnedBytes(bts []byte) *ValueBuilder

OwnedBytes sets an owned string value from the given byte slice. When Build is called, the resulting Value will be backed by this string rather than any accumulated indices.

func (*ValueBuilder) OwnedString

func (b *ValueBuilder) OwnedString(s string) *ValueBuilder

OwnedString sets an owned string value. When Build is called, the resulting Value will be backed by this string rather than any accumulated indices.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL