tokenize

package

v0.8.0 Latest Latest Go to latest Published: Sep 6, 2026 License: MIT Imports: 6 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/git-pkgs/licenses

Links

Open Source Insights

Documentation ¶

Overview ¶

Package tokenize converts license text into normalized integer tokens.

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type ID ¶

type ID uint32

ID identifies a normalized word in a Vocabulary.

const Unknown ID = 0

Unknown is the ID assigned to words absent from a Vocabulary.

type IDTokens ¶ added in v0.4.0

type IDTokens struct {
	IDs           []ID
	UnknownAfter  []uint32
	StopwordAfter []uint32
	Start         int
	End           int
}

IDTokens contains known token IDs, omitted-word positions, and the byte range spanning the known tokens. UnknownAfter and StopwordAfter store the number of known tokens preceding each omitted word.

type Offset ¶

type Offset struct {
	Start int
	End   int
}

Offset is a half-open byte range in the original input.

func KnownTokenOffsetsAppend ¶ added in v0.8.0

func KnownTokenOffsetsAppend(
	input []byte,
	unknownAfter []uint32,
	stopwordAfter []uint32,
	offsets []Offset,
) []Offset

KnownTokenOffsetsAppend returns byte ranges for known words using the omitted-word positions returned by TokenizeIDsAppend.

func TokenOffsets ¶ added in v0.4.0

func TokenOffsets(input []byte, tokenCount int) []Offset

TokenOffsets returns every normalized word's byte range. tokenCount is a capacity hint and does not limit the number of returned offsets.

func TokenOffsetsAppend ¶ added in v0.8.0

func TokenOffsetsAppend(input []byte, offsets []Offset) []Offset

TokenOffsetsAppend is TokenOffsets writing into offsets[:0].

type Tokens ¶

type Tokens struct {
	IDs     []ID
	Offsets []Offset
}

Tokens contains token IDs and their corresponding input byte ranges.

type Vocabulary ¶

type Vocabulary struct {
	// contains filtered or unexported fields
}

Vocabulary is an immutable mapping from normalized words to integer IDs.

func NewVocabulary ¶

func NewVocabulary(texts [][]byte) (*Vocabulary, error)

NewVocabulary builds a deterministic vocabulary from texts. IDs are assigned in normalized lexical order and start at one, leaving zero for Unknown.

func NewVocabularyFromWords ¶

func NewVocabularyFromWords(words []string) (*Vocabulary, error)

NewVocabularyFromWords loads an already normalized and sorted vocabulary.

func NewVocabularyFromWordsWithStopwords ¶ added in v0.8.0

func NewVocabularyFromWordsWithStopwords(
	words []string,
	stopwords []uint32,
) (*Vocabulary, error)

NewVocabularyFromWordsWithStopwords loads normalized words and marks the one-based IDs in stopwords for omission from token sequences.

func NewVocabularyWithStopwords ¶ added in v0.8.0

func NewVocabularyWithStopwords(texts [][]byte, stopwords []string) (*Vocabulary, error)

NewVocabularyWithStopwords builds a vocabulary that omits stopwords from token sequences while retaining their IDs.

func (*Vocabulary) Len ¶

func (v *Vocabulary) Len() int

Len returns the number of known words, excluding Unknown.

func (*Vocabulary) Lookup ¶

func (v *Vocabulary) Lookup(word string) (ID, bool)

Lookup returns the ID for a normalized word.

func (*Vocabulary) Tokenize ¶

func (v *Vocabulary) Tokenize(input []byte) Tokens

Tokenize normalizes input and maps every word to an ID and byte range.

func (*Vocabulary) TokenizeIDs ¶ added in v0.4.0

func (v *Vocabulary) TokenizeIDs(input []byte) IDTokens

TokenizeIDs normalizes input and maps known words to IDs. Start and End span the first through last known token without retaining every token offset.

func (*Vocabulary) TokenizeIDsAppend ¶ added in v0.8.0

func (v *Vocabulary) TokenizeIDsAppend(
	input []byte,
	ids []ID,
	unknownAfter []uint32,
	stopwordAfter []uint32,
	wordScratch *[]byte,
) IDTokens

TokenizeIDsAppend is TokenizeIDs writing into ids[:0] and the position buffers. wordScratch, when provided, is reused for case normalization and may be grown.

func (*Vocabulary) Word ¶

func (v *Vocabulary) Word(id ID) string

Word returns the normalized word for id, or an empty string for an unknown ID.

func (*Vocabulary) Words ¶

func (v *Vocabulary) Words() []string

Words returns the normalized words in ID order, excluding Unknown.

type Word ¶

type Word struct {
	Text  string
	Start int
	End   int
}

Word is a normalized word and its byte range in the original input.

func Words ¶

func Words(input []byte) []Word

Words returns normalized words and their byte ranges without mapping IDs.

Source Files ¶

View all Source files

tokenize.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL