Documentation
¶
Overview ¶
Package tokenize converts license text into normalized integer tokens.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ID ¶
type ID uint32
ID identifies a normalized word in a Vocabulary.
const Unknown ID = 0
Unknown is the ID assigned to words absent from a Vocabulary.
type IDTokens ¶ added in v0.4.0
IDTokens contains token IDs and the byte range spanning all input tokens.
type Offset ¶
Offset is a half-open byte range in the original input.
func TokenOffsets ¶ added in v0.4.0
TokenOffsets returns every normalized word's byte range. tokenCount is a capacity hint and does not limit the number of returned offsets.
type Vocabulary ¶
type Vocabulary struct {
// contains filtered or unexported fields
}
Vocabulary is an immutable mapping from normalized words to integer IDs.
func NewVocabulary ¶
func NewVocabulary(texts [][]byte) (*Vocabulary, error)
NewVocabulary builds a deterministic vocabulary from texts. IDs are assigned in normalized lexical order and start at one, leaving zero for Unknown.
func NewVocabularyFromWords ¶
func NewVocabularyFromWords(words []string) (*Vocabulary, error)
NewVocabularyFromWords loads an already normalized and sorted vocabulary.
func (*Vocabulary) Len ¶
func (v *Vocabulary) Len() int
Len returns the number of known words, excluding Unknown.
func (*Vocabulary) Lookup ¶
func (v *Vocabulary) Lookup(word string) (ID, bool)
Lookup returns the ID for a normalized word.
func (*Vocabulary) Tokenize ¶
func (v *Vocabulary) Tokenize(input []byte) Tokens
Tokenize normalizes input and maps every word to an ID and byte range.
func (*Vocabulary) TokenizeIDs ¶ added in v0.4.0
func (v *Vocabulary) TokenizeIDs(input []byte) IDTokens
TokenizeIDs normalizes input and maps every word to an ID. Start and End span the first through last token without retaining an offset for every token.
func (*Vocabulary) Word ¶
func (v *Vocabulary) Word(id ID) string
Word returns the normalized word for id, or an empty string for an unknown ID.
func (*Vocabulary) Words ¶
func (v *Vocabulary) Words() []string
Words returns the normalized words in ID order, excluding Unknown.
Source Files