gse

package
v0.8.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 7, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package gse Go efficient multilingual NLP and text segmentation

Index

Constants

View Source 
const (
	// RatioWord ratio words and letters
	RatioWord float32 = 1.5
	// RatioWordFull full ratio words and letters
	RatioWordFull float32 = 1
)
View Source 
const (
	// Version get the gse version
	Version = "v1.0.1.705, Green Lake!"
)

Variables

View Source 
var StopWordMap = map[string]bool{
	" ": true,
}

StopWordMap the default stop words.

View Source 
var (
	// ToLower set alpha to lowercase
	ToLower = true
)

Functions

func DictPaths 

func DictPaths(dictDir, filePath string) (files []string)

DictPaths get the dict's paths

func FilterEmoji 

func FilterEmoji(text string) (new string)

FilterEmoji filter the emoji

func FilterHtml 

func FilterHtml(text string) string

FilterHtml filter the html tag

func FilterLang 

func FilterLang(text, lang string) (new string)

FilterLang filter the language

func FilterSymbol 

func FilterSymbol(text string) (new string)

FilterSymbol filter the symbol

func FindAllOccs 

func FindAllOccs(data []byte, searches []string) map[string][]int

FindAllOccs find the all search byte start in data

func GetVersion 

func GetVersion() string

GetVersion get the version of gse

func IsJp 

func IsJp(segText string) bool

IsJp is Japan char return true

func Join 

func Join(text []Text) string

Join is better string splicing

func Range 

func Range(text string) (new []string)

Range range text to []string

func RangeText 

func RangeText(text string) (new string)

RangeText range text to string

func SplitNum 

func SplitNum(text string) []string

SplitNum cut string by num to []string

func SplitNums 

func SplitNums(text string) string

SplitNums cut string by num to string

func ToSlice 

func ToSlice(segs []Segment, searchMode ...bool) (output []string)

ToSlice converts a segments to slice return string slice

func ToString 

func ToString(segs []Segment, searchMode ...bool) (output string)

ToString converts a segments slice to string return the string 

 two output modes:

	normal mode (searchMode=false）
	search mode（searchMode=true）

default searchMode=false search mode is used search engine, and will output more results

Types

type AnalyzeToken 

type AnalyzeToken struct {
	// the start of the segment in the text
	Start int
	End   int

	Position int
	Len      int

	Type string

	Text string
	Freq float64
	Pos  string
}

AnalyzeToken analyze the segment info structure

type Dictionary 

type Dictionary struct {
	Tokens []Token // the all tokens in the dictionary, to traverse
	// contains filtered or unexported fields
}

Dictionary struct implements a string double array trie. one segment maybe in leaf node or not

func NewDict 

func NewDict() *Dictionary

NewDict a new dictionary trie

func (*Dictionary) AddToken 

func (dict *Dictionary) AddToken(token Token) error

AddToken add a token to the dictionary

func (*Dictionary) Find 

func (dict *Dictionary) Find(word []byte) (float64, string, bool)

Find find the word in the dictionary is non-existent and the word's frequency and pos

func (*Dictionary) FindTFIDF 

func (dict *Dictionary) FindTFIDF(word []byte) (float64, float64, bool)

func (*Dictionary) LookupTokens 

func (dict *Dictionary) LookupTokens(
	words []Text, tokens []*Token) (numOfTokens int)

LookupTokens finds tokens and words in the dictionary, matching the given pattern and returns the number of tokens

func (*Dictionary) MaxTokenLen 

func (dict *Dictionary) MaxTokenLen() int

MaxTokenLen the maximum length of the dictionary

func (*Dictionary) NumTokens 

func (dict *Dictionary) NumTokens() int

NumTokens the number of tokens in the dictionary

func (*Dictionary) RemoveToken 

func (dict *Dictionary) RemoveToken(token Token) error

RemoveToken remove token in dictionary

func (*Dictionary) TotalFreq 

func (dict *Dictionary) TotalFreq() float64

TotalFreq the total frequency of the dictionary

func (*Dictionary) Value 

func (dict *Dictionary) Value(word []byte) (val, id int, err error)

Value find word in the dictionary return the word's value and id

type Prob 

type Prob struct {
	B, E, M, S map[rune]float64
}

Prob define the hmm model struct

type SegPos 

type SegPos struct {
	Text, Pos string
}

SegPos type a POS struct

func ToPos 

func ToPos(segs []Segment, searchMode ...bool) (output []SegPos)

ToPos converts a segments slice to []SegPos

type Segment 

type Segment struct {
	Position int
	// contains filtered or unexported fields
}

Segment a segment in the text

func (*Segment) End 

func (s *Segment) End() int

End return the end byte position of the segment (not including this)

func (*Segment) Start 

func (s *Segment) Start() int

Start returns the start byte position of the segment

func (*Segment) Token 

func (s *Segment) Token() *Token

Token return the segment token information

type Segmenter 

type Segmenter struct {
	Dict     *Dictionary
	Load     bool
	DictSep  string
	DictPath string

	// NotLoadHMM option load the default hmm model config (Chinese char)
	NotLoadHMM bool

	// AlphaNum set splitTextToWords can add token
	// when words in alphanum
	// set up alphanum dictionary word segmentation
	AlphaNum bool
	Alpha    bool
	Num      bool

	// LoadNoFreq load not have freq dict word
	LoadNoFreq bool
	// MinTokenFreq load min freq token
	MinTokenFreq float64
	// TextFreq add token frequency when not specified freq
	TextFreq string

	// SkipLog set skip log print
	SkipLog bool
	MoreLog bool

	// SkipPos skip PosStr pos
	SkipPos bool

	NotStop bool
	// StopWordMap the stop word map
	StopWordMap map[string]bool

	// CorpusAverLen the average length of corpus
	CorpusAverLen float64
}

Segmenter define the segmenter structure

func New 

func New(files ...string) (seg Segmenter, err error)

New return a new gse segmenter

func NewEmbed 

func NewEmbed(dict ...string) (seg Segmenter, err error)

NewEmbed return new gse segmenter by embed dictionary

func (*Segmenter) AddStop 

func (seg *Segmenter) AddStop(text string)

AddStop add a token to the StopWord dictionary.

func (*Segmenter) AddStopArr 

func (seg *Segmenter) AddStopArr(text ...string)

AddStopArr add array stop token to stop dictionaries

func (*Segmenter) AddToken 

func (seg *Segmenter) AddToken(text string, freq float64, pos ...string) error

AddToken add a new text to the token

func (*Segmenter) AddTokenForce 

func (seg *Segmenter) AddTokenForce(text string, freq float64, pos ...string) (err error)

AddTokenForce add new text to token and force time-consuming

func (*Segmenter) Analyze 

func (seg *Segmenter) Analyze(text []string, t1 string, by ...bool) (az []AnalyzeToken)

Analyze analyze the token segment info

func (*Segmenter) CalcToken 

func (seg *Segmenter) CalcToken()

CalcToken calc the segmenter token

func (*Segmenter) Cut 

func (seg *Segmenter) Cut(str string, hmm ...bool) []string

Cut cuts a str into words using accurate mode. Parameter hmm controls whether to use the HMM(Hidden Markov Model) or use the user's model.

seg.Cut(text): 

use the shortest path

seg.Cut(text, false): 

use cut dag not hmm

seg.Cut(text, true): 

use cut dag and hmm mode

func (*Segmenter) CutAll 

func (seg *Segmenter) CutAll(str string) []string

CutAll cuts a str into words using full mode.

func (*Segmenter) CutDAG 

func (seg *Segmenter) CutDAG(str string, reg ...*regexp.Regexp) []string

CutDAG cut string with DAG use hmm and regexp

func (*Segmenter) CutDAGNoHMM 

func (seg *Segmenter) CutDAGNoHMM(str string) []string

CutDAGNoHMM cut string with DAG not use hmm

func (*Segmenter) CutSearch 

func (seg *Segmenter) CutSearch(str string, hmm ...bool) []string

CutSearch cuts str into words using search engine mode.

func (*Segmenter) CutStop 

func (seg *Segmenter) CutStop(str string, hmm ...bool) []string

CutStop cut string and tirm stop

func (*Segmenter) CutStr 

func (seg *Segmenter) CutStr(str []string, separator ...string) (r string)

CutStr cut []string with Cut return string

func (*Segmenter) CutTrim 

func (seg *Segmenter) CutTrim(str string, hmm ...bool) []string

CutTrim cut string and tirm

func (*Segmenter) CutTrimHtml 

func (seg *Segmenter) CutTrimHtml(str string, hmm ...bool) []string

CutTrimHtml cut string trim html and symbol return []string

func (*Segmenter) CutTrimHtmls 

func (seg *Segmenter) CutTrimHtmls(str string, hmm ...bool) string

CutTrimHtmls cut string trim html and symbol return string

func (*Segmenter) CutUrl 

func (seg *Segmenter) CutUrl(str string, num ...bool) []string

CutUrl cut url string trim symbol return []string

func (*Segmenter) CutUrls 

func (seg *Segmenter) CutUrls(str string, num ...bool) string

CutUrls cut url string trim symbol return string

func (*Segmenter) Dictionary 

func (seg *Segmenter) Dictionary() *Dictionary

Dictionary returns the dictionary used by the tokenizer

func (*Segmenter) Empty 

func (seg *Segmenter) Empty() error

Empty empty the seg dictionary

func (*Segmenter) EmptyStop 

func (seg *Segmenter) EmptyStop() error

EmptyStop empty the stop dictionary

func (*Segmenter) Find 

func (seg *Segmenter) Find(str string) (float64, string, bool)

Find find word in dictionary return word's freq, pos and existence

func (*Segmenter) FindTFIDF 

func (seg *Segmenter) FindTFIDF(str string) (float64, float64, bool)

FindTFIDF find word in dictionary return word's freq, inverseFreq and existence

func (*Segmenter) GetCorpusPath 

func (seg *Segmenter) GetCorpusPath(files ...string) []string

GetCorpusPath get the corpus path

func (*Segmenter) GetCurrentFilePath 

func (seg *Segmenter) GetCurrentFilePath() string

GetCurrentFilePath get the current file path

func (*Segmenter) GetIdfPath 

func (seg *Segmenter) GetIdfPath(files ...string) []string

GetIdfPath get the idf path

func (*Segmenter) GetTfIdfPath 

func (seg *Segmenter) GetTfIdfPath(files ...string) []string

GetTfIdfPath get the tfidf path

func (*Segmenter) HMMCut 

func (seg *Segmenter) HMMCut(str string, reg ...*regexp.Regexp) []string

HMMCut cut sentence string use HMM with Viterbi

func (*Segmenter) HMMCutMod 

func (seg *Segmenter) HMMCutMod(str string, prob ...map[rune]float64) []string

HMMCutMod cut sentence string use HMM with Viterbi

func (*Segmenter) Init 

func (seg *Segmenter) Init()

Init initializes the segmenter config

func (*Segmenter) IsStop 

func (seg *Segmenter) IsStop(s string) bool

IsStop check the word is a stop word.

func (*Segmenter) LoadCorpusAverLen 

func (seg *Segmenter) LoadCorpusAverLen(files ...string) (corpusTotal float64, err error)

LoadCorpusAverLen load the average length of corpus

func (*Segmenter) LoadDict 

func (seg *Segmenter) LoadDict(files ...string) error

LoadDict load the dictionary from the file

The format of the dictionary is (one for each participle): 

participle text, frequency, part of speech

And you can option the dictionary separator by seg.DictSep = ","

Can load multiple dictionary files, the file name separated by "," or ", " the front of the dictionary preferentially load the participle, 

such as: "user_dictionary.txt,common_dictionary.txt"

When a participle appears both in the user dictionary and in the `common dictionary`, the `user dictionary` is given priority.

func (*Segmenter) LoadDictEmbed 

func (seg *Segmenter) LoadDictEmbed(dict ...string) (err error)

LoadDictEmbed load the dictionary by embed file

func (*Segmenter) LoadDictMap 

func (seg *Segmenter) LoadDictMap(dict []map[string]string) error

LoadDictMap load dictionary from []map[string]string

func (*Segmenter) LoadDictStr 

func (seg *Segmenter) LoadDictStr(dict string) error

LoadDictStr load the dictionary from dict path

func (*Segmenter) LoadModel 

func (seg *Segmenter) LoadModel(prob ...map[rune]float64)

LoadModel load the hmm model (default is Chinese char)

Use the user's model: 

seg.LoadModel(B, E, M, S map[rune]float64)

func (*Segmenter) LoadStop 

func (seg *Segmenter) LoadStop(files ...string) error

LoadStop load stop word files add token to map

func (*Segmenter) LoadStopArr 

func (seg *Segmenter) LoadStopArr(dict []string)

LoadStopArr load stop word by []string

func (*Segmenter) LoadStopEmbed 

func (seg *Segmenter) LoadStopEmbed(dict ...string) (err error)

LoadStopEmbed load the stop dictionary from embed file

func (*Segmenter) LoadStopStr 

func (seg *Segmenter) LoadStopStr(dict string) error

LoadStopStr load the stop dictionary from dict path

func (*Segmenter) LoadTFIDFDict 

func (seg *Segmenter) LoadTFIDFDict(files []*types.LoadDictFile) error

LoadTFIDFDict load tfidf dict for cal tfidf & bm25

func (*Segmenter) LoadTFIDFDictStr 

func (seg *Segmenter) LoadTFIDFDictStr(dictFile *types.LoadDictFile) error

LoadTFIDFDictStr load the TFIDF dictionary from dict path

func (*Segmenter) ModeSegment 

func (seg *Segmenter) ModeSegment(bytes []byte, searchMode ...bool) []Segment

ModeSegment segment using search mode if searchMode is true

func (*Segmenter) Pos 

func (seg *Segmenter) Pos(s string, searchMode ...bool) []SegPos

Pos return text and pos array

func (*Segmenter) PosStr 

func (seg *Segmenter) PosStr(str []SegPos, separator ...string) (r string)

PosStr cut []SegPos with Pos return string

func (*Segmenter) PosTrim 

func (seg *Segmenter) PosTrim(str string, search bool, pos ...string) []SegPos

PosTrim cut string pos and trim

func (*Segmenter) PosTrimArr 

func (seg *Segmenter) PosTrimArr(str string, search bool, pos ...string) (re []string)

PosTrimArr cut string return pos.Text []string

func (*Segmenter) PosTrimStr 

func (seg *Segmenter) PosTrimStr(str string, search bool, pos ...string) string

PosTrimStr cut string return pos.Text string

func (*Segmenter) ReAddToken 

func (seg *Segmenter) ReAddToken(text string, freq float64, pos ...string) error

ReAddToken remove and add token again

func (*Segmenter) Read 

func (seg *Segmenter) Read(file string) error

Read read the dict file

func (*Segmenter) ReadCorpus 

func (seg *Segmenter) ReadCorpus(file string) (corpusAverLen float64, err error)

func (*Segmenter) ReadN 

func (seg *Segmenter) ReadN(reader *bufio.Reader) (size int, text, freqText, pos string, fsErr error)

ReadN read the tokens by '\n'

func (*Segmenter) ReadNTFIDF 

func (seg *Segmenter) ReadNTFIDF(reader *bufio.Reader) (size int, text, freqText, idfText string, fsErr error)

ReadNTFIDF read the tokens with tfidf by '\n'

func (*Segmenter) ReadTFIDF 

func (seg *Segmenter) ReadTFIDF(file string) error

ReadTFIDF read the dict file

func (*Segmenter) Reader 

func (seg *Segmenter) Reader(reader *bufio.Reader, files ...string) error

Reader load dictionary from io.Reader

func (*Segmenter) ReaderTFIDF 

func (seg *Segmenter) ReaderTFIDF(reader *bufio.Reader, files ...string) error

ReaderTFIDF load tfidf dictionary from io.Reader

func (*Segmenter) RemoveStop 

func (seg *Segmenter) RemoveStop(text string)

RemoveStop remove a token from the StopWord dictionary.

func (*Segmenter) RemoveToken 

func (seg *Segmenter) RemoveToken(text string) error

RemoveToken remove token in dictionary

func (*Segmenter) Segment 

func (seg *Segmenter) Segment(bytes []byte) []Segment

Segment use the shortest path to segment the text

input parameter：

bytes UTF8 text []byte

output：

[]Segment return segments result

func (*Segmenter) Size 

func (seg *Segmenter) Size(size int, text, freqText string) (freq float64)

Size frequency is calculated based on the size of the text

func (*Segmenter) Slice 

func (seg *Segmenter) Slice(s string, searchMode ...bool) []string

Slice use modeSegment segment return []string using search mode if searchMode is true

func (*Segmenter) SplitTextToWords 

func (seg *Segmenter) SplitTextToWords(text Text) []Text

SplitTextToWords splits a string to token words

func (*Segmenter) Stop 

func (seg *Segmenter) Stop(s []string) (r []string)

Stop trim []string stop word

func (*Segmenter) String 

func (seg *Segmenter) String(s string, searchMode ...bool) string

Slice use modeSegment segment return string using search mode if searchMode is true

func (*Segmenter) SuggestFreq 

func (seg *Segmenter) SuggestFreq(words ...string) float64

SuggestFreq suggest the words frequency return a suggested frequency of a word cutted to short words.

func (*Segmenter) ToToken 

func (seg *Segmenter) ToToken(text string, freq float64, pos ...string) Token

ToToken make the text, freq and pos to token structure

func (*Segmenter) Trim 

func (seg *Segmenter) Trim(s []string) (r []string)

Trim trim []string exclude symbol, space and punct

func (*Segmenter) TrimPos 

func (seg *Segmenter) TrimPos(s []SegPos) (r []SegPos)

TrimPos trim SegPos not symbol, space and punct

func (*Segmenter) TrimPosPunct 

func (seg *Segmenter) TrimPosPunct(se []SegPos) (re []SegPos)

TrimPosPunct trim SegPos not space and punct

func (*Segmenter) TrimPunct 

func (seg *Segmenter) TrimPunct(s []string) (r []string)

TrimPunct trim []string exclude space and punct

func (*Segmenter) TrimSymbol 

func (seg *Segmenter) TrimSymbol(s []string) (r []string)

TrimSymbol trim []string exclude symbol, space and punct

func (*Segmenter) TrimWithPos 

func (seg *Segmenter) TrimWithPos(se []SegPos, pos ...string) (re []SegPos)

TrimWithPos trim some seg with pos

func (*Segmenter) Value 

func (seg *Segmenter) Value(str string) (int, int, error)

Value find word in dictionary return word's value

type Text 

type Text []byte

Text a string type，used to parse text 1. a word, such as "world" or "boundary", in English a word is a word 2. a participle, such as "world" a.k.a. "population" 3. a text, such as "the world has seven billion people"

func SplitWords 

func SplitWords(text Text) []Text

SplitWords splits a string to token words

type Token 

type Token struct {
	// contains filtered or unexported fields
}

Token define a segment token structure

func (*Token) Equals 

func (token *Token) Equals(str string) bool

Equals compare str split tokens

func (*Token) Freq 

func (token *Token) Freq() float64

Freq returns the frequency in the dictionary token

func (*Token) Pos 

func (token *Token) Pos() string

Pos returns the part of speech in the dictionary token

func (*Token) Segments 

func (token *Token) Segments() []*Segment

Segments will segment further subdivisions of the text of this participle, the participle has two subclauses.

Subclauses can also have further subclauses forming a tree structure, which can be traversed to get all the detailed subdivisions of the participle, which is mainly Used by search engines to perform full-text searches on a piece of text.

func (*Token) Text 

func (token *Token) Text() string

Text return the text of the segment

Source Files

View all Source files

Directories

Path Synopsis
Package hmm is the Golang HMM cut module
 Package hmm is the Golang HMM cut module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.