fulltext

package
v0.0.0-debug-20260702 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	FulltextRelevancyAlgo       = "ft_relevancy_algorithm"
	FulltextRelevancyAlgo_bm25  = "BM25"
	FulltextRelevancyAlgo_tfidf = "TF-IDF"
)
View Source 
const BM25_B = 0.75
View Source 
const BM25_K1 = 1.5
View Source 
const DOC_LEN_WORD = "__DocLen" //

Variables

View Source 
var (
	TEXT        = 0
	STAR        = 1
	PLUS        = 2
	MINUS       = 3
	LESSTHAN    = 4
	GREATERTHAN = 5
	RANKLESS    = 6
	GROUP       = 7
	PHRASE      = 8
	JOIN        = 9
)
View Source 
var LOWER_BIT_MASK = uint64(0xffffff)

24 bits low bits - offset in partition (16MB) high bits - partition id

View Source 
var LOWER_BIT_SHIFT = uint64(24)

Functions

func GenTextSql 

func GenTextSql(p *Pattern, mode int64, idxtbl string, parser string) (string, error)

GenTextSql that support ngram in boolean mode TEXT is a word. For english, it is fine to have a simple filter word = 'word". For Chinese, word should be tokenize into ngram

func GetOp 

func GetOp(op rune) int

func GetPartitionAddr 

func GetPartitionAddr(partid uint64, offset uint64) uint64

convert partition id and offset into address

func GetPartitionId 

func GetPartitionId(addr uint64) uint64

get partition ID from address

func GetPartitionOffset 

func GetPartitionOffset(addr uint64) uint64

get offset from address

func GetResultCountFromPattern 

func GetResultCountFromPattern(ps []*Pattern) int

func IsSubExpression 

func IsSubExpression(pattern string) bool

func OperatorToString 

func OperatorToString(op int) string

func PatternListToString 

func PatternListToString(ps []*Pattern) string

func PatternListToStringWithPosition 

func PatternListToStringWithPosition(ps []*Pattern) string

func PatternToSql 

func PatternToSql(ps []*Pattern, mode int64, idxTable string, parser string, algo FullTextScoreAlgo) (string, error)

API for generate SQL from pattern

func PatternToString 

func PatternToString(pattern string, mode int64) (string, error)

func PatternToStringWithPosition 

func PatternToStringWithPosition(pattern string, mode int64) (string, error)

func PhraseCountSQL 

func PhraseCountSQL(ps []*Pattern, mode int64, idxTable string) (string, bool, error)

func PhraseTopKBM25SQL 

func PhraseTopKBM25SQL(ps []*Pattern, mode int64, idxTable string, idfSq float64, avgDocLen float64, nkeywords int, limit uint64) (string, bool, error)

func PhraseTopKSQL 

func PhraseTopKSQL(ps []*Pattern, mode int64, idxTable string, limit uint64) (string, bool, error)

func SingleKeywordTopKBM25SQL 

func SingleKeywordTopKBM25SQL(ps []*Pattern, mode int64, idxTable string, avgDocLen float64, limit uint64) (string, bool, error)

func SingleKeywordTopKSQL 

func SingleKeywordTopKSQL(ps []*Pattern, mode int64, idxTable string, limit uint64) (string, bool, error)

func SqlBoolean 

func SqlBoolean(ps []*Pattern, mode int64, idxtbl string, parser string) (string, error)

Generate SQL in boolean mode

func SqlPhrase 

func SqlPhrase(ps []*Pattern, mode int64, idxtbl string, withIndex bool) (string, error)

Generate SQL in phrase mode. It is the same for natural language mode and phrase search in boolean mode

Types

type FixedBytePool 

type FixedBytePool struct {
	// contains filtered or unexported fields
}

FixedBytePool

func NewFixedBytePool 

func NewFixedBytePool(proc *process.Process, dsize uint64, partition_cap uint64, mem_limit uint64) *FixedBytePool

FixedBytePool

func (*FixedBytePool) Close 

func (pool *FixedBytePool) Close()

Close the pool and cleanup memory and temp files

func (*FixedBytePool) FreeItem 

func (pool *FixedBytePool) FreeItem(addr uint64) error

FreeItem call partition.FreeItem

func (*FixedBytePool) GetItem 

func (pool *FixedBytePool) GetItem(addr uint64) ([]byte, error)

Getitem return item with partitions. If requested partition was spilled, unspill()

func (*FixedBytePool) NewItem 

func (pool *FixedBytePool) NewItem() (addr uint64, b []byte, err error)

NewItem will find a available partition for NewItem. Usually the last item of partition slice If memory in use exceed the memory limit, spill If no avaiable partition, create a new partition

func (*FixedBytePool) Spill 

func (pool *FixedBytePool) Spill() error

spill will find LRU partitions to spill and will double the number of partitions to spill for the next time

func (*FixedBytePool) String 

func (pool *FixedBytePool) String() string

type FixedBytePoolIterator 

type FixedBytePoolIterator struct {
	// contains filtered or unexported fields
}

FixedBytePoolIterator to tranverse the data in the pool

func NewFixedBytePoolIterator 

func NewFixedBytePoolIterator(p *FixedBytePool) *FixedBytePoolIterator

Iterator

After hash build/aggregate, all data will reside in partitions. You can use FixedBytePoolIterator to tranverse all data in partitions. We don't provide you the hash key here. just the values

Iterator will free up the memory once it finish traverse a partition with partition.Close(). If not doing this, we will have OOM due to keep all partitions in memory. When partition was spilled into disk, iterator will read the file into memory.

func (*FixedBytePoolIterator) Next 

func (it *FixedBytePoolIterator) Next() ([]byte, error)

Get next []byte from the pool. If no more data, return nil []byte and nil error

type FullTextBooleanOperator 

type FullTextBooleanOperator int

Boolean mode search string parsing

type FullTextParserParam 

type FullTextParserParam struct {
	Parser string `json:"parser"`
	Async  string `json:"async"`
}

Parser parameters

type FullTextScoreAlgo 

type FullTextScoreAlgo int
const (
	ALGO_BM25 FullTextScoreAlgo = iota
	ALGO_TFIDF
)

func GetScoreAlgo 

func GetScoreAlgo(proc *process.Process) (FullTextScoreAlgo, error)

type Lru 

type Lru struct {
	// contains filtered or unexported fields
}

Least recently use

type Partition 

type Partition struct {
	// contains filtered or unexported fields
}

Partition which is able to spill/unspill. Data must be fixed size when init

func NewPartition 

func NewPartition(proc *process.Process, id uint64, capacity uint64, dsize uint64) (*Partition, error)

New Partition with capacity, fixed data size

func (*Partition) Close 

func (part *Partition) Close()

Close the partition

func (*Partition) FreeItem 

func (part *Partition) FreeItem(offfset uint64) (uint64, error)

FreeItem simply reduce reference count by one and free the data when refcnt == 0

func (*Partition) GetItem 

func (part *Partition) GetItem(offset uint64) ([]byte, error)

GetItem with offset

func (*Partition) Id 

func (part *Partition) Id() uint64

func (*Partition) LastUpdate 

func (part *Partition) LastUpdate() time.Time

check last update of the partition. Spill always choose LRU partitions

func (*Partition) NewItem 

func (part *Partition) NewItem() (addr uint64, b []byte, err error)

NewItem will return the []byte and address and set full is true when partition is full for next item

func (*Partition) Spill 

func (part *Partition) Spill() error

Spill to spill the memory into disk and free up memory

func (*Partition) Spilled 

func (part *Partition) Spilled() bool

check partition is already spilled

func (*Partition) Unspill 

func (part *Partition) Unspill() error

read the spill file to memory and remove the temp file the state of the partition should be the same with spill/unspill so that partition can still perform NewItem() and GetItem() after unspill()

type Pattern 

type Pattern struct {
	Text     string
	Operator int
	Children []*Pattern
	Position int32
	Index    int32
}

Pattern works on both Natural Language and Boolean mode

func CreatePattern 

func CreatePattern(pattern string, parser string) (*Pattern, error)

first character is a operator

func ParsePattern 

func ParsePattern(pattern string, mode int64, parser string) ([]*Pattern, error)

Parse search string into list of patterns

func ParsePatternInBooleanMode 

func ParsePatternInBooleanMode(pattern string, parser string) ([]*Pattern, error)

Parse the search string in boolean mode

func ParsePatternInNLMode 

func ParsePatternInNLMode(pattern string, parser string) ([]*Pattern, error)

Parse search string in natural language mode

func ParsePhrase 

func ParsePhrase(pattern string, parser string) ([]*Pattern, error)

ParsePhrase splits a quoted-phrase body into TEXT children for a PHRASE node. With parser="gojieba" the phrase is segmented through jieba so the children match how the index stores Chinese words; otherwise the legacy whitespace split is used.

Children carry BytePos as Position so SqlPhrase's positional JOIN (kw[i].pos - kw[0].pos = ps[i].Position - ps[0].Position) is consistent with what fulltext_index_tokenize writes into the index.

func PatternOptimizeJoin 

func PatternOptimizeJoin(ps []*Pattern) []*Pattern

func (*Pattern) Combine 

func (p *Pattern) Combine(s *SearchAccum, docvec []uint8, aggcnt []int64, arg, result []float32) ([]float32, error)

Combine two score maps into single map. max(float32) will return when same doc_id (key) exists in both arg and result.

func (*Pattern) Eval 

func (p *Pattern) Eval(accum *SearchAccum, docvec []uint8, docLen int64, aggcnt []int64, weight float32, result []float32) ([]float32, error)

Eval() function to evaluate the previous result from Eval and the current pattern (with data from datasource) and return map[doc_id]float32

func (*Pattern) EvalLeaf 

func (p *Pattern) EvalLeaf(s *SearchAccum, docvec []uint8, docLen int64, aggcnt []int64, weight float32, result []float32) ([]float32, error)

Eval leaf node. compute the tfidf from the data in WordAccums and return result as map[doc_id]float32

func (*Pattern) EvalMinus 

func (p *Pattern) EvalMinus(s *SearchAccum, docvec []uint8, aggcnt []int64, arg, result []float32) ([]float32, error)

Minus operation. Remove the result when doc_id is present in argument e.g. (+ (text apple)) (- (text banana))

func (*Pattern) EvalOR 

func (p *Pattern) EvalOR(s *SearchAccum, docvec []uint8, aggcnt []int64, arg, result []float32) ([]float32, error)

OR operation. Either apple and banana can be the result e.g. (text apple) (text banana)

func (*Pattern) EvalPlusOR 

func (p *Pattern) EvalPlusOR(s *SearchAccum, docvec []uint8, aggcnt []int64, arg, result []float32) ([]float32, error)

Eval Plus OR. The previous result from Eval() is a Plus Operator and current Pattern is a Text or Star. e.g. (+ (text apple)) (text banana)

func (*Pattern) EvalPlusPlus 

func (p *Pattern) EvalPlusPlus(s *SearchAccum, docvec []uint8, aggcnt []int64, arg, result []float32) ([]float32, error)

Eval Plus Plus operation. Basically AND operation between input argument and result from the previous Eval() e.g. (+ (text apple)) (+ (text banana))

func (*Pattern) GetLeafText 

func (p *Pattern) GetLeafText(operator int) []string

Get the text of leaf nodes with operator specified. Either TEXT or STAR as operator.

func (*Pattern) GetWeight 

func (p *Pattern) GetWeight() float32

Get the weight for compute the TFIDF LESSTHAN is lower the ranking GREATERTHAN is higher the ranking RANKLESS is to discourage the ranking but not delete such as Minus. weight is negative to discourage the ranking.

func (*Pattern) String 

func (p *Pattern) String() string

Pattern

func (*Pattern) StringWithPosition 

func (p *Pattern) StringWithPosition() string

func (*Pattern) Validate 

func (p *Pattern) Validate() error

validate the Pattern

type SearchAccum 

type SearchAccum struct {
	SrcTblName string
	TblName    string
	Mode       int64
	Pattern    []*Pattern
	Params     string
	Nrow       int64
	Nkeywords  int
	AnyPlus    bool

	ScoreAlgo FullTextScoreAlgo
	AvgDocLen float64
}

Search accumulator is to parse the search string into list of pattern and each pattern will associate with WordAccum by pattern.Text

func NewSearchAccum 

func NewSearchAccum(srctbl string, tblname string, pattern string, mode int64, params string, scoreAlgo FullTextScoreAlgo) (*SearchAccum, error)

Init Search Accum

func (*SearchAccum) Eval 

func (s *SearchAccum) Eval(docvec []uint8, docLen int64, aggcnt []int64) ([]float32, error)

Evaluate the search string

func (*SearchAccum) PatternAnyPlus 

func (s *SearchAccum) PatternAnyPlus() bool

type SqlNode 

type SqlNode struct {
	Index    int32
	Label    string
	Sql      string
	IsJoin   bool
	Children []*SqlNode
}

fulltext SQL generation

For natural language mode, it is a phrase search. The search will be AND operation and with position as filter.

For boolean mode, the rule are the followings:

  • operation JOIN contains a list of + operators which is single TEXT or STAR values
  • operator + is considered as AND Operation. + Operators can be a single TEXT/STAR or Group. Operator + with children Group cannot be optimized to under JOIN.
  • operator - is not considered as AND operation because NOT filter is slow in SQL. Change the formula from A INTERSECT (NOT B) into A UNION (A INTERSECT B) and process the negative filter afterwards.
  • other operators are OR operation.

SQL generation from boolean search string is a Set theory.

e.g. search string +hello -world

the search string will convert into Pattern/Plan

((+ (TEXT hello)) (- (TEXT world)))

with set theory, we can formulae the above plans into

A\B = A - (A INTERSECT B)

Since NOT filter is slow in SQL, we change the plan into A UNION (A INTERSECT B) and process the negative later.

The result SQL will be

A UNION ALL (A JOIN B)

In case there are multiple + operators like the search string "+A +B -(<C >D)" JOIN will be used to optimize the SQL with Pattern/Plan

(((JOIN (+ (TEXT A)) (+ (TEXT B))) (- (GROUP (< (TEXT C)) (> (TEXT D)))))

With plan above, we can formula the SQL like belows

(A INTERSECT B) UNION ((A INTERSECT B) INTERSECT C) UNION ((A INTERSECT B) INTERSECT D)

WITH t0 (A JOIN B) (t0) UNION ALL (t0 JOIN C) UNION ALL (t0 JOIN D)

(A JOIN B) is the result contain both A and B. Consider it is the appearance of A and B (denoted as index 0 in SQL)). (t0 JOIN C) is the result contain both A, B and C. Consider it is the appearance of A, B and C (denoted as index 1 in SQL). (t0 JOIN D) is the result contain both A, B and D. Consider it is the appearance of A, B and D (denoted as index 2 in SQL).

For TD-IDF calculation,

Instead of caculating the TD-IDF from the words A, B, C and D, we will calculate the score from word groups (A & B) -> "0" , (A & B & C) -> "1", (A & B & D) -> "2". The ordering of the result remain the same and of course the score is different.

In case + operator with GROUP such as "+(A B) ~C -D",

Although operator + with GROUP cannot be optimized with JOIN Pattern node, we can still optimize the SQL with JOIN like below

(+ (GROUP (TEXT A) (TEXT B))) (~ (TEXT C)) (- (TEXT D)))

The generated SQL is like

A UNION B UNION (A INTERSECT C) UNION (A INTERSECT D) UNION (B INTERSECT C) UNION (B INTERSECT D)

(A) UNION ALL (B) UNION ALL (A JOIN C) UNION ALL (A JOIN D) UNION ALL (B JOIN C) UNION ALL (B JOIN D)

In case multiple + operator with single values and + operator with GROUP like "+A +B +(C D) E"

The plan like the followings: ((JOIN (+ (TEXT A)) (+ (TEXT B))) (+ (GROUP (TEXT C) (TEXT D))) (TEXT E))

To simplify the SQL, (+ (GROUP (TEXT C) (TEXT D))) will not considered as JOIN.

(A INTERSECT B) UNION (A INTERSECT B INTERSECT C) UNION (A INTEREST B INTERSECT D) UNION (A INTERSECT B INTERSECT E)

WITH t0 (A JOIN B) (t0) UNION ALL (t0 JOIN C) UNION ALL (t0 JOIN D) UNION ALL (t0 JOIN E)

IMPORTANT NOTE: In SQL, please don't use ORDER BY, UNION to generate the SQL. ORDER BY is slow and UNION will cause OOM.

func GenJoinPlusSql 

func GenJoinPlusSql(p *Pattern, mode int64, idxtbl string, parser string) ([]*SqlNode, error)

PLUS node as JOIN. Index is from TEXT/STAR node children of PLUS node can be a single TEXT/STAR or GROUP. In case of GROUP, mutiple SqlNodes will be generated.

func GenJoinSql 

func GenJoinSql(p *Pattern, mode int64, idxtbl string, parser string) ([]*SqlNode, error)

JOIN node. Index is from JOIN node. Index of TEXT/STAR is invalid Generate a JOIN subsql in children and union sql in Sql. subsql t0 is the table for all JOIN

func GenSql 

func GenSql(p *Pattern, mode int64, idxtbl string, joinsql []*SqlNode, isJoin bool, parser string) ([]*SqlNode, error)

generate the sql with the pattern. isJoin flag is true, geneate the SQL in JOIN mode (only for JOIN and PLUS node) if joinsql is not NILL, all the OR TEXT/STAR node will be joined with joinsql

Source Files

View all Source files

Directories

Path Synopsis
Package plugin is the fulltext index plugin registration point.
 Package plugin is the fulltext index plugin registration point.
compile
Package compile implements the fulltext plugin's compile-layer (DDL) hooks.
 Package compile implements the fulltext plugin's compile-layer (DDL) hooks.
idxcron
Package idxcron is fulltext's idxcron hook implementation.
 Package idxcron is fulltext's idxcron hook implementation.
iscp
Package iscp provides fulltext's ISCP hook layer.
 Package iscp provides fulltext's ISCP hook layer.
plan
Package plan implements the fulltext plugin's plan-layer hooks.
 Package plan implements the fulltext plugin's plan-layer hooks.
runtime
Package runtime holds the fulltext index's catalog-side metadata.
 Package runtime holds the fulltext index's catalog-side metadata.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.