Documentation
¶
Index ¶
Constants ¶
const (
MAX_TOKEN_SIZE = 23
)
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type JiebaTokenizer ¶
type JiebaTokenizer struct {
// contains filtered or unexported fields
}
func NewJiebaTokenizer ¶
func NewJiebaTokenizer(useHmm bool) (*JiebaTokenizer, error)
NewJiebaTokenizer constructs a JiebaTokenizer backed by gojieba. When useHmm is true the HMM model is used to discover unknown words during segmentation; when false only dictionary-based segmentation is performed.
Returns an error if dictionary files are missing or gojieba initialization fails.
The returned tokenizer holds C resources that are released either when Free is called explicitly or when the value is garbage collected.
func SharedJiebaTokenizer ¶
func SharedJiebaTokenizer(useHmm bool) (*JiebaTokenizer, error)
SharedJiebaTokenizer returns a process-wide JiebaTokenizer. Two singletons are maintained — one with HMM enabled and one without — and each is loaded lazily on first use (~1s for dictionary loading). The returned tokenizer is safe for concurrent Tokenize calls and must not be Free'd.
If dictionary files are missing or gojieba initialization otherwise fails, the error is cached and returned on every subsequent call for the same useHmm value — initialization is not retried.
Choose useHmm by intent:
- false at index build time: dictionary-only segmentation gives stable, reproducible tokens that don't drift across deployments.
- true at query time: HMM new-word discovery broadens recall for terms not in the dictionary.
func (*JiebaTokenizer) Free ¶
func (t *JiebaTokenizer) Free()
Free releases the underlying gojieba resources. Subsequent calls are no-ops. Free is a no-op for the shared tokenizer returned by SharedJiebaTokenizer.
type SimpleTokenizer ¶
type SimpleTokenizer struct{}
SimpleTokenizer holds no per-call state; concurrent Tokenize calls on the same instance are safe.
func NewSimpleTokenizer ¶
func NewSimpleTokenizer() *SimpleTokenizer
type Tokenizer ¶
Tokenizer yields a sequence of (Token, error) pairs. Implementations may report errors before any token is emitted (e.g. input validation), or mid-stream for tokenizers that can fail partway through the input. Callers must check err on each iteration and stop on the first non-nil err.
Source Files