scanner

package module

v0.0.1 Latest Latest Go to latest Published: May 2, 2025 License: MIT Imports: 6 Imported by: 0

Details

Text Scanners compatible with bufio.Scanner.

View Source

const EndOfSentence = ".!?"

Default end of sentence

This section is empty.

func NewSentencer(eos string, r io.Reader) *bufio.Scanner

Create a scanner that slices input stream by end of sentence

func NewSlicer(delim string, r io.Reader) *bufio.Scanner

Create a scanner that slices input stream by fixed delimiter

type Chunker struct {
	Scanner
	// contains filtered or unexported fields
}

func NewChunker(size int, s Scanner) *Chunker

func (s *Chunker) Scan() bool

func (s *Chunker) Text() string

type Identity struct {
	io.Reader
	// contains filtered or unexported fields
}

func NewIdentity(r io.Reader) *Identity

func (r *Identity) Err() error

func (r *Identity) Scan() bool

func (r *Identity) Text() string

type Scanner interface {
	Scan() bool
	Text() string
	Err() error
}

Scanner is an interface similar to bufio.Scanner. It defines core functionality defined by this library.

type Sentencer []byte

func (s Sentencer) Split(data []byte, atEOF bool) (advance int, token []byte, err error)

type Slicer []byte

func (s Slicer) Split(data []byte, atEOF bool) (advance int, token []byte, err error)

bufio.SplitFunc for fixed delimiter.