text

func (*TextEngine) ChunkerFor ¶

func (e *TextEngine) ChunkerFor(fileSize int64) chunker.Chunker

ChunkerFor selects the chunking strategy for a text file. Files smaller than textWholeFileThreshold return nil, signalling the caller to store the whole file as a single chunk. Larger files use FastCDC with the text-tiered defaults (4/8/16 KB — tuned for source code and structured text).

func (*TextEngine) DetectByExtension ¶

func (e *TextEngine) DetectByExtension(path string) bool

DetectByExtension checks if the path's extension or basename is a known text type.

func (*TextEngine) DetectByHeuristic ¶

func (e *TextEngine) DetectByHeuristic(path string, header []byte) bool

DetectByHeuristic is the last-resort content sniffing used for extensionless or unknown-extension files. A header is treated as text only when ALL of the following hold:

• it is non-empty and at least 4 bytes long,
• it does not start with a known image/video magic signature,
• it contains no NUL bytes (0x00),
• its control-byte ratio is at most 10%.

Control bytes are 0x01-0x1F (excluding \t, \n, \r) and 0x7F (DEL). High bytes (0x80-0xFF) are NOT counted as control bytes because valid UTF-8 text legitimately contains them. The 10% threshold catches raw binary data that happens to omit 0x00 (e.g. byte sequences 1..255, which are ~11% control bytes) while allowing text with occasional control characters.

Headers shorter than 4 bytes are too short for reliable heuristic detection (a single 0xFF byte would pass the control-ratio check because 0x80-0xFF are not counted as control bytes). Let the fallback binary engine handle them instead.

func (*TextEngine) DetectByMagic ¶

func (e *TextEngine) DetectByMagic(header []byte) bool

DetectByMagic checks file header signatures. Plain text has no unified magic byte signature, but files beginning with a Byte Order Mark (BOM) — UTF-8 (EF BB BF), UTF-16 big-endian (FE FF), or UTF-16 little-endian (FF FE) — are unambiguously text.

func (*TextEngine) Diff ¶

func (e *TextEngine) Diff(ctx context.Context, oldPath string, oldReader io.Reader, newPath string, newReader io.Reader) (string, error)

Diff produces a unified diff between two text files using Myers diff. Content is read streaming from oldReader/newReader via a line scanner; the Myers algorithm still requires both line arrays in memory, but the file bytes are never buffered whole. The context is threaded into the Myers/Hirschberg loops so a cancelled context aborts the diff promptly.

Inputs exceeding maxDiffLines fall back to a simple "files differ" message to avoid the O(N*M) Myers DP, which would be prohibitively slow for very large files. Identical content is short-circuited before the threshold check so that large but unchanged files produce no diff.

func (*TextEngine) Metadata ¶

func (e *TextEngine) Metadata() *core.FileMetadata

Metadata returns the file metadata for text files.

func (*TextEngine) Name ¶

func (e *TextEngine) Name() string

Name returns "text".

func (*TextEngine) Preview ¶

func (e *TextEngine) Preview(header []byte, size int64, reader io.Reader, maxLines int) (string, error)

Preview returns the first maxLines of text content, read streaming from reader. Only the requested number of lines are consumed, so previewing a huge text file does not load it whole.