Documentation
¶
Overview ¶
Package safetext converts raw process output bytes into model-safe text (spec "docs/specs/long-running-command-supervision.md", "Output capture and storage": "Model-visible text passes through safe-text normalization: invalid UTF-8 is replaced deterministically; disallowed terminal control sequences are escaped or removed; binary detection is reported; normalization is reported").
This package is pure text processing: it has no knowledge of processes, owners, handles, cursors, or storage, and it must stay that way -- the process package (process/render.go) is the only place that wires this package's output to a process's identity and cursor-addressed reads. safetext is stdlib-only.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func LooksBinary ¶
LooksBinary reports whether data looks like binary data rather than text, so a caller (process/render.go) can route it through base64 instead of inline safe text. This is a heuristic, not a proof: it combines a NUL-byte density check with a Shannon-entropy check over the byte-value distribution (task text: "NUL-byte density and/or Shannon-entropy-style byte-distribution check").
func Truncate ¶
Truncate trims data to at most limit bytes without splitting a multi-byte UTF-8 sequence in half. data is assumed to already be valid UTF-8 (as Normalize's own output always is), so any RuneError observed while backing off from the cut point can only mean the cut fell inside a multi-byte sequence, never a genuinely invalid byte -- backing off by at most utf8.UTFMax-1 bytes always reaches a valid boundary. A non-positive limit returns data unchanged (no cap requested); limit == 0 returns an empty slice.
Types ¶
type Normalizer ¶
type Normalizer struct {
// contains filtered or unexported fields
}
Normalizer incrementally converts a raw byte stream, arriving in arbitrary chunks, into model-safe text. It:
- replaces invalid UTF-8 byte sequences deterministically with the standard U+FFFD replacement character;
- strips C0 control bytes (0x00-0x1F) and C1 control codepoints (U+0080-U+009F, however encoded) except approved whitespace (space, tab, newline, and carriage return -- see the isApprovedWhitespace doc comment for why \r is included);
- recognizes and removes ANSI/terminal CSI (`ESC [ ... final-byte`), OSC (`ESC ] ... BEL` or `ESC ] ... ESC \`), and DCS (`ESC P ... ESC \`) sequences as complete units, even when a sequence is split across two separate Normalize calls.
The zero Normalizer is ready to use. A Normalizer carries state between calls (in-progress escape sequence, a truncated trailing UTF-8 byte sequence) and assumes its Normalize calls are fed strictly in stream order with no gaps or rewinds; it is NOT safe for concurrent use, and re-feeding an overlapping or out-of-order chunk produces undefined (though never unsafe -- no raw control byte can leak) results. Each distinct byte stream (e.g. one process's combined output) needs its own Normalizer.
func (*Normalizer) Normalize ¶
func (n *Normalizer) Normalize(chunk []byte) []byte
Normalize consumes chunk -- the next slice of a byte stream, in stream order -- and returns the model-safe text produced from it. Bytes that belong to an in-progress escape sequence or a truncated UTF-8 rune carried over from a previous call (or held back for a future one) are never included in the returned slice's bytes as raw/partial data; they are either completed and dropped (escape sequences are removed entirely) or reconstructed and decoded (a completed multi-byte rune) on a later call.
Source Files