sanitize

package

v1.3.2 Latest Latest Go to latest Published: Mar 20, 2026 License: MIT Imports: 2 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/thebtf/engram

Links

Open Source Insights

Documentation ¶

Overview ¶

Package sanitize provides text sanitization for session backfill processing.

Index ¶

Constants
Variables
func Text(text string) string
func TextWithLimit(text string, maxBlockSize int) string

Constants ¶

View Source

const DefaultMaxBlockSize = 3000

DefaultMaxBlockSize is the maximum size of a single text block before truncation.

Variables ¶

View Source

var (
	// Base64Regex matches large base64/hex payloads (500+ chars).
	Base64Regex = regexp.MustCompile(`(?m)[A-Za-z0-9+/=]{500,}`)

	// SystemReminderRegex matches <system-reminder>...</system-reminder> blocks.
	SystemReminderRegex = regexp.MustCompile(`(?s)<system-reminder>.*?</system-reminder>`)
)

Patterns for content that should be stripped before LLM processing.

Functions ¶

func Text ¶

func Text(text string) string

Text sanitizes a text block by removing system reminders, base64 payloads, and truncating overly long blocks.

func TextWithLimit ¶

func TextWithLimit(text string, maxBlockSize int) string

TextWithLimit sanitizes text with a custom truncation limit.

Types ¶

This section is empty.

Source Files ¶

View all Source files

sanitize.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL