pruner

package
v0.0.0-...-8acab51 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 26, 2026 License: MIT Imports: 33 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var DefaultMemorySignals = []string{

	"prefer", "like", "dislike", "hate", "love", "favorite", "choose", "decided",
	"always", "never", "usually", "want", "need",

	"my name", "i am", "i'm", "i live", "i work", "my job", "my email",
	"birthday", "age", "family",

	"todo", "remember", "don't forget", "remind", "deadline", "schedule",
	"important", "critical", "urgent",

	"api key", "password", "token", "config", "setting", "version",
	"install", "setup", "deploy",

	"喜欢", "不喜欢", "偏好", "选择", "决定", "总是", "从不", "通常",
	"我叫", "我是", "住在", "工作", "邮箱", "生日",
	"记住", "别忘了", "提醒", "截止", "重要", "紧急",
	"密码", "密钥", "配置", "设置", "版本", "安装", "部署",
}

DefaultMemorySignals are words/phrases that indicate memorable content.

View Source
var ErrOnnxNotReady = errors.New("onnx pruner model not ready")

ErrOnnxNotReady is returned when the ONNX model is still loading. Callers should fall back to local/IR pruning or passthrough.

View Source
var Stopwords = map[string]bool{
	"a": true, "an": true, "and": true, "are": true, "as": true, "at": true,
	"be": true, "by": true, "do": true, "for": true, "from": true,
	"has": true, "have": true, "he": true, "her": true, "his": true,
	"how": true, "if": true, "in": true, "is": true, "it": true, "its": true,
	"me": true, "my": true, "no": true, "not": true, "of": true, "on": true,
	"or": true, "our": true, "she": true, "so": true, "that": true,
	"the": true, "their": true, "them": true, "then": true, "there": true,
	"these": true, "they": true, "this": true, "to": true, "up": true,
	"us": true, "was": true, "we": true, "were": true, "what": true,
	"when": true, "which": true, "who": true, "will": true, "with": true,
	"you": true, "your": true,

	"的": true, "了": true, "在": true, "是": true, "我": true, "有": true,
	"和": true, "就": true, "不": true, "人": true, "都": true, "一": true,
	"一个": true, "上": true, "也": true, "很": true, "到": true, "说": true,
	"要": true, "去": true, "你": true, "会": true, "着": true, "没有": true,
	"看": true, "好": true, "自己": true, "这": true,
}

Stopwords is a set of common English words to filter from non-code content.

Functions

func CacheKey

func CacheKey(code, query string) uint64

CacheKey computes a uint64 hash from content and query using FNV-1a. Uses unsafe string→bytes to avoid allocation for large content strings.

func CompactMarkdown

func CompactMarkdown(s string) string

CompactMarkdown strips noise from markdown text to reduce token usage:

  • Removes HTML comments (<!-- ... -->)
  • Collapses 3+ consecutive blank lines into 2 (preserves paragraph breaks)
  • Strips trailing whitespace from each line
  • Removes empty placeholder sections (heading followed only by blank lines before next heading)

func EstimateTokens

func EstimateTokens(text string) int

EstimateTokens provides a rough token count estimate. Uses ~4 chars/token for ASCII, ~1.5 chars/token for CJK. Optimized: fast path for pure ASCII, sampling for mixed content.

func IsCodeContent

func IsCodeContent(content string, minLines int) bool

IsCodeContent returns true if content appears to be source code and exceeds the minimum line threshold.

func IsPrunerDisabled

func IsPrunerDisabled(ctx context.Context) bool

IsPrunerDisabled checks whether pruning is disabled for this request context.

func MarkdownToText

func MarkdownToText(s string) string

MarkdownToText converts markdown-like content into compact plain text for prompt context:

  • Runs CompactMarkdown first
  • Removes common markdown markers (headings/lists/quotes/fences/emphasis)
  • Converts links/images to readable text
  • Collapses blank lines to at most one

func MarkdownToTextMinimal

func MarkdownToTextMinimal(s string) string

MarkdownToTextMinimal is an ultra-compact mode: converts markdown to plain text, then collapses all whitespace/newlines into single spaces.

func NormalizeBackendName

func NormalizeBackendName(_ string) string

NormalizeBackendName canonicalizes backend selection for public APIs. Pruner backend switching is hidden; only local backend is exposed.

func SignalWordScore

func SignalWordScore(text string, signals []string) float64

SignalWordScore returns a boost score (0-1) based on signal words found in text. Uses diminishing returns: 1 match = 0.5, 2 = 0.75, 3+ = ~1.0.

func SplitSentences

func SplitSentences(text string) []string

SplitSentences splits text into sentences, handling both English and Chinese. Requires whitespace after punctuation to avoid breaking emails/URLs.

func TextTokenize

func TextTokenize(text string) []string

TextTokenize splits text into lowercase tokens with stopword removal. Designed for non-code content (docs, logs, prose). Handles English and Chinese.

func UnifiedTokenize

func UnifiedTokenize(text string, ct ContentType) []string

UnifiedTokenize dispatches to the appropriate tokenizer based on content type.

func WithPruneStats

func WithPruneStats(ctx context.Context, stats *RequestPruneStats) context.Context

WithPruneStats returns a new context with pruning stats attached.

func WithPrunerDisabled

func WithPrunerDisabled(ctx context.Context, disabled bool) context.Context

WithPrunerDisabled marks whether pruning should be bypassed for this request.

Types

type APIHandler

type APIHandler struct {
	// contains filtered or unexported fields
}

APIHandler provides HTTP handlers for pruner management.

func NewAPIHandler

func NewAPIHandler(mw *Middleware, cfg *Config, mm *PrunerModelManager) *APIHandler

NewAPIHandler creates a new pruner API handler.

func (*APIHandler) CancelModelDownload

func (h *APIHandler) CancelModelDownload(c echo.Context) error

CancelModelDownload cancels the current model download. POST /api/v1/proxy/pruner/model/cancel

func (*APIHandler) GetConfig

func (h *APIHandler) GetConfig(c echo.Context) error

GetConfig returns the current pruner configuration. GET /api/v1/proxy/pruner/config

func (*APIHandler) GetModelStatus

func (h *APIHandler) GetModelStatus(c echo.Context) error

GetModelStatus returns the model download status. GET /api/v1/proxy/pruner/model/status

func (*APIHandler) GetStats

func (h *APIHandler) GetStats(c echo.Context) error

GetStats returns pruning statistics. GET /api/v1/proxy/pruner/stats

func (*APIHandler) HealthCheck

func (h *APIHandler) HealthCheck(c echo.Context) error

HealthCheck checks if the pruner backend is available. GET /api/v1/proxy/pruner/health

func (*APIHandler) RegisterRoutes

func (h *APIHandler) RegisterRoutes(g *echo.Group)

RegisterRoutes registers pruner API routes.

func (*APIHandler) SetMiddleware

func (h *APIHandler) SetMiddleware(mw *Middleware)

SetMiddleware replaces the middleware reference (used when restoring saved state).

func (*APIHandler) SetOnBackendChange

func (h *APIHandler) SetOnBackendChange(fn func(backend string))

SetOnBackendChange sets a callback invoked when the backend changes.

func (*APIHandler) SetOnMiddlewareCreated

func (h *APIHandler) SetOnMiddlewareCreated(fn func(mw *Middleware))

SetOnMiddlewareCreated sets a callback invoked when the middleware is lazily created (e.g. when the user enables pruning for the first time via API).

func (*APIHandler) SetOnToggle

func (h *APIHandler) SetOnToggle(fn func(enabled bool))

SetOnToggle sets a callback invoked when the enabled state changes.

func (*APIHandler) StartModelDownload

func (h *APIHandler) StartModelDownload(c echo.Context) error

StartModelDownload starts downloading the ONNX pruner model. POST /api/v1/proxy/pruner/model/download

func (*APIHandler) UpdateConfig

func (h *APIHandler) UpdateConfig(c echo.Context) error

UpdateConfig updates the pruner configuration (partial update). PUT /api/v1/proxy/pruner/config

type BM25Backend

type BM25Backend struct {
	// contains filtered or unexported fields
}

BM25Backend implements Backend using BM25 scoring with segment indexing.

func NewBM25Backend

func NewBM25Backend(cfg Config) *BM25Backend

NewBM25Backend creates a new BM25-based pruning backend.

func (*BM25Backend) Close

func (b *BM25Backend) Close() error

Close is a no-op for the BM25 backend.

func (*BM25Backend) Health

func (b *BM25Backend) Health(_ context.Context) error

Health always returns nil for the BM25 backend.

func (*BM25Backend) Prune

func (b *BM25Backend) Prune(ctx context.Context, req PruneRequest) (*PruneResponse, error)

Prune implements Backend using the BM25 + segment pipeline.

type BM25Scorer

type BM25Scorer struct {
	K1 float64 // term frequency saturation (default 1.2)
	B  float64 // length normalization (default 0.75)
}

BM25Scorer implements Scorer using Okapi BM25.

func NewBM25Scorer

func NewBM25Scorer(k1, b float64) *BM25Scorer

NewBM25Scorer creates a BM25 scorer with the given parameters.

func (*BM25Scorer) Score

func (s *BM25Scorer) Score(query string, segments []Segment) []ScoredSegment

Score computes BM25 relevance scores for each segment against the query.

type BPETokenizer

type BPETokenizer struct {
	// contains filtered or unexported fields
}

BPETokenizer implements byte-level BPE tokenization compatible with HuggingFace tokenizers. It loads vocab.json + merges.txt (Qwen3/GPT-style BPE).

func LoadBPETokenizer

func LoadBPETokenizer(vocabPath, mergesPath string) (*BPETokenizer, error)

LoadBPETokenizer loads a BPE tokenizer from vocab.json and merges.txt files.

func (*BPETokenizer) BuildPrunerInput

func (t *BPETokenizer) BuildPrunerInput(query, code string, maxLength int) (inputIDs, attentionMask []int64, codeStart, codeEnd int)

BuildPrunerInput constructs input tensors for the pruner model. Returns input_ids, attention_mask, and the start/end offsets of code tokens.

func (*BPETokenizer) Encode

func (t *BPETokenizer) Encode(text string) []int

Encode tokenizes text into token IDs.

func (*BPETokenizer) PadID

func (t *BPETokenizer) PadID() int

PadID returns the padding token ID.

func (*BPETokenizer) VocabSize

func (t *BPETokenizer) VocabSize() int

VocabSize returns the vocabulary size.

type Backend

type Backend interface {
	Prune(ctx context.Context, req PruneRequest) (*PruneResponse, error)
	Health(ctx context.Context) error
	Close() error
}

Backend defines the inference backend interface for context pruning.

func NewBackend

func NewBackend(cfg Config) (Backend, error)

NewBackend creates a Backend based on the config.

type Config

type Config struct {
	Enabled       bool    `yaml:"enabled"`
	Backend       string  `yaml:"backend"`    // public: "local" only; internal tests may still use bm25/ir
	RemoteURL     string  `yaml:"remote_url"` // deprecated (SWE remote backend removed)
	ModelDir      string  `yaml:"model_dir"`  // deprecated (ONNX cross-encoder removed)
	Threshold     float64 `yaml:"threshold"`
	MinLines      int     `yaml:"min_lines"`
	TimeoutMs     int     `yaml:"timeout_ms"`
	CacheCapacity int     `yaml:"cache_capacity"` // LRU cache slots (default 256)
}

Config holds pruner configuration.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns the default pruner configuration (enabled by default).

type ContentType

type ContentType int

ContentType identifies the type of content being pruned.

const (
	ContentCode    ContentType = iota // Source code (Go, Python, JS, etc.)
	ContentDoc                        // Documentation, markdown, prose
	ContentLog                        // Log output, structured text
	ContentData                       // JSON, YAML, structured data
	ContentUnknown                    // Fallback — treated as non-code
)

func DetectContentType

func DetectContentType(content string, minLines int) ContentType

DetectContentType classifies content into code, documentation, logs, data, or unknown. It samples only the first 100 lines for keyword/pattern detection to avoid scanning the entire content (which can be very large).

type DiskCache

type DiskCache struct {
	// contains filtered or unexported fields
}

DiskCache provides SQLite-backed cache persistence with TTL expiration. Reuses the same SQLite driver (mattn/go-sqlite3) as the proxy's CCCache.

func NewDiskCache

func NewDiskCache(dbPath string, ttl time.Duration) *DiskCache

NewDiskCache creates a SQLite-backed disk cache at the given path. If dbPath is empty or the database cannot be opened, returns a no-op cache.

func (*DiskCache) Cleanup

func (d *DiskCache) Cleanup() int64

Cleanup removes expired entries.

func (*DiskCache) Close

func (d *DiskCache) Close() error

Close closes the database connection.

func (*DiskCache) Get

func (d *DiskCache) Get(key string) ([]ScoredSegment, bool)

Get retrieves scored segments from disk cache.

func (*DiskCache) Put

func (d *DiskCache) Put(key string, segments []ScoredSegment)

Put stores scored segments to disk cache.

type HybridBackend

type HybridBackend struct {
	// contains filtered or unexported fields
}

HybridBackend runs local and neural backends in parallel, picking the better result.

func NewHybridBackend

func NewHybridBackend(local, neural Backend, cfg Config) *HybridBackend

NewHybridBackend creates a hybrid backend that runs both backends concurrently.

func (*HybridBackend) Close

func (h *HybridBackend) Close() error

Close closes both backends.

func (*HybridBackend) Health

func (h *HybridBackend) Health(ctx context.Context) error

Health returns healthy if either backend is healthy.

func (*HybridBackend) Prune

Prune runs both backends in parallel and returns the result with better compression.

type IRPruner

type IRPruner struct {
	// contains filtered or unexported fields
}

IRPruner implements Backend using BM25 scoring with unified segmentation. Handles both code and non-code content automatically.

func NewIRPruner

func NewIRPruner(cfg Config) *IRPruner

NewIRPruner creates a new IR-based pruning backend.

func (*IRPruner) Close

func (p *IRPruner) Close() error

Close is a no-op for the IR pruner.

func (*IRPruner) Health

func (p *IRPruner) Health(_ context.Context) error

Health always returns nil for the IR pruner.

func (*IRPruner) Prune

func (p *IRPruner) Prune(ctx context.Context, req PruneRequest) (*PruneResponse, error)

Prune implements Backend. It detects content type, segments, scores, and prunes.

type LRUScoreCache

type LRUScoreCache struct {
	// contains filtered or unexported fields
}

LRUScoreCache wraps ecache2 for scored segment caching. Uses uint64 keys with FNV hash — ecache2 does direct value sharding (no rehash).

func NewScoreCache

func NewScoreCache(capacity int) *LRUScoreCache

NewScoreCache creates an LRU cache with the given capacity.

func (*LRUScoreCache) Get

func (c *LRUScoreCache) Get(key uint64) ([]ScoredSegment, bool)

Get retrieves scored segments from the cache.

func (*LRUScoreCache) Put

func (c *LRUScoreCache) Put(key uint64, segments []ScoredSegment)

Put stores scored segments in the cache.

type LocalBackend

type LocalBackend struct {
	// contains filtered or unexported fields
}

LocalBackend implements Backend using a pure-Go scoring algorithm. Inspired by SWE-Pruner's line-level approach, it uses TF-IDF relevance scoring combined with structural heuristics to identify and prune low-relevance code lines without requiring a neural model.

Reference: https://arxiv.org/abs/2601.16746

func NewLocalBackend

func NewLocalBackend(cfg Config) *LocalBackend

NewLocalBackend creates a new Go-native pruning backend.

func (*LocalBackend) Close

func (b *LocalBackend) Close() error

Close is a no-op for the local backend.

func (*LocalBackend) Health

func (b *LocalBackend) Health(_ context.Context) error

Health always returns nil for the local backend.

func (*LocalBackend) Prune

Prune scores each line and removes low-relevance lines.

type Middleware

type Middleware struct {
	// contains filtered or unexported fields
}

Middleware intercepts proxy requests and prunes code content in tool messages.

func NewMiddleware

func NewMiddleware(backend Backend, cfg Config, stats *Stats) *Middleware

NewMiddleware creates a new pruner middleware.

func (*Middleware) Enabled

func (m *Middleware) Enabled() bool

Enabled returns whether the middleware is active.

func (*Middleware) GetStats

func (m *Middleware) GetStats() *Stats

GetStats returns the current pruning statistics.

func (*Middleware) ProcessRequest

func (m *Middleware) ProcessRequest(ctx context.Context, body []byte) ([]byte, error)

ProcessRequest scans the request body for tool messages containing code, prunes them, and returns the modified body. Returns the original body unchanged if no pruning was applied or on any error. If a *RequestPruneStats is attached to ctx via WithPruneStats, it will be populated.

func (*Middleware) SetBackend

func (m *Middleware) SetBackend(b Backend) Backend

SetBackend swaps the pruning backend at runtime.

func (*Middleware) SetEnabled

func (m *Middleware) SetEnabled(enabled bool)

SetEnabled toggles the middleware on or off at runtime.

type ModelDownloadProgress

type ModelDownloadProgress struct {
	File       string  `json:"file"`
	FileIndex  int     `json:"file_index"`
	TotalFiles int     `json:"total_files"`
	Downloaded int64   `json:"downloaded"`
	Total      int64   `json:"total"`
	Percentage float64 `json:"percentage"`
	SpeedHuman string  `json:"speed_human"`
	ETA        string  `json:"eta"`
}

ModelDownloadProgress tracks download progress.

type OnnxBackend

type OnnxBackend struct {
	// contains filtered or unexported fields
}

OnnxBackend implements Backend using ONNX Runtime for local neural pruning. Model loading is async — Prune() returns ErrOnnxNotReady until loaded.

func NewOnnxBackend

func NewOnnxBackend(cfg Config, modelDir string) (*OnnxBackend, error)

NewOnnxBackend creates a new ONNX-based pruning backend. Model loading is deferred — call StartAsync() or it auto-starts on first Prune(). modelDir should contain: model.onnx, vocab.json, merges.txt

func (*OnnxBackend) Close

func (b *OnnxBackend) Close() error

Close releases ONNX runtime resources.

func (*OnnxBackend) Health

func (b *OnnxBackend) Health(_ context.Context) error

Health checks if the ONNX session is loaded.

func (*OnnxBackend) IsReady

func (b *OnnxBackend) IsReady() bool

IsReady returns true if the ONNX model is loaded and ready for inference.

func (*OnnxBackend) Prune

func (b *OnnxBackend) Prune(ctx context.Context, req PruneRequest) (*PruneResponse, error)

Prune implements Backend using ONNX neural inference. Returns ErrOnnxNotReady if the model is still loading.

func (*OnnxBackend) StartAsync

func (b *OnnxBackend) StartAsync()

StartAsync begins model loading in the background.

type PruneRequest

type PruneRequest struct {
	Code        string      `json:"code"`    // Backward compat alias
	Content     string      `json:"content"` // Preferred: raw content (code or text)
	Query       string      `json:"query,omitempty"`
	Threshold   float64     `json:"threshold,omitempty"`
	ContentType ContentType `json:"content_type,omitempty"` // Hint (auto-detected if 0)
}

PruneRequest is the input to the pruning engine.

func (PruneRequest) GetContent

func (r PruneRequest) GetContent() string

GetContent returns the content to prune, preferring Content over Code for backward compat.

type PruneResponse

type PruneResponse struct {
	PrunedCode      string      `json:"pruned_code"`
	PrunedContent   string      `json:"pruned_content"`
	ContentType     ContentType `json:"content_type"`
	Score           float64     `json:"score"`
	OriginalLines   int         `json:"original_lines"`
	KeptLines       int         `json:"kept_lines"`
	PrunedLines     int         `json:"pruned_lines"`
	OriginalTokens  int         `json:"original_tokens"`
	PrunedTokens    int         `json:"pruned_tokens"`
	CompressionRate float64     `json:"compression_rate"`
	LatencyMs       float64     `json:"latency_ms"`
}

PruneResponse is the output from the pruning engine.

type PrunerFileStatus

type PrunerFileStatus struct {
	Filename   string `json:"filename"`
	Downloaded bool   `json:"downloaded"`
	Size       string `json:"size"`
}

PrunerFileStatus shows per-file download status.

type PrunerModelInfo

type PrunerModelInfo struct {
	Filename string   `json:"filename"`
	URL      string   `json:"url"`
	Mirrors  []string `json:"-"` // fallback URLs (hf-mirror, modelscope, etc.)
	Size     string   `json:"size"`
}

PrunerModelInfo describes a downloadable pruner model file.

type PrunerModelManager

type PrunerModelManager struct {
	// contains filtered or unexported fields
}

PrunerModelManager manages pruner model downloads via the unified downloader.

func NewPrunerModelManager

func NewPrunerModelManager(modelDir string) *PrunerModelManager

NewPrunerModelManager creates a new model manager.

func (*PrunerModelManager) CancelDownload

func (m *PrunerModelManager) CancelDownload()

CancelDownload cancels the current download.

func (*PrunerModelManager) Download

func (m *PrunerModelManager) Download(ctx context.Context) error

Download starts downloading all model files via the unified downloader.

func (*PrunerModelManager) GetStatus

func (m *PrunerModelManager) GetStatus() PrunerModelStatus

GetStatus returns the current model status.

func (*PrunerModelManager) IsReady

func (m *PrunerModelManager) IsReady() bool

IsReady returns true if all model files are downloaded.

func (*PrunerModelManager) ModelDir

func (m *PrunerModelManager) ModelDir() string

ModelDir returns the model directory path.

type PrunerModelStatus

type PrunerModelStatus struct {
	Ready       bool                   `json:"ready"`
	Downloading bool                   `json:"downloading"`
	State       string                 `json:"state,omitempty"`
	Error       string                 `json:"error,omitempty"`
	Progress    *ModelDownloadProgress `json:"progress,omitempty"`
	Files       []PrunerFileStatus     `json:"files,omitempty"`
}

PrunerModelStatus represents the model state.

type RemoteBackend

type RemoteBackend struct {
	// contains filtered or unexported fields
}

RemoteBackend calls an external SWE-Pruner FastAPI service.

func NewRemoteBackend

func NewRemoteBackend(baseURL string, client *http.Client) *RemoteBackend

NewRemoteBackend creates a new remote backend.

func (*RemoteBackend) Close

func (r *RemoteBackend) Close() error

Close releases resources (no-op for remote backend).

func (*RemoteBackend) Health

func (r *RemoteBackend) Health(ctx context.Context) error

Health checks if the remote SWE-Pruner service is available.

func (*RemoteBackend) Prune

Prune sends code to the remote SWE-Pruner service and returns pruned output.

type RequestPruneStats

type RequestPruneStats struct {
	Pruned         bool `json:"pruned"`
	MessagesPruned int  `json:"messages_pruned"`
	TokensBefore   int  `json:"tokens_before"`
	TokensAfter    int  `json:"tokens_after"`
}

RequestPruneStats holds per-request pruning statistics.

func GetPruneStats

func GetPruneStats(ctx context.Context) *RequestPruneStats

GetPruneStats retrieves per-request pruning stats from context, or nil.

type ScoredSegment

type ScoredSegment struct {
	Segment Segment
	Score   float64
}

ScoredSegment pairs a segment with its relevance score.

func BoostScores

func BoostScores(scored []ScoredSegment) []ScoredSegment

BoostScores applies rule-based weight boosting to scored segments. Structural segments (functions, classes) get a multiplier boost. Import/include segments get an additional boost.

func BoostScoresWithQuery

func BoostScoresWithQuery(scored []ScoredSegment, query string) []ScoredSegment

BoostScoresWithQuery applies rule-based weight boosting with query-aware name matching. Segments whose name matches query terms get a significant boost.

func SelectTopK

func SelectTopK(scored []ScoredSegment, tokenBudget int) []ScoredSegment

SelectTopK selects the highest-scoring segments that fit within the token budget. Segments are returned sorted by their original position (StartLine).

type Scorer

type Scorer interface {
	Score(query string, segments []Segment) []ScoredSegment
}

Scorer is the pluggable scoring interface.

type Segment

type Segment struct {
	StartLine  int
	EndLine    int
	Kind       SegmentKind
	Name       string
	NameTokens []string // pre-tokenized segment name
	Content    string
	Tokens     []string // pre-tokenized content
	TokenCount int      // pre-computed token estimate (0 = not set)
}

Segment represents a code unit (function, class, block, or line range).

func AutoSegmentize

func AutoSegmentize(content string, ct ContentType) []Segment

AutoSegmentize dispatches to the correct segmenter based on content type.

func NewSegment

func NewSegment(startLine, endLine int, kind SegmentKind, name, content string, tokens []string) Segment

NewSegment creates a Segment with pre-computed token count.

func Segmentize

func Segmentize(code string) []Segment

Segmentize splits code into semantic segments (functions, classes, blocks). It detects boundaries using regex patterns for multiple languages, then groups remaining lines into SegmentLines blocks.

func SegmentizeData

func SegmentizeData(text string) []Segment

SegmentizeData splits JSON/YAML content into top-level key segments. For JSON objects, each top-level key becomes a segment.

func SegmentizeLogs

func SegmentizeLogs(text string) []Segment

SegmentizeLogs splits log output into groups separated by blank lines.

func SegmentizeParagraphs

func SegmentizeParagraphs(text string) []Segment

SegmentizeParagraphs splits non-code text into heading and paragraph segments. Headings are lines starting with # (markdown). Paragraphs are separated by blank lines.

type SegmentKind

type SegmentKind int

SegmentKind identifies the type of segment.

const (
	// Code segments
	SegmentFunction SegmentKind = iota
	SegmentClass
	SegmentBlock
	SegmentLines

	// Non-code segments
	SegmentParagraph
	SegmentHeading
	SegmentSentence
	SegmentLogGroup
	SegmentDataKey
)

type Stats

type Stats struct {
	// contains filtered or unexported fields
}

Stats tracks cumulative pruning metrics (thread-safe).

func NewStats

func NewStats() *Stats

NewStats creates a new Stats instance.

func (*Stats) Record

func (s *Stats) Record(r *PruneResponse)

Record adds a single prune result to the cumulative stats.

func (*Stats) RecordPassthrough

func (s *Stats) RecordPassthrough()

RecordPassthrough records a request that was not pruned.

func (*Stats) Snapshot

func (s *Stats) Snapshot() StatsSnapshot

Snapshot returns a point-in-time copy of the pruning stats.

type StatsSnapshot

type StatsSnapshot struct {
	TotalRequests       int64   `json:"total_requests"`
	PrunedRequests      int64   `json:"pruned_requests"`
	PassthroughRequests int64   `json:"passthrough_requests"`
	TotalTokensBefore   int64   `json:"total_tokens_before"`
	TotalTokensAfter    int64   `json:"total_tokens_after"`
	TokensSaved         int64   `json:"tokens_saved"`
	AvgCompressionRate  float64 `json:"avg_compression_rate"`
	AvgLatencyMs        float64 `json:"avg_latency_ms"`
}

Snapshot returns a point-in-time copy of the stats.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL