Documentation
¶
Overview ¶
Package quality implements an offline compression-quality benchmark harness.
It runs sample prompts through tok at several tiers and reports, per tier:
- compression ratio (tokens out / tokens in; lower = more compression)
- char retention (chars out / chars in)
- a lexical-fidelity proxy (ROUGE-1 unigram recall of key terms between the original and the compressed text)
Because we cannot run a downstream QA model offline, the fidelity proxy stands in for end-task quality: it measures how many of the original's informative terms survive compression. A "baseline (no-op)" arm is emitted alongside the tok arms as a placeholder for a real LLMLingua comparison until one is wired in.
Everything here is offline: no network, no model calls.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Rouge1Recall ¶
Rouge1Recall computes ROUGE-1 recall of key terms: the fraction of the original's distinct informative unigrams that also appear in the compressed text. Recall (not F1) is the right proxy here — compression is allowed to drop tokens, and we care how much source signal survives.
func WriteMarkdown ¶
func WriteMarkdown(w io.Writer, summaries []TierSummary, results []Result) error
WriteMarkdown emits a human-readable report: a per-tier summary table followed by per-sample detail. The baseline (no-op) row is clearly labeled.
Types ¶
type Result ¶
type Result struct {
SampleID string
Tier string
InTokens int
OutTokens int
Ratio float64 // OutTokens / InTokens (lower = more compression)
CharRet float64 // len(out) / len(in)
Fidelity float64 // ROUGE-1 key-term recall in [0,1]
}
Result holds the measured metrics for one (sample, tier) pair.
type Sample ¶
type Sample struct {
ID string `json:"id"`
Category string `json:"category"`
Prompt string `json:"prompt"`
Description string `json:"description"`
}
Sample is one benchmark prompt. The shape matches benchmarks/samples.json.
func LoadSamples ¶
LoadSamples reads a samples JSON file (the benchmarks/samples.json shape).
func LongBenchSamples ¶
func LongBenchSamples() []Sample
LongBenchSamples returns a few LongBench-style prose/QA entries that exercise the prose-compression paths the shell-command samples.json does not. They are embedded so the harness stays fully offline.
type Tier ¶
Tier names a tok pipeline profile to evaluate. The "baseline (no-op)" arm is represented by a nil Option slice and passes input through unchanged.
func DefaultTiers ¶
func DefaultTiers() []Tier
DefaultTiers returns the arms evaluated by the harness: a no-op baseline (LLMLingua placeholder) plus a spread of tok tiers from light to maximal.
type TierSummary ¶
type TierSummary struct {
Tier string
AvgRatio float64
AvgCharRet float64
AvgFidelity float64
Samples int
}
TierSummary aggregates results across all samples for a single tier.
func Summarize ¶
func Summarize(results []Result, tiers []Tier) []TierSummary
Summarize aggregates per-tier averages across samples, preserving the tier ordering supplied in tiers.
Source Files
Directories