Help us shape the future of Go! Take the 2026 Go Developer Survey

chunk

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source
const (
	DefaultChunkLines             = 600
	DefaultChunkOverlap           = 20
	DefaultChunkMinLines          = 1500
	DefaultChunkTokenThreshold    = 30000
	DefaultChunkConcurrency       = 3
	DefaultSynthesisLineThreshold = 240
)

Chunking is a latency tool for large specs, not the normal path: a single review sees the whole spec at once and costs one call. The defaults keep auto mode off for anything a single call handles comfortably.

View Source
const TagSynthesis = "synthesis"

Variables

This section is empty.

Functions

func BuildChunkTask

func BuildChunkTask(ch Chunk, lineCount int, withLines bool) (string, error)

BuildChunkTask returns the task for reviewing one chunk. It follows the shared prefix built by llm.BuildSpecPrefix.

Normally that prefix holds the whole numbered spec: the reviewer reads every section, so a term defined elsewhere or a rule stated in another section is not mistaken for a gap, and reports only on the chunk's own lines. With withLines set the spec was too large to share and the prefix leaves it out, so the task carries the chunk's own numbered lines instead.

func BuildSynthesisTask

func BuildSynthesisTask(input SynthesisInput) (string, error)

BuildSynthesisTask returns the task for the cross-section pass. It follows the shared prefix, so synthesis reads the whole spec and the preflight findings there; the task lists what the chunk reviews found, one line each.

func EstimateTokens

func EstimateTokens(text string) int

func ParseChunkResponse

func ParseChunkResponse(raw string, lineCount int, ch Chunk) (*schema.Report, error)

ParseChunkResponse reads a complete chunk review response. See parseChunkResponse for what is kept and what is dropped.

func ParseSynthesisResponse

func ParseSynthesisResponse(raw string, lineCount int) (*schema.Report, error)

ParseSynthesisResponse reads a complete synthesis response.

func ShouldChunk

func ShouldChunk(lineCount, estimatedTokens int, cfg Config) bool

ShouldChunk reports whether a spec should be reviewed in chunks. estimatedTokens is the estimated size of the spec alone: context files and prompt instructions do not make a spec any harder to review in one call.

func ShouldRunSynthesis

func ShouldRunSynthesis(chunkingEnabled bool, lineCount int, findingCount int, threshold int) bool

func ValidateConfig

func ValidateConfig(cfg Config) error

Types

type Chunk

type Chunk struct {
	ID          string
	Path        string
	LineStart   int
	LineEnd     int
	ContextFrom int
	ContextTo   int
	HeadingPath []string
	Numbered    string
}

type ChunkResult

type ChunkResult struct {
	Chunk  Chunk
	Report *schema.Report
	Model  string
}

func ReviewChunks

func ReviewChunks(ctx context.Context, provider llm.Provider, s *spec.Spec, plan Plan, cfg ExecutorConfig) ([]ChunkResult, error)

type Config

type Config struct {
	Mode                   Mode
	ChunkLines             int
	ChunkOverlap           int
	ChunkMinLines          int
	ChunkTokenThreshold    int
	ChunkConcurrency       int
	SynthesisLineThreshold int
}

func WithDefaults

func WithDefaults(cfg Config) Config

type ExecutorConfig

type ExecutorConfig struct {
	SystemPrompt string
	// Prefix is the shared user-message prefix from llm.BuildSpecPrefix. Every
	// chunk call sends it unchanged, so it is cached after the first.
	Prefix string
	// LongCache is passed through to llm.Request.LongCache.
	LongCache bool
	// WithLines puts each chunk's numbered lines in its task, for a Prefix
	// that leaves the spec out because it is too large to share.
	WithLines bool
	MaxTokens int
	Effort    string
	// EnforceSchema asks the provider to constrain responses to the review
	// schema.
	EnforceSchema bool
	// NoPatches leaves patches out of the requested output.
	NoPatches   bool
	Concurrency int
	Verbose     bool
	ErrWriter   io.Writer
}

type Heading

type Heading struct {
	Level int
	Text  string
	Line  int
}

func ExtractHeadings

func ExtractHeadings(lines []string) []Heading

type MergeGroup

type MergeGroup struct {
	IssueIDs []string `json:"issue_ids"`
	Reason   string   `json:"reason"`
}

MergeGroup names chunk findings that report the same defect.

type MergeInput

type MergeInput struct {
	ChunkResults []ChunkResult
	Preflight    []schema.Issue
	OriginalSpec string
}

type MergeResult

type MergeResult struct {
	Issues    []schema.Issue
	Questions []schema.Question
	Patches   []schema.Patch
	// DroppedFindings counts findings the chunk and synthesis parsers left
	// out as invalid.
	DroppedFindings int
}

func ApplySynthesis

func ApplySynthesis(merged MergeResult, synthesis *SynthesisResult, specText, specPath string) (MergeResult, *schema.SynthesisMeta)

ApplySynthesis folds the result of the cross-section pass into merged:

  • A retraction removes a chunk finding when its quote is found in the spec. Preflight findings are never retracted. Every retraction applied is recorded with the spec text that justified it.
  • Each merge group keeps the first of its findings, which is the most severe, and folds the others into it.
  • New findings are added, deduplicated against the rest.

Findings are then renumbered and patches re-pointed at the new numbers; a patch for a removed finding is dropped. specText is the spec the model was shown and specPath its evidence path.

func MergeReports

func MergeReports(input MergeInput) MergeResult

type Mode

type Mode string
const (
	ModeAuto Mode = "auto"
	ModeOn   Mode = "on"
	ModeOff  Mode = "off"
)

type Plan

type Plan struct {
	Chunks    []Chunk
	Headings  []Heading
	Sections  []Section
	LineCount int
}

func PlanSpec

func PlanSpec(s *spec.Spec, cfg Config) (Plan, error)

type Retraction

type Retraction struct {
	ID        string `json:"id"`
	LineStart int    `json:"line_start"`
	LineEnd   int    `json:"line_end"`
	Quote     string `json:"quote"`
	Reason    string `json:"reason"`
}

Retraction names a chunk finding the spec answers, and the text that answers it.

type Section

type Section struct {
	LineStart   int
	LineEnd     int
	HeadingPath []string
}

func BuildSections

func BuildSections(lines []string, headings []Heading) []Section

type SynthesisConfig

type SynthesisConfig struct {
	SystemPrompt string
	// Prefix is the shared user-message prefix from llm.BuildSpecPrefix, the
	// same one the chunk calls sent, so synthesis reads the spec from cache.
	Prefix string
	// LongCache is passed through to llm.Request.LongCache.
	LongCache bool
	// SpecShown reports that Prefix holds the whole spec. It is false when the
	// spec was too large to share, and synthesis then works from the findings.
	SpecShown bool
	MaxTokens int
	Effort    string
	// EnforceSchema asks the provider to constrain the response to the review
	// schema.
	EnforceSchema bool
	// NoPatches leaves patches out of the requested output.
	NoPatches     bool
	LineThreshold int
	Enabled       bool
}

type SynthesisInput

type SynthesisInput struct {
	Merged MergeResult
	// SpecShown reports that the shared prefix holds the whole spec.
	SpecShown bool
}

type SynthesisResult

type SynthesisResult struct {
	// Report holds the new cross-section findings.
	Report *schema.Report
	// Merges are groups of chunk findings that report one defect.
	Merges []MergeGroup
	// Retractions are chunk findings the spec answers.
	Retractions []Retraction
}

SynthesisResult is what the cross-section pass returned.

func RunSynthesis

func RunSynthesis(ctx context.Context, provider llm.Provider, s *spec.Spec, preflight []schema.Issue, merged MergeResult, cfg SynthesisConfig) (*SynthesisResult, string, error)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL