processing

package

v0.1.0-round1 Latest Latest Go to latest Published: Aug 25, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/Merge42-SyncBase/syncbase-was

Links

Open Source Insights

Documentation ¶

Overview ¶

Package processing implements SyncBase's document-processing module.

Package processing coordinates the fenced document-processing pipeline.

Index ¶

func ChunkPages(pages []knowledge.PageText) []knowledge.Chunk
func ChunkPagesWithCounter(pages []knowledge.PageText, counter TokenCounter) ([]knowledge.Chunk, error)
func ChunkPagesWithProfile(pages []knowledge.PageText, counter TokenCounter, profile knowledge.Profile) ([]knowledge.Chunk, error)
type Processor
- func New(runs runStore, artifacts artifactStore, parser pdfParser, embedder embedder, ...) *Processor
- func (p *Processor) Process(ctx context.Context, run knowledge.ClaimedRun) error
type TokenCounter

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func ChunkPages ¶

func ChunkPages(pages []knowledge.PageText) []knowledge.Chunk

ChunkPages creates deterministic page-bounded chunks.

func ChunkPagesWithCounter ¶

func ChunkPagesWithCounter(pages []knowledge.PageText, counter TokenCounter) ([]knowledge.Chunk, error)

ChunkPagesWithCounter applies the default production token contract while preserving page boundaries. New processing code should pass its immutable profile to ChunkPagesWithProfile instead.

func ChunkPagesWithProfile ¶

func ChunkPagesWithProfile(
	pages []knowledge.PageText,
	counter TokenCounter,
	profile knowledge.Profile,
) ([]knowledge.Chunk, error)

ChunkPagesWithProfile applies the profile's token sizing contract while preserving page boundaries. Forced splits retain exactly the profile's configured overlap, so the persisted profile and generated chunks cannot silently diverge.

Types ¶

type Processor ¶

type Processor struct {
	// contains filtered or unexported fields
}

Processor advances one claimed run from parsing through active-version publication.

func New ¶

func New(
	runs runStore,
	artifacts artifactStore,
	parser pdfParser,
	embedder embedder,
	profile knowledge.Profile,
) *Processor

New returns a processor that coordinates a fenced parse-to-activate run.

func (*Processor) Process ¶

func (p *Processor) Process(ctx context.Context, run knowledge.ClaimedRun) error

Process executes a claimed run and records recoverable stage checkpoints.

type TokenCounter ¶

type TokenCounter func(string) (int, error)

TokenCounter returns the number of tokens produced by the pinned processing tokenizer for one passage.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL