documentreader

package
v1.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type APIError

type APIError struct {
	Status int
	Body   string
}

APIError is returned for non-2xx docling-serve responses.

func (*APIError) Error

func (e *APIError) Error() string

type Chunk

type Chunk struct {
	Filename    string         `json:"filename"`
	ChunkIndex  int            `json:"chunk_index"`
	Text        string         `json:"text"`
	RawText     string         `json:"raw_text"`
	NumTokens   *int           `json:"num_tokens"`
	Headings    []string       `json:"headings"`
	Captions    []string       `json:"captions"`
	PageNumbers []int          `json:"page_numbers"`
	DocItems    []string       `json:"doc_items"`
	Metadata    map[string]any `json:"metadata"`
}

Chunk is one docling-serve document-aware chunk.

type ChunkOptions

type ChunkOptions struct {
	MaxTokens         int
	Tokenizer         string
	MergePeers        *bool
	UseMarkdownTables *bool
	UseMarkdownImages *bool
	ImagePlaceholder  string
	IncludeRawText    *bool
	ConvertOptions    ConvertOptions
}

ChunkOptions tunes docling-serve's hybrid chunker.

type ChunkParser

type ChunkParser func(rawBody []byte) ([]Chunk, error)

ChunkParser translates a server's raw chunk-endpoint response body into seshat's own []Chunk shape. Client's own parseChunkResponse is one example.

type ConversionResult

type ConversionResult struct {
	// Markdown is the full document rendered as Markdown.
	Markdown string

	// Images are the pictures extracted from the document (inline figures, diagrams).
	// Each image carries a base64-encoded data URI so the agent can view them.
	Images []ExtractedImage

	// PageCount is the number of pages (0 if unknown).
	PageCount int
}

ConversionResult is what we get back from docling-serve for a single file.

type ConvertOptions

type ConvertOptions struct {
	FromFormats        []string
	ToFormats          []string
	DoOCR              *bool
	ForceOCR           *bool
	DoTableStructure   *bool
	TableMode          string
	PDFBackend         string
	Pipeline           string
	OCRLang            []string
	ImageExportMode    string
	IncludePageImages  *bool
	IncludeImages      *bool
	AbortOnError       *bool
	DocumentTimeoutSec *float64
	ImagesScale        *float64
}

ConvertOptions tunes docling-serve conversion. Zero values keep server defaults, which is the safest behavior across docling-serve versions.

type ConvertParser

type ConvertParser func(rawBody []byte) (*ConversionResult, error)

ConvertParser translates a server's raw convert-endpoint response body into seshat's own ConversionResult shape. Client's own parseResponse (docling-serve's JSON shape) is one example; a different server gets its own parser doing the equivalent field-by-field translation.

type Converter

type Converter interface {
	HealthChecker
	ConvertFile(ctx context.Context, filePath string) (*ConversionResult, error)
	ConvertBytes(ctx context.Context, data []byte, filename string) (*ConversionResult, error)
	ConvertURL(ctx context.Context, docURL string) (*ConversionResult, error)
}

Converter is anything that can convert documents to markdown. *DoclingClient (this package) is the default implementation, backed by docling-serve - every consumer in internal/tools and internal/pdfsmart depends on this interface rather than *DoclingClient directly, so a host application can supply its own implementation (e.g. one backed by a different service) via pkg/sdk.ClientConfig.DocumentConverter instead of being forced onto docling-serve's wire format.

type DoclingClient

type DoclingClient struct {
	// contains filtered or unexported fields
}

DoclingClient calls a running docling-serve instance to convert documents to markdown.

func NewDoclingClient

func NewDoclingClient(baseURL string, opts ...Option) *DoclingClient

NewDoclingClient creates a client pointing at a docling-serve base URL. baseURL is typically "http://localhost:5001". Defaults to a 120s per-request timeout - pass WithTimeout to override it.

func (*DoclingClient) ChunkHybrid

func (c *DoclingClient) ChunkHybrid(ctx context.Context, filename string, r io.Reader, opts ChunkOptions) ([]Chunk, error)

ChunkHybrid sends a document to docling-serve's hybrid chunk endpoint.

func (*DoclingClient) ChunkHybridBytes

func (c *DoclingClient) ChunkHybridBytes(ctx context.Context, data []byte, filename string, opts ChunkOptions) ([]Chunk, error)

ChunkHybridBytes chunks an in-memory document without writing it to disk.

func (*DoclingClient) ChunkHybridFile

func (c *DoclingClient) ChunkHybridFile(ctx context.Context, filePath string, opts ChunkOptions) ([]Chunk, error)

ChunkHybridFile sends filePath to docling-serve's hybrid chunk endpoint.

func (*DoclingClient) ConvertBytes

func (c *DoclingClient) ConvertBytes(ctx context.Context, data []byte, filename string) (*ConversionResult, error)

ConvertBytes converts an in-memory document without writing it to disk first. filename is used only to hint the MIME type to docling (e.g. "report.pdf").

func (*DoclingClient) ConvertBytesWithOptions

func (c *DoclingClient) ConvertBytesWithOptions(ctx context.Context, data []byte, filename string, opts ConvertOptions) (*ConversionResult, error)

ConvertBytesWithOptions converts an in-memory document with conversion options.

func (*DoclingClient) ConvertFile

func (c *DoclingClient) ConvertFile(ctx context.Context, filePath string) (*ConversionResult, error)

ConvertFile sends filePath to docling-serve and returns the markdown + images.

func (*DoclingClient) ConvertFileWithOptions

func (c *DoclingClient) ConvertFileWithOptions(ctx context.Context, filePath string, opts ConvertOptions) (*ConversionResult, error)

ConvertFileWithOptions sends filePath to docling-serve with conversion options.

func (*DoclingClient) ConvertURL

func (c *DoclingClient) ConvertURL(ctx context.Context, docURL string) (*ConversionResult, error)

ConvertURL fetches and converts a remote document (e.g. an arXiv PDF URL).

func (*DoclingClient) IsAvailable

func (c *DoclingClient) IsAvailable(ctx context.Context) bool

IsAvailable does a cheap health check against the running service.

type ExtractedImage

type ExtractedImage struct {
	// Filename is the suggested on-disk name (e.g. "image_0001.png").
	Filename string
	// MimeType is the image MIME type (e.g. "image/png").
	MimeType string
	// Base64 is the raw base64-encoded image bytes (no data-URI prefix).
	Base64 string
}

ExtractedImage is one picture found inside the converted document.

type GenericClient

type GenericClient struct {
	// contains filtered or unexported fields
}

GenericClient is a configurable client for any document-intelligence HTTP server that accepts a multipart file upload and returns JSON - the shape docling-serve, seshat-intelligence, and similar services all share, even though each names its own endpoints and JSON fields differently.

Unlike Client (docling-serve-specific: hardcoded paths, a fixed JSON shape, docling-serve's own conversion/chunking option names), every protocol detail here is supplied by the caller at construction time via GenericConfig: paths, the multipart file field name, and functions translating that server's own JSON into the same ConversionResult/Chunk types Client produces. The rest of seshat (internal/rag.HybridDocumentChunker, internal/pdfsmart, the file-read/convert tools) depends only on the Converter/HybridChunker interfaces, so it works unchanged regardless of which concrete backend a host configures.

A concrete backend's own adapter is typically small: paths + a parser function mapping its response fields onto ConversionResult/Chunk, then NewGenericClient. It doesn't need to live in this package, or even this repo - GenericClient (and the ConvertParser/ChunkParser it's built from) are exported specifically so a private downstream repo (e.g. seshat-backend, for its own seshat-intelligence deployment) can build one without needing a change here first.

func NewGenericClient

func NewGenericClient(cfg GenericConfig) (*GenericClient, error)

NewGenericClient validates cfg and builds a GenericClient. Returns an error rather than panicking or silently producing an unusable client, since a misconfigured backend adapter (e.g. forgetting ParseConvert) would otherwise only fail at first use, far from the actual mistake.

func (*GenericClient) ChunkHybridBytes

func (g *GenericClient) ChunkHybridBytes(ctx context.Context, data []byte, filename string, opts ChunkOptions) ([]Chunk, error)

ChunkHybridBytes satisfies HybridChunker. The options go to the server as the form fields GenericConfig.ChunkFields gives them; without that mapping the file is sent alone, and the server chunks as it likes.

func (*GenericClient) ChunkHybridFile

func (g *GenericClient) ChunkHybridFile(ctx context.Context, filePath string, opts ChunkOptions) ([]Chunk, error)

ChunkHybridFile mirrors Client's own convenience method of the same name (see ChunkHybridBytes's doc comment for why opts is unused here).

func (*GenericClient) ConvertBytes

func (g *GenericClient) ConvertBytes(ctx context.Context, data []byte, filename string) (*ConversionResult, error)

func (*GenericClient) ConvertFile

func (g *GenericClient) ConvertFile(ctx context.Context, filePath string) (*ConversionResult, error)

func (*GenericClient) ConvertURL

ConvertURL is not supported by GenericClient - a URL-based conversion endpoint's request shape (JSON body vs. docling-serve's own {"sources":[...]}) is specific enough per server that it isn't worth generalizing here without a second real backend that needs it. Returns a clear error rather than silently failing a caller that assumed every Converter supports it.

func (*GenericClient) IsAvailable

func (g *GenericClient) IsAvailable(ctx context.Context) bool

type GenericConfig

type GenericConfig struct {
	// BaseURL is the server's base URL, e.g. "http://localhost:5100" for a
	// local seshat-intelligence instance. Required.
	BaseURL string

	// FileField is the multipart form field name the server expects the
	// uploaded file under. docling-serve uses "files"; seshat-intelligence
	// uses "file" (FastAPI's UploadFile parameter is named `file` in its
	// route signature, and FastAPI binds the multipart field name to the
	// parameter name). Required.
	FileField string

	// ConvertPath is the endpoint ConvertFile/ConvertBytes POST the file
	// to, e.g. "/v1/documents" for seshat-intelligence (vs docling-serve's
	// "/v1/convert/file"). Leave empty (with ParseConvert nil) to leave
	// conversion unsupported - ConvertFile/ConvertBytes/ConvertURL then
	// return a clear error instead of attempting a request.
	ConvertPath  string
	ParseConvert ConvertParser

	// ChunkPath is the endpoint ChunkHybridBytes/ChunkHybridFile POST the
	// file to, e.g. "/v1/documents/chunks" for seshat-intelligence (vs
	// docling-serve's "/v1/chunk/hybrid/file"). Leave empty (with
	// ParseChunk nil) to leave chunking unsupported.
	ChunkPath  string
	ParseChunk ChunkParser

	// ChunkFields maps the ChunkOptions of a call (the size the chunks should have, for one) onto the form fields the
	// server's chunk endpoint takes, by name. docling-serve has its own mapping (see ChunkOptions); an arbitrary
	// server has its own names, or none. Leave nil to send the file alone.
	ChunkFields func(ChunkOptions) map[string]string

	// HealthPath is the endpoint IsAvailable GETs. Defaults to "/health" -
	// true for both docling-serve and seshat-intelligence today, but
	// overridable for a server that names it differently.
	HealthPath string

	// The remaining fields mirror Client's own Options (see their doc
	// comments in client.go) - a plain config struct here instead of the
	// same functional-options pattern, since a concrete backend's adapter
	// typically builds one GenericConfig value up front rather than
	// composing it incrementally.
	HTTPClient       *http.Client
	Timeout          time.Duration
	APIKey           string
	TenantID         string
	UserAgent        string
	MaxResponseBytes int64
	Retry            RetryConfig

	// HealthCacheTTL caches IsAvailable results for this duration, mirroring
	// WithHealthCacheTTL. nil (the zero value - easy to get by accident by
	// simply not setting this field) keeps Client's own default (5s), not
	// "disabled" - a pointer, not a plain time.Duration, specifically so an
	// unset field can't be silently misread as an explicit request to
	// disable caching. Pass a pointer to 0 to actually disable it.
	HealthCacheTTL *time.Duration
}

GenericConfig configures a GenericClient. BaseURL and FileField are always required; ConvertPath/ParseConvert and ChunkPath/ParseChunk are each optional as a pair - configure whichever capabilities the target server actually offers (see NewGenericClient's validation).

type HealthChecker

type HealthChecker interface {
	IsAvailable(ctx context.Context) bool
}

HealthChecker reports whether a document-intelligence backend is reachable. Embedded by both interfaces below rather than duplicated on each.

type HybridChunker

type HybridChunker interface {
	HealthChecker
	ChunkHybridBytes(ctx context.Context, data []byte, filename string, opts ChunkOptions) ([]Chunk, error)
}

HybridChunker is anything that can produce document-aware hybrid chunks. *DoclingClient (this package) is the default implementation, backed by docling-serve's hybrid chunk endpoint - internal/rag.HybridDocumentChunker depends on this interface rather than *DoclingClient directly, so a caller assembling its own rag.Service can supply a different backend.

type Option

type Option func(*DoclingClient)

Option configures a DoclingClient at construction time.

func WithAPIKey

func WithAPIKey(key string) Option

WithAPIKey sets the X-Api-Key header sent to docling-serve.

func WithHTTPClient

func WithHTTPClient(client *http.Client) Option

WithHTTPClient replaces the HTTP client used for docling-serve calls.

func WithHealthCacheTTL

func WithHealthCacheTTL(ttl time.Duration) Option

WithHealthCacheTTL caches IsAvailable results for the given duration.

func WithMaxResponseBytes

func WithMaxResponseBytes(n int64) Option

WithMaxResponseBytes adjusts the response body safety limit.

func WithRetry

func WithRetry(maxAttempts int, baseDelay, maxDelay time.Duration) Option

WithRetry enables retries for transient failures on replayable requests.

func WithTenantID

func WithTenantID(id string) Option

WithTenantID sets the X-Tenant-Id header sent to docling-serve.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout overrides the default 120s per-request HTTP timeout. Pass a larger budget for deployments that expect large or scan-heavy documents (see defaultTimeout's doc comment for why the default is often too short), or a smaller one for latency-sensitive callers that would rather fail fast than wait.

func WithUserAgent

func WithUserAgent(userAgent string) Option

WithUserAgent sets the User-Agent header sent to docling-serve.

type RetryConfig

type RetryConfig struct {
	MaxAttempts int
	BaseDelay   time.Duration
	MaxDelay    time.Duration
}

RetryConfig controls best-effort retries for replayable docling-serve calls.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL