docparser

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2026 License: Apache-2.0, MIT Imports: 59 Imported by: 0

Documentation

Overview

Package docparser turns uploaded files and URLs into Markdown plus the images they contain, which is the input the chunking and multimodal stages downstream expect.

A parse request names a parser engine, and every engine is one entry in the registry:

  • engines.go declares the catalog — each engine's name, description, supported file types, availability probe, and the reader that does the work.
  • engine_registry.go turns an engine name into a reader (NewReader) and merges the local catalog with the engines the Python docreader reports over its ListEngines RPC (ListAllEngines).

Readers differ in where the parsing happens: in this process (SimpleFormatReader, AnydocReader), in the docreader service over gRPC or HTTP (GRPCDocumentReader, HTTPDocumentReader), or in a remote API (MinerU, PaddleOCR-VL, WeKnora Cloud). They all return types.ReadResult, so the rest of the package — image resolution and storage, table normalization — is shared regardless of which engine ran.

Index

Constants

View Source 
const (
	// BuiltinEngineName is the DocReader (Python) parser suite.
	BuiltinEngineName = "builtin"
	// SimpleEngineName is Go-native handling of text formats and images.
	SimpleEngineName = "simple"
	// AnydocEngineName is the in-process anydoc office-document converter.
	AnydocEngineName = "anydoc"
	// WeKnoraCloudEngineName is the hosted WeKnora Cloud document reader.
	WeKnoraCloudEngineName = "weknoracloud"
	// MinerUEngineName is a self-hosted MinerU service.
	MinerUEngineName = "mineru"
	// MinerUCloudEngineName is the MinerU Cloud API.
	MinerUCloudEngineName = "mineru_cloud"
	// PaddleOCRVLEngineName is a self-hosted PaddleOCR-VL pipeline.
	PaddleOCRVLEngineName = "paddleocr_vl"
	// PaddleOCRVLCloudEngineName is the PaddleOCR-VL AI Studio cloud API.
	PaddleOCRVLCloudEngineName = "paddleocr_vl_cloud"
)

Engine names, as stored in knowledge base parser rules and shown in the UI.

View Source 
const (
	PathRead        = "/read"
	PathListEngines = "/list-engines"
)

Variables

View Source 
var (
	ErrEmptyJSONContent   = errors.New("empty JSON content")
	ErrInvalidJSONContent = errors.New("invalid JSON content")
)

Errors returned by ValidateJSONContent.

View Source 
var ErrAmbiguousInlineImage = errors.New(
	"cannot determine the end of a malformed multiline inline image; repair or remove the image")

ErrAmbiguousInlineImage means the image cannot be separated from following lines without guessing. Callers should keep the source and stop ingestion.

View Source 
var ErrCSVMarkdownTooLarge = errors.New("CSV conversion exceeds maximum Markdown size")

ErrCSVMarkdownTooLarge marks deterministic CSV expansion failures. Callers should report the failure without retrying the same document.

Functions

func HasUnresolvedInlineImagePayload added in v0.9.0 

func HasUnresolvedInlineImagePayload(markdown string) bool

HasUnresolvedInlineImagePayload guards the chunking boundary against data URIs that a parser could not recognize or resolve. It deliberately ignores ordinary prose mentioning base64 and non-image base64 examples.

func IsAudioFormat 

func IsAudioFormat(fileType string) bool

IsAudioFormat returns true if the file type is a recognized audio format.

func IsImageFormat 

func IsImageFormat(fileType string) bool

IsImageFormat returns true if the file type is a recognized image format.

func IsSimpleFormat 

func IsSimpleFormat(fileType string) bool

IsSimpleFormat returns true if the file type can be handled by the Go SimpleFormatReader.

func ListAllEngines 

func ListAllEngines(
	docreaderConnected bool, overrides map[string]string, remoteEngines []types.ParserEngineInfo,
) []types.ParserEngineInfo

ListAllEngines returns the merged engine list: locally registered engines plus engines discovered from the remote docreader via ListEngines RPC.

Merge rules:

  • Local engines are always included, with Go-side availability checks.
  • For a remote engine whose name matches a local one, the remote's file_types and description take precedence (the remote service is authoritative for its own capabilities).
  • Remote engines not present locally are appended as-is, enabling auto-discovery of newly added docreader engines without Go changes.

func NewReader 

func NewReader(
	ctx context.Context, engine, fileType string, isURL bool, deps ReaderDeps,
) (interfaces.DocReader, error)

NewReader builds the reader for an engine.

An empty engine name means "no explicit choice": simple formats are handled in Go and everything else goes to the docreader service. An unknown name is routed to the docreader too, so engines that only exist in the Python service keep working without a Go-side registration.

func NormalizeHTMLTables 

func NormalizeHTMLTables(md string) string

NormalizeHTMLTables rewrites inline HTML <table> blocks embedded in OCR markdown output. PaddleOCR-VL emits tables as HTML with per-cell text-align styles, which (1) waste tokens on layout markup and (2) are not recognized by the chunker's table-protection logic, so large tables get split mid-row.

Each table block is converted to a GFM Markdown table when possible. Tables that use rowspan/colspan (which Markdown cannot express), or that the converter cannot turn into a valid GFM table (cell <br>, nested lists, …), fall back to presentational-attribute stripping plus one row per line so the chunker can still split on "\n". Tables inside fenced code blocks are left unchanged.

func PingMinerU 

func PingMinerU(endpoint, apiKey string) (bool, string)

PingMinerU checks if the self-hosted MinerU service is reachable. For MinerU 4.0 it also verifies the configured API key, since /v1/health itself is served without authentication unless sitting behind an authenticating gateway.

func PingMinerUCloud 

func PingMinerUCloud(apiKey string) (bool, string)

PingMinerUCloud checks if the MinerU Cloud API is reachable with the given API key.

func PingPaddleOCRVL 

func PingPaddleOCRVL(endpoint string) (bool, string)

PingPaddleOCRVL checks whether a self-hosted PaddleOCR-VL service is reachable.

func PingPaddleOCRVLCloud 

func PingPaddleOCRVLCloud(token string) (bool, string)

PingPaddleOCRVLCloud checks whether the cloud token is present (the API has no lightweight health endpoint, so we only validate configuration here).

func RegisterEngine 

func RegisterEngine(e EngineRegistration)

RegisterEngine adds an engine to the local registry. Called from init().

func StripMarkdownImages 

func StripMarkdownImages(markdown string) string

StripMarkdownImages removes complete ![alt](destination) constructs, including destinations and titles that contain parentheses. Leftover prose is preserved so callers can tell whether a chunk still has extractable text after image placeholders are gone.

func UnwrapLinkedImages 

func UnwrapLinkedImages(markdown string) string

UnwrapLinkedImages replaces all [![alt](img_url)](link_url) occurrences in the markdown with just ![alt](img_url), stripping the outer link wrapper. This should be called before any image-extraction regex so that only the flat ![alt](url) form needs to be handled.

func ValidateJSONContent 

func ValidateJSONContent(data []byte) error

ValidateJSONContent reports whether data is well-formed JSON after stripping a leading UTF-8 BOM. Upload paths call this before enqueueing parse work so JSONC / truncated payloads fail immediately instead of after async retries.

Types

type AnydocReader 

type AnydocReader struct {
	// contains filtered or unexported fields
}

AnydocReader converts office documents to markdown in this process, using the anydoc converter instead of the Python docreader service.

Embedded images are rendered in place from the document model. PDFs with no text layer or a structure anydoc rejects are handed to the configured fallback reader (typically the builtin docreader). PPTX files with EMF/WMF media use the docreader's MarkItDown path to rasterize those embedded images.

func NewAnydocReader 

func NewAnydocReader(overrides map[string]string, fallback interfaces.DocReader) *AnydocReader

NewAnydocReader builds a reader. The "anydoc_extract_images" override turns off image extraction. fallback may be nil.

func (*AnydocReader) Read 

func (r *AnydocReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

Read converts the document carried by the request.

type EngineRegistration 

type EngineRegistration interface {
	Name() string
	Description() string
	FileTypes(docreaderConnected bool) []string
	CheckAvailable(docreaderConnected bool, overrides map[string]string) (available bool, reason string)
	// NewReader builds the reader for one parse request. Returning an error
	// means this engine cannot serve the request (missing credentials,
	// unreachable service); the caller reports it rather than silently
	// parsing with something else.
	NewReader(ctx context.Context, deps ReaderDeps) (interfaces.DocReader, error)
}

EngineRegistration is what every locally registered parser engine provides: the metadata the engine list shows, and the reader that does the parsing. Remote-only engines (e.g. markitdown) live in the Python docreader, are discovered through its ListEngines RPC, and never register here — the registry routes them to the docreader client by default.

type GRPCDocumentReader 

type GRPCDocumentReader struct {
	// contains filtered or unexported fields
}

GRPCDocumentReader implements DocumentReader over gRPC.

func NewGRPCDocumentReader 

func NewGRPCDocumentReader(addr string) (*GRPCDocumentReader, error)

func (*GRPCDocumentReader) Close 

func (p *GRPCDocumentReader) Close() error

func (*GRPCDocumentReader) IsConnected 

func (p *GRPCDocumentReader) IsConnected() bool

func (*GRPCDocumentReader) ListEngines 

func (p *GRPCDocumentReader) ListEngines(ctx context.Context, overrides map[string]string) ([]types.ParserEngineInfo, error)

func (*GRPCDocumentReader) PreprocessImage added in v0.9.0 

func (p *GRPCDocumentReader) PreprocessImage(ctx context.Context, image []byte) ([]byte, bool, error)

PreprocessImage straightens and crops a photographed page before OCR runs. The docreader does the work; a page it declines to improve comes back as fallback=true with no bytes, which the caller reads as "use the original".

func (*GRPCDocumentReader) Read 

func (p *GRPCDocumentReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

func (*GRPCDocumentReader) Reconnect 

func (p *GRPCDocumentReader) Reconnect(addr string) error

type HTTPDocumentReader 

type HTTPDocumentReader struct {
	// contains filtered or unexported fields
}

HTTPDocumentReader implements DocumentReader over HTTP/JSON.

func NewHTTPDocumentReader 

func NewHTTPDocumentReader(baseURL string) (*HTTPDocumentReader, error)

func (*HTTPDocumentReader) Close 

func (p *HTTPDocumentReader) Close() error

func (*HTTPDocumentReader) IsConnected 

func (p *HTTPDocumentReader) IsConnected() bool

func (*HTTPDocumentReader) ListEngines 

func (p *HTTPDocumentReader) ListEngines(ctx context.Context, overrides map[string]string) ([]types.ParserEngineInfo, error)

func (*HTTPDocumentReader) PreprocessImage added in v0.9.0 

func (p *HTTPDocumentReader) PreprocessImage(_ context.Context, _ []byte) ([]byte, bool, error)

PreprocessImage is not offered over the HTTP transport: page straightening lives in the docreader service, and this reader speaks to a different surface. It reports that as a verdict — fallback=true, no error — so the caller's single fallback (OCR the original bytes) is the only outcome, with nothing logged that looks like a failure.

func (*HTTPDocumentReader) Read 

func (p *HTTPDocumentReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

func (*HTTPDocumentReader) Reconnect 

func (p *HTTPDocumentReader) Reconnect(addr string) error

type ImageResolver 

type ImageResolver struct {
	// TenantID for storage path namespacing
	TenantID uint64
}

ImageResolver reads images from a DocReader ReadResult (inline bytes only) and saves them via FileService, replacing markdown references with unified URLs.

func NewImageResolver 

func NewImageResolver() *ImageResolver

NewImageResolver creates a resolver.

func (*ImageResolver) ResolveAndStore 

func (r *ImageResolver) ResolveAndStore(
	ctx context.Context,
	result *types.ReadResult,
	fileSvc interfaces.FileService,
	tenantID uint64,
) (updatedMarkdown string, images []StoredImage, err error)

ResolveAndStore reads images from the convert result, persists them via fileSvc, and replaces markdown references with provider:// URLs. It returns the updated markdown and a list of stored images.

func (*ImageResolver) ResolveBareBase64Content 

func (r *ImageResolver) ResolveBareBase64Content(
	ctx context.Context,
	markdown string,
	fileSvc interfaces.FileService,
	tenantID uint64,
) (updatedMarkdown string, images []StoredImage, err error)

ResolveBareBase64Content finds remaining bare data URIs and base64 image content in the markdown text, decodes and stores them, and replaces with image references. This acts as a catch-all after the standard markdown and HTML resolvers.

func (*ImageResolver) ResolveDataURIImages 

func (r *ImageResolver) ResolveDataURIImages(
	ctx context.Context,
	markdown string,
	fileSvc interfaces.FileService,
	tenantID uint64,
) (updatedMarkdown string, images []StoredImage, err error)

ResolveDataURIImages finds embedded data:image/*;base64 images in markdown, decodes and stores each valid image, and replaces each reference with the returned provider URL. A per-image decoded-size limit still applies.

func (*ImageResolver) ResolveHTMLDataURIImages 

func (r *ImageResolver) ResolveHTMLDataURIImages(
	ctx context.Context,
	markdown string,
	fileSvc interfaces.FileService,
	tenantID uint64,
) (updatedMarkdown string, images []StoredImage, err error)

ResolveHTMLDataURIImages finds <img src="data:image/*;base64,..."> tags in markdown, decodes the images, stores them via fileSvc, and replaces each tag with a markdown image reference using the storage URL.

func (*ImageResolver) ResolveRelativeHTMLImages 

func (r *ImageResolver) ResolveRelativeHTMLImages(
	ctx context.Context,
	markdown string,
	fileSvc interfaces.FileService,
	tenantID uint64,
	refMap map[string]types.ImageRef,
	savedRefs map[string]StoredImage,
) (updatedMarkdown string, images []StoredImage, err error)

ResolveRelativeHTMLImages finds HTML <img> tags whose src points at a relative document image reference, stores the corresponding bytes via fileSvc, and replaces only the src attribute value with the storage URL.

func (*ImageResolver) ResolveRemoteImages 

func (r *ImageResolver) ResolveRemoteImages(
	ctx context.Context,
	markdown string,
	fileSvc interfaces.FileService,
	tenantID uint64,
) (updatedMarkdown string, images []StoredImage, err error)

ResolveRemoteImages downloads remote http(s) images into storage and points the document at the stored copies. Both Markdown image syntax and HTML <img> tags with a quoted src are covered.

type MinerUCloudReader 

type MinerUCloudReader struct {
	// contains filtered or unexported fields
}

MinerUCloudReader calls the MinerU Cloud API (mineru.net) to read/convert documents. Flow: POST /file-urls/batch → PUT file → poll GET /extract-results/batch/{batch_id}.

func NewMinerUCloudReader 

func NewMinerUCloudReader(overrides map[string]string) *MinerUCloudReader

NewMinerUCloudReader creates a reader from ParserEngineOverrides.

func (*MinerUCloudReader) Read 

func (c *MinerUCloudReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

type MinerUReader 

type MinerUReader struct {
	// contains filtered or unexported fields
}

MinerUReader calls a self-hosted MinerU API to read/convert documents. MinerU >= 4.0 is driven through the V1 API; older servers through the legacy /file_parse endpoint. The protocol is detected per request, so an in-place MinerU upgrade needs no configuration change.

func NewMinerUReader 

func NewMinerUReader(overrides map[string]string) *MinerUReader

NewMinerUReader creates a reader from ParserEngineOverrides.

func (*MinerUReader) Read 

func (c *MinerUReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

type PaddleOCRVLCloudReader 

type PaddleOCRVLCloudReader struct {
	// contains filtered or unexported fields
}

PaddleOCRVLCloudReader calls the PaddleOCR-VL AI Studio cloud API. Flow: POST /jobs (multipart) → poll GET /jobs/{id} → download result JSONL, then fetch each referenced image URL.

func NewPaddleOCRVLCloudReader 

func NewPaddleOCRVLCloudReader(overrides map[string]string) *PaddleOCRVLCloudReader

NewPaddleOCRVLCloudReader creates a reader from ParserEngineOverrides.

func (*PaddleOCRVLCloudReader) Read 

func (c *PaddleOCRVLCloudReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

type PaddleOCRVLReader 

type PaddleOCRVLReader struct {
	// contains filtered or unexported fields
}

PaddleOCRVLReader calls a self-hosted PaddleOCR-VL pipeline service (the full document-parsing API, not the bare VLM inference server).

Flow: POST {endpoint}/layout-parsing with base64 file → synchronous JSON response containing per-page markdown + inline base64 images.

func NewPaddleOCRVLReader 

func NewPaddleOCRVLReader(overrides map[string]string) *PaddleOCRVLReader

NewPaddleOCRVLReader creates a reader from ParserEngineOverrides.

func (*PaddleOCRVLReader) Read 

func (c *PaddleOCRVLReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

type ReaderDeps 

type ReaderDeps struct {
	// Overrides holds tenant-level engine configuration (service endpoints,
	// API keys), as produced by ParserEngineConfig.ToOverridesMap.
	Overrides map[string]string
	// Remote is the docreader client. Nil when the service is not connected.
	Remote interfaces.DocReader
	// WeKnoraCloudCredentials resolves the tenant's WeKnora Cloud
	// credentials. It is a function rather than a value because resolving
	// them can hit the database, which most engines never need. Nil, or a
	// nil return, means the tenant has not configured them.
	WeKnoraCloudCredentials func(ctx context.Context) *types.WeKnoraCloudCredentials
}

ReaderDeps carries everything an engine may need to build its reader but cannot construct itself: tenant configuration, tenant credentials, and the shared docreader connection.

type SimpleFormatReader 

type SimpleFormatReader struct{}

SimpleFormatReader handles simple file formats and images directly in Go, bypassing the Python docreader service.

func (*SimpleFormatReader) Read 

func (b *SimpleFormatReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

Read reads simple format files and returns markdown.

type StoredImage 

type StoredImage struct {
	OriginalRef string // reference in the original markdown
	ServingURL  string // provider:// URL (e.g. local://images/xxx.png, minio://bucket/key)
	MimeType    string
	// Inline marks Base64 images stored from inline document content, so only
	// resources created by this resolver are eligible for partial-save cleanup.
	Inline bool
	// NewlyStored marks a resource created by this resolver invocation, including
	// document-parser ImageRefs, so guard failures can clean up unbound uploads.
	NewlyStored bool
	// SkipMultimodal keeps a saved inline image out of OCR/caption fan-out once
	// its syntax-specific legacy budget is exhausted. The image still remains in
	// the document and resource bindings.
	SkipMultimodal bool
	// SourceLocators place the image in the original file (e.g. the scanned
	// page it was rendered from). Filled by ingestion after resolution.
	SourceLocators types.SourceLocators
}

StoredImage describes an image that has been saved to storage.

type WeKnoraCloudSignedDocumentReader 

type WeKnoraCloudSignedDocumentReader struct {
	// contains filtered or unexported fields
}

WeKnoraCloudSignedDocumentReader implements the docreader HTTP protocol with WeKnoraCloud signing.

func NewWeKnoraCloudSignedDocumentReader 

func NewWeKnoraCloudSignedDocumentReader(appID, apiKey string) (*WeKnoraCloudSignedDocumentReader, error)

func (*WeKnoraCloudSignedDocumentReader) IsConnected 

func (p *WeKnoraCloudSignedDocumentReader) IsConnected() bool

func (*WeKnoraCloudSignedDocumentReader) ListEngines 

func (p *WeKnoraCloudSignedDocumentReader) ListEngines(ctx context.Context, overrides map[string]string) ([]types.ParserEngineInfo, error)

func (*WeKnoraCloudSignedDocumentReader) PreprocessImage added in v0.9.0 

func (p *WeKnoraCloudSignedDocumentReader) PreprocessImage(_ context.Context, _ []byte) ([]byte, bool, error)

PreprocessImage is not offered by the cloud-signed reader: page straightening runs in the local docreader service, while this reader talks to WeKnoraCloud. It reports that as a verdict — fallback=true, no error — so the caller's only fallback (OCR the original bytes) needs no special case and nothing looks like a failure in the logs.

func (*WeKnoraCloudSignedDocumentReader) Read 

func (p *WeKnoraCloudSignedDocumentReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error)

func (*WeKnoraCloudSignedDocumentReader) Reconnect 

func (p *WeKnoraCloudSignedDocumentReader) Reconnect(addr string) error

Source Files

View all Source files

Directories

Path Synopsis
Package anydoc converts office documents to Markdown inside the Go process, without the Python docreader service.
 Package anydoc converts office documents to Markdown inside the Go process, without the Python docreader service.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.