Documentation
¶
Overview ¶
Package document turns an attached document into text inside this process.
This is the native parser engine. It reaches no provider and spends no money, which is the whole reason it exists: a document that already carries a text layer needs no model to read it, and sending one to a recognition provider would pay for work this process can do.
The package is a leaf. It holds no provider client, no transport, and no Starport concept beyond the bytes it was handed and the text it read back. A dependency in either direction would let the meaning of a request decide what a document says, and the import graph test holds the rule.
The bytes arriving here are a caller's own upload, so every read is bounded. A page count bound and an elapsed time bound cap the work, a decode failure inside the reader becomes a typed refusal rather than a panic, and the stored-byte bound that caps the input itself belongs to internal/files.
Index ¶
Constants ¶
const CacheKeyVersion = 1
CacheKeyVersion identifies the encoding a key was built under. Raise it when a field joins the key: entries a running gateway holds keep their own prefix, and none of them is read back under an encoding that did not write it.
const DefaultCacheWindow = time.Hour
DefaultCacheWindow is how long a read stays reusable when an operator configures no window.
A document's text does not change, so the window is not about correctness. It bounds how much text a gateway holds for conversations that ended, which is why an hour is long enough to cover a conversation and short enough that an idle deployment holds nothing.
Variables ¶
var ( // ErrCacheStoreRequired reports a cache built with no byte store. ErrCacheStoreRequired = errors.New("extraction cache store is required") // ErrIncompleteCacheKey reports a key missing a field that scopes it. // Serving an entry under a partial key would cross an account, an engine, // or a catalog generation, so an incomplete key never reads and never // writes. ErrIncompleteCacheKey = errors.New("extraction cache key is incomplete") // ErrCorruptCacheRecord reports stored data this schema cannot read. ErrCorruptCacheRecord = errors.New("extraction cache record is invalid") )
var ( // ErrEmptyDocument reports a document that carries no bytes. ErrEmptyDocument = errors.New("document carries no bytes") // ErrUnsupportedFormat reports a container the native engine cannot read. // A caller reaching this has attached something no engine here handles, // which is different from a document whose text this engine cannot find. ErrUnsupportedFormat = errors.New("document format is not read by the native engine") // ErrFormatMismatch reports bytes that are not the container the caller // declared. Reading them anyway would let a caller drive this engine to a // parser it did not name. ErrFormatMismatch = errors.New("document bytes are not the declared format") // ErrMalformedDocument reports bytes the reader could not parse. The // reader panics on some malformed input, and this turns that into an // answer the caller can act on. ErrMalformedDocument = errors.New("document is malformed") // ErrPageBudgetExceeded reports a document with more pages than the // configured bound. The refusal comes before any page is read, so a // thousand-page upload costs the gateway one header parse. ErrPageBudgetExceeded = errors.New("document exceeds the page budget") // ErrTimeBudgetExceeded reports an extraction that ran past its elapsed // time bound. A document can be small and still be expensive to read. ErrTimeBudgetExceeded = errors.New("document extraction exceeded its time budget") // ErrRecognitionFailed reports that the recognition engine did not return // every page of a document. It lives beside the native engine's refusals // because a caller reads one document vocabulary, not one per engine, and // the two engines fail the same request for the same reason: the gateway // was asked to turn a document into text and did not. // // The native engine never raises it. The seam that ordered the recognition // read does, because it is the seam that knows how many pages the document // holds and therefore whether the answer is the whole document. ErrRecognitionFailed = errors.New("document recognition failed") )
Functions ¶
func ContentHash ¶
ContentHash names a document by its bytes.
It is the only part of the key that is about the document itself, and it is what makes the cache safe: two requests share an entry when they carry the same bytes, whatever the caller named the file or claimed its format was.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache holds one document's text for reuse inside a window.
func NewCache ¶
func NewCache(store CacheStore, clock CacheClock, window time.Duration) (*Cache, error)
NewCache returns a cache over the given store. An unset window takes the default.
func (*Cache) Get ¶
Get returns the text one engine already read out of these bytes.
The window is checked here as well as at the store, because a store is free to keep a value past the lifetime it was written with. The gateway's own answer to how long text stays reusable does not depend on which store a deployment chose.
type CacheClock ¶
CacheClock supplies the time a record is stamped and compared against.
type CacheKey ¶
type CacheKey struct {
// AccountID is the account that paid for the read. An entry never crosses
// accounts: one account's upload is not another account's to read back.
AccountID string
// ContentHash is the digest of the document bytes.
ContentHash string
// Engine is the parser engine the caller named.
Engine string
// Generation is the catalog generation in force when the read ran.
Generation string
}
CacheKey names one document read.
Every field scopes the entry, and dropping any one of them serves text that answers a different question. The account is who paid for the read. The content hash is which bytes were read. The engine is which reader produced the text, because a scanned page reads as nothing natively and as its contents under recognition. The generation is which catalog was in force, because that is what decides which offerings serve the recognition operation at all.
type CacheStore ¶
type CacheStore interface {
Get(ctx context.Context, key string) ([]byte, bool, error)
Set(ctx context.Context, key string, value []byte, ttl time.Duration) error
}
CacheStore is the byte store the cache holds entries in.
It is the narrowest contract that expresses a window: a write states how long its value stays readable. This package names no store implementation, because a leaf that named one would decide where a deployment keeps its data.
type Extraction ¶
type Extraction struct {
// Text is every page's text in page order, separated by a blank line.
Text string
// Pages holds one entry per page in the document, in page order. Its
// length is the document's page count, because the page budget refuses
// an oversized document before any page is read.
Pages []Page
// Scanned reports that no page carried a usable text layer. This is the
// signal that the document needs the recognition engine: the native read
// succeeded and found nothing a model could use.
Scanned bool
}
Extraction is what the native engine read out of one document.
func (Extraction) PageCount ¶
func (e Extraction) PageCount() int
PageCount returns the number of pages the document holds.
type Extractor ¶
type Extractor struct {
// contains filtered or unexported fields
}
Extractor reads text out of a document inside this process.
It holds bounds and nothing else. There is no client, no cache, and no account: PLG5 owns the cache and scopes it to the account that paid, and this seam stays the part that turns bytes into text.
func NewExtractor ¶
NewExtractor returns an extractor bounded by the given limits. An unset bound takes its default.
func (*Extractor) Extract ¶
Extract reads the document's text layer.
The order of the checks is the point. A document is refused for its bytes before it is opened, refused for its page count before a page is read, and only then read page by page against the deadline. Each step that runs is paid for by the step before it having narrowed the input.
A document that opens and yields no usable text is not an error. It is a scanned document, and reporting it as one is what lets the recognition engine take over.
type Format ¶
type Format string
Format names a document container the native engine can read.
const FormatPDF Format = "pdf"
FormatPDF is the one container this engine reads. Starmap spells the modality "pdf", and the OpenRouter wire word for it is "file".
type Input ¶
type Input struct {
// Data holds the document bytes.
Data []byte
// Format is the container the caller declared, spelled either as a bare
// name such as "pdf" or as a media type such as "application/pdf". An
// empty value asks this engine to read the container out of the bytes.
Format string
// Filename is the caller's own name for the document. It never decides
// the container, because a name is not evidence about bytes. It travels
// so a caller with several attachments learns which one a refusal names.
Filename string
}
Input is one document handed to the native engine.
type Limits ¶
type Limits struct {
// MaxPages refuses a document with more pages than this.
MaxPages int
// MaxDuration refuses an extraction that runs longer than this.
MaxDuration time.Duration
}
Limits bounds one extraction.
func DefaultLimits ¶
func DefaultLimits() Limits
DefaultLimits returns the bounds this gateway applies when an operator configures none.
The page bound is generous for a document a caller attaches to a chat turn and small enough that one upload cannot occupy a worker. The time bound is well past what a healthy read of that many pages takes, so it fires on a document built to be slow rather than on a large one.
type Page ¶
type Page struct {
// Number is the page's one-based position in the document.
Number int
// Text is the text this page carried, with trailing space removed from
// each line. It is empty when the page carried no text layer.
Text string
// Scanned reports that this page carried no usable text layer, so a
// model reading the document natively would see nothing here.
Scanned bool
}
Page is one page of a document after extraction.
type Reading ¶
type Reading struct {
// Text is the document's text as the named engine read it.
Text string `json:"text"`
// Pages is how many pages the document holds.
Pages int `json:"pages"`
// Offering is the recognition model that produced the text. It is empty
// when the native engine read the document, which is also how a reader
// tells a free read from a paid one.
Offering string `json:"offering,omitempty"`
}
Reading is what one engine read out of one document.
It is smaller than an Extraction on purpose. A cached read answers what the model receives and what the account owes, and neither needs the per-page text layer that decided which engine ran.
Source Files