Documentation
¶
Overview ¶
Package extract turns an uploaded file into agent-readable text. It is the inbound mirror of pkg/channels/channelassets: a MIME type selects a registered backend, so a new format is a registration rather than a branch in a caller.
Extractors run in the extractord pod against untrusted user uploads. They must never perform network I/O — that pod has no egress and no credentials by design. A backend whose underlying library requires a filesystem path may use os.TempDir() and must remove what it writes; nothing else on disk is theirs to touch. Malformed input must produce an error, never a panic.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultLimits = Limits{ MaxUncompressedTotal: 64 << 20, MaxMemberBytes: 25 << 20, MaxMembers: 256, MaxRatio: 100, MaxWallClock: 30 * time.Second, }
DefaultLimits is the shipped policy, and the single place it is written down. Every field is non-zero on purpose: a zero bound reads as "unlimited" to any check of the form `n > lim`, which is the one way this struct could fail open.
var ErrArchiveBomb = errors.New("extract: archive exceeds a decompression bound")
ErrArchiveBomb is a decompression bound trip: uncompressed total, per-member size, ratio, or wall clock.
PERMANENT. The same archive trips it again every time, so a caller must never word it as a temporary failure — "try sending it again" is advice that cannot succeed, and telling a user that is worse than telling them nothing.
var ErrArchiveMalformed = errors.New("extract: not a readable archive")
ErrArchiveMalformed is input that is not a readable archive of its claimed type. Also permanent, and deliberately distinct from ErrArchiveBomb: one means "we refused to open this", the other "there was nothing to open".
var ErrTooLarge = errors.New("extract: input too large")
ErrTooLarge is returned when input exceeds an extractor's own limit.
Functions ¶
func ContainerMIMEs ¶
func ContainerMIMEs() []string
ContainerMIMEs lists the container vocabulary, sorted. For tests and readiness output.
func ExplodableMIMEs ¶
func ExplodableMIMEs() []string
ExplodableMIMEs lists every claimed container MIME, sorted. Used by extractord's readiness output and by tests asserting the registered set.
func IsContainerMIME ¶
IsContainerMIME reports whether mime names a container format. Parameters are ignored, matching For and ExploderFor.
func Register ¶
func Register(e Extractor)
Register claims every MIME in e.MIMEs(). It panics on a duplicate claim: two extractors silently competing for one type would make which one runs depend on package-import order, which is not a thing to debug in production.
func RegisterExploder ¶
func RegisterExploder(e Exploder)
RegisterExploder claims every MIME in e.MIMEs(). It panics on a duplicate claim, for the same reason Register does: two backends silently competing would make which one runs depend on package-import order.
The registry is separate from the extractor registry rather than shared, because the two answer different questions about the same MIME — a container has an exploder and no extractor, and a caller must never get one where it asked for the other.
Types ¶
type Exploder ¶
type Exploder interface {
// MIMEs returns the exact MIME types this exploder claims. Registration
// panics on a duplicate claim, so two backends cannot silently fight.
MIMEs() []string
// Explode calls yield once per surviving member, in archive order.
//
// Takes io.ReaderAt + size rather than io.Reader because a container
// format needs random access to read its directory; a backend fed a stream
// buffers to os.TempDir() and removes what it writes.
//
// Implementations MUST enforce lim while streaming, and MUST NOT recurse
// into a member that is itself an archive — nesting depth is 0, so a
// zip-of-zips cannot re-enter. An error returned by yield aborts the walk
// and is returned unchanged.
Explode(r io.ReaderAt, size int64, lim Limits, yield func(Member) error) (Summary, error)
}
Exploder turns one container format into its members.
Bound by every rule in this package's doc: no network, no filesystem beyond os.TempDir(), malformed input produces an error rather than a panic.
func ExploderFor ¶
ExploderFor returns the exploder claiming mime. Parameters are ignored, so "application/zip; charset=binary" resolves to the "application/zip" backend.
type Extractor ¶
type Extractor interface {
// MIMEs returns the exact MIME types this extractor claims. Registration
// panics on a duplicate claim, so two backends cannot silently fight.
MIMEs() []string
// Extract reads r to EOF and returns the text. Implementations must not
// perform network or filesystem I/O.
Extract(r io.Reader) (Result, error)
}
Extractor converts one family of MIME types into text.
type Limits ¶
type Limits struct {
// MaxUncompressedTotal caps the whole archive's expanded size.
MaxUncompressedTotal int64
// MaxMemberBytes caps a single member.
MaxMemberBytes int64
// MaxMembers caps how many members are yielded. Reaching it TRUNCATES —
// Summary.Truncated — rather than failing the archive: a partial support
// bundle is still useful, provided the caller is told it is partial.
MaxMembers int
// MaxRatio caps expanded÷compressed, evaluated per member AND across the
// archive, so neither one enormous member nor a thousand modest ones gets
// through on the other's budget.
MaxRatio int
// MaxWallClock bounds the whole explode. It covers the bomb that stays
// under every byte bound by decompressing slowly, which no size check can
// see.
MaxWallClock time.Duration
}
Limits is the exploder's refusal policy.
Every field is enforced WHILE STREAMING members. A bound checked after expansion has already lost: the bytes exist by then, and the process it would have protected is already the one that has to survive them.
func ParseLimits ¶
ParseLimits resolves the archive bounds an AgentClass asks for against the ceiling a deployment will actually serve.
CLAMPS, never substitutes. A class TIGHTENING a bound is a legitimate operator choice; a class loosening one past the ceiling is not, and an unclamped path would let an AgentClass edit raise the limit on a shared extractord that many classes use. Every value returned is therefore min(requested, ceiling), and an omitted value is the ceiling itself.
Called from two places on purpose: the attachments capability parses it at apply time so a malformed config fails where an operator sees it, and the operator parses it again at upload time because that is where the value is actually used. One parser, so the two can never disagree about what a config means.
type Member ¶
type Member struct {
// Name is the sanitized, archive-relative path. Never absolute, never
// containing "..", and still UNTRUSTED text: it reaches an agent's context
// and must be treated as such by whoever renders it.
Name string
// Size is the uncompressed size actually produced, not the size the
// archive's header claimed.
Size int64
// MIME is SNIFFED from the member's first bytes, never derived from Name.
//
// This is where the archive path differs from a direct upload, where the
// channel supplies a MIME we have no better source for: here WE assign it,
// so trusting an attacker-chosen extension would let a filename decide
// which parser runs downstream.
MIME string
// Open returns the member's bytes. The caller owns closing it. Reading
// past a limit returns ErrArchiveBomb mid-stream.
Open func() (io.ReadCloser, error)
}
Member is one entry in an exploded archive.
type Result ¶
type Result struct {
// Text is the extracted plain text, suitable for an agent to read.
Text string
// Pages is a page/slide count when the format has one, else 0. Reported to
// the agent in the attachment manifest so it can judge scale before reading.
Pages int
}
Result is what an extractor produces.
type SkipReason ¶
type SkipReason string
SkipReason is why one entry inside an archive was refused.
A CLOSED vocabulary, and skips are reported as COUNTS PER REASON rather than as a list of names. A member name is file content, and this package runs in a pod deliberately given no way to send file content anywhere — a diagnostic naming the entries it rejected would be exactly that channel, reopened for convenience.
const ( // SkipNonRegular is a directory, symlink, hardlink, device, or anything // else that is not a regular file. Skipped rather than followed: a symlink // inside an archive is a traversal primitive, not a file. SkipNonRegular SkipReason = "non_regular" // SkipTraversalName is a name that escapes the archive root. SkipTraversalName SkipReason = "traversal_name" // SkipOverMemberBytes is one member past MaxMemberBytes. The member is // dropped; the archive continues, because one oversized file in a bundle // does not make the other 200 unreadable. SkipOverMemberBytes SkipReason = "over_member_bytes" // SkipNameCollision is a second member normalizing to a name already // taken. Dropped rather than overwritten, so a hostile archive cannot hide // a file behind a benign one. SkipNameCollision SkipReason = "name_collision" )
type Summary ¶
type Summary struct {
// Members is how many were yielded.
Members int
// Truncated is set when a bound stopped the walk early.
Truncated bool
// TruncatedReason names the bound, for a caller to state plainly.
TruncatedReason string
// TruncatedMember names the member whose COPY was cut, when the walk
// stopped inside one rather than between them.
//
// Without it a member cut mid-copy is indistinguishable from a whole one:
// its part is already on the wire, and the consumer stores it, runs text
// extraction over it and lists it as complete. The archive-level Truncated
// flag says SOMETHING was cut; this says WHICH, which is what a consumer
// needs to mark that one row partial instead of guessing.
//
// Empty when the walk stopped BETWEEN members (the member-count bound), or
// when nothing was truncated at all.
TruncatedMember string
// Skipped counts refused entries by reason. Counts, never names — see
// SkipReason.
Skipped map[SkipReason]int
}
Summary reports what a whole archive did.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package extractordclient is the HTTP client internal/cmd/operator dials extractord (internal/cmd/extractord) with — the network counterpart to pkg/platform/extract's MIME registry, which extractord wraps as a zero-egress HTTP service.
|
Package extractordclient is the HTTP client internal/cmd/operator dials extractord (internal/cmd/extractord) with — the network counterpart to pkg/platform/extract's MIME registry, which extractord wraps as a zero-egress HTTP service. |
|
Package tabula adapts github.com/tsawler/tabula (MIT; pure Go on the build path used here — OCR support lives behind the library's own "ocr" build tag, which pulls in a CGO Tesseract binding we never enable) to extract.Extractor.
|
Package tabula adapts github.com/tsawler/tabula (MIT; pure Go on the build path used here — OCR support lives behind the library's own "ocr" build tag, which pulls in a CGO Tesseract binding we never enable) to extract.Extractor. |
|
Package text is the extract.Extractor for formats that are already text: no document-parsing library buys anything over reading to EOF and coercing to valid UTF-8.
|
Package text is the extract.Extractor for formats that are already text: no document-parsing library buys anything over reading to EOF and coercing to valid UTF-8. |
|
Package ziparchive is the extract.Exploder for ZIP containers.
|
Package ziparchive is the extract.Exploder for ZIP containers. |