decode

package
v0.7.41 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package decode is the runner image's `decode` role: it turns the documents under one directory into plain text under another, before any agent sees them, so a run reads text and no model ever runs a decoder. Each family of file has one fixed decoder with a fixed argv — pdftotext for PDF, pandoc (in its sandbox mode) for DOCX, ODT, EPUB and RTF, tesseract's OCR for images, and this package's own spreadsheet and slide readers, plus Tika for the long tail. An archive's members are unpacked and decoded in turn (archive.go). Before any decoder reads a file, every file and member is checked by ClamAV in one clamscan run, and a file it finds is never decoded (clamav.go). A git repository's internals, everything under a .git directory, are listed and checked but never detected or decoded (InRepository). Nothing a document contains reaches an argv but its own path. Any other UTF-8 text (code, markup, configuration, data) passes as it is, byte for byte, told without a JVM (detect). The caller runs this in a container with no network and no capabilities, the inputs mounted read-only.

This package is part of the embedded runner-image build context and so imports nothing outside the standard library.

Index

Constants

View Source
const (
	Schema     = "decode/v1"
	ReportFile = "decode.json"
)

Schema is the report's format; ReportFile is where Decode writes it, beside the texts.

View Source
const (
	MaxDepth        = 3
	DefaultMaxFiles = 2000
	// DefaultSourceMaxFiles is an input source's own cap, its archive
	// members included: the fetch step refused
	// a larger source already.
	DefaultSourceMaxFiles = 20000
)

The decode's fixed bounds on archives, which the host holds a report to: how deep archives nest and are still unpacked, and how many files and members a decode takes in all.

View Source
const OriginalsDir = "originals"

OriginalsDir is where a decode asked to keep members copies their own bytes, under the output.

View Source
const ScanSchema = "agentry.decode.scan/v2"

ScanSchema names the scan record's shape.

Variables

This section is empty.

Functions

func BinaryMagic

func BinaryMagic(b []byte) bool

BinaryMagic says whether a file's first bytes are an executable's, a library's, Java's, WebAssembly's, an OLE container's or a PDF's. A source file that opens so is checked whatever its detected type: an Outlook message is an OLE container under a type of its own, and the host holds a file left unchecked to these same bytes.

func ExecutableMagic

func ExecutableMagic(b []byte) bool

ExecutableMagic says whether a file's first bytes are an executable's, a library's, Java's or WebAssembly's: the bytes the host holds a file left unchecked as not executable to.

func ExecutableRisky

func ExecutableRisky(mime, p string) bool

ExecutableRisky says whether a file of this detected type, at this report path, is checked when only executables are.

func InRepository

func InRepository(rel string) bool

InRepository says whether a report path names a file inside a .git directory: some segment before the last is ".git". Its internals (loose objects, packs, the index, refs, hooks) are kept as they are: listed, checked by ClamAV, never detected, decoded or unpacked. A run that wants a repository's history reads it with git; a decode of zlib objects only cost a detector start or two per object. A file named .git (a worktree's or submodule's pointer) is decoded like any other. The host holds a report to the same rule.

func InSource

func InSource(rel string, sources []string) bool

InSource says whether a report path is inside one of the sources' folders.

func PlainMemberPath

func PlainMemberPath(p string) bool

PlainMemberPath says whether a member's path, as a report names it below its archive, is one the decode could have written: already clean and within memberName's rules. The host holds a report to it, since the container that wrote the report read the archive.

func RecordOf

func RecordOf(files []File, checked bool) []byte

RecordOf is the scan record of files as the check left them: checked says signatures were mounted. It is one JSON line, as the runner writes it.

func SourceRisky

func SourceRisky(mime, p string) bool

SourceRisky says whether a file of this detected type, at this report path, is checked in an input source.

func ValidMIME

func ValidMIME(s string) bool

ValidMIME says whether s is a bare media type, as the report may carry it; the host holds the report to the same rule.

func ValidSignature

func ValidSignature(s string) bool

ValidSignature says whether a signature name is within that grammar; the host holds a report to it.

Types

type Check

type Check string

Check is what the malware check made of one file.

const (
	// CheckClean: clamscan read it and found nothing.
	CheckClean Check = "clean"
	// CheckFound: clamscan found a signature in it (Signature names it).
	CheckFound Check = "found"
	// CheckUnchecked: it was not checked, for the reason Unchecked gives.
	CheckUnchecked Check = "unchecked"
)

The checks.

type Family

type Family string

Family is the kind of document a file is, which picks its decoder.

const (
	FamilyText         Family = "text"
	FamilyPDF          Family = "pdf"
	FamilyDocument     Family = "document"
	FamilyPresentation Family = "presentation"
	FamilySpreadsheet  Family = "spreadsheet"
	FamilyImage        Family = "image"
	FamilyArchive      Family = "archive"
	FamilyUnknown      Family = "unknown"
)

The families.

type File

type File struct {
	// Path is the input's slash path relative to the input root.
	Path   string `json:"path"`
	Family Family `json:"family"`
	// Tool is the decoder that ran: pdftotext, pandoc, tesseract, or
	// agentry-runner for this package's own readers, or tika; for an
	// archive, what read it: agentry-runner, bsdcat or bsdtar. A file read
	// as text names none; its Markdown view names its own (MarkdownBy).
	Tool    string  `json:"tool,omitempty"`
	Outcome Outcome `json:"outcome"`
	// Output is the text's slash path relative to the output root. A
	// member's text is written there whether it was decoded or is text
	// already, since the member itself stays in the decode container.
	Output string `json:"output,omitempty"`
	// Bytes is the size of the text written.
	Bytes int64 `json:"bytes,omitempty"`
	// Markdown is the slash path, relative to the output root, of a
	// second view of a file read as text: HTML converted to Markdown by
	// MarkdownBy (pandoc), at exactly <path>.md. The file itself stays
	// its own view; a failed conversion leaves this empty and the file read
	// as it is. MarkdownBytes is the size of the Markdown written.
	Markdown      string `json:"markdown,omitempty"`
	MarkdownBy    string `json:"markdownBy,omitempty"`
	MarkdownBytes int64  `json:"markdownBytes,omitempty"`
	Reason        string `json:"reason,omitempty"`
	// In is the archive a member was unpacked from, empty for a file as
	// staged.
	In string `json:"in,omitempty"`
	// Original is where a member's own bytes were copied, relative to the
	// output root, when the decode was asked to keep members
	// (Limits.KeepMembers): exactly OriginalsDir/<path>.
	Original string `json:"original,omitempty"`
	// Size is a member's unpacked size.
	Size int64 `json:"size,omitempty"`
	// Why is why an archive or a member was not read in full, in the
	// closed vocabulary of archive.go, or WhyFound, or WhyRepository.
	Why Why `json:"why,omitempty"`
	// Check is what the malware check made of it (clamav.go); empty for a
	// file never written to be read (a link, a refused member, past the
	// cap). Signature names what was found; Unchecked says why it was not
	// checked.
	Check     Check     `json:"check,omitempty"`
	Signature string    `json:"signature,omitempty"`
	Unchecked Unchecked `json:"unchecked,omitempty"`
	// MIME is the type the file was detected as, and SHA256 its content's
	// digest, lower-case hex: for every file read, archives and members
	// included, so the host can look the risky ones up on Hybrid Analysis
	// by hash. The host checks a staged file's digest
	// against the file; only a member's is the decode's word alone.
	MIME   string `json:"mime,omitempty"`
	SHA256 string `json:"sha256,omitempty"`
}

File is one input's line in the report.

func (File) HasText

func (f File) HasText() bool

HasText says the decode wrote a text for it at Output: a file it decoded, or a member read as text, whose text is copied out since the member itself stays in the decode container.

type Limits

type Limits struct {
	// MaxFiles refuses the files past this many (default 2,000, a release
	// archive's own cap).
	MaxFiles int
	// ScratchDir is where a decoder writes before its text is cut to size
	// and copied out, so no more than MaxOutputBytes of a file's text ever
	// reaches the output directory (default the system temporary
	// directory: in the decode container, a size-limited tmpfs).
	ScratchDir string
	// PerFileSeconds stops a decoder that runs longer (default 120).
	PerFileSeconds int
	// MaxOutputBytes cuts a text past this size (default 10 MiB, a whole
	// knowledge pack).
	MaxOutputBytes int64
	// MaxExpandedBytes refuses an Office file whose parts expand past this
	// (default 64 MiB), so a zip bomb costs a bounded read.
	MaxExpandedBytes int64
	// UnpackDir is where archives are unpacked (default ScratchDir: in the
	// decode container, a tmpfs of its own the size of MaxUnpackedBytes).
	UnpackDir string
	// MaxUnpackedBytes is what every archive in the decode may unpack in
	// all, counted as each member streams (default 512 MiB).
	MaxUnpackedBytes int64
	// KeepMembers copies each member that was read as text or decoded, as
	// it was in its archive, to OriginalsDir/<path> under the output, within
	// MaxOutputBytes: for a caller that keeps every document as given (the
	// Knowledge Creator, the Voice Writer).
	KeepMembers bool
	// Record, when set, takes the scan record (ScanRecord) once clamscan
	// has answered and before any decoder starts: the runner's stdout.
	Record io.Writer
	// ClamDB is the directory of ClamAV signatures the malware check reads
	// (the host's bundle, mounted read-only); empty, nothing is checked and
	// every file says so.
	ClamDB string
	// Sources names the top-level folders that are input sources. A source's
	// files count against SourceMaxFiles, apart from the uploads' MaxFiles,
	// and only its executable and binary risky files are checked by ClamAV
	// (SourceRisky); the rest say so (UncheckedNotRisky).
	Sources []string
	// SourceMaxFiles is each source's cap (default 20,000).
	SourceMaxFiles int
	// ExecutablesOnly checks only the executables among every file, an
	// upload's and a source's alike (ExecutableRisky); the rest say so
	// (UncheckedNotExecutable). An archive is still checked whole.
	ExecutablesOnly bool
}

Limits bound a decode; zero values take the defaults.

type Outcome

type Outcome string

Outcome is what happened to one file.

const (
	// OutcomeDecoded: text was written for it.
	OutcomeDecoded Outcome = "decoded"
	// OutcomeText: it is text already and is read as it is.
	OutcomeText Outcome = "text"
	// OutcomeUndecodable: no decoder applies, or the decoder failed.
	OutcomeUndecodable Outcome = "undecodable"
	// OutcomeRefused: not a regular file, or past the file cap; for an
	// archive or a member, Why says which.
	OutcomeRefused Outcome = "refused"
	// OutcomeUnpacked: an archive whose members follow it in the report,
	// each named <archive>/<member>; Why, when set, says why only part of
	// it was unpacked.
	OutcomeUnpacked Outcome = "unpacked"
)

The outcomes.

type Report

type Report struct {
	Schema string `json:"schema"`
	Files  []File `json:"files"`
}

Report is what one Decode did, file by file, in path order.

func Decode

func Decode(ctx context.Context, in, out string, tools Tools, lim Limits) (Report, error)

Decode decodes every regular file under in into out and writes the report to out/decode.json. A file's failure is its outcome; the error is for the roots alone. The file cap counts an archive's members with the files as staged. It goes in three passes: every file is detected and every archive unpacked, the members kept until the end; then everything is checked by ClamAV at once; then each file the check did not find is decoded.

type ScanFile

type ScanFile struct {
	Path      string    `json:"path"`
	Check     Check     `json:"check"`
	Signature string    `json:"signature,omitempty"`
	Unchecked Unchecked `json:"unchecked,omitempty"`
	SHA256    string    `json:"sha256,omitempty"`
}

ScanFile is one file's check as the record states it: its report path, the check, the signature's name when it was found and the name is within the grammar, why it was not checked, and its content's SHA-256 as the check read it, so a copy kept of it later can be held to the bytes that were checked.

type ScanRecord

type ScanRecord struct {
	Schema  string     `json:"schema"`
	Checked bool       `json:"checked"`
	Files   []ScanFile `json:"files"`
}

ScanRecord is the malware check's result as the runner states it the moment clamscan has answered and before any decoder starts: whether signatures were mounted, and every file's check. The report written at the end is the work of the same process that then ran the decoders, so a decoder a file exploited could rewrite another file's check there; the host takes the first record the runner wrote instead, and holds the report to it.

What the record cannot vouch for: detection (file, Tika's detector) and the unpackers (the zip and tar readers, bsdtar, bsdcat) read each file before the scan, as the same uid, so a file that exploits one of them could reach the runner's stdout through /proc and write a record first.

type Tools

type Tools struct {
	PDFToText, Pandoc, Tesseract, File, Tika, Bsdtar, Bsdcat, Clamscan string
}

Tools are the absolute paths of the detectors and external decoders; zero values take the runner image's.

type Unchecked

type Unchecked string

Unchecked is the closed vocabulary of why a file was not checked.

const (
	// UncheckedNoSignatures: the host mounted no signatures, so clamscan
	// did not run.
	UncheckedNoSignatures Unchecked = "no-signatures"
	// UncheckedTooLarge: the file, or what clamscan unpacked from it, is
	// past clamscan's limits (Heuristics.Limits.Exceeded).
	UncheckedTooLarge Unchecked = "too-large"
	// UncheckedFailed: clamscan could not read it, failed or was stopped
	// before it answered for it.
	UncheckedFailed Unchecked = "failed"
	// UncheckedNotRisky: a file in an input source that is not an
	// executable or binary risky type, which only those are checked in a
	// source (SourceRisky).
	UncheckedNotRisky Unchecked = "not-risky"
	// UncheckedNotExecutable: a file that is not an executable, in a decode
	// that checks only executables (Limits.ExecutablesOnly).
	UncheckedNotExecutable Unchecked = "not-executable"
)

The reasons.

type Why

type Why string

Why is the closed vocabulary of why an archive or a member was not read in full. The host turns each into its own fixed words, so nothing an archive or a decoder says reaches a run.

const (
	// WhyEncrypted: the archive or the member is encrypted, and nothing
	// can check what it holds.
	WhyEncrypted Why = "encrypted"
	// WhyDamaged: the runner's own reader found it damaged, or not the
	// archive it appeared to be.
	WhyDamaged Why = "damaged"
	// WhyUnreadable: libarchive could not read it; it may be encrypted or
	// damaged, and libarchive does not always say which.
	WhyUnreadable Why = "unreadable"
	// WhyTooDeep: an archive nested deeper than the limit is not unpacked.
	WhyTooDeep Why = "too-deep"
	// WhyTooMany: unpacking stopped at the file cap.
	WhyTooMany Why = "too-many"
	// WhyTooLarge: unpacking stopped at the unpacked-bytes budget.
	WhyTooLarge Why = "too-large"
	// WhyUnsafePath: a member's name is absolute, climbs out with "..", or
	// is not a plain name (a backslash, a control or reordering mark,
	// invalid UTF-8, past the length bounds), so nothing in the archive is
	// unpacked.
	WhyUnsafePath Why = "unsafe-path"
	// WhyLink: a symbolic or hard link is never unpacked.
	WhyLink Why = "link"
	// WhySpecial: a device, a FIFO or another special file is never
	// unpacked.
	WhySpecial Why = "special"
	// WhyConflict: the archive names the path twice, or as a file and a
	// directory.
	WhyConflict Why = "conflict"
	// WhyMethod: the member is compressed by a method the runner does not
	// read.
	WhyMethod Why = "method"
)

The reasons.

const WhyFound Why = "found"

WhyFound: a file clamscan found a signature in is never decoded.

const WhyRepository Why = "repository"

WhyRepository: the file is inside a git repository's .git directory, whose internals are never decoded (InRepository).

const WhySourceImage Why = "source-image"

WhySourceImage: an image in an input source is not read with OCR. A codebase's screenshots and icons run to hundreds, tesseract takes seconds each, and the whole decode has 30 minutes (a desktop repository's 896 screenshots alone would take about 40); the image stays readable as its original.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL