Documentation
¶
Overview ¶
Package decode is the runner's decode step: the documents a run is given are turned into plain text before any agent starts, by the runner image's own `decode` role in an image that holds the decoders and content detectors (the files image, pulled by digest, with the runner binary added), in a container with no network and no capabilities, the documents mounted read-only. No model runs a decoder, and nothing a document contains reaches a decoder's argv but its own path. The catalogue of what provides each decoder is named once, here, for every caller.
Index ¶
- Constants
- Variables
- func PullFiles(ctx context.Context, d dockerx.Client, pol *policy.Policy, logw io.Writer) (string, error)
- func ScriptFake(f *dockerx.Fake, runnerImage, filesImage string)
- func Skipped(docs []Document) []string
- func UnpackedInPart(f runnerdecode.File) string
- func UnreadReason(f runnerdecode.File) string
- type Document
- type Image
- type Options
- type Pages
- type Result
- func Check(img Image, in, out string) (*Result, error)
- func CheckRecorded(img Image, in, out string, opt Options, stdout []byte) (*Result, error)
- func CheckSources(img Image, in, out string, sources []string) (*Result, error)
- func CheckWith(img Image, in, out string, opt Options) (*Result, error)
- func Run(ctx context.Context, d dockerx.Client, img Image, in, out string, ...) (*Result, error)
- func RunSources(ctx context.Context, d dockerx.Client, img Image, in, out string, ...) (*Result, error)
- func RunWith(ctx context.Context, d dockerx.Client, img Image, in, out string, ...) (*Result, error)
- func (r *Result) Checks() string
- func (r *Result) CopyMarkdown(path, dst string) error
- func (r *Result) CopyOriginal(path, dst string) error
- func (r *Result) CopyText(path, dst string) error
- func (r *Result) Counts() string
- func (r *Result) Decoded() int
- func (r *Result) Flags() map[string]string
- func (r *Result) Found() []runnerdecode.File
- func (r *Result) HandedOver() string
- func (r *Result) Line() string
- func (r *Result) LineAs(when string) string
- func (r *Result) LookUp(ctx context.Context, c *hybrid.Client)
- func (r *Result) Malicious() []runnerdecode.File
- func (r *Result) OriginalSize(path string) (int64, bool)
- func (r *Result) Readable() []runnerdecode.File
- func (r *Result) Refusal() string
- func (r *Result) Refused() bool
- func (r *Result) Stage(givenAs map[string]string, dst string) ([]Document, []string, error)
- func (r *Result) StageWith(givenAs map[string]string, dst string, opt StageOptions) ([]Document, []string, error)
- func (r *Result) Text() int
- func (r *Result) Undecodable() []string
- type Spec
- type StageOptions
- type Tool
Constants ¶
const ( // MaxTextBytes is the most of one decoded text the host reads: the // decoder's own cut, a whole knowledge pack. MaxTextBytes = 10 << 20 // Timeout ends a whole decode. Timeout = 30 * time.Minute )
The bounds on what the host reads back from the decode container.
const DocumentsFile = "documents.json"
DocumentsFile is the list of what a run was given, at the root of what it reads (the Knowledge Creator's inputs; a hosted run's).
const FakeFilesPackages = "pandoc-cli-3.6.4-r0\n"
FakeFilesPackages is the package record ScriptFake has the files image answer: one decoder's version, as a real files image records it.
const FilesPackagesFile = "/etc/agentry/files/packages"
FilesPackagesFile is where the files image records the version apk resolved for each of its packages, one name-version a line.
const MaxDocumentBytes = 256 << 20
MaxDocumentBytes bounds one document a decode reads back.
const PagesTimeout = 15 * time.Minute
PagesTimeout ends the whole pages role: twelve pages at their limit, and the reads before them.
Variables ¶
var Catalogue = []Tool{ {Command: "pandoc"}, {Command: "pdftotext"}, {Command: "pdftoppm", Pages: true}, {Command: "pdfinfo", Pages: true}, {Command: "pdffonts", Pages: true}, {Command: "tesseract"}, {Command: "java"}, {Command: "file"}, {Command: "bsdtar"}, {Command: "bsdcat"}, {Command: "clamscan"}, }
Catalogue is every external decoder, named once for every caller. It is the application's, not the operator's exec catalogue: a decode step is the runner's own, like its proxy, and runs no command a QuickAgent declared. The decode image is minimized to exactly these and the runner.
Functions ¶
func PullFiles ¶
func PullFiles(ctx context.Context, d dockerx.Client, pol *policy.Policy, logw io.Writer) (string, error)
PullFiles pulls the files image the policy names, by digest, unless it is here, and answers its reference: Ensure's pull alone, which a sandbox host runs before any run (`agentry files ensure`), so the first run with inputs after an update does not download it.
func ScriptFake ¶ added in v0.7.12
ScriptFake makes f play the decode step of a run with inputs, for a test of anything that drives the engine with a dockerx.Fake: the decode image built from runnerImage and filesImage is present, so Ensure neither pulls nor builds; the files image's package record answers; and the decode container states its scan record on stdout before it decodes anything, as the runner image does, then writes a report that decodes no file. The real CheckWith then records each staged file as given and not read, and the run goes on.
The scan record says the files were checked against signatures exactly when the container was given them (the /clamav mount), so an image with signatures and one without both pass the real check. Call it after anything that replaces f.Foreground, f.ExistingImages or f.ImageFiles wholesale.
func Skipped ¶
Skipped names the documents Stage listed as unread: the built-ins' account of what a build did not read.
func UnpackedInPart ¶
func UnpackedInPart(f runnerdecode.File) string
UnpackedInPart is why an archive that was unpacked was unpacked only in part, in the decode's fixed words, or "" when it was unpacked whole.
func UnreadReason ¶
func UnreadReason(f runnerdecode.File) string
UnreadReason is why a file was not read, in fixed words chosen by its outcome and tool. The report's own reason is never passed on: it can carry a decoder's error output, which can quote the document, and a run shows the reason to the model (list_inputs, read_input, a session's notice).
Types ¶
type Document ¶
type Document struct {
// Name is the document as it was staged, and as a knowledge pack cites
// it: [src: doc:<Name>].
Name string `json:"name"`
// GivenAs is the path it was given as, when that differs.
GivenAs string `json:"givenAs,omitempty"`
// Read is the file to read for it: the document itself when it is
// text, else the text decoded from it. Empty when it could not be read.
Read string `json:"read,omitempty"`
Family runnerdecode.Family `json:"family"`
// DecodedBy names the decoder when the text was decoded.
DecodedBy string `json:"decodedBy,omitempty"`
// Unread says why a document could not be read, in the decode's words:
// the answer for an undecodable file, the same for every caller.
Unread string `json:"unread,omitempty"`
}
Document is one line of documents.json.
type Image ¶
type Image struct {
Tag string
Files string
Packages []string
// Signatures is the ClamAV bundle Run mounts for the malware check
// (clamdb.Ensure); none, every file is reported unchecked.
Signatures clamdb.DB
}
Image is a built decode image, the files image it was built on, and the package versions that image recorded.
func Ensure ¶
func Ensure(ctx context.Context, d dockerx.Client, runnerImage string, pol *policy.Policy, ctxDir string, logw io.Writer) (Image, error)
Ensure builds the decode image when it is missing: it pulls the files image the policy names, by digest, then builds the offline layer on it, and reads back the versions the files image recorded.
type Options ¶
type Options struct {
// Sources are held to the source rule: their own file cap, and only
// their executable and binary risky files checked.
Sources []string
// ExecutablesOnly checks only the executables among every file, and
// asks Hybrid Analysis about those alone.
ExecutablesOnly bool
// KeepMembers has the decode copy out each archive member it read, as
// it was in its archive, for a caller that keeps every document as
// given (the built-ins' Stage).
KeepMembers bool
}
Options are what a decode is asked beyond its inputs: which top-level folders are input sources, and whether only executables are checked for malware.
type Pages ¶ added in v0.7.26
type Pages struct {
Facts docpages.Facts
// Full are the rendered pages at docpages.RenderPPI, page 1 first, and
// View the same pages at docpages.ViewPPI, each a PNG.
Full, View [][]byte
// Media are a Word document's pictures, each a PNG.
Media [][]byte
// Image is the decode image that read it.
Image Image
}
Pages is what the runner image's pages role read of the Report Designer's document, held to its rules: the facts, and each page and picture as this process encoded it again from its decoded pixels, so no chunk or metadata the renderer or the document wrote survives.
func CheckPages ¶ added in v0.7.26
CheckPages reads what the pages role wrote to out and holds it to its rules: the facts to docpages.Facts.Validate and to kind, each page and picture a regular file reached by no link, within its cap, decoded and encoded again. A PDF's colours are the ones its rendered pages show, found here.
func RunPages ¶ added in v0.7.26
func RunPages(ctx context.Context, d dockerx.Client, img Image, doc string, kind docpages.Kind, out string, lim policy.Limits, stderr io.Writer) (*Pages, error)
RunPages reads the one document at doc, of kind, with the pages role in the decode image: no network, no capabilities, the document's folder mounted read-only, the pages rendered in a bounded /tmp and written to out, a folder the caller made for it. A document the role could not read is a Pages with its Facts.Failure; the error is the container's own failure, or what it wrote breaking the rules.
type Result ¶
type Result struct {
Report runnerdecode.Report
Image Image
// In holds the documents as given, each at its File.Path; Out the
// texts, each at its File.Output.
In, Out string
// Lookups is each risky file's Hybrid Analysis lookup by path (LookUp); nil
// until LookUp ran, and then the receipt says nothing of a lookup.
Lookups map[string]hybrid.Result
// Sources are the top-level folders that are input sources:
// their files are held to the source rule.
Sources []string
// ExecutablesOnly says the decode checked only the executables
// (Options.ExecutablesOnly).
ExecutablesOnly bool
// KeepMembers says the decode was asked to copy out members'
// originals (Options.KeepMembers).
KeepMembers bool
}
Result is one decode: the report the container wrote, checked, and the image that wrote it. Its files are read only through its own methods, which hold every read to the two directories.
func Check ¶
Check reads the report a decode wrote to out and holds it to what was staged under in (checkReport). It does not hold the report's malware checks to the scan record the decode stated first: a decode given signatures is read through CheckRecorded, which Run uses.
func CheckRecorded ¶
CheckRecorded is CheckWith held to the scan record the decode wrote to its stdout before any decoder ran (firstScanRecord, applyScan): what Run does after the container, and what a caller that decodes some other way does with that stdout. A decode given signatures must have stated one.
func CheckSources ¶
CheckSources is Check for a decode over input sources (RunSources).
func Run ¶
func Run(ctx context.Context, d dockerx.Client, img Image, in, out string, lim policy.Limits, stderr io.Writer) (*Result, error)
Run decodes every file under in into out. Both must be directories the caller made for this decode — they are relabelled for the container on a daemon with SELinux, which is a write to the host's labels and so never done to a directory a person owns — and in should hold copies of the documents, not the originals.
func RunSources ¶
func RunSources(ctx context.Context, d dockerx.Client, img Image, in, out string, lim policy.Limits, sources []string, stderr io.Writer) (*Result, error)
RunSources is Run over inputs that hold input sources, each a top-level folder of in named in sources: a source's files count against its own cap, and only its executable and binary risky files are checked; the report is held to the same rule.
func RunWith ¶
func RunWith(ctx context.Context, d dockerx.Client, img Image, in, out string, lim policy.Limits, opt Options, stderr io.Writer) (*Result, error)
RunWith is Run with its Options.
func (*Result) Checks ¶
Checks is the malware check's clause of the receipt: the signatures' date and bundle, what was clean, found and not checked, and whether the signatures were stale or could not be refreshed.
func (*Result) CopyMarkdown ¶
CopyMarkdown copies the Markdown view of the file read as text at path (HTML, converted by pandoc) to dst.
func (*Result) CopyOriginal ¶
CopyOriginal copies the document at path, as given, to dst: a file as staged, or a member of an archive whose own bytes the decode kept.
func (*Result) CopyText ¶
CopyText copies the text decoded from the document at path to dst, or, for an archive member read as text, the text copied out for it.
func (*Result) Counts ¶
Counts is what happened to the files: how many were decoded, read as text, and not decoded, naming the first few of those. The files inside a .git directory are counted apart, unnamed: a clone holds hundreds, and they would crowd out the documents that could not be read.
func (*Result) Flags ¶
Flags is why each flagged file was flagged, by path, in the receipt's words: ClamAV found a signature in it, or Hybrid Analysis's verdict on it is malicious. Empty when none was.
func (*Result) Found ¶
func (r *Result) Found() []runnerdecode.File
Found is every file ClamAV found a signature in: the run is refused before the agent starts.
func (*Result) HandedOver ¶
HandedOver is the receipt's sentence for flagged files a run that runs no commands was given anyway: never decoded, as uploaded and with no execute bit. Empty when none was flagged.
func (*Result) Line ¶
Line is the receipt's decode line: the mechanism, the versions, and what happened to the files.
func (*Result) LineAs ¶
LineAs is Line led by its own account of when the decode happened: a session's files added mid-run were decoded before the agent was told of them, not before the run.
func (*Result) LookUp ¶
LookUp asks Hybrid Analysis about each file of a risky type, archive members included, by its SHA-256 alone, from this process and never a container. A file ClamAV found is refused already and is not asked about, nor is a file inside a .git directory; a risky file with no digest is not checked. c nil or with no key asks nothing, and each risky file is not checked, no key.
func (*Result) Malicious ¶
func (r *Result) Malicious() []runnerdecode.File
Malicious is every file Hybrid Analysis gave a malicious verdict: the run is refused before the agent starts.
func (*Result) OriginalSize ¶
OriginalSize is the size of the document named path as given: a file as staged, or a member's kept original; false when it has none.
func (*Result) Readable ¶
func (r *Result) Readable() []runnerdecode.File
Readable are the documents that were read as text or decoded into it.
func (*Result) Refusal ¶
Refusal is the sentence a refused run, or a session's refused file, gives: each file ClamAV found and its signature, in the decode's own words, and each file Hybrid Analysis gave a malicious verdict, with its threat score.
func (*Result) Refused ¶
Refused says whether a check refuses what was decoded: ClamAV found something, or Hybrid Analysis's verdict on a file is malicious.
func (*Result) Stage ¶
Stage lays out what a run reads from this decode, in dst (a directory the caller made): each readable document's text at its Read path, and documents.json naming every document — those that could not be read too, with why. It returns the documents and the paths to stage under /inputs. Two documents that would be read as one file are refused rather than one hiding the other. An archive is unread and its members are left out; StageWith lists them.
func (*Result) StageWith ¶
func (r *Result) StageWith(givenAs map[string]string, dst string, opt StageOptions) ([]Document, []string, error)
StageWith is Stage taking an archive's members as opt says. An archive none of whose members is read is listed, unread, as it was given.
func (*Result) Undecodable ¶
Undecodable names the files neither decoded nor read as text: those no decoder reads, those a decoder failed on, those refused, and those inside a .git directory.
type Spec ¶
type Spec struct {
// Tag is the local tag, which moves with the whole build.
Tag string
// Files is the files image, name:tag@sha256:digest.
Files string
// Runner is the local runner image whose binary is copied in.
Runner string
}
Spec is the decode image this machine builds: the files image, pinned by digest and pulled, with this CLI's runner binary copied in and the result minimized to the decoders and the runner. The build installs nothing (the base is pulled first, by digest, and the runner image is local), so it cannot fail as a run-time `apk add` did.
func ImageSpec ¶
ImageSpec is the decode image for this runner image and files image. The tag is a hash of the whole build, so a new runner image, a new files image or a new recipe in a later CLI each get an image of their own and never reuse a stale one.
func (Spec) Dockerfile ¶
Dockerfile is the decode image's build: the files image, the runner binary, and the minimize step, each in exec form. The files image has no entrypoint, so the runner is named as this one's: a decode container's argv is the runner's (`decode --in /in --out /out`).
type StageOptions ¶
type StageOptions struct {
// Members lists each member of an archive as a document of its own,
// named <archive>/<member>, and not the archive, since a file and a
// folder of its members cannot share a name. A member is read by its
// text, as the decode copied it out.
Members bool
// Originals also requires each member's own bytes, as the decode kept
// them (Options.KeepMembers), for a caller that keeps every document as
// given: a member whose bytes were not kept is unread.
Originals bool
// Named, when set, says whether a member's name can be one of the
// caller's documents (a knowledge pack's citation carries only plain
// names); a member it refuses is unread.
Named func(path string) bool
}
StageOptions are how the built-ins take an archive's members.
type Tool ¶
type Tool struct {
Command string
// Pages says the command is the pages role's, which reads the Report
// Designer's document, and no decoder: the decode line leaves it out.
Pages bool
}
Tool is one external decoder: the command the decode image keeps. The files image provides it (agentry-dockerimages build/agentry-files, whose Dockerfile names the Alpine packages); this CLI pins that image by digest.